This week we definitely left Red Queen territory for a visit to the Land of Gods Who First Make Men Mad.
After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.
The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.
The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.
One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).
As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.
In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.
There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.
(And the weatherman says there is 45% chance of a third record windstorm tonight.) 😭
CenturyLink will be performing software upgrades on its equipment serving our network sometime between 11 PM Thursday, April 6, and 5 AM Friday, April 7. The outage should be no more than ten minutes.
Our network will experience one or more brief outages between 7:30 and 9:30 AM tomorrow (Friday, 1/6), as CenturyLink pulls the trigger on a major speed increase for our service. Continue reading
There will be a total service outage of up to an hour sometime between 11 PM and midnight next Tuesday (4/26), while CenturyLink replaces networking equipment at Beardsley that suffers from a known manufacturer defect. We will post updates if CenturyLink revises this schedule.
We’ve just been informed by CenturyLink that our gateway to the internet will be offline for five minutes, sometime between midnight tonight and 6 AM tomorrow (Friday), to correct a trouble light at Beardsley caused by the upgrade work performed during theiir scheduled outage of the 24th.
CenturyLink has just informed us that they will be taking our network gateway offline for up to 90 minutes sometime between midnight and 6 AM on Wednesday, February 24 (two weeks away), to perform an upgrade on CenturyLink’s own network. This means that the greater internet will be unreachable by our subscribers during this 90-minute period.
Last Friday, our consultant told us he couldn’t commit for sure as to whether the limiting factor in the speed of our gateway was CenturyLink’s equipment or our own, and recommended we upgrade to beefier equipment just to get off the edge of doubt. So after a flurry of express equipment orders, we now have one of MikroTik’s burliest routers (nine cores, rated for 28Gbps), fully configured and in position at our gateway, ready to accept a cutover of our new double-speed circuit.
Tomorrow, we plan to recable our existing tower equipment into the new router (a rigorous task, involving tall ladders and large hole saws). Because our network is configured as a giant ring, we can disconnect and transfer one leg at a time, which should allow us to cut over the bulk of our subscribers with no more than a 30-second service interruption.
Those few subscribers being served directly by our downtown Wickenburg AP on the gateway tower itself will be an exception, as there is only a single path cable that we must disconnect, reroute, and reconnect. We hope to keep their outage period down to no more than 15 minutes.
When we’re done, we will either be operating at our full gateway capacity, or we won’t. In the latter case we already have a come-together meeting scheduled with CenturyLink early Monday morning.
Wish us godspeed.
Our new equipment at Rio Vista Tower, like our new equipment at Mockingbird tower, has been meeting and exceeding all our expectations. We’re seeing greatly improved signal strength and effective bandwidth to every one of our subscribers on those towers.
According to CenturyLink’s schedule (and supported by visual verification today), new fiber has been pulled to the utility pole nearest our Wickenburg gateway. The next week or so will be spent establishing a physical service entrance to the building, and configuring and testing the new CenturyLink fiber equipment. We are hoping to be able to cut over to the new gateway sometime during the first full week of October. This gateway will provide us twice the bandwidth to the greater Internet as we currently have, and will be expandable for even more speed beyond that. We’re looking forward to this improvement, and we know you are too!
Thank you for using Grand Avenue Broadband.