Service interruption 1 AM tonight

We will be reinitializing and re-securing our main gateway router tonight, in an attempt to correct some strange observed behavior on its part that may possibly be due to either corruption or infiltration. We hope this may also relieve some of the prime-time slowdown behavior we’ve encountered within the past month or so.

We expect our connection to the Internet to be interrupted for about 30 minutes sometime within the window of 01:00 to 02:00 tonight (Sunday morning the 30th). As always, our recorded message at 623-640-7883 will be kept updated for you to verify the status of our network.

A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  ?


Planned outage, Feb. 24 (wee hours)

CenturyLink has just informed us that they will be taking our network gateway offline for up to 90 minutes sometime between midnight and 6 AM on Wednesday, February 24 (two weeks away), to perform an upgrade on CenturyLink’s own network. This means that the greater internet will be unreachable by our subscribers during this 90-minute period.

Once more unto the breach

Last Friday, our consultant told us he couldn’t commit for sure as to whether the limiting factor in the speed of our gateway was CenturyLink’s equipment or our own, and recommended we upgrade to beefier equipment just to get off the edge of doubt. So after a flurry of express equipment orders, we now have one of MikroTik’s burliest routers (nine cores, rated for 28Gbps), fully configured and in position at our gateway, ready to accept a cutover of our new double-speed circuit.

Tomorrow, we plan to recable our existing tower equipment into the new router (a rigorous task, involving tall ladders and large hole saws). Because our network is configured as a giant ring, we can disconnect and transfer one leg at a time, which should allow us to cut over the bulk of our subscribers with no more than a 30-second service interruption.

Those few subscribers being served directly by our downtown Wickenburg AP on the gateway tower itself will be an exception, as there is only a single path cable that we must disconnect, reroute, and reconnect. We hope to keep their outage period down to no more than 15 minutes.

When we’re done, we will either be operating at our full gateway capacity, or we won’t. In the latter case we already have a come-together meeting scheduled with CenturyLink early Monday morning.

Wish us godspeed.