A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  😭

 

Morristown: Brief outage today (Wed 4/19)

We’ve determined that the radio card on Morristown Tower West is souring, delivering unacceptable performance to subscribers, and requires replacement. We expect a replacement unit to arrive today, and the replacement activity to occur between 3-5 PM, requiring a brief service outage of around 15 minutes. The West AP serves Morristown subscribers west of the ADOT traffic sign on eastbound US 60 out to the end of Quiet Hills Road (except for subscribers in that area served by Skip-In Ranch Tower). Subscribers served by Morristown South and East APs will not be interrupted.

Brief outage Friday midnight

This announcement affects only subscribers of our towers located in Morristown and Wittmann. Subscribers of towers located in Wickenburg (including “postal Wickenburg”) will not be affected.

We are replacing the Network Operations Center router in Morristown with a faster and more powerful model. There will be a brief network interruption on the order of 10-30 minutes just after midnight Friday night (i.e., Saturday morning) to cut over power and cabling to the new router.

Ill Wind

bent-wmSubscribers homed to our central Morristown Tower may experience some slowdowns, dropouts, or other unusual behavior over the next day or two, due to interference from the windmill atop our transmitter tower, whose entire head canted over last night such that the tail intermittently swings into the path of one or the other of our three sector antennas.

Last week, after the tail itself (though not the entire head) had bent down into a similar position, we called windmill specialists out to dismount, repair, and remount the entire head assembly. The repair obviously didn’t last, and we have put in an order for an urgent followup service call.

We hope you’ll bear with us in the interim, and we thank you for your patience as we get this issue resolved.

Skip-In Ranch: Unscheduled outages

Our Skip-In Ranch tower host is having major residential renovation performed. This has led to a half dozen or so outages to Skip-In Ranch tower subscribers over the past several weeks, as workmen turn off circuit breakers, work on the roof in the vicinity of our mast, or remove and replace our indoor equipment for framing, drywall, and painting work. (Subscribers of our Morristown towers may experience micro-outages when any existing sessions through the Skip-In Ranch long link reroute through the Mockingbird long link.)

We’ve successfully educated the foreman about what breakers and components interrupt our service, to the point where at least the workmen no longer disable us indefinitely. Although we still don’t get advance notification of these outages, they do take pains to keep them as brief as possible. We’re all hoping that this renovation progresses quickly and completes soon.

Planned outage: Morristown West

Our Morristown West access point will be taken out of service for approximately 20 minutes tomorrow morning (Saturday, October 3) within the period from 9:00 and 10:00 AM, to replace a radio card that is losing receive sensitivity. The Morristown West AP serves Morristown Tower subscribers located generally “clockwise” of the line between the tower and Fenway Park Orchards.

Construction-related congestion

As you know if you have been keeping up with this blog, we have a new fiber circuit on order from CenturyLink that will double the bandwidth at our Wickenburg gateway to the Internet. While we’re awaiting that installation, we have been planning ahead to ensure that we will be able to distribute this new bandwidth evenly across our entire network without artificial bottlenecks.

Late this week, in preparation, we replaced several of the radios driving our long links between Morristown and Wickenburg with new and faster units. The new radio links are now operating at multiples of the speed of the old, and well over the speeds necessary to distribute the new bandwidth clear to Circle City.

However, despite the speed of the new radio links themselves, the effect on our overall speed on the Morristown side of our network seems to have been negative. We’re extremely concerned (and puzzled) by this result. Although we’ve been able to make a number of tuning changes to evade the worst of these performance issues, we haven’t yet identified an underlying cause.

We’re continuing to work this problem over the weekend, with consulting help as available, as long as necessary to ensure that this temporary disruption is solved expeditiously. We apologize for a less than seamless enhancement experience, and look forward to being able to deliver our upcoming bandwidth increase to you as efficiently as possible.

Progress with Reliability Upgrades

With four subscriber unit upgrades and a bit of creative rerouting on North Vulture Mine Road this past weekend, it appears that we have eliminated (or at least greatly reduced) the retransmit cascade failures affecting all users of the Mockingbird West tower. Return signal and bandwidth from the upgraded units to our tower is now capable of several times rated contract speed. If you’re still experiencing slowdowns or outages, please inform us.

We’re awaiting an equipment restock delivery (mid-week) to address one problem subscriber site that is regularly interrupting the Morristown East sector, and one or two units that require the same improvement at Rio Vista Hills. After next weekend, we hope the cascade failure problem will be a thing of the past (or at least the far, far future).

Intermittent Service Cure Plan

Simultaneously with the arrival of the current cold weather, we began receiving automated connectivity failure reports against a number of our subscribers, mainly those being served by our Mockingbird West tower, with Rio Vista, and Morristown East users in place and show positions. They were difficult to pin down, primarily because the metrics reported by the towers didn’t support any particular diagnosis. We replaced the radio unit at Mockingbird West in case the receiver discrimination was going sour, but gained zero relief.

After several weeks of thrashing and several hours of attention from consultants, we now believe we understand the problem, and have an action plan in place to address it. Continue reading