Upgrade trifecta

Dave works his magic on our 251st Avenue tower.

Today’s upgrades were uniformly satisfying.

We added significant bandwidth to feed Skip-In Ranch and all the towers in Morristown; plus additional bandwidth to Mockingbird and Rio Vista towers; plus increased bandwidth between the Morristown hub and the 251st Avenue and Castle Hot Springs towers.

In addition, we installed a new northward-facing AP on the 251st Avenue tower to serve the growing number of subscribers located on the east side of Castle Hot Springs Road who can’t access their nearest neighborhood tower (Castle Hot Springs) due to an intervening hill. These subscribers are now benefiting from 8-10dB better signal (about eight times the previous signal), as well as the additional bandwidth available at their tower.

We also corrected a mysterious configuration gremlin that was causing decreased signal to our subscribers in Grantham Ranch.

We hope today’s work will make your Internet experience more enjoyable.

Morristown upgrade Friday morning; possible Mockingbird upgrade Friday afternoon

Just a brief notification that the upgrade work to the WIckenburg/Morristown link is scheduled for tomorrow morning (Friday) between 9:00 and noon.

Depending entirely on time available, it is possible that similar upgrade work may be performed on the Wickenburg/Mockingbird link sometime in the afternoon, to provide even more bandwidth there. This would temporarily affect network performance to all the towers mentioned in our previous notification, with the addition of Mockingbird and Rio Vista towers.

Thank you for being a part of Grand Avenue Broadband.

 

Morristown Upgrade, Stage II

Tower climber

Dave installs Stage I…

The equipment needed for stage II of the Morristown bandwidth upgrade arrives this week, and we plan to be installing it on either Thursday or Friday morning.

This is the gear needed to complete the “full link” upgrade of our feed from Wickenburg, providing additional bandwidth to our east-end subscribers over and above the “half link” upgrade installed four weeks ago.

Subscribers served by east-end towers may see some short service burps and lower performance for roughly a 20-minute period sometime between the hours of 9:00 AM and noon, as our network compensates for the temporarily missing link by routing Wickenburg traffic over slower alternate routes.

Subscribers of these towers will be affected:

  • Morristown
  • Skip-In Ranch
  • Castle Hot Springs
  • 251st Avenue
  • Dixileta

We’ll post a brief notice like this one once we are certain of the day of installation.

Work continues Wednesday 20th

Last Wednesday’s installment in our bandwidth improvement effort surpassed our expectations. Despite the fact that we had only enough time to deploy half the planned link (due to the structural enhancements we had to add to the tower to handle the heavier equipment), we managed to increase bandwidth to Mockingbird tower significantly via its backup link, plus we determined that the leftover half of the high-speed link could be more effectively employed elsewhere. 

Tomorrow, we hope to be able to install improved equipment on the inbound side of both our Skip-In link (the main line to Morristown and all points east) and our primary link to Mockingbird Tower. As this involves some high-tower work, it will be slower, and some of the transitions may be less than smooth.

Although our existing redundant paths help ensure that your Internet availability will never be entirely severed, you may experience some “brownouts” as our system determines in real time that primary paths have gone black, and temporarily switches to secondary paths that will serve some of our towers somewhat more slowly. Short of unplanned disaster, the perturbations should not persist past 5 PM.

Thank you for your patience as we work to bring speedier Internet access to our ever-growing network of subscribers.

251st Avenue tower outage Wednesday

Due to a recent real-estate transfer of the residence at which our 251st Avenue tower is located, we have been instructed to remove our tower.

We have already secured a commitment from a neighbor and long-time subscriber to assume the duties of neighborhood tower host, so we will be moving the tower to its new location during the day tomorrow (Wednesday). This will result in an outage of two to three hours for those subscribers and satellite towers served by 251st Avenue Tower, as the physical tower itself must be disconnected, detached, uprooted, transported, re-erected, cabled for power in its new location, and have its inter-tower feed links properly realigned.

After the relocation, we’ll be re-aiming the roof units of our closest-in subscribers first, then the farther-out subscribers as necessary. Our furthest-out subscribers in the vicinity of Circle City will likely require no re-aiming at all, though all signal levels will be compared before and after the relocation.

We apologize for this service interruption, and wish to assure you that we will be performing the relocation expeditiously, in order to resume service as promptly as possible.

A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  ?

 

Tower Maintenance: 251st Ave.

We will be increasing the capacity of 251st Avenue tower this Tuesday, December 12, between 9 AM and 2 PM. Users served by this tower, as well as by Dixileta and Circle City towers, may experience brief (<10 minute) outage periods as equipment is detached and replaced. Users served by Castle Hot Springs tower may experience mild slowdowns as data is shunted through that tower to serve 251st Avenue during times when the primary link units to 251st Avenue are being replaced.

Skip-In Ranch: Unscheduled outages

Our Skip-In Ranch tower host is having major residential renovation performed. This has led to a half dozen or so outages to Skip-In Ranch tower subscribers over the past several weeks, as workmen turn off circuit breakers, work on the roof in the vicinity of our mast, or remove and replace our indoor equipment for framing, drywall, and painting work. (Subscribers of our Morristown towers may experience micro-outages when any existing sessions through the Skip-In Ranch long link reroute through the Mockingbird long link.)

We’ve successfully educated the foreman about what breakers and components interrupt our service, to the point where at least the workmen no longer disable us indefinitely. Although we still don’t get advance notification of these outages, they do take pains to keep them as brief as possible. We’re all hoping that this renovation progresses quickly and completes soon.

Construction-related congestion

As you know if you have been keeping up with this blog, we have a new fiber circuit on order from CenturyLink that will double the bandwidth at our Wickenburg gateway to the Internet. While we’re awaiting that installation, we have been planning ahead to ensure that we will be able to distribute this new bandwidth evenly across our entire network without artificial bottlenecks.

Late this week, in preparation, we replaced several of the radios driving our long links between Morristown and Wickenburg with new and faster units. The new radio links are now operating at multiples of the speed of the old, and well over the speeds necessary to distribute the new bandwidth clear to Circle City.

However, despite the speed of the new radio links themselves, the effect on our overall speed on the Morristown side of our network seems to have been negative. We’re extremely concerned (and puzzled) by this result. Although we’ve been able to make a number of tuning changes to evade the worst of these performance issues, we haven’t yet identified an underlying cause.

We’re continuing to work this problem over the weekend, with consulting help as available, as long as necessary to ensure that this temporary disruption is solved expeditiously. We apologize for a less than seamless enhancement experience, and look forward to being able to deliver our upcoming bandwidth increase to you as efficiently as possible.