Mockingbird testing progresses

Mockingbird TowerCondensed version: We’ve narrowed down the Mockingbird Tower problem to RF noise that spikes periodically. Our tower equipment and configuration are clean and validated. We now need to perform after-midnight testing over the next few days that will result in brief (10-20 minute) interruptions of service for a number of the subscribers on this tower.

Details and analysis follow for the technically curious. Continue reading

Mockingbird Improved

As of about an hour ago, we believe we have found our issue with Mockingbird Tower Central, and taken steps to provide immediate improvement. Although speeds are not at the level they were three weeks ago (due to suspected antenna hardware damage and pending part arrival and replacement), they should be greatly improved over the degraded performance seen over the past week. Please drop us an email if you are not seeing adequate performance for your equipment.

Technical details follow for the curious.


Continue reading

Mockingbird Performance Issues

Our performance problem with Mockingbird Tower unfortunately wasn’t affected by replacing the radio driver on Tuesday.

We’ve since spent two very late nights (until 4 AM) attempting to locate the true cause of this problem. We’ve checked for queueing and routing problems, limitations on link capacity, RF issues (such as improper equipment orientation, local interference, and — our perennial bugaboo — a subscriber with degrading signal strength whose retry requests overwhelm the tower and destroy performance for all other tower users). We’ve temporarily backed out recent configuration changes to see if any of them triggered the issue. Despite our efforts, the problem continues to defy explanation.

On Wednesday, we contracted with the largest MikroTik consulting firm in the US to look at this problem, as well as the challenge of delivering increased bandwidth to all our subscribers on all towers. We’ve been assured that their engineers will begin work on these tasks on Monday.

We’re sorry to have no interim fix available for this problem. As other potential causes suggest themselves to us, we will continue to test them, and either repair them or eliminate them from consideration. We want you to know that if you are a Mockingbird Tower subscriber, we are not ignoring you — we are aware of your speed issues and we are working on them.

Thank you for using Grand Avenue Broadband.

A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  😭

 

Leave the light on for us

Just another reminder to our seasonal subscribers not to shut the power off on their rooftop receivers, if at all possible, when they leave for the season.

Our network is always undergoing improvements in performance and reliability, as well as expansion. Certain of these improvements require reaching into all our subscribers’ roof units and making coordinated changes between them and the towers that serve them. If you turn your equipment off for the summer season, we won’t be able to update your unit. When you return here, you may find your account’s configuration six months behind everyone else’s, and your service may not be immediately operational.

Your internet equipment uses less power then an incandescent night-light or a single Christmas-tree bulb. Please leave it powered up. Thank you!

Brief outage Friday midnight

This announcement affects only subscribers of our towers located in Morristown and Wittmann. Subscribers of towers located in Wickenburg (including “postal Wickenburg”) will not be affected.

We are replacing the Network Operations Center router in Morristown with a faster and more powerful model. There will be a brief network interruption on the order of 10-30 minutes just after midnight Friday night (i.e., Saturday morning) to cut over power and cabling to the new router.

Repaving the Information Highway

In a week or three, CenturyLink should be completing our recent order to double our gateway speed, resulting in our having more than three times the network bandwidth we had only one month ago.

Unfortunately for us, just having all this new speed available at the gateway doesn’t automatically make it accessible to all our subscribers.

For the past several weeks, a phrase has been looping through my mind from Larry Niven’s “A World Out of Time.” Tasked with a 200-year intergalactic voyage, the bulk of which he must spend in cryogenic stasis, the story’s hero is subjected to a rigorous dietary regimen to ensure his surviving the journey: before each cold-sleep session, it is crucial for him to “grow fat… and exercise to distribute the fat.”

This passage echoes in my head because it’s a perfect description of our current challenge. Continue reading