Brief Downtown Tower outage Tuesday

Recently, we determined that our Downtown Wickenburg Tower was exhibiting impaired performance depending on how long it was continuously operational. While awaiting the manufacturer’s diagnosis, we put a script in place to reboot this unit in the wee hours every four days, to keep it operating at peak.

The manufacturer replied today with the suggestion that reinitializing the unit from scratch should return it to proper operation. We’ll be performing this procedure around noon on Tuesday (tomorrow). This will result in a service outage of 10-15 minutes, which will affect only users directly served by Downtown Tower. We hope this will result in improved performance for our Downtown subscribers.

Wednesday: interruptions for performance improvements

Since our last post, we’ve been hard at work on a new system to deliver more bandwidth to our four neighborhood towers in west Wickenburg, and to ramp our popular Mockingbird Tower up to the speeds necessary to support our increasing subscriber levels.

Subscribers of all our towers from Mockingbird Road westward may encounter some brief (5-20 minute) network interruptions this coming Wednesday at staggered times of the day, as we perform certain of the equipment upgrades described below.

Here are some details about what we’ve been doing, how far along we’ve gotten, and what we hope to accomplish Wednesday. Continue reading

Possible Diamondback work today

There’s about a 70% chance that we will have an opportunity this afternoon (Wednesday 1/31) to be able to perform an equipment upgrade at Diamondback Tower. Subscribers served by that tower may encounter several brief (about two minute) outages, along with possibly somewhat slower performance from the temporary mast we’ll be setting up to provide service while the original tower is being re-engineered. This work will affect only subscribers served by Diamondback Tower.

Nasty KRACK

This week’s networking crisis is that black-hats have found another way to violate your privacy. Called the KRACK exploit, it allows hackers to spoof WiFi protocol so as to break its encryption and read your traffic.

It’s worth pointing out that this is not just a zero-day bug in some manufacturer’s implementation—it’s a defect in the WiFi standard itself, and all WiFi (802.11) encryption is vulnerable to it. It does require a hacker to be close enough to have physical access to your radio signal.

We wanted to let you know what we were doing about it, and also what you should be doing about it yourself. Continue reading

251st Avenue tower outage Wednesday

Due to a recent real-estate transfer of the residence at which our 251st Avenue tower is located, we have been instructed to remove our tower.

We have already secured a commitment from a neighbor and long-time subscriber to assume the duties of neighborhood tower host, so we will be moving the tower to its new location during the day tomorrow (Wednesday). This will result in an outage of two to three hours for those subscribers and satellite towers served by 251st Avenue Tower, as the physical tower itself must be disconnected, detached, uprooted, transported, re-erected, cabled for power in its new location, and have its inter-tower feed links properly realigned.

After the relocation, we’ll be re-aiming the roof units of our closest-in subscribers first, then the farther-out subscribers as necessary. Our furthest-out subscribers in the vicinity of Circle City will likely require no re-aiming at all, though all signal levels will be compared before and after the relocation.

We apologize for this service interruption, and wish to assure you that we will be performing the relocation expeditiously, in order to resume service as promptly as possible.

“Big Sleep” to blanket Mockingbird tonight

After last night’s testing, we suspect the most probable cause of the performance problem at Mockingbird is a bad customer roof unit radio hashing our tower. We’ve exhausted all the minimally invasive tests we can run. Starting tonight, we’re running a Big Sleep test on all our Mockingbird subscriber units. Continue reading

Mockingbird Update

After several abortive ordering attempts (in which “in stock” on a website listing apparently translated to “let us check our warehouse” once the order was placed), we located a dealer with a replacement antenna actually in stock. We have been verbally assured that our replacement unit shipped yesterday, and are awaiting written confirmation including a tracking number. In the absence of nasty surprises, we plan to be replacing the Mockingbird central antenna this weekend.

We are also going to be subjecting the Rio Vista antenna (same manufacturer and model) to testing to determine if it shows any early symptoms of the same failure mode, in case a prophylactic replacement is indicated.

A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  ?