Mockingbird Update

After several abortive ordering attempts (in which “in stock” on a website listing apparently translated to “let us check our warehouse” once the order was placed), we located a dealer with a replacement antenna actually in stock. We have been verbally assured that our replacement unit shipped yesterday, and are awaiting written confirmation including a tracking number. In the absence of nasty surprises, we plan to be replacing the Mockingbird central antenna this weekend.

We are also going to be subjecting the Rio Vista antenna (same manufacturer and model) to testing to determine if it shows any early symptoms of the same failure mode, in case a prophylactic replacement is indicated.

Mockingbird Improved

As of about an hour ago, we believe we have found our issue with Mockingbird Tower Central, and taken steps to provide immediate improvement. Although speeds are not at the level they were three weeks ago (due to suspected antenna hardware damage and pending part arrival and replacement), they should be greatly improved over the degraded performance seen over the past week. Please drop us an email if you are not seeing adequate performance for your equipment.

Technical details follow for the curious.


Continue reading

Mockingbird Update

The battle continues on the Mockingbird speed issue.

In the past two days, our consultant pointed us towards two problems that allowed us to more than double the bandwidth capacity between our gateway and the Mockingbird Tower. Maddeningly, this increased bandwidth availability is still not making its way out to actual tower subscribers, and we are still at a loss as to why not. (Note that this is not happening on our other towers.) Our next step is to install some more sophisticated metering software at our gateway for our consultant to use to analyze traffic and routing issues within our network.

Today the new, faster, weatherized router model we’ve been awaiting (for months) to order, to replace the mid-2013 board currently operating Mockingbird Tower, finally hit US distributors’ shelves. We express-ordered one, which should arrive late on Friday. We plan to get it installed over the weekend, possibly solving our problem in whole or in part. (At the very worst, it will allow us to rule out a router hardware problem.) This installation will involve a brief outage, as cables have to be moved en masse from one unit to the other.

We wanted to let you know how the work is progressing, and that the Mockingbird speed issue is still our top priority… to the point where we have suspended all new subscriber installations to Mockingbird Tower until we have got it beat.

Thanks for your patience while we conquer this problem.

Mockingbird Performance Issues

Our performance problem with Mockingbird Tower unfortunately wasn’t affected by replacing the radio driver on Tuesday.

We’ve since spent two very late nights (until 4 AM) attempting to locate the true cause of this problem. We’ve checked for queueing and routing problems, limitations on link capacity, RF issues (such as improper equipment orientation, local interference, and — our perennial bugaboo — a subscriber with degrading signal strength whose retry requests overwhelm the tower and destroy performance for all other tower users). We’ve temporarily backed out recent configuration changes to see if any of them triggered the issue. Despite our efforts, the problem continues to defy explanation.

On Wednesday, we contracted with the largest MikroTik consulting firm in the US to look at this problem, as well as the challenge of delivering increased bandwidth to all our subscribers on all towers. We’ve been assured that their engineers will begin work on these tasks on Monday.

We’re sorry to have no interim fix available for this problem. As other potential causes suggest themselves to us, we will continue to test them, and either repair them or eliminate them from consideration. We want you to know that if you are a Mockingbird Tower subscriber, we are not ignoring you — we are aware of your speed issues and we are working on them.

Thank you for using Grand Avenue Broadband.

Mockingbird Interruption

Subscribers homed to Mockingbird Tower (both Central and West) will experience a short interruption of service tomorrow (Tuesday) morning, as we rotate the mast down to replace the radio and pigtails for the Mockingbird Central antenna in order to attempt to alleviate our recent performance issues. We will also be correcting the tower’s cant at this time.

Should the problem turn out to be in the physical antenna itself (a much lower probability, but possible), time will be required to locate and ship a replacement for this unit, which appears to be in short supply due to a manufacturer merger. If this is the case, we’ll keep you informed via this blog.

A Red Queen Marathon

After two nights of record windstorms, today (Sunday) was devoted to an impromptu “tower tour,” driving around our service area and checking for visual confirmation of proper aim on the majority of our towers.

The Easy Street input feed dish had been blown clockwise about 70°. (The Easy Street neighborhood itself was pretty slammed, with roads strewn with wash silt and downed trees; our host’s flagpole was actually bent in half by the winds!) Without a roof man, given that the dish was far above my reach, I attempted a rough re-aim job by “pokin’ a stick at it” (our 16′ telescoping pole) from a ladder, with surprisingly satisfactory results.

The output feed from the NOC to 251st Avenue had also been blown off kilter, and was re-aimed in the same manner. The mast has gone a bit wobbly over time, which will be addressed when we can schedule Dave, our roof man, to fine tune and lock in all of today’s crude readjustments.

One thing these storms made quite apparent is a need for more UPS power capability than we currently enjoy. Although the NOC, Constellation Tower, and our Wickenburg gateway are hardened by indoor UPS systems that provide them about 90 minutes of emergency power, most of our towers have no UPS. During the last two nights of power outages in central Morristown and Castle Hot Springs, all our Morristown tower subscribers were taken offline immediately because no power was available to our windmill tower—even though the NOC continued operating (and serving 251st Avenue, which suffered no extended power outage either night).

As funding allows, we plan to begin introducing indoor UPS systems at host-based towers, which is technically simple—weatherproof UPS units for the standalone towers (Morristown, Mockingbird, and Rio Vista) are expected to be more problematic, but a search for suitable equipment is on our immediate agenda.

In addition, it’s clear that the sheer amount of equipment installed at the NOC and the gateway locations deserves beefier UPS units that can provide more than 90 minutes of outage protection, since when these locations go dark, many or all towers lose service.

There’s nothing like weather problems to trigger a Red Queen Marathon: a situation where you must run as fast as you can just to stay in the same place. We’re going to be running one of our own over the next few months, to better prepare ourselves for future monsoon devilry.

(And the weatherman says there is 45% chance of a third record windstorm tonight.)  😭

 

Easy Street Tower Problems

The last two nights of record monsoon windstorms have caused us an unusual number of headaches.

The latest: the feed radio at the Easy Street neighborhood tower appears to have been blown a bit off aim by the recent high winds.  Easy Street subscribers may see slower than usual network performance until we can dispatch a technician to reaim the unit.

Broadband on the Bounce

As part of the process of analyzing a strange signal performance problem affecting a handful of Mockingbird tower subscribers, we have determined that the recent heat wave has actually warped the Mockingbird mast. The bend is causing eastbound signal to impact the terrain in Rio Vista Hills, creating destructive interference, increasing packet retries, and slowing down internet speeds to a few subscribers located on or near the 295th Avenue corridor in Rio Vista Hills and served by the Mockingbird tower. This problem does not affect other subscribers of Mockingbird Tower, nor does it affect subscribers in Rio Vista Hills served by the Rio Vista tower.

We are scheduling maintenance on the tower as heat forecasts and crew schedules allow. We hope to have the signal anomaly corrected by the middle to the end of next week.

Mockingbird, Rio Vista traffic affected

What appears to be a power supply problem at Mockingbird Tower is currently affecting network speeds to subscribers of Mockingbird West access point and Rio Vista Tower. Subscribers of Mockingbird Central access point are not currently affected, but may become so before the repair is scheduled. We will be running a repair crew out to the tower at around noon tomorrow.  Details follow. Continue reading