After last night’s testing, we suspect the most probable cause of the performance problem at Mockingbird is a bad customer roof unit radio hashing our tower. We’ve exhausted all the minimally invasive tests we can run. Starting tonight, we’re running a Big Sleep test on all our Mockingbird subscriber units. Continue reading
Condensed version: We’ve narrowed down the Mockingbird Tower problem to RF noise that spikes periodically. Our tower equipment and configuration are clean and validated. We now need to perform after-midnight testing over the next few days that will result in brief (10-20 minute) interruptions of service for a number of the subscribers on this tower.
Details and analysis follow for the technically curious. Continue reading
After several abortive ordering attempts (in which “in stock” on a website listing apparently translated to “let us check our warehouse” once the order was placed), we located a dealer with a replacement antenna actually in stock. We have been verbally assured that our replacement unit shipped yesterday, and are awaiting written confirmation including a tracking number. In the absence of nasty surprises, we plan to be replacing the Mockingbird central antenna this weekend.
We are also going to be subjecting the Rio Vista antenna (same manufacturer and model) to testing to determine if it shows any early symptoms of the same failure mode, in case a prophylactic replacement is indicated.
As of about an hour ago, we believe we have found our issue with Mockingbird Tower Central, and taken steps to provide immediate improvement. Although speeds are not at the level they were three weeks ago (due to suspected antenna hardware damage and pending part arrival and replacement), they should be greatly improved over the degraded performance seen over the past week. Please drop us an email if you are not seeing adequate performance for your equipment.
Technical details follow for the curious.
Today’s status for Mockingbird Tower subscribers…
Our performance problem with Mockingbird Tower unfortunately wasn’t affected by replacing the radio driver on Tuesday.
We’ve since spent two very late nights (until 4 AM) attempting to locate the true cause of this problem. We’ve checked for queueing and routing problems, limitations on link capacity, RF issues (such as improper equipment orientation, local interference, and — our perennial bugaboo — a subscriber with degrading signal strength whose retry requests overwhelm the tower and destroy performance for all other tower users). We’ve temporarily backed out recent configuration changes to see if any of them triggered the issue. Despite our efforts, the problem continues to defy explanation.
On Wednesday, we contracted with the largest MikroTik consulting firm in the US to look at this problem, as well as the challenge of delivering increased bandwidth to all our subscribers on all towers. We’ve been assured that their engineers will begin work on these tasks on Monday.
We’re sorry to have no interim fix available for this problem. As other potential causes suggest themselves to us, we will continue to test them, and either repair them or eliminate them from consideration. We want you to know that if you are a Mockingbird Tower subscriber, we are not ignoring you — we are aware of your speed issues and we are working on them.
Thank you for using Grand Avenue Broadband.
The last two nights of record monsoon windstorms have caused us an unusual number of headaches.
The latest: the feed radio at the Easy Street neighborhood tower appears to have been blown a bit off aim by the recent high winds. Easy Street subscribers may see slower than usual network performance until we can dispatch a technician to reaim the unit.
As part of the process of analyzing a strange signal performance problem affecting a handful of Mockingbird tower subscribers, we have determined that the recent heat wave has actually warped the Mockingbird mast. The bend is causing eastbound signal to impact the terrain in Rio Vista Hills, creating destructive interference, increasing packet retries, and slowing down internet speeds to a few subscribers located on or near the 295th Avenue corridor in Rio Vista Hills and served by the Mockingbird tower. This problem does not affect other subscribers of Mockingbird Tower, nor does it affect subscribers in Rio Vista Hills served by the Rio Vista tower.
We are scheduling maintenance on the tower as heat forecasts and crew schedules allow. We hope to have the signal anomaly corrected by the middle to the end of next week.
What appears to be a power supply problem at Mockingbird Tower is currently affecting network speeds to subscribers of Mockingbird West access point and Rio Vista Tower. Subscribers of Mockingbird Central access point are not currently affected, but may become so before the repair is scheduled. We will be running a repair crew out to the tower at around noon tomorrow. Details follow. Continue reading