After “rejuvenating” the queues on the bulk of our towers by hand yesterday, our bandwidth usage at peak hour reached a new all-time high of 25% more than our previous record. This is gateway bandwidth that has always been “available” for users, and we’re very pleased now to see that it has at last become accessible.
We’ve also installed the scripting code to perform automated queue rejuvenation monthly on all towers, so this OS bug should not be silently biting us ever again.
Recently, we’ve been buffaloed by a handful of problems where subscribers were reporting severely inadequate bandwidth, despite our active metering showing oodles of bandwidth available all the way down to their in-house WiFi units. We may have identified the cause. Continue reading →
We added significant bandwidth to feed Skip-In Ranch and all the towers in Morristown; plus additional bandwidth to Mockingbird and Rio Vista towers; plus increased bandwidth between the Morristown hub and the 251st Avenue and Castle Hot Springs towers.
In addition, we installed a new northward-facing AP on the 251st Avenue tower to serve the growing number of subscribers located on the east side of Castle Hot Springs Road who can’t access their nearest neighborhood tower (Castle Hot Springs) due to an intervening hill. These subscribers are now benefiting from 8-10dB better signal (about eight times the previous signal), as well as the additional bandwidth available at their tower.
We also corrected a mysterious configuration gremlin that was causing decreased signal to our subscribers in Grantham Ranch.
We hope today’s work will make your Internet experience more enjoyable.
The equipment needed for stage II of the Morristown bandwidth upgrade arrives this week, and we plan to be installing it on either Thursday or Friday morning.
This is the gear needed to complete the “full link” upgrade of our feed from Wickenburg, providing additional bandwidth to our east-end subscribers over and above the “half link” upgrade installed four weeks ago.
Subscribers served by east-end towers may see some short service burps and lower performance for roughly a 20-minute period sometime between the hours of 9:00 AM and noon, as our network compensates for the temporarily missing link by routing Wickenburg traffic over slower alternate routes.
Subscribers of these towers will be affected:
Castle Hot Springs
We’ll post a brief notice like this one once we are certain of the day of installation.
Our recent project to improve the bandwidth of both the Mockingbird/Rio Vista and Skip-In Ranch/Morristown backhaul routes has been successful and gratifying. Installation of improved equipment on the Wickenburg end of both links turned out to be sufficient to reach our immediate target, leaving a future opportunity for further improvement when necessary by installing matching upgrades at the far ends of both links.
Today’s project to improve signal and bandwidth to our newer customers in Grantham Ranch and the Rincon corridor east of the river was also particularly successful. That area is now served by a new dedicated directional access point located at our downtown gateway.
In spite of all the technical challenges, we are continually installing improvements to help keep our Internet service one you would be proud to recommend. Thank you for your patience and for your support.
Tomorrow afternoon (Wednesday the 13th) we will be upgrading equipment on Mockingbird Tower in conjunction with the first phase of our bandwidth improvement project. Subscribers homed to Mockingbird and Rio Vista towers may experience slower response and possibly some brief outages on the order of 10-15 minutes in the period between noon and 5 PM, as the tower may need to be rotated down out of service to replace radios at height. These interruptions should affect subscribers of those two towers only.
Our downtown feed radio to Mockingbird Tower is now definitely misbehaving. Despite showing good signal and quality, it is not delivering anywhere near the bandwidth it should be. We suspect the unit is undergoing a slow failure mode.
This problem affects subscribers of Mockingbird and Rio Vista towers only.
We have switched to an easier frequency for the ailing radio to handle, and have also activated the backup link between our airport tower and Mockingbird in order to steer more bandwidth into Mockingbird and Rio Vista.
We already have equipment in transit, ordered earlier to replace this unit, scheduled to arrive Wednesday. We expect to have it configured and ready for deployment before Friday close of business. Some brief (~10 minutes) interruptions in service may occur near the end of the week as the new equipment is cut into service.
This situation is receiving our priority attention. We apologize for the degradation.
With the growth of our network and increasing frequency competition in Wickenburg, where our gateway is located, we’ve been experiencing some congestion issues delivering services to our subscribers east of the river at certain time periods. This congestion is most noticeable during what networkers call “the Netflix Hour,” or roughly 7:30 to 10:30 each night, and is especially troublesome for subscribers of our Mockingbird and main Morristown towers.
We suspect that this situation is being caused by performance limitations of the radio units running our “long links” eastward between Wickenburg and Morristown. However, despite applying all our standard remedies, we’ve been unable to make a dent in the network’s bottom-line performance; so we may be wrong about the basic cause, or the proper solution, or the issue may simply be an effect of multiple causes.
To address this issue, we’ve engaged a MikroTik consultant to analyze our network performance, identify problem areas, and propose solutions, which may involve anything from retuning to replacement of our current link equipment with other models, brands, or frequencies.
MikroTik consultants are like fairy godmothers: they can perform astonishing magic on your behalf, but first you need to find one who isn’t busy godmothering another Cinderella. We’ve been working since late November to get one on board to look at our network, and we’re happy to be able to announce that the bibbledy-bobbledy-boo is finally scheduled to begin this Thursday, the 24th.
Subscribers may experience temporary decreased performance issues or even minor outages as out consultant adjusts configuration settings and performs testing. Hopefully, there will be no major impact on your service while this work is in progress.
Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.
Our faith was betrayed.
We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.