With the growth of our network and increasing frequency competition in Wickenburg, where our gateway is located, we’ve been experiencing some congestion issues delivering services to our subscribers east of the river at certain time periods. This congestion is most noticeable during what networkers call “the Netflix Hour,” or roughly 7:30 to 10:30 each night, and is especially troublesome for subscribers of our Mockingbird and main Morristown towers.
We suspect that this situation is being caused by performance limitations of the radio units running our “long links” eastward between Wickenburg and Morristown. However, despite applying all our standard remedies, we’ve been unable to make a dent in the network’s bottom-line performance; so we may be wrong about the basic cause, or the proper solution, or the issue may simply be an effect of multiple causes.
To address this issue, we’ve engaged a MikroTik consultant to analyze our network performance, identify problem areas, and propose solutions, which may involve anything from retuning to replacement of our current link equipment with other models, brands, or frequencies.
MikroTik consultants are like fairy godmothers: they can perform astonishing magic on your behalf, but first you need to find one who isn’t busy godmothering another Cinderella. We’ve been working since late November to get one on board to look at our network, and we’re happy to be able to announce that the bibbledy-bobbledy-boo is finally scheduled to begin this Thursday, the 24th.
Subscribers may experience temporary decreased performance issues or even minor outages as out consultant adjusts configuration settings and performs testing. Hopefully, there will be no major impact on your service while this work is in progress.
We’re experiencing an issue with some of our towers in which the speed to subscribers degrades as equipment uptime increases. Towers identified so far have included Skip-In Ranch Tower, Downtown Tower, and all three APs on Morristown Tower. In all cases, power-cycling the radio unit restores full performance.
Our equipment manufacturer, MikroTik, suspects a memory leak in their OS, and has asked us to provide them with diagnostic dumps from the affected equipment, which we are in the process of assembling for them today.
While we wait for a response from MikroTik, we’ll be setting up an automatic powercycle schedule on the affected towers. This will result in an approximately four-minute interruption of service between 2:00-4:00 AM several times each week at these towers. We hope to have this periodic refresh cycle operational by Monday.
Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.
Our faith was betrayed.
We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.
This week our tower bandwidth improvement project is for all practical purposes complete. Read on to learn about the improvements we’ve made to our performance in Wickenburg and Morristown… plus a little bit of nostalgia in celebration of our 11th anniversary. Continue reading →
Since our last post, we’ve been hard at work on a new system to deliver more bandwidth to our four neighborhood towers in west Wickenburg, and to ramp our popular Mockingbird Tower up to the speeds necessary to support our increasing subscriber levels.
Subscribers of all our towers from Mockingbird Road westward may encounter some brief (5-20 minute) network interruptions this coming Wednesday at staggered times of the day, as we perform certain of the equipment upgrades described below.
Here are some details about what we’ve been doing, how far along we’ve gotten, and what we hope to accomplish Wednesday. Continue reading →
There’s about a 70% chance that we will have an opportunity this afternoon (Wednesday 1/31) to be able to perform an equipment upgrade at Diamondback Tower. Subscribers served by that tower may encounter several brief (about two minute) outages, along with possibly somewhat slower performance from the temporary mast we’ll be setting up to provide service while the original tower is being re-engineered. This work will affect only subscribers served by Diamondback Tower.
As a result of a long-term project accomplished earlier this week, subscribers are now seeing faster speeds and improved response times. Most subscribers are experiencing sustained speeds exceeding their contract limits (some are seeing double their contract limit), with an additional speed boost layered on top whenever possible for brief, non-sustained transactions such as web surfing. Continue reading →
I’m happy to report that, as far as we have been able to determine, all our speed-related problems at Mockingbird Tower are now resolved. Assuming this announcement doesn’t generate a spate of contrary responses, our next task will be to arrange for incremental UPS protection at our towers, beginning with the ones that are most highly subscribed; and working with our consultants to assure that we can deliver to our subscribers all the untapped bandwidth currently available at our Internet gateway.
For the technically curious, a brief description of the source of our speed problems appears below. (Warning: unpaved roads ahead!) Continue reading →
We found and corrected the source of the frequent four-second timeout pauses. Mockingbird subscribers will now at least be seeing consistent data delivery, just not yet at the speeds we would like to deliver. What the tower is delivering right now should be fine for browsing, mail, and even a Netflix movie, but heavy data demands like Windows/Mac updates will still take longer than usual.
We are continuing to work on the speed problem with our dedicated and knowledgeable consultants.
After last night’s testing, we suspect the most probable cause of the performance problem at Mockingbird is a bad customer roof unit radio hashing our tower. We’ve exhausted all the minimally invasive tests we can run. Starting tonight, we’re running a Big Sleep test on all our Mockingbird subscriber units. Continue reading →