With the growth of our network and increasing frequency competition in Wickenburg, where our gateway is located, we’ve been experiencing some congestion issues delivering services to our subscribers east of the river at certain time periods. This congestion is most noticeable during what networkers call “the Netflix Hour,” or roughly 7:30 to 10:30 each night, and is especially troublesome for subscribers of our Mockingbird and main Morristown towers.
We suspect that this situation is being caused by performance limitations of the radio units running our “long links” eastward between Wickenburg and Morristown. However, despite applying all our standard remedies, we’ve been unable to make a dent in the network’s bottom-line performance; so we may be wrong about the basic cause, or the proper solution, or the issue may simply be an effect of multiple causes.
To address this issue, we’ve engaged a MikroTik consultant to analyze our network performance, identify problem areas, and propose solutions, which may involve anything from retuning to replacement of our current link equipment with other models, brands, or frequencies.
MikroTik consultants are like fairy godmothers: they can perform astonishing magic on your behalf, but first you need to find one who isn’t busy godmothering another Cinderella. We’ve been working since late November to get one on board to look at our network, and we’re happy to be able to announce that the bibbledy-bobbledy-boo is finally scheduled to begin this Thursday, the 24th.
Subscribers may experience temporary decreased performance issues or even minor outages as out consultant adjusts configuration settings and performs testing. Hopefully, there will be no major impact on your service while this work is in progress.
We will be reinitializing and re-securing our main gateway router tonight, in an attempt to correct some strange observed behavior on its part that may possibly be due to either corruption or infiltration. We hope this may also relieve some of the prime-time slowdown behavior we’ve encountered within the past month or so.
We expect our connection to the Internet to be interrupted for about 30 minutes sometime within the window of 01:00 to 02:00 tonight (Sunday morning the 30th). As always, our recorded message at 623-640-7883 will be kept updated for you to verify the status of our network.
CenturyLink has informed us of a possible interruption of service of between one and three hours to our entire network, sometime between midnight and 6:00 AM on the morning of Tuesday, January 8. The interruption is for “scheduled maintenance to upgrade equipment in order to add network capacity and prevent future service interruptions to customer services.”
This far in advance of the scheduled date, there is a good chance that CenturyLink may reschedule this interruption one or more times before it actually occurs. We will report any changes to the scheduled date and time as notice is received.
We will be replacing the aging and increasingly problematic service radio atop Downtown Tower late tomorrow morning (Wednesday 10/10) with a new and faster unit. Expect an outage of 10-15 minutes as the old unit is unbolted and the new unit secured. This outage will affect only users served directly by Downtown Wickenburg Tower — subscribers served by all other towers will be unaffected.
We’re experiencing an issue with some of our towers in which the speed to subscribers degrades as equipment uptime increases. Towers identified so far have included Skip-In Ranch Tower, Downtown Tower, and all three APs on Morristown Tower. In all cases, power-cycling the radio unit restores full performance.
Our equipment manufacturer, MikroTik, suspects a memory leak in their OS, and has asked us to provide them with diagnostic dumps from the affected equipment, which we are in the process of assembling for them today.
While we wait for a response from MikroTik, we’ll be setting up an automatic powercycle schedule on the affected towers. This will result in an approximately four-minute interruption of service between 2:00-4:00 AM several times each week at these towers. We hope to have this periodic refresh cycle operational by Monday.
We placed an order for new armored power cabling for Mockingbird Tower late last Tuesday (UPS three-day) with a new cable dealer who loudly advertises “same-day shipping.” They didn’t actually ship our order until Friday evening (without telling us)… so the cable, originally planned to arrive Monday (Tuesday at latest), still hasn’t shown up and we don’t know what time it will arrive.
This has forced us to postpone today’s tower upgrade to next Wednesday, the 25th. With luck, we’ll be able to dodge monsoon power outages for one extra week.
This Wednesday (7/18) around mid-day, we’re going to be upgrading the power supply and power cabling at Mockingbird Tower. We’re going to continue running on the existing cabling until we get everything laid out and ready to go, at which time we’ll cut over to the new power feed. If everything goes as planned, there will be no interruption in tower service at all; if it doesn’t, subscribers served directly by Mockingbird Tower might experience at most a two-minute outage as the tower resets. (No other subscribers will be affected by this maintenance.)
Part of the upgrade includes a battery backup system designed to be able to operate the tower for over an hour, to ride out any short-term monsoon-induced power outages that may affect Mockingbird Hill. In addition, relieving our power crunch on this tower should also help improve speeds to our Rio Vista tower subscribers.
Recently, we determined that our Downtown Wickenburg Tower was exhibiting impaired performance depending on how long it was continuously operational. While awaiting the manufacturer’s diagnosis, we put a script in place to reboot this unit in the wee hours every four days, to keep it operating at peak.
The manufacturer replied today with the suggestion that reinitializing the unit from scratch should return it to proper operation. We’ll be performing this procedure around noon on Tuesday (tomorrow). This will result in a service outage of 10-15 minutes, which will affect only users directly served by Downtown Tower. We hope this will result in improved performance for our Downtown subscribers.
We’ve determined from subscriber feedback and subsequent testing that the east sector of Morristown Tower is not functioning properly and is delivering insufficient bandwidth. We’ll be replacing that unit Wednesday morning sometime between 9:15 and 10:30. This will involve an outage of 10-15 minutes, that should affect only Morristown Tower subscribers east of the 267th Avenue alignment.
Since our last post, we’ve been hard at work on a new system to deliver more bandwidth to our four neighborhood towers in west Wickenburg, and to ramp our popular Mockingbird Tower up to the speeds necessary to support our increasing subscriber levels.
Subscribers of all our towers from Mockingbird Road westward may encounter some brief (5-20 minute) network interruptions this coming Wednesday at staggered times of the day, as we perform certain of the equipment upgrades described below.
Here are some details about what we’ve been doing, how far along we’ve gotten, and what we hope to accomplish Wednesday. Continue reading →