CenturyLink has informed us of a possible interruption of service of between one and three hours to our entire network, sometime between midnight and 6:00 AM on the morning of Tuesday, January 8. The interruption is for “scheduled maintenance to upgrade equipment in order to add network capacity and prevent future service interruptions to customer services.”
This far in advance of the scheduled date, there is a good chance that CenturyLink may reschedule this interruption one or more times before it actually occurs. We will report any changes to the scheduled date and time as notice is received.
We will be replacing the aging and increasingly problematic service radio atop Downtown Tower late tomorrow morning (Wednesday 10/10) with a new and faster unit. Expect an outage of 10-15 minutes as the old unit is unbolted and the new unit secured. This outage will affect only users served directly by Downtown Wickenburg Tower — subscribers served by all other towers will be unaffected.
We’re experiencing an issue with some of our towers in which the speed to subscribers degrades as equipment uptime increases. Towers identified so far have included Skip-In Ranch Tower, Downtown Tower, and all three APs on Morristown Tower. In all cases, power-cycling the radio unit restores full performance.
Our equipment manufacturer, MikroTik, suspects a memory leak in their OS, and has asked us to provide them with diagnostic dumps from the affected equipment, which we are in the process of assembling for them today.
While we wait for a response from MikroTik, we’ll be setting up an automatic powercycle schedule on the affected towers. This will result in an approximately four-minute interruption of service between 2:00-4:00 AM several times each week at these towers. We hope to have this periodic refresh cycle operational by Monday.
This morning’s trunk cutover failed to take place, with CenturyLink making the attempt to switch circuits but ultimately delivering no traffic. Next-tier engineers at CenturyLink are scheduled to be diagnosing the issue, and we’ll be setting up a new cutover time once they inform us they have the problem figured out.
Tomorrow morning (Thursday, August 23), between 6:00 and 7:30 AM, CenturyLink will be moving our gateway service (our trunk connection to the greater Internet) onto a different fiber link. We expect an outage blip of about 15-30 seconds during that time period, barring complications. If complications occur, we will revert to the original fiber link as expeditiously as possible and reschedule the cutover. We regret the short notice, but we received this scheduling information only this morning.
We placed an order for new armored power cabling for Mockingbird Tower late last Tuesday (UPS three-day) with a new cable dealer who loudly advertises “same-day shipping.” They didn’t actually ship our order until Friday evening (without telling us)… so the cable, originally planned to arrive Monday (Tuesday at latest), still hasn’t shown up and we don’t know what time it will arrive.
This has forced us to postpone today’s tower upgrade to next Wednesday, the 25th. With luck, we’ll be able to dodge monsoon power outages for one extra week.
Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.
Our faith was betrayed.
We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.
This week our tower bandwidth improvement project is for all practical purposes complete. Read on to learn about the improvements we’ve made to our performance in Wickenburg and Morristown… plus a little bit of nostalgia in celebration of our 11th anniversary. Continue reading →
Recently, we determined that our Downtown Wickenburg Tower was exhibiting impaired performance depending on how long it was continuously operational. While awaiting the manufacturer’s diagnosis, we put a script in place to reboot this unit in the wee hours every four days, to keep it operating at peak.
The manufacturer replied today with the suggestion that reinitializing the unit from scratch should return it to proper operation. We’ll be performing this procedure around noon on Tuesday (tomorrow). This will result in a service outage of 10-15 minutes, which will affect only users directly served by Downtown Tower. We hope this will result in improved performance for our Downtown subscribers.
We’ve determined from subscriber feedback and subsequent testing that the east sector of Morristown Tower is not functioning properly and is delivering insufficient bandwidth. We’ll be replacing that unit Wednesday morning sometime between 9:15 and 10:30. This will involve an outage of 10-15 minutes, that should affect only Morristown Tower subscribers east of the 267th Avenue alignment.