We placed an order for new armored power cabling for Mockingbird Tower late last Tuesday (UPS three-day) with a new cable dealer who loudly advertises “same-day shipping.” They didn’t actually ship our order until Friday evening (without telling us)… so the cable, originally planned to arrive Monday (Tuesday at latest), still hasn’t shown up and we don’t know what time it will arrive.
This has forced us to postpone today’s tower upgrade to next Wednesday, the 25th. With luck, we’ll be able to dodge monsoon power outages for one extra week.
This Wednesday (7/18) around mid-day, we’re going to be upgrading the power supply and power cabling at Mockingbird Tower. We’re going to continue running on the existing cabling until we get everything laid out and ready to go, at which time we’ll cut over to the new power feed. If everything goes as planned, there will be no interruption in tower service at all; if it doesn’t, subscribers served directly by Mockingbird Tower might experience at most a two-minute outage as the tower resets. (No other subscribers will be affected by this maintenance.)
Part of the upgrade includes a battery backup system designed to be able to operate the tower for over an hour, to ride out any short-term monsoon-induced power outages that may affect Mockingbird Hill. In addition, relieving our power crunch on this tower should also help improve speeds to our Rio Vista tower subscribers.
Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.
Our faith was betrayed.
We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.
Recently, we determined that our Downtown Wickenburg Tower was exhibiting impaired performance depending on how long it was continuously operational. While awaiting the manufacturer’s diagnosis, we put a script in place to reboot this unit in the wee hours every four days, to keep it operating at peak.
The manufacturer replied today with the suggestion that reinitializing the unit from scratch should return it to proper operation. We’ll be performing this procedure around noon on Tuesday (tomorrow). This will result in a service outage of 10-15 minutes, which will affect only users directly served by Downtown Tower. We hope this will result in improved performance for our Downtown subscribers.
Since our last post, we’ve been hard at work on a new system to deliver more bandwidth to our four neighborhood towers in west Wickenburg, and to ramp our popular Mockingbird Tower up to the speeds necessary to support our increasing subscriber levels.
Subscribers of all our towers from Mockingbird Road westward may encounter some brief (5-20 minute) network interruptions this coming Wednesday at staggered times of the day, as we perform certain of the equipment upgrades described below.
Here are some details about what we’ve been doing, how far along we’ve gotten, and what we hope to accomplish Wednesday. Continue reading →
There’s about a 70% chance that we will have an opportunity this afternoon (Wednesday 1/31) to be able to perform an equipment upgrade at Diamondback Tower. Subscribers served by that tower may encounter several brief (about two minute) outages, along with possibly somewhat slower performance from the temporary mast we’ll be setting up to provide service while the original tower is being re-engineered. This work will affect only subscribers served by Diamondback Tower.
This week’s networking crisis is that black-hats have found another way to violate your privacy. Called the KRACK exploit, it allows hackers to spoof WiFi protocol so as to break its encryption and read your traffic.
It’s worth pointing out that this is not just a zero-day bug in some manufacturer’s implementation—it’s a defect in the WiFi standard itself, and all WiFi (802.11) encryption is vulnerable to it. It does require a hacker to be close enough to have physical access to your radio signal.
We wanted to let you know what we were doing about it, and also what you should be doing about it yourself. Continue reading →
Due to a recent real-estate transfer of the residence at which our 251st Avenue tower is located, we have been instructed to remove our tower.
We have already secured a commitment from a neighbor and long-time subscriber to assume the duties of neighborhood tower host, so we will be moving the tower to its new location during the day tomorrow (Wednesday). This will result in an outage of two to three hours for those subscribers and satellite towers served by 251st Avenue Tower, as the physical tower itself must be disconnected, detached, uprooted, transported, re-erected, cabled for power in its new location, and have its inter-tower feed links properly realigned.
After the relocation, we’ll be re-aiming the roof units of our closest-in subscribers first, then the farther-out subscribers as necessary. Our furthest-out subscribers in the vicinity of Circle City will likely require no re-aiming at all, though all signal levels will be compared before and after the relocation.
We apologize for this service interruption, and wish to assure you that we will be performing the relocation expeditiously, in order to resume service as promptly as possible.
After last night’s testing, we suspect the most probable cause of the performance problem at Mockingbird is a bad customer roof unit radio hashing our tower. We’ve exhausted all the minimally invasive tests we can run. Starting tonight, we’re running a Big Sleep test on all our Mockingbird subscriber units. Continue reading →