Subscribers served by Constellation Tower (this includes subscribers served by Diamondback, County Line, Black Mountain, and Easy Street Towers) will experience several short micro-outages around mid-day, as equipment and cables are swapped to upgrade Constellation Tower to gigabit bandwidths. The outages will affect almost all Wickenburg subscribers west of Burger King (except for those served directly by Mockingbird and Downtown Towers), and should last at most a minute or two; several micro-outages will occur before work is complete.
Watch for a posting later on this week in which we will discuss the infrastructure improvements we’ve been making over the past month to provide you faster service.
As a result of a long-term project accomplished earlier this week, subscribers are now seeing faster speeds and improved response times. Most subscribers are experiencing sustained speeds exceeding their contract limits (some are seeing double their contract limit), with an additional speed boost layered on top whenever possible for brief, non-sustained transactions such as web surfing. Continue reading
I’m happy to report that, as far as we have been able to determine, all our speed-related problems at Mockingbird Tower are now resolved. Assuming this announcement doesn’t generate a spate of contrary responses, our next task will be to arrange for incremental UPS protection at our towers, beginning with the ones that are most highly subscribed; and working with our consultants to assure that we can deliver to our subscribers all the untapped bandwidth currently available at our Internet gateway.
For the technically curious, a brief description of the source of our speed problems appears below. (Warning: unpaved roads ahead!) Continue reading
The battle continues on the Mockingbird speed issue.
In the past two days, our consultant pointed us towards two problems that allowed us to more than double the bandwidth capacity between our gateway and the Mockingbird Tower. Maddeningly, this increased bandwidth availability is still not making its way out to actual tower subscribers, and we are still at a loss as to why not. (Note that this is not happening on our other towers.) Our next step is to install some more sophisticated metering software at our gateway for our consultant to use to analyze traffic and routing issues within our network.
Today the new, faster, weatherized router model we’ve been awaiting (for months) to order, to replace the mid-2013 board currently operating Mockingbird Tower, finally hit US distributors’ shelves. We express-ordered one, which should arrive late on Friday. We plan to get it installed over the weekend, possibly solving our problem in whole or in part. (At the very worst, it will allow us to rule out a router hardware problem.) This installation will involve a brief outage, as cables have to be moved en masse from one unit to the other.
We wanted to let you know how the work is progressing, and that the Mockingbird speed issue is still our top priority… to the point where we have suspended all new subscriber installations to Mockingbird Tower until we have got it beat.
Thanks for your patience while we conquer this problem.
Subscribers homed to Mockingbird Tower (both Central and West) will experience a short interruption of service tomorrow (Tuesday) morning, as we rotate the mast down to replace the radio and pigtails for the Mockingbird Central antenna in order to attempt to alleviate our recent performance issues. We will also be correcting the tower’s cant at this time.
Should the problem turn out to be in the physical antenna itself (a much lower probability, but possible), time will be required to locate and ship a replacement for this unit, which appears to be in short supply due to a manufacturer merger. If this is the case, we’ll keep you informed via this blog.
In a week or three, CenturyLink should be completing our recent order to double our gateway speed, resulting in our having more than three times the network bandwidth we had only one month ago.
Unfortunately for us, just having all this new speed available at the gateway doesn’t automatically make it accessible to all our subscribers.
For the past several weeks, a phrase has been looping through my mind from Larry Niven’s “A World Out of Time.” Tasked with a 200-year intergalactic voyage, the bulk of which he must spend in cryogenic stasis, the story’s hero is subjected to a rigorous dietary regimen to ensure his surviving the journey: before each cold-sleep session, it is crucial for him to “grow fat… and exercise to distribute the fat.”
This passage echoes in my head because it’s a perfect description of our current challenge. Continue reading
Our network will experience one or more brief outages between 7:30 and 9:30 AM tomorrow (Friday, 1/6), as CenturyLink pulls the trigger on a major speed increase for our service. Continue reading
Last Friday, our consultant told us he couldn’t commit for sure as to whether the limiting factor in the speed of our gateway was CenturyLink’s equipment or our own, and recommended we upgrade to beefier equipment just to get off the edge of doubt. So after a flurry of express equipment orders, we now have one of MikroTik’s burliest routers (nine cores, rated for 28Gbps), fully configured and in position at our gateway, ready to accept a cutover of our new double-speed circuit.
Tomorrow, we plan to recable our existing tower equipment into the new router (a rigorous task, involving tall ladders and large hole saws). Because our network is configured as a giant ring, we can disconnect and transfer one leg at a time, which should allow us to cut over the bulk of our subscribers with no more than a 30-second service interruption.
Those few subscribers being served directly by our downtown Wickenburg AP on the gateway tower itself will be an exception, as there is only a single path cable that we must disconnect, reroute, and reconnect. We hope to keep their outage period down to no more than 15 minutes.
When we’re done, we will either be operating at our full gateway capacity, or we won’t. In the latter case we already have a come-together meeting scheduled with CenturyLink early Monday morning.
Wish us godspeed.
If you’ve been wondering for the last week if this is as good as it’s going to get, it isn’t — far from it.
Our new speed increase turned out to be not as plug-and-play as it should have been. We’re getting only about a 10-25% speed increase during peak periods, far short of the 100% increase we ordered. This is true even when we test the circuit directly without any portion of our private network involved. At this point, it appears the speed limitation is occurring somewhere in CenturyLink’s network.
Although we are currently operating on the new fiber circuit, because it’s marginally faster than our old circuit, we have not accepted the cutover as compliant with our order. We are working with CenturyLink as well as an independent consultant to isolate and resolve the speed issues. Our progress has been slowed somewhat by the unavailability of CenturyLink personnel to work this problem.
This speed improvement is our highest-priority issue. We’ll put out short bulletins as necessary, to keep you informed.
We’re scheduling a brief interruption in service this Friday morning (January 8) between 6:00 and 7:00 AM, to execute a cutover to our brand new fiber circuit — replacing our old copper circuit with twice the bandwidth, to feed our growing network and reduce peak-hour congestion.
Since this is only CenturyLink’s second fiber installation in all Wickenburg, we’re crossing our fingers that the cutover will go smoothly with no false starts. If so, we expect to have our internal network tuned to utilize the new speed fully by 11:00 AM or so.
We appreciate your patience in waiting for this long-overdue upgrade, and we will continue to work hard to make sure you enjoy a zippier and more exhilarating Internet experience.