The battle continues on the Mockingbird speed issue.
In the past two days, our consultant pointed us towards two problems that allowed us to more than double the bandwidth capacity between our gateway and the Mockingbird Tower. Maddeningly, this increased bandwidth availability is still not making its way out to actual tower subscribers, and we are still at a loss as to why not. (Note that this is not happening on our other towers.) Our next step is to install some more sophisticated metering software at our gateway for our consultant to use to analyze traffic and routing issues within our network.
Today the new, faster, weatherized router model we’ve been awaiting (for months) to order, to replace the mid-2013 board currently operating Mockingbird Tower, finally hit US distributors’ shelves. We express-ordered one, which should arrive late on Friday. We plan to get it installed over the weekend, possibly solving our problem in whole or in part. (At the very worst, it will allow us to rule out a router hardware problem.) This installation will involve a brief outage, as cables have to be moved en masse from one unit to the other.
We wanted to let you know how the work is progressing, and that the Mockingbird speed issue is still our top priority… to the point where we have suspended all new subscriber installations to Mockingbird Tower until we have got it beat.
Thanks for your patience while we conquer this problem.
Subscribers homed to Mockingbird Tower (both Central and West) will experience a short interruption of service tomorrow (Tuesday) morning, as we rotate the mast down to replace the radio and pigtails for the Mockingbird Central antenna in order to attempt to alleviate our recent performance issues. We will also be correcting the tower’s cant at this time.
Should the problem turn out to be in the physical antenna itself (a much lower probability, but possible), time will be required to locate and ship a replacement for this unit, which appears to be in short supply due to a manufacturer merger. If this is the case, we’ll keep you informed via this blog.
In a week or three, CenturyLink should be completing our recent order to double our gateway speed, resulting in our having more than three times the network bandwidth we had only one month ago.
Unfortunately for us, just having all this new speed available at the gateway doesn’t automatically make it accessible to all our subscribers.
For the past several weeks, a phrase has been looping through my mind from Larry Niven’s “A World Out of Time.” Tasked with a 200-year intergalactic voyage, the bulk of which he must spend in cryogenic stasis, the story’s hero is subjected to a rigorous dietary regimen to ensure his surviving the journey: before each cold-sleep session, it is crucial for him to “grow fat… and exercise to distribute the fat.”
This passage echoes in my head because it’s a perfect description of our current challenge. Continue reading
Our network will experience one or more brief outages between 7:30 and 9:30 AM tomorrow (Friday, 1/6), as CenturyLink pulls the trigger on a major speed increase for our service. Continue reading
Last Friday, our consultant told us he couldn’t commit for sure as to whether the limiting factor in the speed of our gateway was CenturyLink’s equipment or our own, and recommended we upgrade to beefier equipment just to get off the edge of doubt. So after a flurry of express equipment orders, we now have one of MikroTik’s burliest routers (nine cores, rated for 28Gbps), fully configured and in position at our gateway, ready to accept a cutover of our new double-speed circuit.
Tomorrow, we plan to recable our existing tower equipment into the new router (a rigorous task, involving tall ladders and large hole saws). Because our network is configured as a giant ring, we can disconnect and transfer one leg at a time, which should allow us to cut over the bulk of our subscribers with no more than a 30-second service interruption.
Those few subscribers being served directly by our downtown Wickenburg AP on the gateway tower itself will be an exception, as there is only a single path cable that we must disconnect, reroute, and reconnect. We hope to keep their outage period down to no more than 15 minutes.
When we’re done, we will either be operating at our full gateway capacity, or we won’t. In the latter case we already have a come-together meeting scheduled with CenturyLink early Monday morning.
Wish us godspeed.
If you’ve been wondering for the last week if this is as good as it’s going to get, it isn’t — far from it.
Our new speed increase turned out to be not as plug-and-play as it should have been. We’re getting only about a 10-25% speed increase during peak periods, far short of the 100% increase we ordered. This is true even when we test the circuit directly without any portion of our private network involved. At this point, it appears the speed limitation is occurring somewhere in CenturyLink’s network.
Although we are currently operating on the new fiber circuit, because it’s marginally faster than our old circuit, we have not accepted the cutover as compliant with our order. We are working with CenturyLink as well as an independent consultant to isolate and resolve the speed issues. Our progress has been slowed somewhat by the unavailability of CenturyLink personnel to work this problem.
This speed improvement is our highest-priority issue. We’ll put out short bulletins as necessary, to keep you informed.
We’re scheduling a brief interruption in service this Friday morning (January 8) between 6:00 and 7:00 AM, to execute a cutover to our brand new fiber circuit — replacing our old copper circuit with twice the bandwidth, to feed our growing network and reduce peak-hour congestion.
Since this is only CenturyLink’s second fiber installation in all Wickenburg, we’re crossing our fingers that the cutover will go smoothly with no false starts. If so, we expect to have our internal network tuned to utilize the new speed fully by 11:00 AM or so.
We appreciate your patience in waiting for this long-overdue upgrade, and we will continue to work hard to make sure you enjoy a zippier and more exhilarating Internet experience.
Back in June, we realized that our network was growing so aggressively that we needed to make immediate plans to increase our bandwidth to the greater Internet.
After assessing our options, and losing a bid to gain our additional bandwidth at a tower on the Morristown end of our network (which we would have preferred just for the redundancy), we placed an order in mid-July with CenturyLink, our current bandwidth provider, to double our gateway bandwidth in Wickenburg as soon as possible.
It took CenturyLink until late August even to give us an installation completion target date: October 27. The date was a source of anxiety to us, as we expected a much more timely completion and feared that our subscribers might well be “hitting the wall” by that time.
The October estimate turned out to be a pleasant fantasy, as after October 8 all CenturyLink progress came to an abrupt halt. As of last week (Christmas), dirt was still being pushed around at various downtown intersections in an attempt to get our new fiber laid. Meanwhile, our subscribers began topping out our gateway during the “Netflix Hour” by the last week of October, and throughout much of the rest of the day a month after that.
Today (New Year’s Eve), we were informed by CenturyLink that their fiber laying and splicing work is finally complete, and now we are to wait to be contacted by a technician “sometime in the next few days” to schedule equipment installation.
Here’s fervently hoping we will be able to wave goodbye to our bandwidth congestion problems by mid-January. We apologize for any sluggish response times you have been encountering, and we appreciate your patience while we work this upgrade to completion.
Our new equipment at Rio Vista Tower, like our new equipment at Mockingbird tower, has been meeting and exceeding all our expectations. We’re seeing greatly improved signal strength and effective bandwidth to every one of our subscribers on those towers.
According to CenturyLink’s schedule (and supported by visual verification today), new fiber has been pulled to the utility pole nearest our Wickenburg gateway. The next week or so will be spent establishing a physical service entrance to the building, and configuring and testing the new CenturyLink fiber equipment. We are hoping to be able to cut over to the new gateway sometime during the first full week of October. This gateway will provide us twice the bandwidth to the greater Internet as we currently have, and will be expandable for even more speed beyond that. We’re looking forward to this improvement, and we know you are too!
Thank you for using Grand Avenue Broadband.
Our consultant in Latvia did a dynamite job last week on our long-link radios. They’re now operating at a comfortable multiple of the new gateway speed on order from CenturyLink. (We still have a final link radio to replace on our downtown Wickenburg gateway tower to obtain an even stronger link, and we hope to be able to complete that task this coming weekend.)
The next set of equipment planned for some upgrade lovin’ is the Rio Vista Hills service tower. Continue reading