Morristown upgrade Friday morning; possible Mockingbird upgrade Friday afternoon

Just a brief notification that the upgrade work to the WIckenburg/Morristown link is scheduled for tomorrow morning (Friday) between 9:00 and noon.

Depending entirely on time available, it is possible that similar upgrade work may be performed on the Wickenburg/Mockingbird link sometime in the afternoon, to provide even more bandwidth there. This would temporarily affect network performance to all the towers mentioned in our previous notification, with the addition of Mockingbird and Rio Vista towers.

Thank you for being a part of Grand Avenue Broadband.

 

Morristown Upgrade, Stage II

Tower climber

Dave installs Stage I…

The equipment needed for stage II of the Morristown bandwidth upgrade arrives this week, and we plan to be installing it on either Thursday or Friday morning.

This is the gear needed to complete the “full link” upgrade of our feed from Wickenburg, providing additional bandwidth to our east-end subscribers over and above the “half link” upgrade installed four weeks ago.

Subscribers served by east-end towers may see some short service burps and lower performance for roughly a 20-minute period sometime between the hours of 9:00 AM and noon, as our network compensates for the temporarily missing link by routing Wickenburg traffic over slower alternate routes.

Subscribers of these towers will be affected:

  • Morristown
  • Skip-In Ranch
  • Castle Hot Springs
  • 251st Avenue
  • Dixileta

We’ll post a brief notice like this one once we are certain of the day of installation.

Work continues Wednesday 20th

Last Wednesday’s installment in our bandwidth improvement effort surpassed our expectations. Despite the fact that we had only enough time to deploy half the planned link (due to the structural enhancements we had to add to the tower to handle the heavier equipment), we managed to increase bandwidth to Mockingbird tower significantly via its backup link, plus we determined that the leftover half of the high-speed link could be more effectively employed elsewhere. 

Tomorrow, we hope to be able to install improved equipment on the inbound side of both our Skip-In link (the main line to Morristown and all points east) and our primary link to Mockingbird Tower. As this involves some high-tower work, it will be slower, and some of the transitions may be less than smooth.

Although our existing redundant paths help ensure that your Internet availability will never be entirely severed, you may experience some “brownouts” as our system determines in real time that primary paths have gone black, and temporarily switches to secondary paths that will serve some of our towers somewhat more slowly. Short of unplanned disaster, the perturbations should not persist past 5 PM.

Thank you for your patience as we work to bring speedier Internet access to our ever-growing network of subscribers.

Performance improvement cycle begins Thursday

With the growth of our network and increasing frequency competition in Wickenburg, where our gateway is located, we’ve been experiencing some congestion issues delivering services to our subscribers east of the river at certain time periods. This congestion is most noticeable during what networkers call “the Netflix Hour,” or roughly 7:30 to 10:30 each night, and is especially troublesome for subscribers of our Mockingbird and main Morristown towers.

We suspect that this situation is being caused by performance limitations of the radio units running our “long links” eastward between Wickenburg and Morristown. However, despite applying all our standard remedies, we’ve been unable to make a dent in the network’s bottom-line performance; so we may be wrong about the basic cause, or the proper solution, or the issue may simply be an effect of multiple causes.

To address this issue, we’ve engaged a MikroTik consultant to analyze our network performance, identify problem areas, and propose solutions, which may involve anything from retuning to replacement of our current link equipment with other models, brands, or frequencies. 

MikroTik consultants are like fairy godmothers: they can perform astonishing magic on your behalf, but first you need to find one who isn’t busy godmothering another Cinderella. We’ve been working since late November to get one on board to look at our network, and we’re happy to be able to announce that the bibbledy-bobbledy-boo is finally scheduled to begin this Thursday, the 24th.

Subscribers may experience temporary decreased performance issues or even minor outages as out consultant adjusts configuration settings and performs testing. Hopefully, there will be no major impact on your service while this work is in progress.

Tower slowdowns

We’re experiencing an issue with some of our towers in which the speed to subscribers degrades as equipment uptime increases. Towers identified so far have included Skip-In Ranch Tower, Downtown Tower, and all three APs on Morristown Tower.  In all cases, power-cycling the radio unit restores full performance.

Our equipment manufacturer, MikroTik, suspects a memory leak in their OS, and has asked us to provide them with diagnostic dumps from the affected equipment, which we are in the process of assembling for them today.

While we wait for a response from MikroTik, we’ll be setting up an automatic powercycle schedule on the affected towers. This will result in an approximately four-minute interruption of service between 2:00-4:00 AM several times each week at these towers. We hope to have this periodic refresh cycle operational by Monday.

Yes, we’re working on performance problems

Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.

Our faith was betrayed.

We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.

A great week,
 and a great 11 years

This week our tower bandwidth improvement project is for all practical purposes complete. Read on to learn about the improvements we’ve made to our performance in Wickenburg and Morristown… plus a little bit of nostalgia in celebration of our 11th anniversary. Continue reading

Wednesday: interruptions for performance improvements

Since our last post, we’ve been hard at work on a new system to deliver more bandwidth to our four neighborhood towers in west Wickenburg, and to ramp our popular Mockingbird Tower up to the speeds necessary to support our increasing subscriber levels.

Subscribers of all our towers from Mockingbird Road westward may encounter some brief (5-20 minute) network interruptions this coming Wednesday at staggered times of the day, as we perform certain of the equipment upgrades described below.

Here are some details about what we’ve been doing, how far along we’ve gotten, and what we hope to accomplish Wednesday. Continue reading

Possible Diamondback work today

There’s about a 70% chance that we will have an opportunity this afternoon (Wednesday 1/31) to be able to perform an equipment upgrade at Diamondback Tower. Subscribers served by that tower may encounter several brief (about two minute) outages, along with possibly somewhat slower performance from the temporary mast we’ll be setting up to provide service while the original tower is being re-engineered. This work will affect only subscribers served by Diamondback Tower.