Morristown upgrade Friday morning; possible Mockingbird upgrade Friday afternoon

Just a brief notification that the upgrade work to the WIckenburg/Morristown link is scheduled for tomorrow morning (Friday) between 9:00 and noon.

Depending entirely on time available, it is possible that similar upgrade work may be performed on the Wickenburg/Mockingbird link sometime in the afternoon, to provide even more bandwidth there. This would temporarily affect network performance to all the towers mentioned in our previous notification, with the addition of Mockingbird and Rio Vista towers.

Thank you for being a part of Grand Avenue Broadband.

 

Morristown Upgrade, Stage II

Tower climber

Dave installs Stage I…

The equipment needed for stage II of the Morristown bandwidth upgrade arrives this week, and we plan to be installing it on either Thursday or Friday morning.

This is the gear needed to complete the “full link” upgrade of our feed from Wickenburg, providing additional bandwidth to our east-end subscribers over and above the “half link” upgrade installed four weeks ago.

Subscribers served by east-end towers may see some short service burps and lower performance for roughly a 20-minute period sometime between the hours of 9:00 AM and noon, as our network compensates for the temporarily missing link by routing Wickenburg traffic over slower alternate routes.

Subscribers of these towers will be affected:

  • Morristown
  • Skip-In Ranch
  • Castle Hot Springs
  • 251st Avenue
  • Dixileta

We’ll post a brief notice like this one once we are certain of the day of installation.

Work continues Wednesday 20th

Last Wednesday’s installment in our bandwidth improvement effort surpassed our expectations. Despite the fact that we had only enough time to deploy half the planned link (due to the structural enhancements we had to add to the tower to handle the heavier equipment), we managed to increase bandwidth to Mockingbird tower significantly via its backup link, plus we determined that the leftover half of the high-speed link could be more effectively employed elsewhere. 

Tomorrow, we hope to be able to install improved equipment on the inbound side of both our Skip-In link (the main line to Morristown and all points east) and our primary link to Mockingbird Tower. As this involves some high-tower work, it will be slower, and some of the transitions may be less than smooth.

Although our existing redundant paths help ensure that your Internet availability will never be entirely severed, you may experience some “brownouts” as our system determines in real time that primary paths have gone black, and temporarily switches to secondary paths that will serve some of our towers somewhat more slowly. Short of unplanned disaster, the perturbations should not persist past 5 PM.

Thank you for your patience as we work to bring speedier Internet access to our ever-growing network of subscribers.

Possible Interruptions Wednesday 13th

Tomorrow afternoon (Wednesday the 13th) we will be upgrading equipment on Mockingbird Tower in conjunction with the first phase of our bandwidth improvement project. Subscribers homed to Mockingbird and Rio Vista towers may experience slower response and possibly some brief outages on the order of 10-15 minutes in the period between noon and 5 PM, as the tower may need to be rotated down out of service to replace radios at height. These interruptions should affect subscribers of those two towers only.

Problems affecting Mockingbird and Rio Vista Towers

Our downtown feed radio to Mockingbird Tower is now definitely misbehaving. Despite showing good signal and quality, it is not delivering anywhere near the bandwidth it should be. We suspect the unit is undergoing a slow failure mode.

This problem affects subscribers of Mockingbird and Rio Vista towers only.

We have switched to an easier frequency for the ailing radio to handle, and have also activated the backup link between our airport tower and Mockingbird in order to steer more bandwidth into Mockingbird and Rio Vista.

We already have equipment in transit, ordered earlier to replace this unit, scheduled to arrive Wednesday. We expect to have it configured and ready for deployment before Friday close of business. Some brief (~10 minutes) interruptions in service may occur near the end of the week as the new equipment is cut into service.

This situation is receiving our priority attention. We apologize for the degradation.

CenturyLink service interruption January 31

CenturyLink has informed us of a planned interruption of service of between one and three hours to our entire network, sometime between midnight and 6:00 AM on the morning of Thursday, January 31. The interruption is for “scheduled maintenance to upgrade routing engines in order to improve network reliability and to prevent future service interruptions to customer services.”

Occasionally, CenturyLink may reschedule this interruption one or more times before it actually occurs. We will report any changes to the scheduled date and time as notice is received.

Service interruption 1 AM tonight

We will be reinitializing and re-securing our main gateway router tonight, in an attempt to correct some strange observed behavior on its part that may possibly be due to either corruption or infiltration. We hope this may also relieve some of the prime-time slowdown behavior we’ve encountered within the past month or so.

We expect our connection to the Internet to be interrupted for about 30 minutes sometime within the window of 01:00 to 02:00 tonight (Sunday morning the 30th). As always, our recorded message at 623-640-7883 will be kept updated for you to verify the status of our network.

CenturyLink service interruption January 8

CenturyLink has informed us of a possible interruption of service of between one and three hours to our entire network, sometime between midnight and 6:00 AM on the morning of Tuesday, January 8. The interruption is for “scheduled maintenance to upgrade equipment in order to add network capacity and prevent future service interruptions to customer services.”

This far in advance of the scheduled date, there is a good chance that CenturyLink may reschedule this interruption one or more times before it actually occurs. We will report any changes to the scheduled date and time as notice is received.

Downtown Tower Upgrade

We will be replacing the aging and increasingly problematic service radio atop Downtown Tower late tomorrow morning (Wednesday 10/10) with a new and faster unit. Expect an outage of 10-15 minutes as the old unit is unbolted and the new unit secured. This outage will affect only users served directly by Downtown Wickenburg Tower — subscribers served by all other towers will be unaffected.

Tower slowdowns

We’re experiencing an issue with some of our towers in which the speed to subscribers degrades as equipment uptime increases. Towers identified so far have included Skip-In Ranch Tower, Downtown Tower, and all three APs on Morristown Tower.  In all cases, power-cycling the radio unit restores full performance.

Our equipment manufacturer, MikroTik, suspects a memory leak in their OS, and has asked us to provide them with diagnostic dumps from the affected equipment, which we are in the process of assembling for them today.

While we wait for a response from MikroTik, we’ll be setting up an automatic powercycle schedule on the affected towers. This will result in an approximately four-minute interruption of service between 2:00-4:00 AM several times each week at these towers. We hope to have this periodic refresh cycle operational by Monday.