Update – Limited service disruption on Constellation

Thursday August 8th 2019, 11:17pm MST

Our engineer has just arrived back from the work on Constellation that was affected our 6 subscribers. These 6 customers are back online, however the performance and throughput is limited. We are researching possible frequency interference and power output settings on the new device to compensate and correct, and will continue to work towards a permanent resolution.

Updates will continue to be provided here until the problem is fully resolved.

Update – Limited service disruption on Constellation

Wednesday August 7th 2019, 10:20am MST

Just a quick update for our 6 subscribers on Constellation which are experiencing issues. Last night around 9pm we identified a few likely components which could be causing this limited disruption, and we ordered some replacement parts to be sent priority overnight. We are anticipating arrival of the shipment today, and will configure and install as soon as everything arrives.

Limited service disruption on Constellation

Wednesday August 7th 2019, 8:05pm MST

We currently have about 6 customers on Constellation which have not come back online after the issues this morning. The issue seems to be isolated to these 6 subscribers, and there does not seem to be any performance or congestion overall with other equipment. With dozens of other customers successfully connected to Constellation, we are working through the possible variables to determine a root cause for these isolated disconnections.

We will keep you updated with new developments, although currently an onsite visit with subscribers and/or to the tower tomorrow morning is looking likely.

Resolved – network and link issues resolved

Wednesday August 7th 2019, 1:35pm MST

First, I want to thank everyone for your patience and understanding this morning during our network issues. The power related issue last night presented itself again this morning, in a pretty catastrophic way. The details are below, but the short version is that we have replaced the faulty power injectors and divided the load between multiple sources.

The long version: Many (most) of our radios have variable signal outputs, which is necessary for fluctuations in the wireless “noise” and interference that’s present everywhere. Many things can affect wireless signals including rain, fog, even high humidity as there is additional moisture in the air, as well as dust and even simply other wireless devices in the area (phones, radios, routers, etc). One of the more interesting phenomenon is actually called “thermal inversion” and occurs during temperature fluctuations in different altitudes of atmosphere. To combat the noise and interference the radios adjust power output levels automatically to compensate.

Last night, we noticed that a radio was “flapping” and the link was somewhat unstable. Further troubleshooting showed that the power injectors powering that radio were barely outputting the correct voltage and amperage to keep the radio and link stable. Our assumption at the time is that the power injector was beginning to go bad, as these wear out over time and their useful life is also affected by external factors such as power spikes, brown-outs, and even static in the air generated during storms. Our plan this morning was of course to replace this device, assuming that this would correct all of the issues. The fault in our logic is that the injector was much closer to the end of it’s useful life than we realized…which we didnt know until this morning. Our “fix” last night was to back down on the power on the radios a bit to remove some of the draw from the power injector. This lowered it’s total power consumption which the injector could handle and the link was stabilized again last night.

This morning however, as the weather conditions changed, the air heated up, and other factors began to interfere with the signal the radios (correctly) increased power output levels to compensate. This then created the over-draw condition on the injector again, only this time at a more severe level than last night. This overdraw caused the injector to overheat, which drops amperage even further. The power output dropped to a level that multiple radios went into a “reset” condition, which wiped their configurations back to default and dropped multiple primary links on our network. This disruption filtered through the network, affecting most towers and services.

As noted above, we have separated primary links onto multiple injectors, so no 2 injectors will see an overload condition/outage/excess power draw at the same time. This will ensure, in the future, if an power injector begins to fail or cant handle the amperage draw required to keep a radio active, it will only affect a single radio/link and not multiples. We’re also looking into other ways to increase power redundancy to the radios, and of course our long term plan is to have other additional links out to the network for further link redundancy.

Now, I think it’s time for our guys to take an early day off, as they’re a bit exhausted, dehydrated, and sun burnt from being on the towers this morning… We couldn’t do what we do without them, and we appreciate their amazing hustle this morning.

Thank you again for your patience, and for all of those that called in to report the issues. We appreciate your calls, as they help quickly isolate and pinpoint any problem areas. We’re always happy to hear from you and take your calls. 928-427-1644

Troubleshooting issues on our Mockingbird link

Tuesday August 6th 2019, 9:25pm MST

We believe the issue with Mockingbird is related to a faulty power injector.  We have taken steps to correct the issue by removing some of the load.  The link is active and stable again, and a permanent fix will be implemented in the morning.

Tuesday August 6rth 2019, 8:27pm MST

About 45 minutes ago our NOC began tracking some issues on or link to Mockingbird, and we are evaluating and reviewing the issue for further details. We will update here as we determine a root cause.

Corrected issues at Mockingbid / Rio Vista / Red Hill

Our Mockingbird tower (which directly feeds Rio Vista and Red Hill) had a visit from some local avian friends this morning, and they left plenty of “evidence” of their time around our tower. It appears a number of them perched on one of the antenna feed horns and their weight shifted the antenna by about 3 degrees downward, sending our signal levels well below acceptable thresholds. We have corrected the aim on this antenna as of about 2:45 pm, and the signals and throughput are back to appropriate levels.

Thank you so much for your continued patience and understanding, as certain things are a bit outside of our control. Please keep us informed of any service related issues you may be having, as this helps us quickly narrow down and pinpoint problem areas: 928-427-1644

Current APS power outage

Saturday August 3rd 2019, 9:49am MST – APS power seems to have been restored to the affected areas, and our County Line tower and affected customers are coming back online.  Please let us know if you continue to have service related issues after this time. Thank you again for your ongoing support and patience.

Saturday August 3rd 2019, 7:30am MST – About 35 minutes ago our NOC began reporting outages across south and south-west Wickenburg. Further troubleshooting and discovery revealed that the outages were likely power related, and APS’s outage page has confirmed this theory.

It appears that there is an outage to a major APS service line, with over 1,000 APS customers affected. They are reporting the affected area as Yucca Dr To Vulture Mine Rd And Black Mountain Rd To Mariposa Dr. ETA to service restoration is 09:45 this morning.

Currently, our County Line tower, and customers serviced by this tower, are offline.

You can find further updates and information on APS’s website: http://outagemap.aps.com/outageviewer/ and we will of course keep you updated as we have them as well.

Thank you so much for your patience, and as always, please don’t hesitate to call with any issues (928) 427-1644 if you are experiencing trouble with your service.

Link Upgrade this afternoon

This afternoon and early evening, we will be working to improve performance on a primary link between our downtown hub and Constellation tower. Expected outages should be minimal, as traffic will flow over secondary links, however this will also put additional burden on our secondary links so you may notice some performance degradation between the hours of 3pm – 7pm. We will perform these upgrades as quickly and with as little disruption as possible, and we thank you for your patience and understanding as we work to improve overall network performance.

Affected areas include:
Constellation, Easy Street, Diamondback, Black Mountain, and County Line towers.

Emergency Maintenance

I wanted to make everyone aware of some emergency maintenance that is currently being performed on core routers across our infrastructure. Engineers are applying firmware fixes and patches to address recently discovered bugs. Outages should be minimal, with each core router being down for about 60 seconds. Some routers need updates in sequence, and therefore may be rebooted twice.

These updates are to address recently discovered security and performance issues, and are critical to the smooth and ongoing performance to the network.

All updates should be completed by 7am this morning.

Thank you for your patience, and please call (928) 427-1644 with any issues or concerns.

Storm related outages and updates

The storm last night was pretty intense for most of Arizona, with power and telecommunication systems affected throughout the state. We tracked outages last night through parts of our network including Castle Hot Springs, Morristown, and sections of West and Northwest Wickenburg including Diamondback and Easy Street.

As power was restored to customers this morning between 2 – 5 am everything, including customer locations, started coming back online.

However, we are still tracking and troubleshooting an issue between Diamondback and Easy Street, with customers on Easy street experiencing slower than normal speeds and increased latency. The Easy Street tower is online through its redundant link, and we are working to correct the issue with the primary link. We will provide more information as we have details.

Thank you so much for your support, and please don’t hesitate to call us at (928) 427-1644 if you are experiencing trouble with your service.