Wednesday August 7th 2019, 1:35pm MST
First, I want to thank everyone for your patience and understanding this morning during our network issues. The power related issue last night presented itself again this morning, in a pretty catastrophic way. The details are below, but the short version is that we have replaced the faulty power injectors and divided the load between multiple sources.
The long version: Many (most) of our radios have variable signal outputs, which is necessary for fluctuations in the wireless “noise” and interference that’s present everywhere. Many things can affect wireless signals including rain, fog, even high humidity as there is additional moisture in the air, as well as dust and even simply other wireless devices in the area (phones, radios, routers, etc). One of the more interesting phenomenon is actually called “thermal inversion” and occurs during temperature fluctuations in different altitudes of atmosphere. To combat the noise and interference the radios adjust power output levels automatically to compensate.
Last night, we noticed that a radio was “flapping” and the link was somewhat unstable. Further troubleshooting showed that the power injectors powering that radio were barely outputting the correct voltage and amperage to keep the radio and link stable. Our assumption at the time is that the power injector was beginning to go bad, as these wear out over time and their useful life is also affected by external factors such as power spikes, brown-outs, and even static in the air generated during storms. Our plan this morning was of course to replace this device, assuming that this would correct all of the issues. The fault in our logic is that the injector was much closer to the end of it’s useful life than we realized…which we didnt know until this morning. Our “fix” last night was to back down on the power on the radios a bit to remove some of the draw from the power injector. This lowered it’s total power consumption which the injector could handle and the link was stabilized again last night.
This morning however, as the weather conditions changed, the air heated up, and other factors began to interfere with the signal the radios (correctly) increased power output levels to compensate. This then created the over-draw condition on the injector again, only this time at a more severe level than last night. This overdraw caused the injector to overheat, which drops amperage even further. The power output dropped to a level that multiple radios went into a “reset” condition, which wiped their configurations back to default and dropped multiple primary links on our network. This disruption filtered through the network, affecting most towers and services.
As noted above, we have separated primary links onto multiple injectors, so no 2 injectors will see an overload condition/outage/excess power draw at the same time. This will ensure, in the future, if an power injector begins to fail or cant handle the amperage draw required to keep a radio active, it will only affect a single radio/link and not multiples. We’re also looking into other ways to increase power redundancy to the radios, and of course our long term plan is to have other additional links out to the network for further link redundancy.
Now, I think it’s time for our guys to take an early day off, as they’re a bit exhausted, dehydrated, and sun burnt from being on the towers this morning… We couldn’t do what we do without them, and we appreciate their amazing hustle this morning.
Thank you again for your patience, and for all of those that called in to report the issues. We appreciate your calls, as they help quickly isolate and pinpoint any problem areas. We’re always happy to hear from you and take your calls. 928-427-1644