Great Balls of Fire!

24 hours ago, a monsoon-season electrical storm rolled through Morristown, Wittmann, and Buckeye. I’m only now getting my breath back from dealing with the aftermath.

Just around 10 PM yesterday (Wednesday), a bolt of lightning plowed into the ground right on our own Ranch property. It was bright, loud, and scary. When the rain had finally subsided around 10:30 PM, I took stock of our network and logged two major problems.

  • First, although we retained communication between our Network Operations Center (NOC) and most of our subscribers, our gateway to the internet was non-operational because our provider’s backhaul equipment on the White Tanks had been affected by the storm. I placed a service call, updated our voice mail recording, and awaited followup.
  • Second, we had entirely lost communication with the windmill tower, and therefore all of the subscribers served by it. Normally, this means a simple power outage at the tower, and it’s usually a small matter of resetting a circuit breaker to get it running again. Sure enough, the house breaker feeding the remote breaker panel at the well was popped… but resetting it didn’t bring the windmill back online. Squishing out to the well panel, I found the individual breakers for the windmill and the automatic entry gate popped, and reset them. But this didn’t bring the windmill back online either. So I trekked out to the windmill to find this stirring sight:

IMG_0199 - Version 2The GFCI outlet at the base of the windmill had literally exploded, taking with it the weatherproof in-service outlet cover. The lightning bolt had struck and baked a 24-foot saguaro 30 feet from the windmill; detonated the outlet box; then traveled to the well panel where it destroyed the well pump timer (blowing out its panel window in the process), the GFCI outlet at the gate, the gate’s vehicle detector, and popped two circuit breakers; then traveled up to the house where it blew the feed breaker for the well panel.

Since our dead gateway made the tower’s condition non-critical, I caught a few hours of sleep, then got up at first light to lay a temporary overground power feed out to the windmill.

Nothing’s ever easy. Every repair uncovered further damage that needed addressing. The outlet inside our equipment’s power box had also violently disintegrated. The power injector that had been plugged into it was dead as well. After cobbling together a working DC power supply out of spare equipment and a waterproof bucket, I found that the main router had also been ruined. (Luckily, that was the end of the chain—the four service radios on the windmill had been insulated from damage by the sacrificial router.)

At about this time in the late morning, the White Tanks equipment came online again, but only our lower-bandwidth Circle City gateway could talk to it. Our network automatically re-routed all subscriber traffic through Circle City, while our provider and I determined that the storm had damaged his main backhaul radio on our roof. A team was dispatched to our location with a replacement radio.

It was about at this time that I was dismayed to discover that the spare router I planned to swap into service at the windmill tower was the wrong model, with insufficient ports to run the slave radios. Our provider came to the rescue by loaning me a spare five-port box, which I hurriedly configured to match the blown unit.

By mid-afternoon, we had a working high-speed gateway link and a working Morristown tower, allowing us to go shopping for electrical parts to repair all the damage, and order replacement routers and power supplies to retire our temporary “solutions.”

Days like today are what keep the adventure in rural internet service. Truth to tell, it could have been worse—I could have had to replace one of the units on top of the windmill.

Leave a Reply

Your email address will not be published. Required fields are marked *