Yes, we’re working on performance problems

Our router manufacturer issued a software upgrade on Monday to close an “exploitable hole” that put the security of our network and your data at risk. Unfortunately, they issued the fix first in the “new features” release chain, and were delayed issuing it in the “current bugfix” release chain. Unwilling to delay, we took a leap of faith and installed the “new features” release to get the security hole closed as promptly as possible.

Our faith was betrayed.

We are currently re-installing the (finally available) “current bugfix” release with the security patch on all the routers in our network. Since this is technically a downgrade, the installation is much less automatic and much more labor-intensive than the original upgrade, needing to be performed manually on upwards of 200 machines, certain of which are barely communicating well enough at the moment to load the software. We ask your patience while we back out the misbehaving wireless software suite installed earlier this week.

Mockingbird recapped

I’m happy to report that, as far as we have been able to determine, all our speed-related problems at Mockingbird Tower are now resolved. Assuming this announcement doesn’t generate a spate of contrary responses, our next task will be to arrange for incremental UPS protection at our towers, beginning with the ones that are most highly subscribed; and working with our consultants to assure that we can deliver to our subscribers all the untapped bandwidth currently available at our Internet gateway.

For the technically curious, a brief description of the source of our speed problems appears below. (Warning: unpaved roads ahead!) Continue reading

Red light means go!

A number of our subscribers for whom we have installed wall-mounted MikroTik mAP access points have called us, concerned that they might have a network problem because a red light was showing on their unit.

The light in question is normal. It indicates that the WiFi unit is supplying power to the roof-mounted radio unit over its ethernet cable (hence the label “PoE out,” “power over ethernet”).  This particular model of access point is the only MikroTIk device we supply to subscribers that offers the PoE pass-through feature, which allows us to power both your WiFi access point and your roof unit with only a single power supply.

In general, different MikroTik units can come equipped with red, blue, green, orange, or yellow LEDs. In some of the higher-end models, the same LEDs can show different colors under different circumstances… but on none of them (to our knowledge) does the color of the LED itself ever indicate an error condition. In all circumstances, MikroTik uses the color red or orange to indicate only that the unit is supplying power to some other unit.

So rest easy—this red light means only that all of your systems are go!

Requiem for a Heavyweight

Laird POE-48i-CI

Laird POE-48i-CI

This is a 48-volt power supply made by Laird Technologies. It is the very first power supply we bought, on February 19, 2007, to power our very first tower in Morristown.

It spent 4½ years cooped up in an unventilated metal box on our windmill tower, where it routinely operated in temperatures over 120°F.

In September of 2011, after we had replaced our original Morristown router, power supply, and omni antenna with a newer, faster router driving three directional sector antennas, the original equipment was retasked to serve Echo Hill — a small pocket-valley neighborhood of three residences. The power supply continued to labor there 24/7 in its new, more-comfortable indoor environment.

At 5:54 on February 11, this power supply gave up the ghost, its little amber light pulsing in an entirely undocumented manner — seven years of continuous operation almost to the day. As my father would have said, “It didn’t owe you nothing.”

It took us 24 hours to get Echo Hill operational again, only because it was the only 48V power supply we owned, and its router was likewise the only router we own that doesn’t work on any of the 24V supplies we have in abundance. We found an 18V Laird supply and instructions for restrapping the router to accept 6-22V. The router — itself a full seven years in an outdoor hotbox — continues to crank packets every bit as well as it did in 2007.

One of the things I really enjoy about the networking business is the robust construction of the equipment I get to work with. The original router of this tale (a MikroTik RouterBoard 532c) was shipped in 2007 with RouterOS 2.7 — it is now running RouterOS 6.9, which still fully supports it.  When did you last own a PC or Mac that operated faultlessly for over seven years, including in extreme harsh environments, and continued to be fully supported by the manufacturer’s OS a full four or more major releases beyond what it originally shipped with?

Our corporate hat is off to the folks at Laird and MikroTik, who continue to make stuff that Just Works.