Recently, we’ve been buffaloed by a handful of problems where subscribers were reporting severely inadequate bandwidth, despite our active metering showing oodles of bandwidth available all the way down to their in-house WiFi units. We may have identified the cause.
The operating system supplied by our router manufacturer enforces speed limits for our standard and premier packages using “queue tables.” Inside the main router at each of our 15 subscriber towers is a queue table, containing a line for each subscriber of that tower, specifying their maximum download and upload rates, plus any “bennies.” They’re simple to set, simple to understand, and simple to inspect.
There’s only one problem: they can lie.
For some reason no one understands, after a queue table ages for a year or more (despite entries being added and removed as subscribers come and go), there is some subtle corruption that occurs that causes some indefinable block of subscribers to be arbitrarily throttled to arbitrary and varying lower rates. The table still claims it’s enforcing the correct max bandwidth, and it doesn’t show the subscriber as being throttled, but that’s the end result. And it’s all invisible to everybody.
This is a bug of long standing in the manufacturer’s OS, one they have never found or fixed, and one whose existence is just too easy to forget. Our standard procedure is to assume our equipment is telling us the truth, and when it doesn’t, we hie off down the wrong paths.
Complicating our analyses is that our raw bandwidth tests to subscriber locations continue to indicate plenty of bandwidth available, because they are independent of the queue mechanism.
It turns out that a workaround for this bug is simple: save the queue contents to a text file, destroy the entire queue table, then recreate them from the text file. Then everything is all clean and good, until at some future time it again isn’t.
Having been bitten by this bug three times now (in the past 12 years), and understanding that the manufacturer is unlikely to issue a fix for it any time soon, we’re putting together a script to perform this operation automatically on every tower, every month.
Meanwhile, we’ve performed the appropriate incantations by hand on the towers hosting subscribers reporting such problems (Mockingbird, Morristown, Easy Street, and 251st Avenue), in hopes it will address our known issues.