Condensed version: We’ve narrowed down the Mockingbird Tower problem to RF noise that spikes periodically. Our tower equipment and configuration are clean and validated. We now need to perform after-midnight testing over the next few days that will result in brief (10-20 minute) interruptions of service for a number of the subscribers on this tower.
Details and analysis follow for the technically curious.
The RF spike noise causes packet loss to or from every subscriber unit trained on the tower, slowing them all down dramatically. We’ve ascertained that none of the other nearby towers is affected by this symptom. We’ve also ascertained that the noise is present and affects Mockingbird Central AP even when Mockingbird West AP is offline, and vice versa, so the other AP on this tower is not the source. But we’ve also ascertained that the noise affects Mockingbird West AP’s performance at the same time and in the same manner as Mockingbird Central AP.
The noise doesn’t vary with the operating channel selected at the tower, which means that either the noise blankets the entire 2.4GHz spectrum, or it’s being caused by one of our own subscriber units that is following the tower frequency. This suggests three most probable sources for the problem:
- Non-WiFi wide-spectrum interference, such as from an arc welder (unlikely, since the noise is 24-hour), a three-phase motor, a faulty ham filter, etc. (This is a good fit for a problem that affects either AP even when the other is entirely disabled.)
- A faulty subscriber unit radio chip violating the WiFi protocol and repeatedly broadcasting out of turn.
- A subscriber unit with poor transmit (to the tower) signal strength issuing sufficient transmission retries to swamp the tower. (If this is the case, there would have to be more than one, as each AP is affected even when the other is offline.)
In order to test possibilities 2 and 3, we need to partition the subscribers into groups, temporarily disconnect each group (in a way that allows us to recover them later), and test for the presence or absence of the noise. This is a time-consuming test that will interrupt service to arbitrary subscribers for arbitrary durations, possibly multiple times. We will be performing this testing this week between the hours of midnight and 3 AM, beginning on Tuesday night.
We apologize for the service interruption, but it is necessary to diagnose and resolve the problem that has been making Mockingbird Tower’s performance unacceptable to our subscribers. Thank you for your patience and understanding.