After last night’s testing, we suspect the most probable cause of the performance problem at Mockingbird is a bad customer roof unit radio hashing our tower. We’ve exhausted all the minimally invasive tests we can run. Starting tonight, we’re running a Big Sleep test on all our Mockingbird subscriber units.
The tests we’ve done so far show that dissociating subscriber units from the tower hasn’t eased the problem; in fact, it intensifies it. That’s counter-intuitive. We believe this indicates that, having nothing else to do, a hypothetical bad unit is hammering on the tower at a fairly high rate (with its disruptive signal), asking to be let back in.
The Big Sleep test involves turning the radios of customer roof units completely off, one by one, so that they’re not demanding anything of the tower at all. When the problem disappears, the theory goes, we will have identified the culprit.
The trick to this test is that once you’ve turned a subscriber unit radio off, you no longer have any way to ask it to turn itself back on. To get around that, we’ve spent all day today writing and testing scripts designed to turn subscriber unit radios on every morning at 3:30 AM, and distributing them by hand to every unit served by Mockingbird Tower.
Starting a little after midnight tonight (Thursday morning), we’re going to turn the subscriber units off one by one, watching to see when our problem disappears. Once off, the subscriber units are going to stay off until 3:30 AM, regardless — without having an active link to them, we can’t bring them up any sooner.
We’ll be leaving a recording on our voice mail tonight for night owls on Mockingbird Tower who haven’t yet subscribed to our service bulletins and therefore won’t receive this.
We apologize for the extended service interruption tonight, but it’s to the point we have to pull out the biggest guns in our arsenal to put this poor performance problem behind us.