Sunday morning at 7:30am.
For Avid and our customer base of commercial users, early Sunday morning is generally quiet. But at least one of our clients, a car wash, is busy preparing for a crucial day.
On this particular Sunday, their internet connection was “down.” They called our support line and I took the call (Avid’s partners occasionally take after-hours calls). It didn’t take me long to determine that their Avid services were up and operating error free. And for most in our industry, that’s where the story would end.
But our client was facing the loss of hundreds of dollars per hour, so I didn’t think the standard “everything’s fine on our end” answer was acceptable.
I got one of our top engineers involved to see if there was something more we could do. He determined the router at their site had been running at 100% since 2 am – not a normal condition. Digging deeper, he saw that this small operation had thousands of Internet sessions to a single IP address – another red flag. Unlike many of our clients, they had someone other than Avid managing their firewall, but still our analysis indicated a serious firewall issue.
We were able to restore the service by rebooting the router remotely. Within about 15 minutes, they were back in service. But we also suspected that the firewall issue would rear its ugly head again, and encouraged the client to get their firewall vendor involved.
Because we understand all aspects of network performance, we’re able to play a key role in trouble resolution even when it “isn’t our problem.” I’m sure their firewall vendor would have been able to address the problem eventually, but on that Sunday morning, time was big money.