We Didn’t Start the Fire, But We Fought It: How We Helped Our Customers Recover from a Security Certificate Mix-Up
Service is often overlooked when it comes to technology companies. The general consensus is that good technology should not break, and if it does, the vendor should fix it. However, technology today is highly interconnected. Sometimes, the fires that we fight are not the ones we started. This was the case on the morning of October 13, 2016, when GlobalSign “fouled up a clean-up of some of their root certificates links,” according to ZDNet, which froze hundreds of thousands of websites.
That Thursday morning, we were alerted by a large media customer that errors were being reported by their users. As usual, our teams jumped into action. We quickly identified that the culprit was the GlobalSign certificate issue. This could have been the end of the story. After all, this was not a problem associated with our service. However, we did realize that it was something that we were in the position to help with. So as usual, we acted. What ensued was a mobilization of different teams within network operations, engineering, accounts, web performance, security operations and DevOps who collaborated closely with all customers that were affected by this issue, to find the best way to minimize impact.
We wiped the entire OCSP cache across our network to stop delivering the wrong cached certificate. This immediately mitigated the problem for many of our customers impacted by the GlobalSign glitch. However, our teams observed that the underlying CDN for GlobalSign was continuing to deliver the wrong certificate. While this was out of our control, we decided to proactively monitor the issue until it was resolved so that we could regenerate the correct OCSP stapled responses as soon as possible.
Meanwhile, some of our customers obtained the new certificate, and due to our unique partnership with DigiCert, we were able to quickly distribute the new file within 15 minutes. Our application support team ensured the stapled responses were served correctly, globally. Nearly all of our customers that were impacted by the GlobalSign glitch had the right certificate within two hours of the initial alert.
We believe that better service does matter in a digital content delivery partner. We are more than happy to leverage our capabilities to help our customers stay up and running or get back to business as soon as possible if there’s an issue, regardless of who caused it in the first place. Our teams worked closely and collaboratively with impacted customers to resolve this incident faster than they thought was possible. We are proud of that, and we believe this quality is among the key differences between our solution and that of our competitors. The fact is, as the internet gets more complex, there will be more and more days like this when you want a fire-fighting team to be on your side when the flames erupt.