Cloudflare, the company managing the performance and security of more than 5 million websites, has today revealed that an exposed security bug was bleeding sensitive data from their customers’ websites. The recently fixed vulnerability was seen returning corrupted web pages by some HTTP requests run through edge servers. It included personal information such as HTTP cookies, authentication tokens, passwords and other sensitive data.
Talking about the same in an official blog post, Cloudflare CTO John Graham-Cumming says,
Our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
The said security bug was discovered by Google Project Zero researcher Tavis Ormandy. He contacted Cloudflare once the team of security analysts at Google found enough evidence of the bug affecting the service. And the content delivery network (CDN) started work on finding the root cause of information leakage, which came out to be their HTML parser chain used to increase the website’s performance.
Cloudflare passes websites through the HTML parser chain to make them ready for distribution through Google AMP and upgrades the HTTP links to HTTPS. But, it was discovered that three of its features weren’t properly implemented with the parser. Hence, access to the three feature using the parser was immediately cut off and a patch was deployed within hours (something which might’ve required months). But, this has also led to information from their own services leaking onto the web. John says,
One obvious piece of information that had leaked was a private key used to secure connections between Cloudflare machines
The vulnerability may not sound severe as of yet but Cloudflare was faced with a dilemma. Why, you ask? Well, there are several facets for this security bug to be called severe for the company. Firstly, the leak may have been active for the last five months, since Sept. 22, before it finally came to light earlier this week. But, the leak is expected to be the most severe in the period between February 13 to 18 as 1 in every 3,300,000 HTTP requests to Cloudflare sites resulted in data exposure.
In his vulnerability report, Tavis talked about how a majority of websites are using Cloudflare’s CDN services:
I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
But, what was even worrisome is the fact that some of this personal information had not only seeped through Cloudflare but also been cached by search engines. This means that the said security bug was in the open and hackers could’ve gained accessed to heaps of info in real-time by making web requests to the customer’s website. Talking about the disclosure of this security bug, John adds,
The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We are disclosing this problem now as we are satisfied that search engine caches have now been cleared of sensitive information. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence.
Cloudflare has since worked closely with all search giants — Google, Bing and Yahoo, to completely scrub the information that leaked on the web and was cached by them. Further, Google researcher Ormandy reported that over 3,400 websites were using the tool that contained the flaw and confirmed these three were among those affected. He didn’t name any of the services in his blog post and said that none of Cloudfare’s customers was intimated of the bug because of the scale of implications. And Cloudflare couldn’t have moved any faster in the remediation process.
