Ever since the “Heartbleed” flaw in encryption protocol OpenSSL was made public on April 7 in the US there have been various questions about who knew what and when.
Fairfax Media has spoken to various people and groups involved and has compiled the below timeline.
If you have further information or corrections – especially information about what occurred prior to March 21 at Google – please email the author: email@example.com. Click here for his PGP key.
All times are in US Pacific Daylight Time
Friday, March 21 or before – Neel Mehta of Google Security discovers Heartbleed vulnerability.
Friday, March 21 10.23 – Bodo Moeller and Adam Langley of Google commit a patch for the flaw (This is according to the timestamp on the patch file Google created and later sent to OpenSSL, which OpenSSL forwarded to Red Hat and others). The patch is then progressively applied to Google services/servers across the globe.
Monday, March 31 or before – Someone tells content distribution network CloudFlare about Heartbleed and they patch against it. CloudFlare later boasts on its blog about how they were able to protect their clients before many others. CloudFlare chief executive officer Matthew Prince would not tell Fairfax how his company found out about the flaw early. “I think the most accurate reporting of events with regard to the disclosure process, to the extent I know them, was written by Danny over at the [Wall Street Journal],” he says. The article says CloudFlare was notified of the bug the week before last and made the recommended fix “after signing a non-disclosure agreement”. In a seperate article, The Verge reports that a CloudFlare staff member “got an alarming message from a friend” which requested that they send the friend their PGP email encryption key as soon as possible. “Only once a secure channel was established and a non-disclosure agreement was in place could he share the alarming news” about the bug, The Verge reported. On April 17, CloudFlare says in a blog that when it was informed it did not know then that it was among the few to whom the bug was disclosed before the public announcement. “In fact, we did not even know the bug’s name. At that time we had simply removed TLS heartbeat functionality completely from OpenSSL…”
Tuesday, April 1 – Google Security notifies “OpenSSL team members” about the flaw it has found in OpenSSL, which later becomes known as “Heartbleed”, Mark Cox at OpenSSL says on social network Google Plus.
Tuesday, April 1 04:09 – “OpenSSL team members” forward Google’s email to OpenSSL’s “core team members”. Cox at OpenSSL says the following on Google Plus: “Original plan was to push [a fix] that week, but it was postponed until April 9 to give time for proper processes.” Google tells OpenSSL, according to Cox, that they had “notified some infrastructure providers under embargo”. Cox says OpenSSL does not have the names of providers Google told or the dates they were told. Google declined to tell Fairfax which partners it had told. “We aren’t commenting on when or who was given a heads up,” a Google spokesman said.
Wednesday, April 2 ~23:30 – Finnish IT security testing firm Codenomicon separately discovers the same bug that Neel Mehta of Google found in OpenSSL. A source inside the company gives Fairfax the time it was found as 09:30 EEST April 3, which converts to 23:30 PDT, April 2.
Thursday, April 3 04:30 – Codenomicon notifies the National Cyber Security Centre Finland (NCSC-FI) about its discovery of the OpenSSL bug. Codenomicon tells Fairfax in a statement that they’re not willing to say whether they disclosed the bug to others. “We have strict [non-disclosure agreements] which do not allow us to discuss any customer engagements. Therefore, we do not want to weigh in on the disclosure debate,” a company spokeswoman says. A source inside the company later tells Fairfax: “Our customers were not notified. They first learned about it after OpenSSL went public with the information.”
Friday, April 4 – Content distribution network Akamai patches its servers. They initially say OpenSSL told them about bug but the OpenSSL core team denies this in an email interview with Fairfax. Akamai updates its blog after the denial – prompted by Fairfax – and Akamai’s blog now says an individual in the OpenSSL community told them. Akamai’s chief security officer, Andy Ellis, tells Fairfax: “We’ve amended the blog to specific [sic] a member of the community; but we aren’t going to disclose our source.” It’s well known a number of OpenSSL community members work for companies in the tech sector that could be connected to Akamai.
Friday, April 4 – Rumours begin to swirl in open source community about a bug existing in OpenSSL, according to one security person at a Linux distribution Fairfax spoke to. No details were apparent so it was ignored by most.
Saturday, April 5 15:13 – Codenomicon purchases the Heartbleed.com domain name, where it later publishes information about the security flaw.
Saturday, April 5 16:51 – OpenSSL (not public at this point) publishes this (since taken offline) to its Git repository.
Sunday, April 6 02:30 – The National Cyber Security Centre Finland asks the CERT Coordination Centre (CERT/CC) in America to be allocated a common vulnerabilites exposure (CVE) number “on a critical OpenSSL issue” without disclosing what exactly the bug is. CERT/CC is located at the Software Engineering Institute, a US government funded research centre operated by Carnegie Mellon University. The centre was created in in 1988 at DARPA’s direction in response to the Morris worm incident.
Sunday, April 6 ~22:56 – Mark Cox of OpenSSL (who also works for Red Hat and was on holiday) notifies Linux distribution Red Hat about the Heartbleed bug and authorises them to share details of the vulnerability on behalf of OpenSSL to other Linux operating system distributions.
Sunday, April 6 22.56 – Huzaifa Sidhpurwala (who works for Red Hat) adds a (then private) bug to Red Hat’s bugzilla.
Sunday, April 6 23.10 – Huzaifa Sidhpurwala sends an email about the bug to a private Linux distribution mailing list with no details about Heartbleed but an offer to request them privately under embargo. Sidhpurwala says in the email that the issue would be made public on April 9. Cox of OpenSSL says on Google Plus: “No details of the issue are given: just affected versions [of OpenSSL]. Vendors are told to contact Red Hat for the full advisory under embargo.”
Sunday, April 6 ~23:10 – A number of people on the private mailing list ask Sidhpurwala, who lives in India, for details about the bug. Sidhpurwala gives details of the issue, advisory, and patch to the operating system vendors that replied under embargo. Those who got a response included SuSE (Monday, April 7 at 01:15), Debian (01:16), FreeBSD (01:49) and AltLinux (03:00). “Some other [operating system] vendors replied but [Red Hat] did not give details in time before the issue was public,” Cox said. Sidhpurwala was asleep during the time the other operating system vendors requested details. “Some of them mailed during my night time. I saw these emails the next day, and it was pointless to answer them at that time, since the issue was already public,” Sidhpurwala says. Those who attempted to ask and were left without a response included Ubuntu (asked at 04:30), Gentoo (07:14) and Chromium (09:15), says Cox.
Prior to Monday, April 7 or early April 7 – Facebook gets a heads up, people familiar with matter tell the Wall Street Journal. Facebook say after the disclosure: “We added protections for Facebook’s implementation of OpenSSL before this issue was publicly disclosed, and we’re continuing to monitor the situation closely.” An article on The Verge suggests Facebook got an encrypted email message from a friend in the same way CloudFlare did.
Monday, April 7 08.19 – The National Cyber Security Centre Finland reports Codenomicon’s OpenSSL “Heartbleed” bug to OpenSSL core team members Ben Laurie (who works for Google) and Mark Cox (Red Hat) via encrypted email.
Monday, April 7 09.11 – The encrypted email is forwarded to the OpenSSL core team members, who then decide, according to Cox, that “the coincidence of the two finds of the same issue at the same time increases the risk while this issue remained unpatched. OpenSSL therefore released updated packages [later] that day.”
Monday, April 7 09:53 – A fix for the OpenSSL Heartbleed bug is committed to OpenSSL’s Git repository (at this point private). Confirmed by Red Hat employee: “At this point it was private.”
Monday, April 7 10:21:29 – A new OpenSSL version is uploaded to OpenSSL’s web server with the filename “openssl-1.0.1g.tgz”.
Monday, April 7 10:27 – OpenSSL publishes a Heatbleed security advisory on its website (website metadata shows time as 10:27 PDT).
Monday, April 7 10:49 – OpenSSL issues a Heartbleed advisory via its mailing list. It takes time to get around.
Monday, April 7 11:00 – CloudFlare posts a blog entry about the bug.
Monday, April 7 12:23 – CloudFlare tweets about its blog post.
Monday, April 7 12:37 – Google’s Neel Mehta comes out of Twitter hiding to tweet about the OpenSSL flaw.
Monday, April 7 ~13:13 – Most of the world finds out about the issue through heartbleed.com.
Monday, April 7 15:01 – Ubuntu comes out with patch.
Monday, April 7 23.45 – The National Cyber Security Centre Finland issues a security advisory on its website in Finnish.
Monday, April 8 ~00:45 – The National Cyber Security Centre Finland issues a security advisory on its website in English.
Tuesday, April 9 – A Red Hat technical administrator for cloud security, Kurt Seifried, says in a public mailing list that Red Hat and OpenSSL tried to coordinate disclosure. But Seifried says things “blew up” when Codenomicon reported the bug too. “My understanding is that OpenSSL made this public due to additional reports. I suspect it boiled down to ‘Group A found this flaw, reported it, and has a reproducer, and now Group B found the same thing independently and also has a reproducer. Chances are the bad guys do as well so better to let everyone know the barn door is open now rather than wait 2 more days’. But there may be other factors I’m not aware [of],” Seifried says.
Wednesday, April 9 – A Debian developer, Yves-Alexis Perez, says on the same mailing list: “I think we would have managed to handle it properly if the embargo didn’t break.”
Wednesday, April 9 – Facebook and Microsoft donate $US15,000 to Neel Mehta via the Internet Bug Bounty program for finding the OpenSSL bug. Mehta gives the funds to the Freedom of the Press Foundation.
Monday, April 14 ~12.30pm – The Guardian reports a mothers forum with 1.5 million users called Mumsnets is impacted by Heartbleed. A “hacker” reportedly breached the admin’s user account.
Monday, April 14 – the Canada Revenue Agency announces social insurance numbers of approximately 900 taxpayers were removed from its systems by someone exploiting the Heartbleed vulnerability.
Wednesday, April 16 – A Canadian teen is arrested for stealing tax data with Heartbleed.
Who knew of heatbleed prior to release? Google (March 21 or prior), CloudFlare (March 31 or prior), OpenSSL (April 1), Codenomicon (April 2), National Cyber Security Centre Finland (April 3), Akamai (April 4 or earlier) and Facebook (no date given).
Who knew hours before public release? SuSE, Debian, FreeBSD and AltLinux.
Who didn’t know until public release? Many, including Amazon Web Services, Twitter, Yahoo, Ubuntu, Cisco, Juniper, Pinterest, Tumblr, GoDaddy, Flickr, Minecraft, Netflix, Soundcloud, Commonwealth Bank of Australia (main website, not net banking website), CERT Australia website, Instagram, Box, Dropbox, GitHub, IFTTT, OKCupid, Wikipedia, WordPress and Wunderlist.
Many thanks to: Nik Cubrilovic, Yves-Alexis Perez, public mailing lists, emails with OpenSSL core team, emails with the National Cyber Security Centre Finland, Google Plus posts, and emails with people who volunteer at Linux distributions.
April 15, 4.14pm AEST: Some Codenomicon dates were wrong. They have been fixed.
April 16, 6.04pm AEST: Added another significant event that occured on Sunday, April 6 02:30 PDT.
April 16, 9.57pm AEST: Added details about the time OpenSSL core team members found out about Heartbleed from Google.
April 18, 6.18pm AEST: Added information about the Canadian tax agency breach, the Canadian teen getting arrested for it, and details from this blog post from CloudFlare.