Preface
The first few days of this vulnerability disclosure were a very interesting ride. It started with movrcx posting his attack on Tor Browser (which you should definitely read). Most people blew him off because they thought he didn’t understand how certificate pinning worked in Firefox and that his setup would inherently bypass pinning by design. The funny thing about that is that they were right (while still being very wrong). Going in, movrcx didn’t have a great understanding of how certificate pinning in Firefox worked and neither did I for that matter. To add, had he done his attack on Firefox instead of Tor Browser, his result would have actually been the same even if a vulnerability didn’t exist. It’s kinda hard to blame anyone for blowing off his attack given the totality of the information available at the time. I almost did too.
The only reason I caught onto anything being weird is that I had a vague memory of Tor Browser hardening how they handled pinning by enabling a stricter level of enforcement. Because of that hardening, movrcx’s attack shouldn’t have worked in Tor Browser using his setup. That led me to dig in and explore the issue more deeply to the point that I could pinpoint the source of the problem, and then submit my own post on the subject to daveaitel’s Daily Dave mailing list to get more eyes on it. Within hours of it going live, the root of the problem was pointed out to me via e-mail by Nicolas Vigier of the Tor Project. Then, over the course of the afternoon it was determined that this problem would actually come back for a few days in November and then for 5 WEEKS in December into January. As of this writing, that’s still the case although Mozilla has vowed to fix their processes to prevent it from actually happening. A bug that presents itself on schedule is pretty interesting.
The point of this writeup is give the down and dirty technical specifics on how and why this attack works in detail. My first post had some speculation in it but the facts are known now and patches have already started rolling out. This will lay out the facts without all of the speculation (unless otherwise specifically noted). Plus, I will show that not only would this vulnerability have come back in the future but, more importantly, that it has already happened a few other times in the last couple of years in both Tor Browser and Firefox without being caught.
Certificate Pinning
Certificate Pinning is the process of forcing a browser to only use certain certificates in the validation of a TLS connection to a certain domain. This is done by either a static certificate pin list included with the browser or using a standard called HTTP Public Key Pinning (HPKP) which allows a site to push down its own pins on the first connection to it.
Mozilla uses Certificate Pinning to protect connections to addons.mozilla.org (AMO), which is used for the updates of most Firefox extensions. The purpose of pinning this domain is to prevent a rogue CA from being able to generate a certificate for AMO that could then be used to perform a man-in-the-middle (MiTM) attack on the extension update process.
HTTP Public Key Pinning (HPKP) has a “chicken or the egg” issue with regards to getting the certificate pin to the user. The protocol is a trust-on-first-use (TOFU) protocol. The pins are sent by a website to the user on the first connection to the site. That means that first connection is not protected by pinning. Worse yet, if an attacker were to MiTM that first connection, they could push their own HPKP pins instead. To combat this, both Chrome and Firefox (IE/Edge does not support HPKP) include a HPKP pre-load list bundled into each release of the browser. The pins are included in there to protect any connection (like the first one to a website) that isn’t protected by pushed HPKP pins. Once a site pushes its own HPKP headers to the user, those pins then override the pre-load list.
You can also put a pin in this pre-load list even if the site you include doesn’t push HPKP headers on the first connection. These are referred to as static pins because they are never overrode by HPKP. This was the case for addons.mozilla.org. Pinning for AMO relied solely on the static pin that was bundled with the browser.
The Vulnerability
When HPKP headers are pushed to the user, they are sent with an expiration date. This is necessary because certificates need to change sometimes because they expire, are compromised, are found to be using insecure signatures, or for any other reason. If the pins didn’t expire, the site would be stuck with the certificate they pinned indefinitely. It’s up to the site to determine what the expiration date should be.
This also means that the static pins need to expire. If the user doesn’t update their browser, and these pins don’t expire, a website that changes it’s certificate in the future would be unable to receive connections from the legacy browsers. That’s why it’s also important to stop enforcing pinning when they expire instead of having it fail.
The vulnerability here is that Mozilla failed to set the expiration date for the static pins and HPKP pre-load list long enough into the future to last until the next release of Firefox. Let’s look specifically at the last line of the code that sets the pin expiration. The specific file is StaticHPKPins.h generated on 25 Jul 2016 and included in the source code for Firefox 45.3.0esr released on 02 Aug 2016 (the current and vulnerable version as of this writing which is set to be replaced by 45.4.0esr on 20 Sep 2016).
static const PRTime kPreloadPKPinsExpirationTime = INT64_C(1472903978258000);
The expiration time is expressed here in epoch time. 1472903978258000 here converts to Sat, 03 Sept 2016 11:59:38 GMT.
To review, StaticHPKPins.h was generated on 25 Jul 2016. Firefox 45.3.0esr was released on 02 Aug 2016. The pins expired on 03 Sept 2016. Firefox 45.4.0esr doesn’t release until 20 Sept 2016. That means from 03 Sep until the user installs the new version of Firefox released on 20 Sept 2016, the static pins are expired and pinning is not enforced.
Tor Browser is based off the ESR release so the expiration dates and lack of pinning enforcement are the same in this case. The regular Firefox release faired a bit better because its pins didn’t expire until a week later on 10 Sep 2016 making it vulnerable for 10 days instead of 17 days.
The Fix
Mozilla has an automated process for updating this expiration date and has blamed this issue on that process failing.
Daniel Veditz from Mozilla Security describes the issue in the Mozilla Bug Report
In a statement obtained by Dan Goodin at ArsTechnica, Mozilla vowed to add HPKP to addons.mozilla.org. They also promised to ensure that the pins won’t expire before the next release can extend the expiration date. Here is their statement:
We investigated this and a fix will be issued in the next Firefox release on Tuesday, September 20. We had fixed an issue with the broken automation on the Developer Edition on September 4, but a certificate pinning had expired for users of our Release and Extended Support Release versions. We will be turning on HPKP on the addons.mozilla.org server itself so that users will remain protected once they have visited the site even if the built-in pins expire. We will be changing our internal processes so built-in certificate pins do not expire prematurely in future releases.
Mozilla seems to have followed through and has indeed started to roll out HPKP by turning it on for versioncheck.addons.mozilla.org.
Jason Thomas from Mozilla confirming the deployment of HPKP in a Bugzilla ticket
Rolling out HPKP now has helped Mozilla protect the current version of Firefox until the next version is released on 20 Sept. It’s not a complete fix but I personally believe it reduces the risk to a point that is well below Firefox’s threat model.
Tor, on the other hand, has a much different threat model. They realized this and decided to take the version they had compiled for the 20 Sept release and release it 4 days early on 16 Sept. They deserve credit for their quick reaction.
What’s Old is New
As I mentioned in the Preface, this bug is scheduled to make a reappearance. This is explained here:
Daniel Veditz of Mozilla Security describes how pins will continue to expire
Had this bug gone un-checked, it would have came back for 3 days in November and for 5 weeks from December into January. This made me wonder how often both Firefox and Tor Browser have been vulnerable to this in the past. It coming up 3 times in 4 months seemed pretty excessive to me.
I brought this up in the #tor-dev IRC channel on OFTC and Matt Nordhoff graciously offered to compile that data for me for use in this write-up. He compiled the info for Firefox Stable, Firefox ESR, Tor Browser Alpha, and Tor Browser Stable. It is an impressive dataset and you should definitely check it out.
mnordhoff/firefox-pin-data_firefox-pin-data - Firefox and Tor Browser pin data_github.com
While Firefox and Tor Browser have both been vulnerable in the past, it turns out that it hasn’t happened that often. Lets look at few of them and pull some dates. Here are the stats for Firefox ESR:
As you can see, with the exception of the current release, the only other time that the pinning has expired in the past for Firefox ESR is from 04 July 2015 to 11 August 2015. However, that’s a pretty significant time window. Also, it appears 38.1.1esr actually shipped with the pins already expired. It wasn’t until the release of 38.2.0esr that pinning was reinforced. There was also only a 4 day window to update between 45.0.2esr and 45.1.0esr .
Now, lets look at Tor Browser Stable:
You’ll notice that Tor Browser had expired pins until the release of 5.0. This is because Tor actually backported the pinning functionality but likely weren’t aware they needed to update the expiration date. It’s an honest mistake and it’s fair to say that Tor Browser didn’t really support static pins until version 5.0. However, outside of that, Tor had not been vulnerable before until this most recent release. They actually faired pretty well.
Conclusion and Acknowledgements
This has been a pretty interesting bug to track down. It shows how failures of process can be very difficult to spot when you’re dealing with a complex codebase and there is no indication that something is failing. I’d be lying if I said luck didn’t play a significant role in the discovery of this bug. If movrcx had tried his attack before 3 Sept or after 20 Sept, it would have failed in his tests. It’s only because he conducted it within that 17 day window that this was discovered.
Mozilla is also looking into how to harden the extension update process itself so an attack like this wouldn’t work even if you were able to break certificate pinning again in the future. This vulnerability was actually only limited to extensions that updated through AMO. Extensions that follow their own update process were unaffected. The HTTPS Everywhere update process in the version included with Tor Browser was not affected because they do updates via EFF.org instead and do signature verification of the updates once they are downloaded. However, the version delivered by AMO is likely affected the same as all of the others.
Here are some acknowledgements I’d like to make:
- movrcx — Without his initial discovery, none of this would have been unwrapped. He was also extremely easy to work with while we were combing through the code to figure out the root cause of this bug.
- Erinn Atwater — Her help while I was trying to figure things out was invaluable. She originally pointed me to the fact that Mozilla was using static pins outside of HPKP.
- daveaitel — For publishing my initial writeup on Daily Dave and being generally responsive to me, even while busy in Germany. I definitely recommend subscribing to his mailing list.
- Nicolas Vigier — For being the first person to point out the expiration date issue to me via e-mail.
- Georg Koppen — His swift reaction and decision to release the next Tor Browser release early should be commended. He quickly understood the issue, understood its impact, and took action.
- Matt Nordhoff — For compiling the stats on Firefox and Tor Browser pins in such an extremely clean and usable way and for his dedication in making sure it was accurate and contained the whole picture.
- #tor-dev on OFTC— These people were extremely responsive once I reached out and their transparency in talking through a problem is not something you see from most development teams.
I hope this was as interesting for you as it was for me. I feel like there are a lot of lessons to be learned from this whole process.