I would like to share a recent experience with bug bounties. I’ve been doing bug bounties for years and I just now am trying-out the big boys on the playground — bugcrowd.com and hackerone.com — and the experience is… well, different.
The incentive is there, I guess, to be able to negotiate with someone which cares on delivery times and holding accountability against the firms you are assessing, but I never felt the need of involving a third party in the process, maybe because I’m not making a living out of it and doing it out of fun and genuine care since usually I’ve been discovering vulnerabilities on websites I’m already an active user on. The extra buck, T-shirt or mere mention on a Hall-of-Fame of some sort, is mostly for bragging rights. I’ll come back to some of my own two-cent level insights at the end of this post — but first, let’s go on and tell you about the vulnerability found on Heroku.
What’s Heroku?
Heroku is pretty amazing, actually. They have an integrated PaaS solution that allows developers host their apps on the cloud with maximum ease.
Large as well as small companies are using Heroku non-stop, so I guessed that’s a good place for interesting vulnerabilities that might affect them all by spotting an organization-indifferent vulnerability.
Login API call, Spot the Response Anomaly
Heroku has several options of interfacing with their service. Web-based is pretty common usage and API follows. There is a Heroku Command Line Interface (CLI) tool which basically is a wrapper to API calls. I used the CLI tool in order to contact the site and started on messing with some inputs and after a time I focused on the login mechanism which looked pretty simple. It is based on a POST request, accompanied by username and password parameters.
At first, when messing with the inputs I figured nothing really matters, I was hoping for an error output or any kind of response changes between real usernames against bogus ones. Even though I didn’t see any of those error text changes — there was one more subtle change in response.
A login request looks something like that:
POST https://api.heroku.com/login HTTP/1.1Host: api.heroku.comUser-Agent: heroku-cli/5.2.39-010a227 (windows-386) go1.6.2Content-Length: 54Accept: application/jsonContent-Type: application/x-www-form-urlencodedAccept-Encoding: gzip
password=fjfjfjfj&username=bogus%40gmail.com
And an appropriate response would be:
HTTP/1.1 404 Not FoundCache-Control: private, no-cacheContent-Type: text/html;charset=utf-8Date: Mon, 29 Aug 2016 16:49:52 GMTRequest-Id: b3ca4a2c-1adf-4244-4cb0-17fbbc9deb06Server: nginx/1.8.1Strict-Transport-Security: max-age=31536000; includeSubDomainsVary: Accept-EncodingX-Content-Type-Options: nosniffX-Frame-Options: SAMEORIGINX-Runtime: 0.018023806X-Xss-Protection: 1; mode=blockContent-Length: 0Connection: keep-alive
So, as said, nothing more than a 404, but on the other hand it is somewhat generous on it’s optional headers — “so what?” you might ask. That makes the whole difference. To be specific, the thing that caught my eye was the “X-Runtime” header.
From it’s name I could only guess what it is that I found. But it needed a bit more probing to be sure about it.
Freeze, you bastard!
First stop is looking at the 1,000,000th time at the HTTP RFCs, noting that there is nothing like it in there — so it must be something added by another component on the web server. A short scan revealed it is most probably derived by a server side component relying on Ruby/Rails.
One of those projects, named Rack, was first to come up in search, that is to say that we can’t be sure of whether that’s the Ruby code that is operating on the site, but we’ll take it as a test case which proved to be sufficient at the end.
For those of you that never heard of it (like myself), it’s GitHub page reads:
Rack provides a minimal, modular, and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.
Peering into it’s source code, the aberrant header is explained within lib/rack/runtime.rb:
Now we can be sufficiently sure of our common-knowledge understanding of ‘X-Runtime’ meaning. It is referring to the time it took the operation to complete.
Are you thinking of what I am thinking?
To confirm this thought we have to get a larger test sample set against different mix of non-existent and existent user names on the system. And take note of timing differences and if there any distinguishable kind of delta.
Existent user names inclusion yeilded the following timing annotation:
X-Runtime: 0.093990008X-Runtime: 0.088857339X-Runtime: 0.09251876X-Runtime: 0.089843447X-Runtime: 0.091845765X-Runtime: 0.101260549
And when using non-existent user names (I’ve used the correct scheme of e-mail address, just to be safe):
X-Runtime: 0.027787351X-Runtime: 0.018023806X-Runtime: 0.020997733X-Runtime: 0.021645124X-Runtime: 0.016645836X-Runtime: 0.021828831X-Runtime: 0.022211143
Old-School Side Channel Timing-Attack — ❤
All of this assessment is funneled into one final conclusion — There is a clear distinctive (~1 order-of-magnitude) difference in times between bogus and true user names on the system.
Why is that even possible? Probably because of cryptography-related operations involved in the process of checking the password against it’s true hashed password.
It is a by-the-book example of a Timing Attack. Once the server receives the parameters from the user it first checks the user to see if it is relevant, if it isn’t the operation comes to an end and respond the 404 to the user. On the other case, if the user name is found to be relevant — it progresses forward-on to check the password given by the user against it’s local hashed copy of the password, which is a very intensive operation in contrast to the short trip to username-list that it followed. Because of that difference in operation time, we can perceive these distinctive routes.
Many applications are doing the same, but mostly because of latency it is somewhat harder to determine a definite conclusion upon a timing-based attack. In this case — the Ruby server is giving us all we need to get to this conclusion — all with one simple POST request. Cool!
How to overcome the attack?
This can be tackled on many levels. Since I’m not aware of Heroku’s considerations on keeping this header on, I will try to cover different approaches as there can be practical limitations of nearly each one of those.
- Remove ‘X-Runtime’ header from responses — this is the most obvious recommendation as that’s the fault that led us to our conclusion, but this is just the end-result of something much inherent — the times computation might be observed on other, less straight-forward ways as the time of delay of the response. What I observed by counting round-trip time from sending a request until receiving a response was not something that is distinguishable by that alone. Runtime value can be used for debugging purposes and I have met with clients in the past that couldn’t just remove debug headers from all or even some subset of responses-if that’s the case they might modify it on a higher web-server level and make an effort to delete the generous line before compiling the final response. Very ugly patch indeed, but practicality overpower elegance sometimes.
- Make all operation times equal — relying on the assumption that Heroku has granular control over the process of user authentication, they might delay the process of checking username and password parameters to an extent of time. This wouldn’t necessarily be projected through X-Runtime if they decide on leaving that in place.
- Add random timings to operations — injecting random length delays into the process can, in effect, make those times erratic. The problem with this approach that on the greater scale in multiple checks over each guess you will get (assuming normal distribution of random-length delays) a central tendency towards a larger result. This, too, doesn’t necessarily be projected through X-Runtime.
- Limit the amount of requests — This is considered a solution in case you are seeing this attack as something that is threatening a customer base as a whole and not as a targeted attack against a small subset of users. By this definition — you should be able to limit the amount of requests from a single source on a given time, this makes the efforts of the less-determined attacker redundant. This is not a bulletproof solution in any case since this usually circumvented by attackers which are using VPN, Anonymization network or a botnet to cover their tracks.
Heroku/BugCrowd’s Response
According to Bugcrowd’s terms — User Enumeration vulnerabilities are not in-scope, and they actually penalized me for that in -1 points (honestly, I should have known better. To my defense it wasn’t clear to me that there is a separate terms not only from the vendor but from BugCrowd as well that need to be adhered.)
The thing is, as I mentioned before — that wasn’t my point, as many of my peers I wasn’t doing this for profit, I was trying to make a site I evaluated for using for my own needs and if happen to be, as in this case, prompt for security vulnerabilities. Whether I get the bounty is the secondary objective, the thing that worried me is that because BugCrowd consider it out-of-scope doesn’t make this vulnerability something we shouldn’t discuss. The terms should be there to be defined that not everything is going to make you a small buck on the side, but once you find something that is out of scope for bugcrowd why should it be considered out-of-scope of mitigation for the vendor?
I truly don’t have anything against Heroku nor BugCrowd, they both are comprised of great, passionate people from my experience, but bureaucracy in this case contradicts progress and I wish those unwritten procedures should be taken care of for the benefit of our community.