Update: GoDaddy have confirmed that they re-verified all the initially revoked certificates and removed the ones which passed from the CRL.
When we learned of this issue, we re-validated every affected certificate. If we were unable to properly validate, we revoked the certificate. That is how we got the total of 8,951 revoked certificates.[…]As soon as we discovered the bug, we ran a report to identify every certificate that didn’t fail the domain validation check during the period the bug was active. We then started scanning websites to see which ones were able to re-pass the proper validation check. If they passed, we removed the certificate from the list. If we were unable to revalidate the certificate, we revoked it. If there was any question if the certificate was properly verified, we revoked it.
I still find this extremely dubious that they will modify the CRL.
On 11th Jan ’17 GoDaddy announced they had made a mistake. GoDaddy offer several means of verifying domain ownership before they will issue a certificate. One of these ways is through HTTP verification: They generate a nonce and you ensure a HTTP request to `yourdomain.com/nonce` responds to their request. What GoDaddy realised was that they had been accepting 404s as well as 200s when using this method, essentially completely invalidating it. To rectify the problem they revoked all the certificates they believed were issued this way, sending emails to the requesters saying it had happened.
One of many revocation notification emails we received on the day
On the face of it this seems like a sensible and prudent thing to be doing given the nature of their business, and I’m not too concerned about how they handled things up to here. What I’m really worried about is what happened next.
Unsure why we had received so many revocation notifications (we issue a lot of certificates for a lot of customers) we called our Premium Account Manager at GoDaddy (we buy a lot of certificates from them) and they explained the situation to us. One of the certificates we were particularly concerned about was a multi-domain certificate, as the customer would not have a fun time re-issuing it (this particular certificate has 29 SANs on it, you can see it for yourself). GoDaddy informed us the HTTP verification method was not available at this time, and may not be for up to a few weeks. Instead we would need to go via another method such as DNS change. Accepting that we called our customer to let them know the bad news.
This is where it gets weird. This certificate was definitely revoked. Browsers complained, and downloading the CRL referenced in the certificate we could plainly see the serial number in the list. I have taken a screengrab from the crt.sh page as well, just in case the results on the page change in the future:
crt.sh page showing clearly the revocation and when it happened
30 minutes after giving the bad news to our customer however, we got a follow up call from them to say “thanks very much, that’s all working now.” A bit confused we looked in to it and, indeed, the certificate was no longer revoked! Didn’t appear on the CRL and the sites were working fine. No modification had been made to DNS for any further verification.
While the outcome of this is good for our customer, it has a raised a huge concern for me with GoDaddy’s revocation policy. The CRL is supposed to be a one way system, and indeed GoDaddy even publish a policy stating the process cannot be reversed.
Revoking your SSL certificate cancels it and immediately removes HTTPS from the website. Depending on your Web host, your website might display errors or become temporarily inaccessible. The process cannot be reversed.
I now consider GoDaddys revocation system highly flawed, and will be seeking a new partner for certificates for our clients.
If you would like to independently verify this yourself then visit a site using the certificate in question here and check the crt.sh revocation record here. If anybody in authority would like to dig further I also have all phone calls between GoDaddy and our client recorded as well as the original emails.