The nightmare for everyone working on Web or Node applications is probably waking up to this:
The media was quick in picking up the story and now there is a massive scandal about Crypto and security and JavaScript — the perfect recipe. And it is more than this: this scandal is fundamentally and legitimately questioning the way we build (open source) software with Node.js.
TL;DR
NPM modules should be multi-signed. Certificates and trust models should be handled on the blockchain. Open source developers sign packages and receive a share based on their contribution to paid software. This is made possible and automatically handled by crypto payments and smart contracts. Instead of publishing rights, new maintainers = potential hackers are only granted counter-signing rights. Automatic signature validation checks in module managers can determine unknown / untrusted new signatures and based on rules and individual threat models accept or reject updates. Such modules can then be audited in a more targeted way.
Disclaimer: Opinions are my own and not the views of my employer
So what did happen?
We find the answer on the event-stream repository on GitHub and NPM:
7 years ago, the developer dominictarr created (with good intentions) an NPM module called ‘event-stream’ which “is a toolkit to make creating and working with streams easy”. The module is extremely successful with ~2M weekly downloads:
82 versions later or two months ago the latest version 4.0.1 was released. But instead of dominictarr a hacker named @right9ctrl published it. And it was not with the intention to “make creating and working with streams easy” but to “steal your crypto currency”. And because nobody would use such a module voluntarily they encrypted the code and hijacked the trust and brand of the event-stream module to sneak malicious code into millions of other project installations.
So how did they get access to the official event-stream NPM account? Did they hack into the account? Did they use social engineering, did they blackmail dominictarr or was he even forced at gunpoint to give out the credentials? ..
No, the hacker simply asked for it via email. And that’s apparently enough to be given full publishing rights to 2M weekly (or 112 million yearly) installations!
Many developers are pi.. upset by the fact that they were recklessly put in danger, that their users were put in danger and people have lost or will lose their money in the future:
And with a lot of pressure from outside, the original developer released an official statement 2 days ago addressing some very valid points:
Hey everyone — this is not just a one off thing, there are likely to be many other modules in your dependency trees that are now a burden to their authors. I didn’t create this code for altruistic motivations, I created it for fun. […]
If it’s not fun anymore, you get literally nothing from maintaining a popular package. […]
So right now, we are in a weird valley where you have a bunch of dependencies that are “maintained” by someone who’s lost interest, or is even starting to burnout, and that they no longer use themselves.
I see two strong solutions to this problem…
1.) Pay the maintainers!! Only depend on modules that you know are definitely maintained!
2.) When you depend on something, you should take part in maintaining it.
Open Source
I have yet to see an industry where it is such a common practice to work for free and give so much of your work away for free as it is the case in the software space.
Open source developers are often doing it for “the fun” or they want to “give back” because they know we wouldn’t be where we are without open source software. However, the responsibilities and expectations we have towards these projects are the same we would have towards paid software. Developers and maintainers are often harassed for not “doing their work” or not responding “in time”. But we forget that most of these devs work on their own schedule and “in time” simply does not exist if you have a main job and want to live a life with a family and maybe a hobby that is not coding. It is often forgotten that they don’t owe anyone anything: npm install cat-ascii-faces --save does not include the right to demand a life-long maintenance service with this “purchase”.
(Malicious) Updates
Maintenance is hard and it is much easier to discontinue a project as soon as the usage statistics or payments drop than providing quality service for the last hundred or thousand remaining hardcore fans of a dying product. That these discontinued projects are becoming a massive security problem should be clear to everyone by now.
To cite Paul Betts, one of Electron’s core developers:
Getting updates right is extremely hard and even more so with the Electron framework that was used by the wallet and attacked in this case.
Ironically, Electron itself has a massive problem with its own dependencies such as a year-old outdated Chromium with publicly documented exploits in earlier versions like 1.8. And many applications that rely on these older Electron versions are putting millions of users at risk just by installing and not updating it.
Another problem with Electron and the reason why the attack was so targeted is that the malicious code running in Electron is escalated to a full remote code execution attack (RCE). In this case the code has full access to the file system and can read keyfiles and send them to a server in kuala lumpur (that actually happened in the attack). fs and net access are possible with Node.js and Electron but would be much more limited in a traditional sandboxed web context where a malicious NPM module would already be bad enough. In the case of the crypto wallet the injected code owns the complete host system as if the hacker is sitting in front of it with full access to keyboard, mouse, file system, camera,.. the attack is not just lucrative if we think about crypto wallets. The attack scenario we saw is not limited to stealing crypto.
So we have the following dilemma: we need frequent updates to patch existing security vulnerabilities but too frequent updates cannot be monitored efficiently and every update could potentially include malicious code. Which means even if we are frequently updating our software (as we should!) what is preventing a malicious NPM package to sneak into the app’s code base and take control over our machines?
NPM module security
Earlier this month, I gave a presentation at the Ethereum Devcon IV conference about browser and wallet security. One of my slides was this one which discussed attack vectors and successful attacks from the past for Electron-based applications.
The slide is based on the awesome blackhat presentation “Electronegativity” and threat model by Luca Carettoni:
NPM modules open up a massive attack surface and I discussed and warned the audience about it 3 weeks ago not knowing that another attack of this kind was just happening at the very moment. NPM or dependency security is not a specific problem to Electron or Node.js (even though it is much worse there). And it is also not new or unknown as we know since “I’m harvesting credit card numbers and passwords from your site. Here’s how”. But we will probably see many more of these attacks in the future.
And unfortunately these attacks won’t be solved by new audit features. And they won’t be solved with better AV integration or version pinning or AI or vulnerability checks and flagging on GitHub. Even though these things are a massive improvement and show that we are finally waking up and becoming more aware of our security issues.
So how can we fix our ecosystem?
In his released statement dominictarr writes:
open source is driven by sharing! It’s great! it worked really well before bitcoin got popular
Funnily enough I believe that blockchain technology is not the problem but the solution and we need two things to create a better ecosystem:
1.) We need an easier way to automatically distribute payments based on certain rules to open source developers (shout-out to Gitcoin)
2.) Code and modules need to be multi-signed and certificates should be inexpensive to get and easy to validate
Both things are traditionally very hard but we pretty much get them for free by using recent advances in “blockchain technology”.
So how would we have avoided the above scenario with blockchain™?
Let’s say we have a project that uses open source modules and has in-app payments with crypto.
One of our dependencies could become a malicious package in the same way we saw it with event-stream. But instead of giving full maintenance or full publishing rights to new collaborators, the original developer would give new maintainers or a CI system only signing rights. So they would sign their releases and after a careful review the original author would countersign stating that he checked and approves these changes.
Later, if our fictional project received an in-app payment there could be some logic within the runtime environment that based on the impact or contribution of this open source module to our project would automatically distribute a couple of cents of each payment to the author who can be determined based on the certificate. With 112 million downloads per year these cents could add up to more significant amounts and actually incentivize maintenance of popular and useful projects.
[With every release and additional signature new authors would of course get more trust for their own signatures. If something is audited and flagged it could have a financial penalty and so on and so forth]
While this sounds somewhat futuristic we could have it already today and we should try to get there as soon as possible.
Other projects have suggested signed modules based on PGP keys and code signing on a module-level sounds like a logical next step. We should adopt what is already common practice in many other software sectors. We also saw that signing the installer for the crypto wallet is not providing any guarantees about the included code and is therefore almost completely useless.
I’m pretty sure there are more ideas for improvements but module signatures and automatically handled micro payments are hopefully becoming a common thing soon and help the ecosystem to create higher quality code and help incentivize more open source developers to contribute because we need them so much.
Based on some feedback that I’ve received I would like to share another article and great project for those who are interested in the topic:
Skantek: Securing NodeJS at Adyen_By Kat Chilton, Senior Front-end Developer, Adyen_medium.com