We've seen nearly weekly attacks against the open source software supply chain. I saw the seeds of this trend start in the mid 2015s as an open source maintainer and I've watched it only get worse over the years. I finally decided to try to solve this problem.
Socket is taking an entirely new approach to one of the hardest problems in security in a stagnant part of the industry that has historically been obsessed with just reporting on known vulnerabilities. Unlike other scanning tools, Socket actually analyzes the package code to characterize the package's behavior. This way, Socket can detect when packages use security-relevant platform capabilities, such as the network, filesystem, or shell.
You can search for any npm package and see issues that've we've flagged for each package. We look for 70 issues (full list here: https://socket.dev/npm/issue) and we put those into a Package Health score. See these examples:
https://socket.dev/npm/package/left-pad
https://socket.dev/npm/package/lodash
Socket looks for indicators present in all of the recent npm supply chain attacks. We're proactively auditing every package on npm to flag these issues.
Separately, we have a GitHub app that you can install. It detects typosquat attacks and leaves a comment on your pull request to let you know you might have installed the wrong package. We're currently working to enable it to leave comments for more of the package issues that we can detect, but we want to get the UX really good on that first, so we've released it and labeled it "beta".
Happy to answer questions.
There are certain ways you can objectively determine the security of a package:
- If it or one of its dependencies has been compromised
- If it and its dependencies are made by a trusted org (e.g. I would trust something from Microsoft more than something from a random developer)
- If it and its dependencies have been audited and verified (this could even be a quality audit, or something like the Elm ecosystem)
- If it and its dependencies are made by you, you can be pretty confident that nobody has tampered with them
Similarly there are ways to determine the quality (e.g. code coverage, usage, whether it has tests). And of course you can see the license.
But state these things directly, don't try to summarize them with numbers. Because ultimately those numbers are almost meaningless and make your tool look fake.
What's interesting is that arbitrary, unit-less scores seem to work well in products like Lighthouse [1] and SSL Labs [2] even though they're also "basically made up" metrics. I'm curious if anyone has ideas about why this may be the case.
To your point about just stating things directly, we also do that too. See https://socket.dev/npm/package/left-pad/issues, for example.
[1]: https://developers.google.com/web/tools/lighthouse [2]: https://www.ssllabs.com/ssltest/index.html
I love the direction you're going with this. Would love to hear more about:
1. Status. Is this a beta product or do you have enough here to sell a service?
2. Roadmap. What are you building out next? What are core capabilities you see being added to Socket?
3. Pricing. How much does it cost? What's your revenue model?
Depending on how you've put this together you may be addressing a serious need in this space that I've only previously seen addressed by stuff like Deno (where you set application capabilities prior to execution) or via virtualization.
Contact details in my profile. If your product does what it says on the tin I'd love to give it a spin.
2. https://socket.dev/roadmap See https://socket.dev/npm/issue for analysis we have developed on a per-package basis. We are working on integrating these with our GitHub App to provide custom tailored project views into this data. More info here https://socket.dev/blog/inside-node-modules
3. It will be a paid product, but is free while it is in beta. We plan on keeping it free for open source.
We are looking into providing support for the Deno ecosystem down the road as well. The capabilities stuff they have is super great, but you lose all of that benefit for every dependency as soon as you turn it off, so we think there is probably room for this kind of analysis there. Hopefully Socket can provide a similar signal that --allow-net provides, but for all of npm!
[1] https://socket.dev/npm/package/lodash/files/4.17.21/package....
[2] https://directory.fsf.org/wiki/License:Expat
If you find more examples like this, please email them to me (see bio) so we can fix them. We appreciate the feedback!
Also "left-pad Supply Chain Security: 0", lmao.
Packages can be finished and do everything they set out to do. The fact they haven't been updated in a while can be a measure of high quality as much as it can be a sign of low quality.
So is Socket.dev open source? On your Github (https://github.com/SocketDev) I cannot find the Github App source code. So being such proponents of open source, why don't you release it as open source?
As a maintainer I know how much time it takes to run an open source project beyond just writing code. There's triaging issues, reviewing PRs, answering questions, adding support for alternative operating systems/runtimes, and more. At this precise moment we just don't have that capacity, although I look forward to the day when we'll be able to share more of the code publicly.
There is both a large need for improvements in npm supply chain security and a market willing to pay for them.
Concerns:
1) npm Open-Source Terms condition four[1] states.
You may access and use data about the security of Packages, such as vulnerability reports, audit status reports, and supplementary security documentation, only for your own personal or internal business purposes. You may not provide others access to, copies of, or use of npm data about the security of Packages, directly or as part of other products or services.
This statement seems vague enough to potentially include your use case. It also seems to include what snyk, jfrog xray, sonatype, and white-source do, so maybe this is not an issue.
2) It appears that this will be an open-core business. What capabilities are you willing to provide in the free/community edition and under which licenses?
3) The website doesn't show pricing. Can you provide details on this?
Questions:
1) What are your thoughts on using reproducible builds[2] plus Diverse Double-Compiling (DDC)[3] on the dependency graph to ensure build artifacts originate from known git repositories? Disclosure, I've been working on this for a few months now.
2) Where do you run your analysis? AWS and DigitalOcean have terms that prevent running high risk code.
3) Do you have examples of previous attacks and how your tooling would handle them?
Best of luck.
[1] https://docs.npmjs.com/policies/open-source-terms#conditions [2] https://reproducible-builds.org/ [3] https://dwheeler.com/trusting-trust/
2 and 3) We're still figuring out the business model, but here's our current plan: Package search and Package Health Scores are free for everyone to use through our website https://socket.dev.
Socket integrations, such as the GitHub App, are free for open source repositories forever. For private repositories, Socket is free while we're in beta, but we'll eventually charge something like ~$20/developer/month for private repos. We're still working out pricing but our #1 aim is to keep it affordable so everyone can get protected.
Question 1) I love this idea! This is something the team is already talking about. We want Socket to report reproducible builds and use them as a positive signal, as well as highlight them as a badge on the package page. For npm packages, lots of them probably already have reproducible builds that we can check by just running `npm install; npm build; npm pack`. I need to think more about DDC and how that would fit it. Perhaps we can chat about it sometime?
2) We're currently doing static analysis, so not actually running the code. Our dynamic analysis isn't ready yet so we'll cross that bridge when we get there.
3) All of the issues that Socket detects were picked with previous npm supply chain attacks in mind. You can see a list packages npm removed for security reasons here: https://socket.dev/npm/category/removed When you view any of these, we show the results of our security analysis. Here is a removed package I just picked at random to give you an idea:
https://socket.dev/npm/package/netlify-swag/files/1.2.0/inde...