Maybe I missed it but the article doesn't mention the even easier way to see this: the activity tab.
It has everything. Any force push to hide ugly prototype code is kept forever which annoys me. I wish we were able to remove stuff from there but the only way to do it is to email support it seems?
Looking at some of my projects, it's entirely empty, or only has a few items, so I suspect it was introduced "recently" and doesn't have data from before then.
Funny thing, we had a similar issue with one of our deployement in the past. It's similar to leaking accidently your password into bash history. Happens more than it should.
1. Not all secrets can be rotated. E.g. I can't just "rotate" my home address, which I prefer to be private.
2. Even for rotatable secrets, "I don't think there is any potential further damage" rests on the assumption that the secret is 100% invalidated everywhere. What if there are obscure and/or neglected systems, possibly outside of your control, that still accept that secret? No system is bug-free. If I can take steps to minimize access to an invalidated secret, I will.
Anyone who puts weight on digging through a project to see if they've ever leaked a secret is guilty of encouraging an antipattern-- the guaranteed outcome is you'll have an organization petrified of shipping anything, in case someone interprets it as bad or a security risk, etc.
All devs should run open-source trufflehog as a precommit hook for all repositories on their local system. It’s not a foolproof solution, but it’s a small time investment to get set up and gives me reasonable assurance that I will not accidentally commit a secret. I’m unsure why this is not more widely considered standard practice.
This isn't true in practice. Even among well educated high performing professionals, mistakes happen. Checklists save lives - in medicine, in aircraft maintenance, in all fields.
People who believe they know what they're doing get overconfident, move fast, and make mistakes. Seasoned woodworkers lose fingers. Experienced doctors lose patients to preventable mistakes. Senior developers wipe the prod database or make a commit they shouldn't.
>In a study of 100 Michigan hospitals, he found that, 30 percent of the time, surgical teams skipped one of these five essential steps: washing hands; cleaning the site; draping the patient; donning surgical hat, gloves, and gown; and applying a sterile dressing. But after 15 months of using Pronovost’s simple checklist, the hospitals “cut their infection rate from 4 percent of cases to zero, saving 1,500 lives and nearly $200 million,”
Pre-commit hooks are client-side only and opt-in; I've always been a big proponent of pre-commit hooks, as the sooner you find an issue the cheaper it is to fix, but over time pre-commit hooks that e.g. run unit tests tend to take longer and longer, and some people want to do rapid-fire commits instead of being a bit more thoughtful about it.
- enforce them on CI too; not useful for secrets but at least you're eventually alerted
- do not run tasks that take more than a second; I do not want my commit commands to not be instant.
- do not prevent bad code from being committed, just enforce formatting; running tests on pre-commit is ridiculous, imagine Word stopping you from saving a file until you fixed all your misspellings.
I moved all my precommit hooks to prepush hooks. I don’t need a spellchecker disrupting my headspace when I’m deep into a problem.
My developer environments are setup to reproduce CI test locally, but if I need to resort to “CI driven development” I can bypass prepush hooks with —-no-verify.
One good (and obviously bad) thing about Subversion was the ability to change history. As admin I was asked numerous times to change a commit message. To point to the correct Jira issue, for instance.
Also easier to enforce pre-commit, since it was done server side.
What I've never understood is, how is this an issue with private repos? Aside from open source projects I can't see the problem with accidentally doing this, even though it is a smell.
In a large messaging app I worked for we self hosted a gitlab instance for this exact reason. I thought it was over the top but now I get it, you can never be too sure.
Different employees in the company have different permissions. If an employee with a lot of access commits a secret, then employees who shouldn't have that much access can take the secret and use it.
Anything that makes the repo less private later (deliberate public release, hack (not just if the repo bit of anything that can connect to it), etc) means the secret is now in the open.
Always cycle credentials after an accident like committing them to source control. Do it immediately, you will forget later. Even if you are 100% sure the repo will never be more public, it is a good habit to form.
It's called private but actually shared with a very large corporation you don't control, likely running on infrastructure they don't control. Due to the CLOUD Act it's also shared with the US government.
Exactly; you should fully expect the NSA to have a copy of these logs as well. It can be very valuable to have secret keys from companies in adversarial countries (including your own).
Example, there's an ICE reporting app now where people can anonymously report ICE sightings... but how anonymous is it really? Users report a location, that can be cross-referenced with location histories and quicky led back to an individual. There may be retaliation to users of this app if the spiral into authoritarianism in the US continues.
Many years ago at my first job after university, I accidentally committed a private key into our internal Git repository. We removed it, because we could not completely rule out the possibility that this repository would be made public to a customer, or to the world, in the future. I think we used filter-repo to get the key out of everywhere.
I got tired of "oops" over time and started abusing environment variables. If you have enough discipline to spend 10 seconds configuring them, you'll never have to worry about magic strings accidentally getting sucked up into source control.
The other upside with environment variables is that they work across projects. Set & forget, assuming you memorized the name. Getting at tokens for OpenAI, AWS, GH, etc., is already a solved problem on my machine.
I understand why a lot of developers don't do this though. Especially on Windows, it takes a somewhat unpleasant # of clicks to get to the UI that manages these things. It's so much faster (relatively speaking) to paste the secret into your code. This kind of trivial laziness can really stack up on you if you aren't careful.
I encrypt any secret strings with a master password that lives either in a TPM module or a file named MASTER_SECRET that is absolutely not added to the Git repo. My standard new project script adds this file to .gitignore and I use a pre-commit hook that stops this file from being committed by accident.
If you push a secret publicly, you should consider it leaked. On GitHub, you have 5 minutes on a non-watched repository (due to the delay) and less than 30 seconds on a watched repository to revoke it before it's been cloned and archived by a third-party. Whether that party is malicious or not, rewriting the Git history will not change anything that the secret is leaked. And you can already rewrite the Git History and garbage collect commits that aren't part of the tree anymore on most providers.
If something got out to the internet, you won't get it back. There is little point in rewriting repo history if you have already made a secret public. Just change the secret as soon as you can.
The person who leaked it and the person/team that can rotate it might be in different silos or timezones etc. Rewriting the history is prudent but not sufficient.
For a long time and probably still today, Google AppEngine kinda encouraged storing secrets in the YAML, which is easy to accidentally git-commit. There's no easy way to pass secrets to your services otherwise, unlike Heroku etc where it's always been a single command to put them into envvars on the jobs.
And can we talk about the predatory pricing model? In AWS one secret service prices a secret for 0.4 dollars a month.
I was appalled when I first saw it, are you going to charge me 5$ a year for storing my 12 bytes?
Readit News
It has everything. Any force push to hide ugly prototype code is kept forever which annoys me. I wish we were able to remove stuff from there but the only way to do it is to email support it seems?
Here it is for the test repo mentioned
https://github.com/SharonBrizinov/test-oops-commit/activity
Looking at some of my projects, it's entirely empty, or only has a few items, so I suspect it was introduced "recently" and doesn't have data from before then.
Picking https://github.com/jellyfin/jellyfin/activity?sort=ASC as a busy example, Activity page has no data prior to 7th March 2023. So it has existed for 2 of GitHub's 17 years of existence.
Deleted Comment
Not if you contact customer support and ask them to garbage collect your repo.
What I do when I accidentally push something I don’t want public:
- Force push;
- Immediately rotate if it’s something like a secret key;
- Contact customer support to gc the repo (and verify the commit is gone afterwards).
(Of course you should consider the damage done the moment you pushed it. The above steps are meant to minimize potential further damage.)
2. Even for rotatable secrets, "I don't think there is any potential further damage" rests on the assumption that the secret is 100% invalidated everywhere. What if there are obscure and/or neglected systems, possibly outside of your control, that still accept that secret? No system is bug-free. If I can take steps to minimize access to an invalidated secret, I will.
In my head, the people who accidentally share secrets are also the people who couldn't setup trufflehog with a precommit.
People who believe they know what they're doing get overconfident, move fast, and make mistakes. Seasoned woodworkers lose fingers. Experienced doctors lose patients to preventable mistakes. Senior developers wipe the prod database or make a commit they shouldn't.
https://hsph.harvard.edu/news/fall08checklist/
>In a study of 100 Michigan hospitals, he found that, 30 percent of the time, surgical teams skipped one of these five essential steps: washing hands; cleaning the site; draping the patient; donning surgical hat, gloves, and gown; and applying a sterile dressing. But after 15 months of using Pronovost’s simple checklist, the hospitals “cut their infection rate from 4 percent of cases to zero, saving 1,500 lives and nearly $200 million,”
Unfortunately, that is impossible: https://trufflesecurity.com/blog/anyone-can-access-deleted-a...
- enforce them on CI too; not useful for secrets but at least you're eventually alerted
- do not run tasks that take more than a second; I do not want my commit commands to not be instant.
- do not prevent bad code from being committed, just enforce formatting; running tests on pre-commit is ridiculous, imagine Word stopping you from saving a file until you fixed all your misspellings.
My developer environments are setup to reproduce CI test locally, but if I need to resort to “CI driven development” I can bypass prepush hooks with —-no-verify.
Also easier to enforce pre-commit, since it was done server side.
- commit secret in currently private repo
- 3 years later share / make public
- forget the secret is in the commit history, and still valid, (and relatedly, having long-lived secrets is less secure)
Sure that might not happen for you, but the chances increase dramatically if you make a habit of commiting secrets.
Always cycle credentials after an accident like committing them to source control. Do it immediately, you will forget later. Even if you are 100% sure the repo will never be more public, it is a good habit to form.
Example, there's an ICE reporting app now where people can anonymously report ICE sightings... but how anonymous is it really? Users report a location, that can be cross-referenced with location histories and quicky led back to an individual. There may be retaliation to users of this app if the spiral into authoritarianism in the US continues.
The other upside with environment variables is that they work across projects. Set & forget, assuming you memorized the name. Getting at tokens for OpenAI, AWS, GH, etc., is already a solved problem on my machine.
I understand why a lot of developers don't do this though. Especially on Windows, it takes a somewhat unpleasant # of clicks to get to the UI that manages these things. It's so much faster (relatively speaking) to paste the secret into your code. This kind of trivial laziness can really stack up on you if you aren't careful.
Checkout to event, commit in clean state with prior log history, overlay the state after the elision and replace git repo?
When I had to retain log and elide state I did things like this in RCS. Getting date/time info right was tricky.
Last time I tried, the default suggestion was Cloud KMS (yeah), now there's some new secret manager that also looks annoying: https://stackoverflow.com/questions/58371905/how-to-handle-s...
So I just hard coded the key. The key was rotated after the presentation.
Does not look very good on a repo.