Tell HN: HN Replies Data Security Incident

I'm not an hnreplies user, but IMO this is exactly what an incident disclosure should look:

  - What happened  
  - The extent of the damage  
  - Why it happened  
  - Why it is less likely to happen again in the future

Software is made by people, and people make mistakes. I don't think anyone is asking for perfect software (particularly not on this forum, and particularly not for free services offered by the community). But a clear picture of the breach and why it occured allows the user to make an informed decision about whether a service is worth using.

Maybe this is evidence of how low the bar is for incident reporting, or maybe its evidence of of "no publicity is bad publicity," but I just signed up for hnreplies.

Great service idea, and thanks for the honest and helpful disclosure!

A_No_Name_Mouse · 5 years ago

Plus it does not contain the classic line "At [...] we take security very seriously" which to me is one of the most stupid things to say when you've just been breached.

Deleted Comment

hsbauauvhabzb · 5 years ago

To be fair, accidents do happen. Nuclear reactors take things pretty seriously but they get breached / melt down occasionally too.

jrs235 · 5 years ago

I agree and it's why we must be prepared for WHEN "it" happens, NOT IF "it" happens.

gabrielsroka · 5 years ago

"I am convinced that there are only two types of companies: those that have been hacked and those that will be. And even they are converging into one category: companies that have been hacked and will be hacked again."

-- Robert Mueller, 2012, when he was Director of the FBI [yes, that Robert Mueller]

https://archives.fbi.gov/archives/news/speeches/combating-th...

> a .git directory on the VPS hosting this site was exposed to the internet by a configuration mistake, providing enough information to allow an attacker to access the site's database.

I had trouble figuring out how they got from point A to point B. My best guess is that a file containing database configuration settings had been committed to the Git repo at some point.

Lessons:

- Don't commit secrets to your Git repo. Store them in config files which are listed in .gitignore. Yes, this means you have to manually copy or write config files when you set up different servers, but those servers are likely to have different database credentials and the like anyway, right? If you've found that you've committed secrets by accident but haven't pushed anywhere yet, revert your branch to before the commit and redo or cherry-pick any subsequent commits safely. If it is too late, change the credentials to access the database or whatever other secrets you've revealed.

- Don't put web-accessible files in the root level of a Git repository. Instead, have a subdirectory contain the web-accessible files, and configure your web server to serve from that. That ensures that the .git directory won't be right next to web-accessible files, and has other benefits as well - you can also easily have subdirectories for docs, CLI scripts, and other things that shouldn't be web-accessible right there in the same repo. But if that isn't an option (or even if it is), configure your web daemon to not serve files from .git, or probably any other file or directory that starts with a dot.

- Edit to add one more: As much as possible, lock down access to your database server. An attacker having your database credentials is bad, but they still won't get far if the database server is configured to only listen to connections from the IP address that the web application server is hosted at, or, if they're on the same server, only over local sockets.

Of course, the same applies to other VCSes besides Git.

darekkay · 5 years ago

> have a subdirectory contain the web-accessible files

You can also separate your git repository from the working directory. I have my git repositories at ~/repositories, with a post-receive hook that checks out the working directories at ~/projects and then _actively_ copies only relevant files to ~/html. Basically a "poor man's CI/CD", with the bonus of having to opt-in to publish any files.

DCoder · 5 years ago

You can also tell git that the .git dir is completely separated from the working dir:

    alias dogit='GIT_WORK_TREE=/path/to/src/dir GIT_DIR=/path/to/dotgit/dir git'
    dogit pull remote treeish

---

I set this up as a "poor man's version control" on a large client's server back in 2016-ish. I was lost for words when last year I saw my colleagues still using Ctrl-R to rerun that alias line.

arlk · 5 years ago

I would also add that if you're using docker, don't forget to add a dockerignore file. I often see people miss this step and end up with exposed secrets and git folder (and poor build times).

mtberatwork · 5 years ago

> Yes, this means you have to manually copy or write config files when you set up different servers, but those servers are likely to have different database credentials and the like anyway, right?

No, you can use GitHub Secrets, AWS Secrets Manager, etc ... or encrypt your config files and add a decrypt step to your build. [1]

[1] https://docs.github.com/en/actions/reference/encrypted-secre...

gempir · 5 years ago

Personally my lazy solution to config files is using ansible-vault.

It basically lets you encrypt your secrets and store them in your git how much you want. You then only need one password when decrypting the vault.

jmknoll · 5 years ago

Cyberdog · 5 years ago

jvolkman · 5 years ago

Just want to say thanks to Dan Grossman for providing this free service. I find it really useful.

gregsadetsky · 5 years ago

Same here, kudos Dan! HN Replies has allowed a lot of interesting conversations to happen that probably would not have taken place otherwise.

Best of luck with everything

zupa-hu · 5 years ago

Came here to say the same. I really appreciate the awesome service. I also really appreciate being upfront with the incident.

I badly hope nobody is going to be offensive here. (Mind you, I’m watching!)

THANK YOU, Dan!

joshstrange · 5 years ago

I also just got this email and it actually fills me with confidence. Everyone makes mistakes and while I'm a lot better about using ENV vars and the like to pass secrets into code, it was wasn't that long ago that I was committing secrets to git because it was way easier. Thankfully this was a rather low-stakes mistake for me as my email and username are already linked (though I understand that is not the case for everyone). I'm also very happy with how Dan handled this. A user reached out to warn him about the breach, he investigated, found the issue, and communicated it to his users. A /very/ different situation from how Ubiquiti handled a much higher-stakes breach.

hnnnnnnng · 5 years ago

>For a period of several days earlier this year, a .git directory on the VPS hosting this site was exposed to the internet by a configuration mistake, providing enough information to allow an attacker to access the site's database.

Storing credentials in git. Ahh a classic!

michaelmcmillan · 5 years ago

Binding the database to 0.0.0.0 is not a good idea.

hamhamed · 5 years ago

Thanks for the free service, email getting leaked is the least of my concerns. Can't get mad, thank you Dan. HN should have this natively

atat7024 · 5 years ago

Promotes rapid responses, which doesn't promote well considered, goal-oriented, and empathetic responses.

prawn · 5 years ago

My identity is clear from my profile link so this is no hassle for me, but I assume there could be at least loose repercussions for people who followed their own, anonymous-but-opinionated accounts?

swyx · 5 years ago

some HN user once observed that this was a fun single point of failure to pwn, mapping HN usernames to emails. people who intend to stay anonymous need to be aware this is a risk when using these services.

Akronymus · 5 years ago

I signed up for the HN replies fully expecting it to be pwned sooner or later. Actually, I assume pretty much every service to be pwned at some point, and sign up according to how it'd impact me. If I don't care at all about the association, I use my real address, if I care somewhat I use a "+something" gmail address.

And if I REALLY don't want to be associated, I use a relay email address.

jedimastert · 5 years ago

TIL hnreplies exists! It's exactly what I've been looking for.

the-dude · 5 years ago

There is no such thing as bad publicity

https://idioms.thefreedictionary.com/there+is+no+such+thing+...