Readit News logoReadit News
jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
3np · 2 days ago
It still says:

> IT NEVER ESCAPED.

You haven't confirmed this (at least from the contents of the article). You did some reasonable spot checks and confirmed/corrected your understanding of the setup. I'd agree that it looks likely that it did not escape or gain persistence on your host but in no way have you actually verified this. If it were me I'd still wipe the host and set up everything from scratch again[0].

Also your part about the container user not being root is still misinformed and/or misleading. The user inside the container, the container runtime user, and whether container is privileged are three different things that are being talked about as one.

Also, see my comment on firewall: https://news.ycombinator.com/item?id=46306974

[0]: Not necessarily drop-everything-you-do urgently but next time you get some downtime to do it calmly. Recovering like this is a good excercise anyway to make sure you can if you get a more critical situation in the future where you really need to. It will also be less time and work vs actually confirming that the host is uncontaminated.

jakelsaunders94 · 2 days ago
I did see your comment on Firewall, and you're right about the escape. It seems safe enough for now. Between the hacking and accidentally hitting the front page of HN it's been a long day.

I'm going to sit down and rewrite the article and take a further look at the container tomorrow.

jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
marwamc · 2 days ago
Hahaha OP could be in deep trouble depending on what types of creds/data they had in that container. I had replied to a child comment but I figure best to reply to OP.

From the root container, depending on volume mounts and capabilities granted to the container, they would enumerate the host directories and find the names of common scripts and then overwrite one such script. Or to be even sneakier, they can append their malicious code to an existing script in the host filesystem. Now each time you run your script, their code piggybacks.

OTOH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here. The $HOME/.{aws,docker,claude,ssh} Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.

jakelsaunders94 · 2 days ago
Nothing in that container luckily, just what Umami needed to run, so no creds at all. Thanks for the info though!
jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
tgtweak · 2 days ago
Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.
jakelsaunders94 · 2 days ago
This is a great shout actually. Thanks for pointing it out!
jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
qingcharles · 2 days ago
As an aside, if you're using a Hetzner VPS for Umami you might be over-specced. I just cut my Hetzner bill by $4/mo by moving my Umami box to one of the free Oracle Cloud VPS after someone on here pointed out the option to me. Depends whether this is a hobby thing or something more serious, but that option is there.
jakelsaunders94 · 2 days ago
I've got a whole Hetzner EX41 bare metal server, as opposed to a VPS. It's gotr like 20 services on it.

But yeah it is massively overspecced. Makes me feel cool load testing my go backend at 8000 requests per second though!

jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
tolerance · 2 days ago
Was dad notified of the security breach? If not he may want to consider switching hosting providers. Dad deserves a proper LLM-free post mortem.
jakelsaunders94 · 2 days ago
Hahaha, I did tell him this afternoon. This is the bloke who has the same password for all his banking apps despite me buying him 1password though. The imminent threat from RCE's just didn't land.
jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
heavyset_go · 2 days ago
I wouldn't trust that boot image or storage again, I'd nuke it for peace of mind.

That said, do you have an image of the box or a container image? I'm curious about it.

jakelsaunders94 · 2 days ago
Yeah I did consider just killing it, I'm going to keep an eye on it for a few days with a gun to it just in case.

I was lucky in that my DB backups were working so all my persistence wax backed up to S3. I think I could stand up another one in an hour.

Unfortunately I didn't keep an image no. I almost didn't have the foresight to investigate before yeeting the whole box into the sun!

jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
seafoamteal · 3 days ago
Hi Jake! Cool article, and it's something I'll keep in mind when I start giving my self-hosted setup a remodel soon. That said, I have to agree with the parent comment and say that the LLM writing style dulled what would otherwise have been a lovely sysadmin detective work article and didn't make me want to explore your site further.

I'm glad you're up to writing more of your own posts, though! I'm right there with you that writing is difficult, and I've definitely got some posts on similar topics up on my site that are overly long and meandering and not quite good, but that's fine because eventually once I write enough they'll hopefully get better.

Here's hoping I'll read more from you soon!

jakelsaunders94 · 3 days ago
Thanks for the encouragement! I find it difficult to write articles beyond simply stating a series of facts.

I tried handwriting https://blog.jakesaunders.dev/schemaless-search-in-postgres/ bit I thought it came off as rambling.

Maybe I'll have a go at redrafting this tomorrow in non LLM-ese.

jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
minitech · 3 days ago
> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

  /tmp/.XIN-unix/javae &
  rm /tmp/.XIN-unix/javae
This article’s LLM writing style is painful, and it’s full of misinformation (is Puppeteer even involved in the vulnerability?).

jakelsaunders94 · 3 days ago
I fixed it, apologies for the misinformation.
jakelsaunders94 commented on I got hacked: My Hetzner server started mining Monero   blog.jakesaunders.dev/my-... · Posted by u/jakelsaunders94
minitech · 3 days ago
> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

  /tmp/.XIN-unix/javae &
  rm /tmp/.XIN-unix/javae
This article’s LLM writing style is painful, and it’s full of misinformation (is Puppeteer even involved in the vulnerability?).

jakelsaunders94 · 3 days ago
Yeah fair, I asked claude to help because honestly this was a little beyond my writing skills. I'm real though. Sorry. Will change

u/jakelsaunders94

KarmaCake day265January 13, 2022View Original