I love the idea of self-hosting, especially since I keep a number of very tiny websites/projects going at any given time, so resources would not really be too much of an issue for me.
What stops me is security. I simply do not know enough about securing a self-hosted site on real hardware in my home and despite actively continuing to learn, it seems like the more I learn about it, the more questions I have. My identity is fairly public at this point, so if I say the wrong thing to the wrong person on HN or whatever, do I need to worry about someone much smarter than me setting up camp on my home network and ruining my life? That may sound really stupid to many of you, but this is the type of anxiety that stops the under-informed from trying stuff like this and turning to services like Akamai/Linode or DO that make things fairly painless in terms of setup, monitoring and protection.
That said, I'm 110% open to reading/watching any resources people have that help teach newbies how to protect their assets when self-hosting.
A few days after a remark on hn, while the thread was still active, I received a mysterious package I didn't order from a weird drop shipping service where the original sender is unknown and undiscoverable to you the recipient. It didn't contain anything bad just a single surgical mask (during covid, common valueless item basically). The message was just that they could find my home address. It was a stupid message since I obviously do not hide my identity on hn. But it means you're not wrong to be careful, both in general, and on hn in particular.
Hmm, my first guess would have been that you have been a target of "brushing" [1].
In a Reddit thread from 2020 [2], multiple people mention that they received surgical masks they did not order.
It's always scary, no matter how innocuous. I'm glad it did not escalate into something else for you!
Without getting too deep into it, there are some things I know how to do with computers that I probably shouldn't, so my thought is this; if I, a random idiot who just happened to learn a few things, can do X, then someone smarter than me who learned how to attack a target in an organized way probably has methods that I cannot even conceive of, can do it easier, and possibly without me even knowing. It's this weird vacillation between paranoia and prudence.
For me, it's really about acknowledging what I know I don't know. I do some free courses, muck about with security puzzles, etc, even try my own experiments on my own machines, but the more I learn, the more I realize I don't know. I suppose that's the draw! The problem is when you learn these things in an unstructured way, it's hard to piece it all together and feel certain that you have covered all your vulnerable spots.
I'm right there with you, except at times I have thrown caution to the wind and made my sites available.
My current setup is to rent a cheap $5/month VPS running nginx. I then reverse ssh from my home to the vps, with each app on a different port. It works great until my electric goes out and comes back on the apps become unavailable. I haven't gotten the restart script to work 100% of the time.
But, I'd love to hear thoughts on security of reverse SSH from those that know.
I do something similar with my home server, but with a WireGuard split tunnel. Much easier to set up and keep active all the time (i.e., on my phone).
Nginx handles proxying and TLSing all HTTP traffic. It also enforces access rules: my services can only be reached from my home subnet or VPN subnet. Everywhere else gets a 403.
Maybe try running your services in docker, I don't know how difficult that would be to implement for you, but if you run it in containers you can get it to start up after an outage pretty reliably.
> A VPS with a software firewall is more than enough.
So you don't self-host at home, right?
I have been considering setting up a physical DMZ at home, with two routers (each with its own firewall), such that my LAN stays unmodified and my server can run between both routers. Then it feels like it would be similar to having a VPS in terms of security, maybe?
I agree - I always wonder should I go overkill and put everything in its own VM for separation? Is it ok to just use containers?
If using Podman, should I use rootless containers (which IMO suck because you can't do macvlan so the container won't easily get its own IP on my home network)? Is it ok to just use rootful Podman with an idmapped user running without root privileges inside the container and drop all unneccessary capabilities? Should I set up another POSIX user before, such that breaking out of the container would in the end just yield access to an otherwise unused UID on the host?
If using systemd-nspawn, do all the above concerns about rootful / rootless hold? Is it a problem that one needs to run systemd-nspawn itself with root? The manpage itself mentions "Like all other systemd-nspawn features, this is not a security feature and provides protection against accidental destructive operations only.", so should I trust nspawn in general?
Or am I just being paranoid and everything should just be running YOLO-style with UID 1000 without any separation?
All of this makes me quite wary about running my paperless-ngx instance with all my important data next to my Vaultwarden with all of my passwords next to any Torrent clients or anything else with unencrypted open ports on the internet. Also keeping everything updated seems to be a full time job by itself.
Isn't 95% of it just blocking every port except the service you want to expose, and then making sure everything is up to date and the service is built in a secure way.
WAF's etc just hide the fact the code in your service is full of holes.
Could you give an example of a guide that helped you self host a system or service by telling you how to do the security? One that shows what information would be missing from a regular setup tutorial?
I'm a security consultant so this is not a problem I have. To me it seems very straightforward and like most things are secure by default (with the exceptions being notorious enough that I'd know of it), so I'm interested in the other perspective
This is a good concern to have. I feel like the emotional currency around self-hosting on tech forums makes too many people excited to talk about self-hosting and forget about practical things like security. Remember: defense in layers.
Things I do:
* Make sure domain WHOIS does not point to me in any way, even if that means using some silly product like "WHOIS GUARD"
* Lock down any and all SSH access. Preferably only allow key-based authentication.
* Secure the communication substrate. For me this means running a Zerotier network which all dependent services listen on. I also try to use Unix sockets for any services colocated on the same operating system and restrict the service to only listen on sockets in a directory specifically accessible by the service.
* Try to control the permission surface of any service as much as possible. Containers can be a bit heavyweight for self-hosting but make this easy. There's alternatively like bubblewrap and firejail as well.
* Make use of services like fail2ban which can automate some of the hunting of bad actors for you.
* Consider hosting a listener for external traffic outside of your infra. For redundancy, load-shedding, and for security I have an external VPS that runs haproxy before routing over Zerotier to my home infrastructure. I enforce rate limits and fail2ban at the VPS so that bad actors get stopped upstream and use none of my home compute or bandwidth. (I also am setting up some redundant caches that live on the VPS so if my home network is down, one of my services can failover.)
* Segregate data into separate databases and make sure services only have access to databases that they need. With Postgres this is really simple with virtual databases being tied to different logins. I have some services that prune databases that run in a cron-like way (but using snooze instead) and they have no outbound net access.
If your network layer is secure and your services follow least-privilege, then you should be fairly in the clear.
You should encrypt and backup your assets regularly. We recently published a tutorial on how to to do so using B2 and Infisical so that your private key doesn't live on the server: https://infisical.com/blog/self-hosting-infisical-homelab
Take a look at sandstorm.org - its set of apps is fairly limited compared to the docker based options but it goes incredibly far wrt security. It was designed by the now head of Cloudflare Workers and pitched as a selfhosting platform for medical and other highly regulated industries. There's still nothing else quite like it
> pitched as a selfhosting platform for medical and other highly regulated industries
From first hearing about Sandstorm since the first open beta 10 years ago (https://news.ycombinator.com/item?id=10147774) and reading about it on/off since then, this is first time I hear anyone pitching it for "medical and other highly regulated industries". Where exactly does this come from?
> There's still nothing else quite like it
Plenty of other similar self-hosted platforms, YunoHost is probably the closest, most mature and most feature-packed alternative to Sandstorm, at least as far as I know,.
I think a normal patched Debian/Ubuntu with ufw rule for port 80/443 and 22, ssh certificate auth only and a simple nginx configuration is still very safe.
Of course there can be security issue on your webserver as well, but for a simple site this setup is learnable in an hour or two and you are ready to go.
You can hook that up on a pie attached to your router or pay a bit to have it hosted somewhere. Domain is perhaps 2-5$ and an TLS cert you can get from Let's Encrypt.
No idea how to put everything into a container that it makes sense. I just run this quite often on small hosted machines elsewhere. I just install everything manually because it takes 5 minutes if you have done it before.
Since my setup is for personal use I just use a VPN. My home router is running OPNsense and this setup wasn't too bad. I also pay my ISP for a static IP address.
It is still my opinion that most businesses do not need Kubernetes, and neither should anyone self-hosting a service at home.
I can see running something in a Docker container, and while I'd advise against containers what ships with EVERYTHING, I'd also advise against using Docker-compose to spin up an ungodly amount of containers for every service.
You shouldn't be running multiple instances of Postgresql, or anything for that matter, at home. Find a service that can be installed using your operating systems package manager and set everything to auto-update.
Whatever you're self-hosting, be it for yourself, family or a few friends, there's absolutely nothing wrong with SQLite, files on disk or using the passwd file as your authentication backend.
If you are self hosting Kubernetes to learn Kubernetes, then by all means go ahead and do so. For actual use, stay away from anything more complex than a Unix around the year 2000.
> You shouldn't be running multiple instances of Postgresql, or anything for that matter, at home.
It's not uncommon with self-hosting services using docker. It makes it easier to try out a new stack and you can mix and match versions of postgresql according to the needs of the software. It's also easier to remove if you decide you don't like the bit of software that you're trying out.
I shared this sentiment. But since I just host some personal fun projects and I got really lazy when it comes to self-hosting, I found great pleasure in just creating the simplest possible docker containers. It just keeps the system super clean and easy to wipe and setup again. My databases are usually just mounted volumes which do reside on the host system
Exactly, my first reaction was "I should write a blog post about why I still use Docker Swarm". I deploy to single node swarms, and it's a zero boiler plate solution. I had to migrate services to another server recently, and it was really painless. Why oh why doesn't Docker Swarm get more love (from its owners/maintainers and users)?....
I'd be interested. Might be a strange question but I'll throw it out there, I seem to have a hard time finding a good way to define my self hosted infrastructure nodes and which containers can run on them, have you run into/have a solution for this? Like I want my database to run on my two beefier machines but some of the other services could run on the mini pcs.
> I deploy to single node swarms, and it's a zero boiler plate solution.
Yup, it's basically like a "Docker Compose Manager" that lets you group containers more easily, since the manifest file format is basically Docker Compose's with just 1-2 tiny differences.
If there's one thing I would like Docker Swarm to have, is to not have to worry about which node creates a volume, I just want the service to always be deployed with the same volume without having to think about it.
That's the one weakness I see for multi-node stacks, the thing that prevents it from being "Docker Compose but distributed". So that's probably the point where I'd recommend maybe taking a look at Kubernetes.
I moved us off docker swarm to GKE some years back. The multi node swarm was quite unstable, and none of the big cloud providers offered managed swarm in the same way they offer managed k8s.
It's a shame I agree because it was nicely integrated with dockers own tooling. Plus I wouldn't have had to learn about k8s :)
I am very interested. I tried to migrate to Swarm, got annoyed at incompatibility with tons of small Docker Compose things, and decided against that. I'd love to read about your setup.
Exactly. I am hosting 30+ services using docker compose and very happy. I don’t want to troubleshoot k8s in the early morning because home assistant is down and light dimmers are not working for some random k8s reason.
Since I've migrated our company to Kubernetes, I almost stopped to worry about anything. It just works. Had much more troubles running it with spaghetti of docker containers and host-installed software on multiple servers, that setup definitely breaks like every week or every month. With Kubernetes I just press "update cluster" in some saturday evening once or twice a year and that's about it, pretty smooth sailing.
All my "smart home" stuff needs is mosquitto on a OpenWrt router and bunch of cgi-bin scripts that can run anywhere. I already went through a phase of setting up tons of services which ended up being turned off when something changed in my life (moving, replacing equipment etc.) never to be resurrected as I couldn't be bothered to redo it without the novelty effect, so I learned from that.
I clicked expecting a list of cool things to self host.
Instead I got a list of ways I would never want to host.
Mankind invented BSD jails so that I do not have to tie myself in a knot of container tooling and abstraction.
Same here. I've had the same setup for decades: A "homelab" server on my LAN for internal hobby projects and a $5 VPS for anything that requires public access. On either of these, I just install the software I need through the OS's package manager. If I need a web server, I install it. If I need ssh, I install it. If I need nfs, I install it. I've never seen any reason to jump into containers or orchestration or any of that complex infrastructure. I know there are a lot of people very excited about adding all of that stuff into the mix, but I've never had a use case that prompted me to even consider it!
I only host 3rd party daemons (nothing custom) and only on my local network (plus Tailscale) so Docker’s great for handling package management and init, since I get up-to-date versions of a far broader set of services than Debian or ubuntu’s repos, clean isolation for easy management, and init/restarts are even all free. Plus it naturally documents what I need to back up (any “mounted” directories)
Docker lets my OS be be boring (and lets me basically never touch it) while having up to date user-facing software. No “well, this new version fixes a bug that’s annoying me, but it’s not in Debian stable… do I risk a 3rd party back port repo screwing up my system or other services, or upgrade the whole OS just to get one newer package, which comes with similar risks?”
I just use shell scripts to launch the services, one script per service. Run the script once, forget about it until I want to upgrade it. Modify the version in the script, take the container down and destroy it (easily automated as part of the scripts, but I haven’t bothered), run the script. Done, forget about it again until next time.
Almost all the commands I run on my server are basic file management, docker stuff, or zfs commands. I could switch distros entirely and hardly even notice. Truly a boring OS.
I totally agree! Containers are nice when your installation is ephemeral, deploying and updating several time in a short period. But using the package manager is as easy as you get.
There is a fairly well known/popular blogger who’s blog I was following because of their self hosting/homelab/nix adventures.
Then they decided to port everything to K8 because of overblown internet drama and I lost all interest. Total shame that a great resource for Nix became yet another K8 fest.
But I just wanted to comment something similar. It's probably heavily dependend on how many services you self-host, but I have 6 services on my VPS and they are just simple podman containers that I just run. Some of them automatically, some of them manually. On top of that a very simple nginx configuration (mostly just subdomains with reverse proxy) and that's it. I don't need an extra container for my nginx, I think (or is there a very good security reason? I have "nothing to hide" and/or lose, but still). My lazy brain thinks as long as I keep nginx up to date with my package manager and my certbot running, ill be fine
I run Kubernetes on my homelab on 4x Raspberry Pi's then use Portainer to manage it. Works quite well, can use Helm charts when available, otherwise I use Portainer and deploy container apps manually. It's really not that bad.
I'm sure you find some joy in it or just like to explore what's possible. But just a reminder: Pieter Levels is running his million dollar businesses on a single VPS (if that's still correct). But yeah, if you like it, why not.
"Premature clustering is the source of all evil" - or something like that.
Writing your first yaml or two is scary & seems intimidating at first .
But after that, everything is cut from the same cloth. Its an escape from the long dark age of every sysadmin forever cooking up whatever whimsy sort of served them at the time, escape from each service having very different management practices around it.
And there's no other community anywhere like Kubernetes. Unbelievably many very good quality very smart helm charts out there, such as https://github.com/bitnami/charts/tree/main/bitnami just ready to go. Really sweet home-ops setups like https://github.com/onedr0p/home-ops that show that once you have a platform under foot, adding more services is really easy, showing an amazing range of home-ops things you might be interested in.
> Last thing I need is Kubernetes at home
Last thing we need is incredibly shitty attitude. Fuck around and find out is the hacker spirit. Its actually not hard if you try, and actually having a base platform where things follow common patterns & practices & you can reuse existing skills & services is kind of great. Everything is amazing but the snivelling shitty whining without even making the tiniest little case for your unkind low-effort hating will surely continue. Low signal people will remain low signal, best avoid.
I'm slowly configuring my new VPS. Here's approach I'm taking:
1. RHEL 9 with Developer Subscription. Installed dnf-automatic, set `reboot = when-changed`, so it's zero effort to reliably apply all updates with daily reboots. One or two minutes of downtime, not a big deal.
2. For services: podman with quadlets. It's RH-flavoured replacement for docker-compose. Not sure, if I like it, but I guess that's the "future", so I'm embracing it. Every service is a custom-built image with common parent to reduce space waste (by reusing base OS layer).
So far I want to run static http (nginx), vaultwarden, postfix and some webmail. May be more in the future.
This setup wastes a lot of disk space for image data, so expect to order few more gigabytes of disk to pay for modern tech.
I am actually worried about the self-hosting pandemic.
We self-hosters will stop flying under the radar.
Wonder how long it will take until our matrix instances
require to be backdoored, our immich are scanning
our pictures with AI.
On an unrelated note, an article of how to rent a VPS in
China would be interesting :)
Isn't that the beauty of self hosting? How could anything be enforced on user-controlled servers? Practically everything self-hosted is open source and how would enforcing anything would even work?
> How could anything be enforced on user-controlled servers?
New laws comes to mind. If a government decides to try to outlaw encryption again, cloud/hosting companies located there wouldn't have a choice but to comply, or give up on the business. The laws could also be made in such way that individuals are responsible for avoiding it, even self-hosters, and if people are using it anyways, be legally held responsible for the potential harms of it.
The problem lies with people who are technical enough to self-host, but might not be confident enough to fork/make changes. Maybe you could switch services, but there's still just enough friction/soft-lock in to actually migrate.
You are right though, it gives significantly more control to users. It's just realising 100% of the benefits that might be trickier.
Matrix server backdoors aren't an issue though? It's about the client where decryption happens. If those aren't required to upload decrypted contents, you can always overlay some encryption protocol like OTR over any chat mechanism. I remember using it on MSN via Pidgin
Don't worry about the servers. Worry about mandated software on the client
I suspect you would have trouble hosting long term in China. I don’t recall the specifics now but IIRC every website hosted in China needs a special government ID which requires getting approval. My memory is hazy but it does feel like one of the poorer choices to host unless you live in mainland. There are many better options in the world that both do not restrict information as well as not requiring paperwork.
> an article of how to rent a VPS in China would be interesting
Given that apparently it's quite difficult to even get a WeChat account without a national ID, I suspect that step 1 is "learn mandarin" and step 2 is "get a Chinese national ID".
I didn't have any problems creating wechat account. May be I was lucky, I don't know, I just typed my phone number and it went pretty smooth, like whatsapp. Also was able to connect my visa card. I did it in the Kazakhstan and then was able to pay in China, no problems. May be they got exception for Kazakhstan specifically, we recently got visa-free travels there.
Also your home modem/router is often tied to your ID and then there is ofc the firewall. IIRC You can get vos hosting and ICP code through Ali Cloud somewhat automagically. Agree it would be nice to give it a try some time.
Did you try? It's a few years ago when I had to create one, but it was just as simple as WhatsApp (just a few more CAPTCHAs). And no VPNs or whatever, straight from a Swiss IP.
I've been through just about everything to get where I am and I've ended up with Hashicorp Nomad and Consul with Traefik, managed by OpenTofu (open-source Terraform).
Things that haven't worked for me:
- Standalone Docker: Doesn't work great on its own. Containers often need to be recreated to modify immutable properties, like the specific image the container is running. To recreate the container, you need to store some state about how it _should_ work elsewhere.
- Quadlet: Too hard to manage clusters of services. Podman has subtle differences to Docker that occasionally cause problems and really tempting features (e.g. rootless) that cause more problems if you try to use them.
- Kubernetes: Waaaay too heavy. Even the "lightweight" distributions like k3s, k0s etc. embed large components of the official distribution, which are still heavy. Part of the embedded metric server for example periodically enumerates every single open file handle in every container. This leads to huge CPU spikes for a feature I don't care about.
With my setup now, I can more or less copy-paste a template into a new file, tweak some strings and have a HTTPS-enabled service available at https://thing.mydomain.mine. This works pretty painlessly even for services that need several volumes to maintain state or need several containers that work together.
Do you run a Nomad cluster? Or just on a single host? This is my desired state, I've set up Nomad a number of times but always get stuck in one place or another. I've gotten much further with Nomad than Kubernetes, but I've kind of always gone back to ol' faithful, writing a docker compose file and running everything that way.
I tried portainer once, it looked nice and had a lot of features... for which I had no use. I always found `docker compose` much easier to use, it is often just an alias and a tab away, where for portainer I would have to open a browser tab and sometimes even touch my mouse!
Otherwise good article. If you want to go rootless (which you should!), Podman is the way to go; but Docker works rootless too, with some modifications [1]. I have found Docker rootless to be reliable and robust on both Debian and Ubuntu. It also solves permissions problems because your rootless user owns files inside and outside the container, whereas with rootful setups all files outside the container are owned by root, which can be a pain.
Also, you don't need Watchtower. Automatic `docker compose pull` can be setup using standard crontab, see [2].
Can I suggest Lightkeeper (I'm the maintainer): https://github.com/kalaksi/lightkeeper. I made it for my own needs to simplify repetitive tasks and provide an efficient view, and it has hotkeys and trying to stay "agile". You can drop to a terminal with a hotkey any time.
If you self host, do not use containers and all those things.
Just use a static site generator like zola or hugo and rsync to a small VPS running caddy or nginx. If you need dynamic thing, there are many frameworks you can just rsync too with little dependencies. Or use PHP, it's not that bad. Just restrict all locations except public ones to your ip in nginx config if you use something like wordpress and you should be fine.
If you have any critical stuff, create a zfs dataset and use that to backup to another VPS using zfs send, there are tools to make it easy, much easier than DB replication.
What I'm reading is not to use containers for a web server, which makes sense because web servers have had vhosts since forever and you can host any number of sites on there independently already
But what about other services, like if you want a database server as well, a mail server, etc.?
I started using containers when I last upgraded hardware and while it's not as beneficial as I had hoped, it's still an improvement to be able to clone one, do a test upgrade, and only then upgrade the original one, as well as being able to upgrade services one by one rather than committing to a huge project where you upgrade the host OS and everything has to come with to the new major version
I manage about 500 servers. Critical services like DNS, mail, tftp, monitoring, routing, firewall... are all running openbsd in N+1 configuration, and in 15 years we had zero issue with that.
Now most servers are app servers, and they all run archlinux. We prepare images and we run them with PXE.
Both those are out of scope for self host.
But, we also have about a dozen of staging, dev, playground servers. And those are just regular installs of arch. We run postgres, redis, apps in many languages... For all that we use systems packages and AUR. DB upgrade? Zfs snapshot, and I follow arch wiki postgres upgrade, takes a few minutes, there is downtime, but it is fine. You mess anything? Zfs rollback. You miss a single file? cd .zfs/snapshots and grab it. I get about 30minutes of cumulated downtime per year on those machines. That's way enough for any self host.
We use arch because we try the latest "toys" on those. If you self host take an LTS distribution and you'll be fine.
That’s when you favor stability and use an LTS OS. You can also isolate workload by using VMs. Containers is nice for the installation part, but the immutability can be a pain.
It seems you're talking about about self-hosting a website or web-app that you are developing for the public to use.
My vision of self-hosting is basically the opposite. I only self-host existing apps and services for my and my family's use. I have a TrueNAS box with a few disks, run Jellyfin for music and shows, run a Nextcloud instance, a restic REST server for backing up our devices, etc. I feel like the OP is more targeted this type of "self hosting".
Readit News
What stops me is security. I simply do not know enough about securing a self-hosted site on real hardware in my home and despite actively continuing to learn, it seems like the more I learn about it, the more questions I have. My identity is fairly public at this point, so if I say the wrong thing to the wrong person on HN or whatever, do I need to worry about someone much smarter than me setting up camp on my home network and ruining my life? That may sound really stupid to many of you, but this is the type of anxiety that stops the under-informed from trying stuff like this and turning to services like Akamai/Linode or DO that make things fairly painless in terms of setup, monitoring and protection.
That said, I'm 110% open to reading/watching any resources people have that help teach newbies how to protect their assets when self-hosting.
[1] https://www.bbb.org/article/news-releases/20509-amazon-brush... [2] https://www.reddit.com/r/tulsa/comments/hpe8s1/just_got_a_su...
Without getting too deep into it, there are some things I know how to do with computers that I probably shouldn't, so my thought is this; if I, a random idiot who just happened to learn a few things, can do X, then someone smarter than me who learned how to attack a target in an organized way probably has methods that I cannot even conceive of, can do it easier, and possibly without me even knowing. It's this weird vacillation between paranoia and prudence.
For me, it's really about acknowledging what I know I don't know. I do some free courses, muck about with security puzzles, etc, even try my own experiments on my own machines, but the more I learn, the more I realize I don't know. I suppose that's the draw! The problem is when you learn these things in an unstructured way, it's hard to piece it all together and feel certain that you have covered all your vulnerable spots.
My current setup is to rent a cheap $5/month VPS running nginx. I then reverse ssh from my home to the vps, with each app on a different port. It works great until my electric goes out and comes back on the apps become unavailable. I haven't gotten the restart script to work 100% of the time.
But, I'd love to hear thoughts on security of reverse SSH from those that know.
Nginx handles proxying and TLSing all HTTP traffic. It also enforces access rules: my services can only be reached from my home subnet or VPN subnet. Everywhere else gets a 403.
Because since my new provider only provides cg-nat, I've been using a cheap server, but actually having the server at home would be nice.
Block port 22, secure SSH with certificates only. Allow port 443 and configure your web server as a reverse proxy with a private backend.
You don't need an IDS, you don't need a WAF and you don't need Cloudflare.
Unless you become the next Facebook that's when you start to become concerned about security.
I've contented myself using TLS client certs on my family's Android phones (which do not work at all on iOS for something like Home Assistant).
So you don't self-host at home, right?
I have been considering setting up a physical DMZ at home, with two routers (each with its own firewall), such that my LAN stays unmodified and my server can run between both routers. Then it feels like it would be similar to having a VPS in terms of security, maybe?
You want VPS-provider firewall. Docker's going to punch holes through your software firewall.
If using Podman, should I use rootless containers (which IMO suck because you can't do macvlan so the container won't easily get its own IP on my home network)? Is it ok to just use rootful Podman with an idmapped user running without root privileges inside the container and drop all unneccessary capabilities? Should I set up another POSIX user before, such that breaking out of the container would in the end just yield access to an otherwise unused UID on the host?
If using systemd-nspawn, do all the above concerns about rootful / rootless hold? Is it a problem that one needs to run systemd-nspawn itself with root? The manpage itself mentions "Like all other systemd-nspawn features, this is not a security feature and provides protection against accidental destructive operations only.", so should I trust nspawn in general?
Or am I just being paranoid and everything should just be running YOLO-style with UID 1000 without any separation?
All of this makes me quite wary about running my paperless-ngx instance with all my important data next to my Vaultwarden with all of my passwords next to any Torrent clients or anything else with unencrypted open ports on the internet. Also keeping everything updated seems to be a full time job by itself.
WAF's etc just hide the fact the code in your service is full of holes.
Ensuring your infra is built in a secure way is as important as ensuring your service is built in a secure way.
I'm a security consultant so this is not a problem I have. To me it seems very straightforward and like most things are secure by default (with the exceptions being notorious enough that I'd know of it), so I'm interested in the other perspective
Things I do:
* Make sure domain WHOIS does not point to me in any way, even if that means using some silly product like "WHOIS GUARD"
* Lock down any and all SSH access. Preferably only allow key-based authentication.
* Secure the communication substrate. For me this means running a Zerotier network which all dependent services listen on. I also try to use Unix sockets for any services colocated on the same operating system and restrict the service to only listen on sockets in a directory specifically accessible by the service.
* Try to control the permission surface of any service as much as possible. Containers can be a bit heavyweight for self-hosting but make this easy. There's alternatively like bubblewrap and firejail as well.
* Make use of services like fail2ban which can automate some of the hunting of bad actors for you.
* Consider hosting a listener for external traffic outside of your infra. For redundancy, load-shedding, and for security I have an external VPS that runs haproxy before routing over Zerotier to my home infrastructure. I enforce rate limits and fail2ban at the VPS so that bad actors get stopped upstream and use none of my home compute or bandwidth. (I also am setting up some redundant caches that live on the VPS so if my home network is down, one of my services can failover.)
* Segregate data into separate databases and make sure services only have access to databases that they need. With Postgres this is really simple with virtual databases being tied to different logins. I have some services that prune databases that run in a cron-like way (but using snooze instead) and they have no outbound net access.
If your network layer is secure and your services follow least-privilege, then you should be fairly in the clear.
From first hearing about Sandstorm since the first open beta 10 years ago (https://news.ycombinator.com/item?id=10147774) and reading about it on/off since then, this is first time I hear anyone pitching it for "medical and other highly regulated industries". Where exactly does this come from?
> There's still nothing else quite like it
Plenty of other similar self-hosted platforms, YunoHost is probably the closest, most mature and most feature-packed alternative to Sandstorm, at least as far as I know,.
Of course there can be security issue on your webserver as well, but for a simple site this setup is learnable in an hour or two and you are ready to go.
You can hook that up on a pie attached to your router or pay a bit to have it hosted somewhere. Domain is perhaps 2-5$ and an TLS cert you can get from Let's Encrypt.
No idea how to put everything into a container that it makes sense. I just run this quite often on small hosted machines elsewhere. I just install everything manually because it takes 5 minutes if you have done it before.
Deleted Comment
https://docs.opnsense.org/manual/how-tos/wireguard-client.ht...
Then on my phone I just flick on the switch and can access all my home services. It's a smidge less convenient, but feels nice and secure.
I can see running something in a Docker container, and while I'd advise against containers what ships with EVERYTHING, I'd also advise against using Docker-compose to spin up an ungodly amount of containers for every service.
You shouldn't be running multiple instances of Postgresql, or anything for that matter, at home. Find a service that can be installed using your operating systems package manager and set everything to auto-update.
Whatever you're self-hosting, be it for yourself, family or a few friends, there's absolutely nothing wrong with SQLite, files on disk or using the passwd file as your authentication backend.
If you are self hosting Kubernetes to learn Kubernetes, then by all means go ahead and do so. For actual use, stay away from anything more complex than a Unix around the year 2000.
It's not uncommon with self-hosting services using docker. It makes it easier to try out a new stack and you can mix and match versions of postgresql according to the needs of the software. It's also easier to remove if you decide you don't like the bit of software that you're trying out.
Deleted Comment
Edit: anyone actually interested in such a post?
Yup, it's basically like a "Docker Compose Manager" that lets you group containers more easily, since the manifest file format is basically Docker Compose's with just 1-2 tiny differences.
If there's one thing I would like Docker Swarm to have, is to not have to worry about which node creates a volume, I just want the service to always be deployed with the same volume without having to think about it.
That's the one weakness I see for multi-node stacks, the thing that prevents it from being "Docker Compose but distributed". So that's probably the point where I'd recommend maybe taking a look at Kubernetes.
No issues whatsoever and it is so easy to manage. It just works!
It's a shame I agree because it was nicely integrated with dockers own tooling. Plus I wouldn't have had to learn about k8s :)
Lord knows why people overcomplicate things with docker/kubernetes/etc.
Docker lets my OS be be boring (and lets me basically never touch it) while having up to date user-facing software. No “well, this new version fixes a bug that’s annoying me, but it’s not in Debian stable… do I risk a 3rd party back port repo screwing up my system or other services, or upgrade the whole OS just to get one newer package, which comes with similar risks?”
I just use shell scripts to launch the services, one script per service. Run the script once, forget about it until I want to upgrade it. Modify the version in the script, take the container down and destroy it (easily automated as part of the scripts, but I haven’t bothered), run the script. Done, forget about it again until next time.
Almost all the commands I run on my server are basic file management, docker stuff, or zfs commands. I could switch distros entirely and hardly even notice. Truly a boring OS.
Then they decided to port everything to K8 because of overblown internet drama and I lost all interest. Total shame that a great resource for Nix became yet another K8 fest.
But I just wanted to comment something similar. It's probably heavily dependend on how many services you self-host, but I have 6 services on my VPS and they are just simple podman containers that I just run. Some of them automatically, some of them manually. On top of that a very simple nginx configuration (mostly just subdomains with reverse proxy) and that's it. I don't need an extra container for my nginx, I think (or is there a very good security reason? I have "nothing to hide" and/or lose, but still). My lazy brain thinks as long as I keep nginx up to date with my package manager and my certbot running, ill be fine
"Premature clustering is the source of all evil" - or something like that.
I think it's a great way to learn Kubernetes if you're interested in that.
Deleted Comment
Writing your first yaml or two is scary & seems intimidating at first .
But after that, everything is cut from the same cloth. Its an escape from the long dark age of every sysadmin forever cooking up whatever whimsy sort of served them at the time, escape from each service having very different management practices around it.
And there's no other community anywhere like Kubernetes. Unbelievably many very good quality very smart helm charts out there, such as https://github.com/bitnami/charts/tree/main/bitnami just ready to go. Really sweet home-ops setups like https://github.com/onedr0p/home-ops that show that once you have a platform under foot, adding more services is really easy, showing an amazing range of home-ops things you might be interested in.
> Last thing I need is Kubernetes at home
Last thing we need is incredibly shitty attitude. Fuck around and find out is the hacker spirit. Its actually not hard if you try, and actually having a base platform where things follow common patterns & practices & you can reuse existing skills & services is kind of great. Everything is amazing but the snivelling shitty whining without even making the tiniest little case for your unkind low-effort hating will surely continue. Low signal people will remain low signal, best avoid.
1. RHEL 9 with Developer Subscription. Installed dnf-automatic, set `reboot = when-changed`, so it's zero effort to reliably apply all updates with daily reboots. One or two minutes of downtime, not a big deal.
2. For services: podman with quadlets. It's RH-flavoured replacement for docker-compose. Not sure, if I like it, but I guess that's the "future", so I'm embracing it. Every service is a custom-built image with common parent to reduce space waste (by reusing base OS layer).
So far I want to run static http (nginx), vaultwarden, postfix and some webmail. May be more in the future.
This setup wastes a lot of disk space for image data, so expect to order few more gigabytes of disk to pay for modern tech.
On an unrelated note, an article of how to rent a VPS in China would be interesting :)
New laws comes to mind. If a government decides to try to outlaw encryption again, cloud/hosting companies located there wouldn't have a choice but to comply, or give up on the business. The laws could also be made in such way that individuals are responsible for avoiding it, even self-hosters, and if people are using it anyways, be legally held responsible for the potential harms of it.
You are right though, it gives significantly more control to users. It's just realising 100% of the benefits that might be trickier.
Don't worry about the servers. Worry about mandated software on the client
Given that apparently it's quite difficult to even get a WeChat account without a national ID, I suspect that step 1 is "learn mandarin" and step 2 is "get a Chinese national ID".
Dead Comment
https://dokku.com/
Self hosting to me is, at the very least having physical access to the machines.
Things that haven't worked for me:
- Standalone Docker: Doesn't work great on its own. Containers often need to be recreated to modify immutable properties, like the specific image the container is running. To recreate the container, you need to store some state about how it _should_ work elsewhere.
- Quadlet: Too hard to manage clusters of services. Podman has subtle differences to Docker that occasionally cause problems and really tempting features (e.g. rootless) that cause more problems if you try to use them.
- Kubernetes: Waaaay too heavy. Even the "lightweight" distributions like k3s, k0s etc. embed large components of the official distribution, which are still heavy. Part of the embedded metric server for example periodically enumerates every single open file handle in every container. This leads to huge CPU spikes for a feature I don't care about.
With my setup now, I can more or less copy-paste a template into a new file, tweak some strings and have a HTTPS-enabled service available at https://thing.mydomain.mine. This works pretty painlessly even for services that need several volumes to maintain state or need several containers that work together.
Otherwise good article. If you want to go rootless (which you should!), Podman is the way to go; but Docker works rootless too, with some modifications [1]. I have found Docker rootless to be reliable and robust on both Debian and Ubuntu. It also solves permissions problems because your rootless user owns files inside and outside the container, whereas with rootful setups all files outside the container are owned by root, which can be a pain.
Also, you don't need Watchtower. Automatic `docker compose pull` can be setup using standard crontab, see [2].
[1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...
[2]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...
I run it alongside portainer because exactly of the compose.yaml file I want to have control over
Just use a static site generator like zola or hugo and rsync to a small VPS running caddy or nginx. If you need dynamic thing, there are many frameworks you can just rsync too with little dependencies. Or use PHP, it's not that bad. Just restrict all locations except public ones to your ip in nginx config if you use something like wordpress and you should be fine.
If you have any critical stuff, create a zfs dataset and use that to backup to another VPS using zfs send, there are tools to make it easy, much easier than DB replication.
But what about other services, like if you want a database server as well, a mail server, etc.?
I started using containers when I last upgraded hardware and while it's not as beneficial as I had hoped, it's still an improvement to be able to clone one, do a test upgrade, and only then upgrade the original one, as well as being able to upgrade services one by one rather than committing to a huge project where you upgrade the host OS and everything has to come with to the new major version
Now most servers are app servers, and they all run archlinux. We prepare images and we run them with PXE.
Both those are out of scope for self host.
But, we also have about a dozen of staging, dev, playground servers. And those are just regular installs of arch. We run postgres, redis, apps in many languages... For all that we use systems packages and AUR. DB upgrade? Zfs snapshot, and I follow arch wiki postgres upgrade, takes a few minutes, there is downtime, but it is fine. You mess anything? Zfs rollback. You miss a single file? cd .zfs/snapshots and grab it. I get about 30minutes of cumulated downtime per year on those machines. That's way enough for any self host.
We use arch because we try the latest "toys" on those. If you self host take an LTS distribution and you'll be fine.
My vision of self-hosting is basically the opposite. I only self-host existing apps and services for my and my family's use. I have a TrueNAS box with a few disks, run Jellyfin for music and shows, run a Nextcloud instance, a restic REST server for backing up our devices, etc. I feel like the OP is more targeted this type of "self hosting".