Ask HN: Can any Hetzner user please explain your workflow on Hetzner?

- Deploy by stopping the server, rsyncing in the changes, and starting the server. The whole thing is automated by script and takes 5 seconds which is acceptable for us. - Run apt upgrade manually biweekly or so. - We use client-side load balancing (the client picks an app server at random) but most cloud providers will give you a load balancer IP that transparently does the same thing (not for free though). - For scaling just manually rent more servers. - For backups we use a cronjob that does the backup and then uploads to MEGA - For security we setup a script that runs iptables-restore but this isn't really all that necessary if you don't run anything that listens on the network (except your own server obviously). - DDoS is handled transparently by our provider.

- It provides a standard procedural format for my shell scripts - It comes with some level of type-checking when I play a script - It makes me actually test that my procedures can bootstrap - The playbook style helps me keep scripts organized - It lets me start from "You have a server." without questioning where it came from. - Ansible, while "bottom-up", lets me bootstrap "top-down" systems like Kubernetes, container registries, etc.

ssh root@hetzner-server-ip "cd my-server && git pull && ./prepare.sh && systemctl restart my.service && journalctl -u my.service -f"

To expand a little bit:

- It's a very small service

- I use sqlite db

- Preparation step before the restart ensures all the deps are downloaded for the new repo state. I.e. "a build step"

- I use simple nginx in front of the web server itself

- Backups are implemented as a cron job that sends my whole db as an email attachment to myself

- journalctl shows how it restarted so I see it's working

danwee · 3 years ago

So, you have the ssh port (22? custom one?) open to the public on the same machine in which you have your web server/nginx? May I ask why? I had one similar very simple service on Digitalocean once, and my setup was:

- bastion host: custom port for ssh open to the public

- virtual private cloud (vpc): inside I put my web-server and my db server. These servers are not accessible to the public, but the bastion host has access to the vpc

- another server for my nginx. This is public and it redirects requests to my web-server in the vpc

I know it sounds overkill! But somehow it gives me the (illusion?) of being more secure. Am I right with this setup or I'm just wasting my time (and money)? I know perhaps that a VPN could be better... but somehow I found the idea of bastion+vpc quite simple and effective.

habibur · 3 years ago

Enable certificate authentication only, and turn off password authentication. And you are safe. A million bots attacking a billion times won't break in.

You don't need to put anymore security over it, like changing ssh port or running fail2ban, unless you want to reduce CPU load spent behind handling automated port scanners and bots.

jeroenhd · 3 years ago

I'm not sure if a bastion host for a single server is all that much better. If OpenSSH allows access to hackers, they'll break into your bastion host and move on from there. All you're really adding is on e more host to forget to patch.

You're only ever more secure if you reduce the attack surface. These days, with WireGuard's simple and secure tunnels, I'd say a VPN may be an improvement, but I'm not all that worried about SSH on my servers. Either disabling password logins or using secure passwords should be fine in most cases.

I personally change the SSH port as well, not really for security but mostly because it keeps the logs clean. Port scans will still happen but you won't get bombarded by thousands of pi@server.com sessions failing every day of the week.

yjftsjthsd-h · 3 years ago

SSH with only public key auth allowed is perfectly safe to have exposed to the internet.

Gasp0de · 3 years ago

I have ssh enabled on all my servers, sometimes port 22 sometimes other ports. I have never had a break in. I use fail2ban but I don't know if that's really necessary. But I use it anyway to secure other services (e.g. wordpress instances) against brute-forcing. It goes without saying that password access and root login should be disabled.

If you want to go to crazy lengths to hide your ssh then do port knocking or something.

vlaaad · 3 years ago

I only login with my SSH keys, so I don't see the problem — I'm protected with cryptography.

ilrwbwrkhv · 3 years ago

I use virtually this same setup and we do around 100,000 users per day.

I only use it for side projects right now, and in the past for a real production application for which "high availability" was not a problem (I could do ocasional maintenance windows out of work hours). Here's how I did it in case it helps you:

> deploy from source repo? Terraform?

I use Dokku (https://dokku.com/), then the workflow is the same as if you'd be using Heroku

> keep software up to date? ex: Postgres, OS

Automattic ubuntu updates + I once a week SSH to it and apt-get update, etc.

> do load balancing? built-in load balancer?

I just don't. I don't need for the load of my projects.

> handle scaling? Terraform?

Just vertical scaling for now. A single powerful server can do great before you might need to add more servers.

> automate backups? ex: databases, storage. Do you use provided backups and snapshots?

I just enable the "backup" feature on their admin panel. Adds 20% to the cost but works great and it's easy.

> maintain security? built-in firewall and DDoS protection?

I only expose the HTTP(s) and SSH ports, and I also have setup fail2ban for bruteforce attacks.

> If there is any open source automation scripts please share.

Dokku.

- Running dokku with Heroku Buildpacks to deploy both from source and to run Docker images behind an ngnix reverse proxy. - Autoupgrade apt's, manually updating the OS. - No load balancing. - No scaling. - Automated backups with restic/rclone to OneDrive. - Hetzner firewall, no DDoS protection.

alex7734 · 3 years ago

Not hetzner but a similar provider:

While this might change if you're super big and have thousands of servers, in my experience simple is best and "just shell scripts" is the simplest solution to most sysadmin problems.

bayesian_horse · 3 years ago

Ansible can be a big step forward from "just shell scripts". I was on the fence for a while, but it does make things easier quite quickly, even for smaller deployments. Even for just documenting wtf is running on the servers and how.

sshine · 3 years ago

What I like about Ansible:

lastofus · 3 years ago

In my experience, Ansible is fantastic for provisioning a machine from bare Linux VM to running service.

OTOH it's quite slow when used for deployments. There's no way you would be getting 5 second deployments with it.

My favorite middle ground between shell scripts and Ansible is Fabric (https://www.fabfile.org/).

marcosdumay · 3 years ago

> Ansible can be a big step forward from "just shell scripts".

It can be. It also can not-be. I'd recommend anybody to start looking once they reach more than 2 machines, but yeah, depending on what you are doing it can add value when you have 1 or 2 machines too.

0xblinq · 3 years ago

nemo136 · 3 years ago

> 50 machines at hetzner

- install machines with ansible (using hetzner scripts for OS install)

- machines communicate over vswitch/vlans, external interfaces disabled whenever possible. Pay attention to the custom mtu trick.

- harden machines, unattended-upgrades mandatory on each machine

- ssh open with IP whitelists from iptables on gateways

- machines organized as k8s clusters, took ~1 year to have everything working cleanly

- everything deployed as k8s resources (kustomize, fluxcd, gitops)

- use keepalived for external IPs with floating IPs for ingress on 3 machines per cluster

Machines are managed as cattle, it takes <1h+ hetzner provisioning time to add as many machines as we need.

Volundr · 3 years ago

> Pay attention to the custom mtu trick.

I wish Hetzner made this more clear up front. Maybe a big red banner on the vSwitch page. I can't count the number of hours I've spent troubleshooting network issues at Hetzner that came down to MTU.

johne20 · 3 years ago

As someone who is trialing Hetzner is there a link to info on this?

mtmail · 3 years ago

https://github.com/hetznercloud/awesome-hcloud/ collects various devops tools for Hetzner Cloud.

e12e · 3 years ago

The recent demo of MRSK from 37signals used Hetzner as the first example:

> Introducing MRSK - 37signals way to deploy

Thttps://www.youtube.com/watch?v=LL1cV2FXZ5I

jasonvorhe · 3 years ago

It's not even close to major public cloud providers, but this is my setup:

* https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne... (Terraform, Kubernetes bootstrap)

* Flux for CI

* nginx-ingress + Hetzner Loadbalancer (thanks to https://github.com/hetznercloud/hcloud-cloud-controller-mana...)

* Hetzner storage volumes (thanks to https://github.com/hetznercloud/csi-driver)

Kube-Hetzner supports Hetzner Cloud loadbalancers and volumes out of the box, though it also supports other components.

cstuder · 3 years ago

For my hobby server: