> 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd.
Also, test that it's properly disabled with something like `ssh -v yourserver : 2>&1 | grep continue`, because there are a surprising number of ways for that to go wrong (did you know that sshd lets you Include multiple config files together in a way that can override the main one? I know that now.)
In situations where I have more than one box, one (usually a little embedded SoC) gets openbsd on it, and is used as a bastion host, for exactly the reasons outlined here.
The problems I hit with using Linux for this were different ten years ago, but, based on this thread, things got worse on that side of the fence.
I think doing the opposite is better usually, never making changes to the main /etc/ssh/sshd_config and always add your own customizations to /etc/ssh/sshd_config.d/; that way you have clear separation of your own and distro configs, which makes life easier especially when upgrading sshd. Although I'll readily admit that I don't myself follow that advise all the time.
Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking.
Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware.
The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider.
Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.
This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes).
The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)
One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday.
I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone.
Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.
> how do you find a reliable, trustworthy datacenter?
drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.
You're right you need to find a company you can trust.
And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)
After years of dealing with colocation, I would never deploy a server without KVM in a data center. The cost of truck rolls is just too damned high! Even with KVM, I have had to make an emergency trip to a data center that is a 3.5 hour drive away due to a hardware errata on circa 2010 Intel Xeon boards where the network port shared between the KVM and the host would lock up under certain rare circumstances. The second time that happened I pulled the system from production.
If you do happen to have a system without on-board KVM, check out NanoKVM, which is a cheap (~$40) option for an add on KVM. It's rather more affordable than PiKVM. https://github.com/sipeed/NanoKVM
Back in the late 90s/early 00s when I was a precocious teenager, I ran a somewhat popular website. At some point it made sense to just buy a 1U rack moubtable server and having it colocated (commercial webhosting was expensive then). I couldn't find anyone to give me a ride to the datacenter, so I took a bus. By the time I got there my arms were numb from carrying the bloody thing.
There was a single security guard, I signed in and he gave me directions and a big keychain. The keys opened most of the rooms and most of the cages around the racks. To this day I remain mystified at the level of trust (or nonchalance) that security guard had in a spotty teenager.
Back in the early 2000s I had a job that required me to enter a certain phone companies' datacenter. But, I couldn't, because I was a consultant and it was an employees-only area. I had permission to make changes to machines in a rack but they didn't allow me to enter. There was a guard who would check badges but who didn't really check if he recognized people, and there were some people who had to enter and leave multipel times in a day. Myself, I couldn't get permission to enter via the correct channels.
With my director's unofficial approval I was allowed to _try_ to enter the datacenter. So I just walked very confidently towards the entrance, nodded to the security guard like all of the regulars who didn't bother showing their badges, and he let me in.
Having run hosting companies from the mid 90s as well, from memory this kind of thing was pretty normal, even in allegedly secure places like London Telehouse.
Quite a few of us in that era were juggling it with being students, so it wouldn't surprise me if the security staff were used to it and expected you to look young enough to be their kid!
" 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this. "
If you blindly followed the directions and got locked out, you can do exactly the same thing with other directions. You were not ready.
And I find this to be mean-spirited ("snarky") and the opposite of instructive. Perhaps I am new, or rusty, and I do not know about SSH keys or remember them. Bang! Trapped! And then how is a newbie to trust the rest of this, much less learn?
Good instruction will tell you why and the consequences of missing a step. Perhaps some options.
But perhaps I want too much, the title does include the word "terrible," after all.
If you are new or rusty, Stop. You are not ready for this.
Seriously. Not knowing or remembering abut ssh keys is way below the minimum baseline of skills that are needed to manage a server connected to the internet.
Managing colo-ed servers when you're only just experienced enough to know about and know how to set up ssh keys without being told _is_ a terrible idea.
> And I find this to be mean-spirited ("snarky") and the opposite of instructive. Perhaps I am new, or rusty, and I do not know about SSH keys or remember them. Bang! Trapped! And then how is a newbie to trust the rest of this, much less learn?
I suspect the point of this article isn’t meant to be a Linux 101 guide, but rather, specifically about their learnings with how to go about colocating a linux server.
> But perhaps I want too much, the title does include the word "terrible," after all.
To be fair, it’s not actually a particularly good guide to how to go about getting your first colo server setup either, which aligns with their title about it being terrible.
The great thing about having unfettered physical access to hardware is that you can easily recover from mistakes like this. No need to rebuild that EC2 instance. No need to beg a hosting company for IPMI access. You can just pull the plug and try again as if were your own PC.
On the other hand, it's nice to be able to take a no-reboot AMI of the machine you're futzing with, risk breaking it, then just re provision from that AMI if you fucked up. And if you're chasing three or four nines of uptime you do the same but provision a new instance and futz with that one while your working one keeps on keeping on. Even big instances are cheap enough by the hour that you can afford to run and extra one while testing potentially breaking changes.
Related to this, is the advice I give everyone for editing the sudoers file:
First, open two superuser terminals. The second one is so if you fuck up the sudoers format so it doesn't parse and you accidentally 'exit' one too many times in the first terminal.
The visudo command is often times available as well. It won’t let you save a malformed sudoers file. Finding other footguns though is an exercise for the reader.
Get something like a PiKVM and drop all of the stuff about being very local since you can find a cheaper overall provider elsewhere and use smart hands once a year either for free or still cheaper than picking the local place you can drive to. Even if you do the things in this guide perfectly it'll break/hang/get misconfigured at some point and the PiKVM (or like) lets you remotely hard boot a box instantly without having to drive or open a ticket. It also enables you to easily reinstall the entire OS remotely if you need to.
If your server/device has an IPMI... get a PiKVM (or like) anyways. Not only will you last more than 2 seconds without being hacked but it'll have more functionality and be much faster.
If you're in the US there are lots of places in the Kansas City area that have ridiculously cheap pricing and it's decently centrally located in the country.
I don't want people to think TinyPilot is a bad option but this is where their marketing really grinds my gears. PiKVM and TinyPilot both have preassembled ordering option.
TinyPilot's compare and contrast point of "To exercise the full functionality of PiKVM, users must install a custom circuit board on top of their motherboard and re-route their power supply's ATX pins." is a complete farce (it's not required) and worded in an intentionally scary way (the only reason TinyPilot doesn't have this requirement it doesn't offer the feature while PiKVM does). The "custom circuit board" is literally a PCB that, optionally, allows you to jumper the inline the remote power controls in a way that the normal power buttons still work too rather than rely on ACPI signalling over USB.
It honestly makes my blood boil to see this underhanded approach works so well... the TinyPilot device is good though, as are some other options. Just keep in mind if you opt to go with ACPI only remote power controls via USB things may still go bad if your system hangs/crashes/gets in a weird power state whereas plugging in the wires to the PiKVM will be no different than holding the power button.
I faced the same problem, until I found a non-profit community-operated housing provider (a registered computer club/society that has been operating since at least the early 2000 who is also a LIR (https://en.wikipedia.org/wiki/Regional_Internet_registry#Loc...), and they give shelter, bandwidth and IP address space to anyone who participates in covering their operational costs.
I gladly do, and it's the best hosting experience I've had so far, and I used to have rented dedicated iron from Server4You, Hetzner, Webtropia and the like from 2005 on. Maybe there's a similar hidden gem in the area you live in, and you just do not know about it? :) Mine flew under my radar for nearly 20 years, and even though I knew they existed, I was not aware they'd colo other peoples' boxen at very fair rates.
Is this provider for local people only, or can people from afar also participate in contributing towards costs in exchange for services? If so, can you share their details? I need someone to house some RIPE objects for me.
> finding colocation space at reasonable price is difficult these days.
The author of the post seems to be living in the bay area. It's easy when the number of nerds per km² is high, disposable income is even higher, and driving a car for 2 hours is considered "next door".
I think for losers like myself, living in the middle of nowhere with low disposable income, the best solution is just to rent out a dedicated Kimsufi box (from OVH), or a server off the Heztner auctions. (Or whatever is the equivalent in North America) It's much much more cost effective than collocating.
In the last year or two there was some HN post about some guys in, IIRC, Minneapolis, that got a rack in a data center and were doing something like this. ISTR their website for the colo was in ASCII art. A few months ago I went looking for it again and couldn't find it. I was tempted to put a machine I have at home over there, but I also have a local place I could see about going into, but just haven't yet. What I really need to do is just buckle down and get my home networking set up so I can run it here using my gigabit fiber, maybe upgrade to 10gig.
> the bare minimum required to run your own hardware in a colocation environment
i remember the look in the admin's eyes when they asked "alright, what kind of hardware are you looking to install?" and I said "oh i have it right here" and pulled two Intel NUCs out of my backpack
> Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).
two multitools minimum, sometimes you need to hold a nut with one while loosening the bolt with the other
the best space is the one that is right next to a Frys/Microcenter/whathaveyou
> "oh i have it right here" and pulled two Intel NUCs out of my backpack
NUCs would be like nirvana after some of the jerry-rigged crap I've seen dragged into facilities. Maybe I'd like them to have a secondary power supply, but then again you had two, but they'd make a fantastic little router for something not too traffic heavy. Lots of utility use cases for something like that in a proper facility.
Readit News
Also, test that it's properly disabled with something like `ssh -v yourserver : 2>&1 | grep continue`, because there are a surprising number of ways for that to go wrong (did you know that sshd lets you Include multiple config files together in a way that can override the main one? I know that now.)
The problems I hit with using Linux for this were different ten years ago, but, based on this thread, things got worse on that side of the fence.
it seriously bothered me that an update automatically re-enabled password authentication. i ended up switching to a different OS.
Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking.
Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware.
The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider.
Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.
The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)
I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone.
Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.
drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
If you do happen to have a system without on-board KVM, check out NanoKVM, which is a cheap (~$40) option for an add on KVM. It's rather more affordable than PiKVM. https://github.com/sipeed/NanoKVM
There was a single security guard, I signed in and he gave me directions and a big keychain. The keys opened most of the rooms and most of the cages around the racks. To this day I remain mystified at the level of trust (or nonchalance) that security guard had in a spotty teenager.
With my director's unofficial approval I was allowed to _try_ to enter the datacenter. So I just walked very confidently towards the entrance, nodded to the security guard like all of the regulars who didn't bother showing their badges, and he let me in.
https://www.youtube.com/watch?v=NiEMcjSQOzg
Quite a few of us in that era were juggling it with being students, so it wouldn't surprise me if the security staff were used to it and expected you to look young enough to be their kid!
" 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this. "
If you blindly followed the directions and got locked out, you can do exactly the same thing with other directions. You were not ready.
Good instruction will tell you why and the consequences of missing a step. Perhaps some options.
But perhaps I want too much, the title does include the word "terrible," after all.
Seriously. Not knowing or remembering abut ssh keys is way below the minimum baseline of skills that are needed to manage a server connected to the internet.
Managing colo-ed servers when you're only just experienced enough to know about and know how to set up ssh keys without being told _is_ a terrible idea.
That’s a strong indicator you won’t be able to support the setup moving forward. What if apt wedges some firewall rule, or the machine starts OOMing?
I suspect the point of this article isn’t meant to be a Linux 101 guide, but rather, specifically about their learnings with how to go about colocating a linux server.
> But perhaps I want too much, the title does include the word "terrible," after all.
To be fair, it’s not actually a particularly good guide to how to go about getting your first colo server setup either, which aligns with their title about it being terrible.
Le sight.
First, open two superuser terminals. The second one is so if you fuck up the sudoers format so it doesn't parse and you accidentally 'exit' one too many times in the first terminal.
"visudo edits the sudoers file in a safe fashion..."
If your server/device has an IPMI... get a PiKVM (or like) anyways. Not only will you last more than 2 seconds without being hacked but it'll have more functionality and be much faster.
If you're in the US there are lots of places in the Kansas City area that have ridiculously cheap pricing and it's decently centrally located in the country.
Thanks, I had never heard of PiKVM!
TinyPilot's compare and contrast point of "To exercise the full functionality of PiKVM, users must install a custom circuit board on top of their motherboard and re-route their power supply's ATX pins." is a complete farce (it's not required) and worded in an intentionally scary way (the only reason TinyPilot doesn't have this requirement it doesn't offer the feature while PiKVM does). The "custom circuit board" is literally a PCB that, optionally, allows you to jumper the inline the remote power controls in a way that the normal power buttons still work too rather than rely on ACPI signalling over USB.
It honestly makes my blood boil to see this underhanded approach works so well... the TinyPilot device is good though, as are some other options. Just keep in mind if you opt to go with ACPI only remote power controls via USB things may still go bad if your system hangs/crashes/gets in a weird power state whereas plugging in the wires to the PiKVM will be no different than holding the power button.
I gladly do, and it's the best hosting experience I've had so far, and I used to have rented dedicated iron from Server4You, Hetzner, Webtropia and the like from 2005 on. Maybe there's a similar hidden gem in the area you live in, and you just do not know about it? :) Mine flew under my radar for nearly 20 years, and even though I knew they existed, I was not aware they'd colo other peoples' boxen at very fair rates.
The author of the post seems to be living in the bay area. It's easy when the number of nerds per km² is high, disposable income is even higher, and driving a car for 2 hours is considered "next door".
I think for losers like myself, living in the middle of nowhere with low disposable income, the best solution is just to rent out a dedicated Kimsufi box (from OVH), or a server off the Heztner auctions. (Or whatever is the equivalent in North America) It's much much more cost effective than collocating.
i remember the look in the admin's eyes when they asked "alright, what kind of hardware are you looking to install?" and I said "oh i have it right here" and pulled two Intel NUCs out of my backpack
> Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).
two multitools minimum, sometimes you need to hold a nut with one while loosening the bolt with the other
the best space is the one that is right next to a Frys/Microcenter/whathaveyou
NUCs would be like nirvana after some of the jerry-rigged crap I've seen dragged into facilities. Maybe I'd like them to have a secondary power supply, but then again you had two, but they'd make a fantastic little router for something not too traffic heavy. Lots of utility use cases for something like that in a proper facility.
yup! it was cheaper to bring two NUCs than try to get N+1 power redundancy set up for "atypical" (not u-rack) systems :)
then they started making 1U NUC rack mounts...