Readit News logoReadit News
adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
sanderjd · 4 days ago
This space of #2 like Lithus is not something I'm very familiar with, so thank you for the comment that piqued my interest!

If you're willing to share, I'm curious who else you would describe as being in this space.

My last decade and a half or so of experience has all been in cloud services, and prior to that it was #3 or #4. What was striking to me when I went to the Lithus website was that I couldn't figure out any details without hitting a "Schedule a Call" button. This makes it difficult for me to map my experiences in using cloud services onto what Lithus offers. Can I use terraform? How does the kubernetes offering work? How does the ML/AI data pipelines work? To me, it would be nice if I could try it out in a very limited way as self-service, or at least read some technical documentation. Without that, I'm left wondering how it works. I'm sure this is a conscious decision to not do this, and for good reasons, but I thought I'd share my impressions!

adamcharnock · 4 days ago
Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list.

We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around.

What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am.

Your team also has full access to the Kubernetes cluster to deploy to as you wish.

I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
belthesar · 4 days ago
From a platform risk perspective, each tenant has dedicated resources, so it's their platform to blow up. If a customer with root access blows up their own system, then the resources from the MSP to fix it are billable, and the after-action meetings would likely include a review of whether that access is appropriate, if additional training is needed to prevent those issues in the future (also billable), or if the customer-provider relationship is the right fit. Will the on-call resource be having a bad time fixing someone else's screw up? Yeah, and having been that guy before, I empathize. The business can and should manage this relationship however, so that it doesn't become an undue burden on their support teams. A customer platform that is always getting broken at 4pm on a Friday when an overzealous customer admin is going in and deciding to run arbitrary kubectl commands takes support capacity away from other customers when a major incident happens, regardless of how much you're making in support billing.
adamcharnock · 4 days ago
This is essentially how it is. Additionally, the reality is that our customers don't often even need to think about using root access, but they have it if they want it. They are putting a lot of trust in us, so we also put trust in them.
adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
wulfstan · 4 days ago
I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros.

Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync.

Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...

adamcharnock · 4 days ago
FYI, AWS offers free Egress when leaving them (because they were forced to be EU regulation, but they chose to offer it globally):

https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...

But. Don't leave it until the last minute to talk to them about this. They don't make it easy, and require some warning (think months, IIRC)

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
Schlagbohrer · 4 days ago
Can someone explain 2 to me. How is a managed private cloud different from full cloud? Like you are still using AWS or Azure but you are keeping all your operation in a bundled, portable way, so you can leave that provider easily at any time, rather than becoming very dependent on them? Is it like staying provider-agnostic but still cloud based?
adamcharnock · 4 days ago
To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof).

It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team.

This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues.

If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment.

Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do.

Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers.

And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck.

Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
weavie · 4 days ago
What is the upper limit of Hertzner? Say you have an AWS bill in the $100s of millions, could Hertzner realistically take on that scale?
adamcharnock · 4 days ago
An interesting question, so time for some 100% speculation.

It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost.

So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business."

I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
Lucasoato · 4 days ago
Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.
adamcharnock · 4 days ago
You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours.

We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings"

(plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
bpavuk · 4 days ago
if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible!

plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

adamcharnock · 4 days ago
Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)
adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
mgaunard · 4 days ago
you're missing 5, what they are doing.

There is a world of difference between renting some cabinets in an Equinix datacenter and operating your own.

adamcharnock · 4 days ago
Fair point!

5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).

There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).

Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?

If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?

I think this is one of the more interesting areas to think about, at least for me!

adamcharnock commented on Don't rent the cloud, own instead   blog.comma.ai/datacenter/... · Posted by u/Torq_boi
adamcharnock · 4 days ago
This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between:

1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based).

2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend.

3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time).

4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.

A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system.

Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team!

[0] https://lithus.eu, adam@

adamcharnock commented on EU–INC – A new pan-European legal entity   eu-inc.org/... · Posted by u/tilt
academia_hack · 19 days ago
It's inordinately difficult and expensive to start an LLC or SA in some EU countries. It's even difficult and expensive to _stop_ an LLC and dissolve it. Huge amount of risk and cost on founders and a huge distraction from running a business.

I think that EU-Inc _could_ be an improvement, but it needs to avoid the committee laundry list of ideas/requirements/form fields that plagues the EU startup ecosystem. My worry is that the end result will require notarized declarations of honour, financial plans stretching decades into the future, 30 page business plan documents, reams of corporate governance documents, and tons of other nonsense to protect against the perceived risk that someone who failed at starting a business once fails a second time.

There needs to be UX requirements on the process from day one against which the end result is judged. (E.g. "a company should be able to register in x days", "a complete application should be no longer than y pages", "application costs should be less than z euros").

adamcharnock · 18 days ago
Agreed. The one bastion of sanity in all this is (/was) the UK. I formed my first company there, 18 years ago, online, in 30 minutes, for around £20.

I then moved to Portugal and started not one but two companies there. The whole process is so clearly setup to discourage people from actually forming companies. Everything from the attitude of all involved ("are you sure?!"), to the practical bureaucracy and costs involved.

I thought perhaps Portugal was just an outlier. But then I moved to Germany. And just wow. Definitely worse. Rounds of paperwork, notary offices, and fees. A process taking weeks. And for a GmbH a minimum investment of €25k. Sure you can form a UG. company for €1, but that effectively just announces "don't trust us, we're tiny" (IMHO).

It is something that really saddens and frustrates me with the EU.

Edit: And sure, you can form an Estonian company. But then you have to try and fly under the radar with regards to the 'permanent establishment' rules.

u/adamcharnock

KarmaCake day3345May 3, 2010
About
Senior SRE and founder

https://lithus.eu - Infrastructure with DevOps included

View Original