rcrowley (u/rcrowley)

rcrowley commented on IO Devices and Latency planetscale.com/blog/io-d... · Posted by u/milar

maayank · 6 months ago

what do you mean by drains?

rcrowley · 6 months ago

AWS, for one example, provide a feed of upcoming "events" in EC2 in which certain instances will need to be rebooted or terminated entirely due to whatever maintenance they're doing on the physical infrastructure.

If you miss a termination event you miss your chance to copy that data elsewhere. Of course, if you're _always_ copying the data elsewhere, you can rest easy.

rcrowley commented on IO Devices and Latency planetscale.com/blog/io-d... · Posted by u/milar

__turbobrew__ · 6 months ago

I think something about distributed storage which is not appreciated in this article:

1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot.

2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and drain your data to a different node, the data goes poof. Separating storage from compute allows the cloud operator to drain and move around compute as needed and since the data is independent from the compute — and the cloud operator manages that data system and draining for that system as well — the operator can manage workload placements without the customer needing to be involved.

rcrowley · 6 months ago

Good points. PlanetScale's durability and reliability are built on replication - MySQL replication - and all the operational software we've written to maintain replication in the face of servers coming and going, network partitions, and all the rest of the weather one faces in the cloud.

Replicated network-attached storage that presents a "local" filesystem API is a powerful way to create durability in a system that doesn't build it in like we have.

rcrowley commented on IO Devices and Latency planetscale.com/blog/io-d... · Posted by u/milar

jhgg · 6 months ago

Metal looks super cool, however at my last job when we tried using instance local SSD's on GCP, there were serious reliability issues (e.g. blocks on the device losing data). Has this situation changed? What machine types are you using?

Our workaround was this: https://discord.com/blog/how-discord-supercharges-network-di...

rcrowley · 6 months ago

Neat workaround! We only started working with GCP Local SSDs in 2024 and can report we haven't experienced read or write failures due to bad sectors in any of our testing.

That said, we're running a redundant system in which MySQL semi-sync replication ensures every write is durable to two machines, each in a different availability zone, before that write's acknowledged to the client. And our Kubernetes operator plus Vitess' vtorc process are working together to aggressively detect and replace failed or even suspicious replicas.

In GCP we find the best results on n2d-highmem machines. In AWS, though, we run on pretty much all the latest-generation types with instance storage.

rcrowley commented on PlanetScale Metal planetscale.com/blog/anno... · Posted by u/rvineet02

CSDude · 6 months ago

For years, I just didn't get why replicated databases always stick with EBS and deal with its latency. Like, replication is already there, why not be brave and just go with local disks? At my previous orgs, where we ran Elasticsearch for temporary logs/metrics storage, I proposed we do exactly that since we didn't even have major reliability requirements. But I couldn't convince them back then, we ended up with even worse AWS Elasticsearch.

I get that local disks are finite, yeah, but I think the core/memory/disk ratio would be good enough for most use cases, no? There are plenty of local disk instances with different ratios as well, so I think a good balance could be found. You could even use local hard disk ones with 20TB+ disks for implementing hot/cold storage.

Big kudos to the PlanetScale team, they're like, finally doing what makes sense. I mean, even AWS themselves don't run Elasticsearch on local disks! Imagine running ClickHouse, Cassandra, all of that on local disks.

rcrowley · 6 months ago

Glad to find a kindred spirit. I proposed PlanetScale Metal (though our CEO Sam gets credit for the name) based on how we ran MySQL and Vitess at Slack for many, many years.

We earn our durability through MySQL replication and redundancy and reap the benefits in low, predictable latency.

rcrowley commented on Terraform best practices for reliability at any scale substrate.tools/blog/terr... · Posted by u/holoway

dharmab · 2 years ago

What do you think about multiple backends? It seems to be working well for me to have a single root module but with a separate backend configuration per environment.

rcrowley · 2 years ago

That does work well for environments because typically you’d run exactly the same code, maybe with different cluster sizes or instance types, in each environment. But it doesn’t work well for isolating two services where the code is significantly or even entirely different.

rcrowley commented on Terraform best practices for reliability at any scale substrate.tools/blog/terr... · Posted by u/holoway

swozey · 2 years ago

Ok I may absolutely have the dumb today I appreciate the response. The way this is worded, because of this line - "Modules are the main way to package and reuse resource configurations with Terraform." reads like - "I have 10 golang apps, they all at a minimum use these same services, this is our "golang root module". But some services might have more or less modules, ie service A uses redis, service B uses kafka without redis.

So in this verbiage, is every single "stack/app" a "root module" and if one of them has a different database/whatever module it's just using different child modules and the child modules are the big differentiator?

Just to kind of prove the root-module argument I'm making here, this post in here is confused on calling a "stack" a module as well https://news.ycombinator.com/item?id=37005949

rcrowley · 2 years ago

Glad we cleared up our terminology! I agree that “root module” risks ambiguity, just like you point out.

I just realized I never responded to the very last point in your original comment. I don’t have, and I don’t think Terraform has, a complete solution to dependencies between root modules. Fortunately, data sources will at least fail if they don’t find what they’re looking for. For me, these failures never come up in production since I’m never using `terraform destroy` there. It does come up in pre-production testing and that’s an area that seems rich with patterns and products on top of Terraform that are controlling order and dependence between root modules.

PS thanks for your work on Terraform and Kubernetes.

rcrowley commented on Terraform best practices for reliability at any scale substrate.tools/blog/terr... · Posted by u/holoway

swozey · 2 years ago

Ok I think we're talking about two separate things here - you're referencing a root module and not a "stack", as in a stack is a full service/application that uses multiple modules to deploy. Your db module, eks module, etc. All independent modules, not combined into one singular module. Say it's sitting in the /terraform/app1/services/db(&)app folders type of scenario.

I think you're talking about putting stateful and stateless objects inside of a single module. So you've got /terraform/modules/mybigapp/main.tf that has your microservice + database inside of it.

If I'm right and that's what you mean that's really interesting I don't think I've ever seen or done that but now I'm curious. I'm pretty sure I've never created an "app1" module with all of its resources.

Am I totally off here?

rcrowley · 2 years ago

I stuck with my typical term, root module, synonymous with how folks are using “stack” and “state” in various parts of this thread.

A module is any directory with Terraform code in it. A root module is one that has a configuration for how to store and lock Terraform state, providers, and possibly tfvars files. Both modules and root modules may reference other modules. You run `terraform init|plan|apply` in root modules.

I think my comment makes sense in that if you mix two services into the same root module (directly or via any amount of modules-instantiating-other-modules) you can end up with changes from two people affecting both services that you can’t easily sever.

Happy to clarify further if I’m still not addressing your original comment.

rcrowley commented on Terraform best practices for reliability at any scale substrate.tools/blog/terr... · Posted by u/holoway

swozey · 2 years ago

Can you elaborate on this? I've never heard of this IAC structure and I'm trying to figure out what the benefit/cons are. Maybe it's just Friday and I'm checked out already.

If you run a terraform apply and only update microservices but you also have your dbs/stateful things in the same stack/app, you're only updating the microservices so how would this affect the db/stateful at all?

On the opposite end - I feel like there would be scenarios where I needed to update the stateful AND stateless services with the same terraform apply. Maybe I'm adding a new cluster and adding a db region/replica/securitygroup and that new cluster needs to point at the new db region.

In your scenario I would have updated microservices trying to reach out to a db in a region that doesn't exist yet because I have to terraform apply two different stacks. How would you deal with a depends_on?

Maybe I'm misunderstanding this.

rcrowley · 2 years ago

(Hi, I’m one of the authors of the article at the root of this thread.)

Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually.

It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.

rcrowley commented on Terraform best practices for reliability at any scale substrate.tools/blog/terr... · Posted by u/holoway

badblock · 2 years ago

Some of this seems like old advice, instead of having directories per environment you should be using workspaces to keep your environments consistent so you don't forget to add your new service to prod.

rcrowley · 2 years ago

(Hi, I’m one of the authors of the article at the root of this thread.)

I’ve gone back and forth on workspaces versus more root modules. On balance, I like having more root modules because I can orient myself just by my working directory instead of both my working directory and workspace. Plus, I feel better about stuffing more dimensions of separation into a directory tree than into workspace names. YMMV.