The Y-axis in that graph is fucking hilarious
Readit Newsericvolp12 commented on In praise of “normal” engineers charity.wtf/2025/06/19/in... · Posted by u/zdw
> Make it easy to do the right thing and hard to do the wrong thing.
This is basically the mantra of every platform team I've worked on. Your goal is to make the easy and obvious solution to engineers' problems the "right" one for the sustainability of software and reliability of services.
Make it easy to ship things that are reliable and manage distributed state well and can scale well and engineers will build better muscle memory for building software in that shape and your whole org will benefit.
This will never stop being true.
ericvolp12 commented on When imperfect systems are good: Bluesky's lossy timelines jazco.dev/2025/02/19/impe... · Posted by u/cyndunlop
I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).
When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.
This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.
ericvolp12 commented on Visualizing 13M Bluesky users joelgustafson.com/posts/2... · Posted by u/joelg
Love this kind of post! I was surprised to see you really can drink from the firehose:
brew install websocat
websocat wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos
Give Jetstream a try instead, it's all JSON for you already:
websocat wss://jetstream2.us-west.bsky.network/subscribe
ericvolp12 commented on How Discord stores trillions of messages (2023) discord.com/blog/how-disc... · Posted by u/jakey_bakey
Nope. Didn't change the schema, mainly added read coalescing and used ICS. I think the big thing is when Scylla is processing a bunch of tombstones it's able to do so in a way that doesn't choke the whole server. Latest Scylla version also can send back partial/empty pages to the client to limit the amount of work per query that is run.
Oh that's pretty neat. Did you just end up being okay with large partitions? I've been really afraid to let partition sizes grow beyond 100k rows even if the rows themselves are tiny but I'm not really sure how much of a real-world performance impact it has. It definitely complicates the data model to break the partitions up though.
ericvolp12 commented on How Discord stores trillions of messages (2023) discord.com/blog/how-disc... · Posted by u/jakey_bakey
Fwiw the benchmarked numbers are for writing very small rows. When doing the messages migration, with no read traffic, and the cluster/compaction settings tuned for writes we only managed approx 3m inserts/sec while fully saturating the Scylla cluster.
Interesting, we've got to 5M+ reads/sec in realistic simulated benchmarks and ~2M reads/sec of real-world-throughput on our clusters that are <10 nodes (though really high density). I don't think I've pushed writes beyond 1M QPS in real-world or simulated loads yet though. Thankfully our partitioning schemes are super well distributed though and our rows are very small (generally 1-5k) so I don't think we'd have a problem hitting some big numbers.
ericvolp12 commented on How Discord stores trillions of messages (2023) discord.com/blog/how-disc... · Posted by u/jakey_bakey
Yes, we wanted to migrate all our data stores away from Cassandra due to stability and performance issues. Moving to something that didn't have those issues (or at least had a different set of less severe issues) while also not having to rewrite a bunch of code was a positive.
Did you guys end up redesigning the partitioning scheme to fit within Scylla's recommended partition sizes? I assume the tombstone issue didn't disappear with a move to Scylla but incremental compaction and/or SCTS might have helped a bunch?
ericvolp12 commented on How Discord stores trillions of messages (2023) discord.com/blog/how-disc... · Posted by u/jakey_bakey
Is there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.
ScyllaDB scales horizontally on a shard-per-core architecture with a ballpark throughput of 12,500 Reads and 12,500 Writes per second per shard. If you're running Scylla across a total of 64 cores (maybe on 4 VMs with 16 vCPUs each), you can get up to 800k Reads 800k Writes per sec of throughput with P99 writes of <500us and p99 reads of <2ms.
You will not be able to get that performance out of Postgres and the write scaling will also be impossible on a non-sharded DB.
If you're a company like Discord and are running dozens (70-something?) of ScyllaDB nodes, likely each with 32 or 64 vCPUs, you've got capacity for 50M+ reads/writes per second across the cluster assuming your read/write workloads are evenly balanced across shards.
A friend of mine had her division's headcount cut by >80% that was all research focused and building instruments for deep space observation. No one is hiring people to do that in the private sector. Dozens of astrophysics PhDs in that division alone are now without work and with no real prospects doing anything related to what they've dedicated their entire lives to (and accepted modest salaries as civil servants to do).