Readit NewsDeleted Comment
The author also mentions a leader adjudicator, which means there is probably some sort of coordination to pick a leader. This raises the question of how a leader is picked, and if leadership changes based on how hot a key is in a given AZ.
This blog series is a great read. Every day Marc drops excellent content and leaves room for questions, which he ends up answering on the following days. Hope more details come next.
> We’ve learned from building and operating large-scale systems for nearly two decades that coordination and locking get in the way of scalability, latency, and reliability for systems of all sizes. In fact, avoiding unnecessary coordination is the fundamental enabler for scaling in distributed systems
I hope there is a follow-up since the points the author only glossed over are important to understanding the architecture and trade-offs. I would like to know about the cross-adjudicator coordination protocol and how the journal works.
From the information available, it seems that DSQL should be pretty fast as long as you keep writes local. Once you add active-active replication and start writing to the same key in different regions the coordination costs should slow the system down significantly (or not - but if that is the case I want to know how they managed to do it).
- How the OS knows it can clean up an inode after a hard link is deleted? The post mentioned inodes don't see hard links
- What does it mean to have a dead/dangling soft link?
Glad to see this one mentioned. In fact, it is so popular that it has become a standard unit of measure in Brazilian recipes. The reason it is so popular is because most beer sold in Brazil comes in 600 ml bottles to be shared by everyone at the table. Beer drinking in Brazil is mostly a social activity.
I somehow started to find they kind of beautiful when I worked at a company that only had pint-sized American glasses at their office. Now most cups in my house have this design. They are dirty cheap and very easy to replace.
It is still my go to player, it works great in Windows 10.
I see where this is coming from, but it's also pretty sad. In my experience, it tends to create environments where engineers are second-class citizens compared to managers or product: we're just responsible for "delivery", but can't independently make any real decisions beyond a tiny scope. Our timespan of discretion becomes measured in days or weeks, while, on the best teams I've seen, it's measured in months, quarters or years.
It's counterintuitive, but you absolutely can have real individual owernship for engineers without creating single points of failure or brittle systems. It's a matter of having other sources of slack, encouraging quality work, and giving people a lot of room to fluidly adjust what they're working on when. Folks still have real ownership and can make longer-term decisions on their own, but they also collaborate in ad hoc ways and share tacit knowledge, so that somebody else can jump in, help out or even take over in a pinch. I'm being a bit vague, but all I can say is that I've seen this before, and I'll know it when I see it again.
In practice, the model I saw did end up with more rewriting than a normal team—but we were still way more productive overall, even accounting for the rewrites! Turns out that rewriting systems incrementally, in self-contained chunks, is an amazing way to both evolve the design and to build up instutitional knowledge and capacity. It looks like waste, but it's actually slack that is crucial to making your system as a whole more flexible, adaptable and resilient. (In fact, that's true of a lot of the "waste" top-down management systems try to reduce—I'm incresingly convinced that anybody trying to optimize software team "utilization" is actively sabotaging the work!)
But I think that is beside the point.
Individuals are not fungible, but team members are - or at least can be, depending on how you structure your teams.
And as your org grows, you want predictability on a team level. Skipping a bunch of reasoning steps, this means having somewhat fungible team members, to give you redundancy.
The engineering parallel here is the tradeoff between resilience and efficiency. You can make a system more reliable by adding redundancy. You make a system more efficient by removing redundancy.