dap (u/dap) - Readit News

dap commented on Pricing Changes for GitHub Actions resources.github.com/acti... · Posted by u/kevin-david

Everyone in this thread has gone absolutely insane. $5/month gets you 41 fucking _hours_ of continuous operation. If you're not utterly abusing the platform, this falls extremely below the threshold of caring. And if not, what the fuck are you even doing with all those hours? The new per-minute charge is less than one millisecond of engineer labor cost.

dap · 5 days ago

Doesn't this depend a lot on how long your actions run? Like, you may have already invested in your own hardware (maybe because your actions use a lot of resources and it's cheaper) and now you have to pay per-minute of action runtime for the API that does the bookkeeping?

dap commented on Vacuum Is a Lie: About Your Indexes boringsql.com/posts/vacuu... · Posted by u/birdculture

anarazel · 6 days ago

> First, although I work at Oxide, please don't think I speak for Oxide. None of this happened at Oxide. It informed some of the choices we made at Oxide and we've talked about that publicly. I try to remember to include the caveat that this information is very dated (and I made that edit immediately after my initial comment above).

I said oxide, because it's come up so frequently and at such length on the oxide podcast... Without that I probably wouldn't have commented here. It's one thing to comment on bad experiences, but at this point it feels like more like bashing. And I feel like an open source focused company should treat other folks working on open source with a bit more, idk, respect (not quite the right word, but I can't come up with a better one right now).

I probably shouldn't have commented on this here. But I read the message after just having spent a Sunday morning looking into a problem and I guess that made more thin skinned than usual.

> For most of that time (and several years earlier), the community members we reached out to were very dismissive, saying either these weren't problems, or they were known problems and we were wrong for not avoiding them, etc.

I agree that the wider community sometimes has/had the issue of excusing away postgres problems. While I try to avoid doing that, I certainly have fallen prey to that myself.

Leaving fandom like stuff aside, there's an aspect of having been told over and over we're doing xyz wrong and things would never work that way, and succeeding (to some degree) regardless. While ignoring some common wisdom has been advantageous, I think there's also plenty where we just have been high on our own supply.

> What remains is me feeling triggered when it feels like users' pain is being casually dismissed.

Was that done in this thread?

dap · 5 days ago

>> What remains is me feeling triggered when it feels like users' pain is being casually dismissed.

> Was that done in this thread?

Well, I raised a general problem around 24/7/365 use cases (rooted in my operational experience, reinforced by the more-current words that I was replying to and the OP) and you called it "tedious", "low-info griping". Yes, that seems pretty dismissive.

(Is it fair? Though I thought the podcast episodes were fairly specific, they probably glossed over details. They weren't intended to be about those issues per se. I did write a pretty detailed post though: https://www.davepacheco.net/blog/2024/challenges-deploying-p... (Note the prominent caveat at the top about the experience being dated.))

You also wrote:

> running an, at the time, outdated postgres, on an outdated OS

Yes, pointing to the fact that the software is old and the OS is unusual (it was never outdated; it was just not Linux) are common ways to quickly dismiss users' problems. If the problems had been fixed in newer versions, that'd be one thing. Many (if not all) of them hadn't been. But also: the reason we were running an old version was precisely that it was a 24/7/365 service and there was no way to update databases without downtime, especially replicated ones, nor a great way to mitigate risk (e.g., a mode for running the new software without updating the on-disk format so that you can go back if it's a disaster). This should be seen as a signal of the problem, not a reason to dismiss it (as I feel like you're doing here). As for the OS, I can only think of one major issue we hit that was OS-specific. (We did make a major misconfiguration related to the filesystem that certainly made many of our issues much worse.)

I get that it sucks to keep hearing about problems from years ago. All of this was on 9.2 - 9.6 -- certainly ancient today. When this comes up, I try to balance sharing my operational experience with the fact that it's dated by just explaining that it's dated. After all, all experience is dated. Readers can ignore it if they want, do some research, or folks in the PostgreSQL world can update me when specific things are no longer a problem. That's how I learned that the single-threaded WAL receiver had been updated, apparently in part because of our work: https://x.com/MengTangmu/status/1828665449850294518 (full thread: https://x.com/MengTangmu/status/1828665439234474350). I'll happily share these updates wherever I would otherwise share my gripes!

dap commented on Avoid UUID Version 4 Primary Keys in Postgres andyatkinson.com/avoid-uu... · Posted by u/pil0u

stickfigure · 6 days ago

This is incredibly database-specific. In Postgres random PKs are bad. But in distributed databases like Cockroach, Google Cloud Datastore, and Spanner it is the opposite - monotonic PKs are bad. You want to distribute load across the keyspace so you avoid hot shards.

dap · 6 days ago

It is, although you can have sharded PostgreSQL, in which case I agree with your assessment that you want random PKs to distribute them.

It's workload-specific, too. If you want to list ranges of them by PK, then of course random isn't going to work. But then you've got competing tensions: listing a range wants the things you list to be on the same shard, but focusing a workload on one shard undermines horizontal scale. So you've got to decide what you care about (or do something more elaborate).

dap commented on Vacuum Is a Lie: About Your Indexes boringsql.com/posts/vacuu... · Posted by u/birdculture

anarazel · 7 days ago

> Every time Postgres advice says to “schedule [important maintenance] during low traffic period” (OP) or “outside business hours”, it reinforces my sense that it’s not suitable for performance-sensitive data path on a 24/7/365 service and I’m not sure it really aims to be.

It's a question of resource margins. If you have regular and predictable windows of low resource utilization, you can afford to run closer to the sun during busy periods, deferring (and amortizing, to some degree) maintenance costs till later. If you have a 24/7/365 service, you need considerably higher safety margins.

Also, there's a lot of terrible advice on the internet, if you haven't noticed.

> (To be fair, running it like that for several years and desperately trying to make it work also gave me that feeling. But I’m kind of aghast that necessary operational maintenance still carries these caveats.)

To be fair, I find oxides' continual low-info griping against postgres a bit tedious. There's plenty weaknesses in postgres, but criticizing postgres based on 10+ year old experiences of running an, at the time, outdated postgres, on an outdated OS is just ... not useful? Like, would it useful to criticize oxides lack of production hardware availability in 2021 or so?

Edit: duplicated word removed

dap · 6 days ago

> It's a question of resource margins.

What you describe is true and very important (more margin lets you weather more disruption), but it's not the whole story. The problem we had was queueing delays mainly due to I/O contention. The disks had the extra IOPS for the maintenance operation, but the resulting latency for all operations was higher. This meant overall throughput decreased when the maintenance was going on. The customer, finally accepting the problem, thought: "we'll just build enough extra shards to account for the degradation". But it just doesn't work like that. If the degradation is 30%, and you reduce the steady-state load on the database by 30%, that doesn't change the fact that when the maintenance is ongoing, even if the disks have the IOPS for the extra load, latency goes up. Throughput will still degrade. What they wanted was predictability but we just couldn't give that to them.

> To be fair, I find oxides' continual low-info griping against postgres a bit tedious. There's plenty weaknesses in postgres, but criticizing postgres based on 10+ year old experiences of running an, at the time, outdated postgres, on an outdated OS is just ... not useful?

First, although I work at Oxide, please don't think I speak for Oxide. None of this happened at Oxide. It informed some of the choices we made at Oxide and we've talked about that publicly. I try to remember to include the caveat that this information is very dated (and I made that edit immediately after my initial comment above).

I admit that some of this has been hard for me personally to let go. These issues dominated my professional life for three very stressful years. For most of that time (and several years earlier), the community members we reached out to were very dismissive, saying either these weren't problems, or they were known problems and we were wrong for not avoiding them, etc. And we certainly did make mistakes! But many of those problems were later acknowledged by the community. And many have been improved -- which is great! What remains is me feeling triggered when it feels like users' pain is being casually dismissed.

I'm sorry I let my crankiness slip into the comment above. I try to leave out the emotional baggage. Nonetheless, I do feel like it's a problem that, intentionally or otherwise, a lot of the user base has absorbed the idea that it's okay for necessary database maintenance to significantly degrade performance because folks will have some downtime in which to run it.*

dap commented on Vacuum Is a Lie: About Your Indexes boringsql.com/posts/vacuu... · Posted by u/birdculture

lfittl · 7 days ago

The article has a section where it estimates index bloat based on comparing the number of index reltuples * 40 bytes (?), compared to the size of the file on disk.

This is problematic, first of all because I don't think the math is right (see [0] for a more comprehensive query that takes into account column sizes), and second because it ignores the effects of B-Tree index deduplication in Postgres 13+: [1]

In my experience, fast bloat estimation queries can work okay for table bloat, but for index bloat I'd recommend instead looking at the change in page density over time (i.e. track relpages divided by reltuples), or just go direct to running pgstatindex outside business hours.

[0]: https://github.com/pgexperts/pgx_scripts/blob/master/bloat/i... [1]: https://www.postgresql.org/docs/current/btree.html#BTREE-DED...

dap · 7 days ago

Every time Postgres advice says to “schedule [important maintenance] during low traffic period” (OP) or “outside business hours”, it reinforces my sense that it’s not suitable for performance-sensitive data path on a 24/7/365 service and I’m not sure it really aims to be. (To be fair, running it like that for several years and desperately trying to make it work also gave me that feeling. But I’m kind of aghast that necessary operational maintenance still carries these caveats.)

dap commented on Vacuum Is a Lie: About Your Indexes boringsql.com/posts/vacuu... · Posted by u/birdculture

dap · 7 days ago

> The exclusive lock is only needed during the final swap phase, and its duration can be configured.

FYI: even a very short operation that requires an exclusive lock can induce significant downtime if there’s anything else that holds a shared lock for extended periods. In [1], there was:

- a wraparound autovacuum (which holds a shared lock for potentially a long time — like hours)

- lots of data path operations wanting a shared lock

- one operation that should have been very brief that merely tried to take an exclusive lock

The result is that the presence of an operation wanting an exclusive lock blocked the data path for the duration of the autovacuum. Major outage.

[1] https://web.archive.org/web/20190320162510/https://www.joyen...

Edit: this was a while ago with v9.2, but I don’t know if any of this behavior has changed.

dap commented on The programmers who live in Flatland blog.redplanetlabs.com/20... · Posted by u/winkywooster

xigoi · 14 days ago

> What business concepts or thoughts can you express in Clojure that you can’t express in Python or Rust?

If you only think about programming languages as a way to make money, the analogy of being stuck in Flatland is perfect.

dap · 13 days ago

I’m sympathetic to looking down on the obsession with money. But there’s something deep and important about the monetary element. Engineering is about solving real-world, practical problems. The cost is a real factor in whether a potential solution is a useful one.

I think the money question is a red herring here. I’d phrase it more like: what problem in a user’s problem space is expressible only like this? And if the only user is the programmer, that’s alright, but feels more aligned with pure academia. That’s important, too! But has a much smaller audience than engineering at large.

dap commented on Cloudflare outage on December 5, 2025 blog.cloudflare.com/5-dec... · Posted by u/meetpateltech

roguecoder · 15 days ago

Yep: it is wild for them to claim that a strongly-typed language would have saved them when it didn't.

Relying on language features instead of writing code well will always eventually backfire.

dap · 15 days ago

You're right that you have to "write code well" to prevent this sort of thing. It's also true that Rust's language features, if you use them, can make this sort of mistake a compile-time error rather than something that only blows up at runtime under the wrong conditions. The problem with their last outage was that somebody explicitly opted out of the tool provided by the language. As you say, that's "not writing code well". But I think you're dismissing the value of the language feature in helping you write code well.

dap commented on Cloudflare outage on December 5, 2025 blog.cloudflare.com/5-dec... · Posted by u/meetpateltech

jpeter · 16 days ago

Unwrap() strikes again

dap · 16 days ago

I guess you’re being facetious but for those who didn’t click through:

> This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur.

dap commented on Is 2026 next year? google.com/search?q=is+20... · Posted by u/kjhughes

greenpizza13 · 18 days ago

The thesis of "What is Intelligence" is based around intelligence being just that.

> Intelligence is the ability to model, predict, and influence one’s future; it can evolve in relation to other intelligences to create a larger symbiotic intelligence.

The book is worth a read. But I don't believe it limits the type of intelligence we have to humans, by definition. Then again, I'm only halfway through the book :).

[https://mitpress.mit.edu/9780262049955/what-is-intelligence/]

dap · 18 days ago

It seems obvious to me that "the ability to model, predict, and influence one’s future" is far more general and capable than "constrained to pattern recognition and prediction of text and symbols." How do you conclude that those are the same?

I do like that definition because it seems to capture what's different between LLMs and people even when they come up with the same answers. If you give a person a high school physics question about projectile motion, they'll use a mental model that's a combination of explicit physical principles and algebraic equations. They might talk to themselves or use human language to work through it, but one can point to a clear underlying model (principles, laws, and formulas) that are agnostic to the human language they're using to work through them.

I realize some people believe (and it could be) that ultimately it really is the same process. Either the LLM does have such a model encoded implicitly in all those numbers or human thought using those principles and formulas is the same kind of statistical walk that the LLM is doing. At the very least, that seems far from clear. This seems reflected in the results like the OP's.