Why does the Harbor VM need 32 cores and 64GB of RAM? Especially if it's only serving 32,000 pulls over 2 months.

Luckily GCP load balancers and Pub/Sub we're generally unaffected, at least for us, which are the 2 products we depend on the most. Messages piled up in Pub/Sub while other GCP products were down but that was it. We did uncover a few spots where we had a GCP product dependency in the critical path that didn't need to be (the actual usage is optional but we don't ignore the error).

In most industries there isn't an expectation that providers are multi-cloud because these multi-region, multi-product outages within a single cloud are rare. Hopefully customers were forgiving in this case and see news headlines validating the extent of the outage.

It's not clear what the actual issue was. My best guess is that Workers KV was in transition to B2 from a combination of GCS and "a storage provider" before. A step in transitioning to B2 involved removing the "a storage provider" and relying just on GCS for some period of time because of consistency concerns. Today's GCS outage was unfortunately timed and they didn't have coverage to support GCS being down.

Separately, their dependence on their own products results in a SPOF where now, theoretically, if B2 goes down so does over half of Cloudflare's product suite. Ideally those outages can be limited to a single region at a time but that's still a massive blast radius from a single service. I completely understand them not wanting to invest in competing products as a form of redundancy though.

We've been storing values on errors using https://github.com/levenlabs/errctx for close to a decade now. We have a separate logging library that stores a KV map but technically you can store anything on errors. I recently just made it unwrap errors to find deeply stored keys.