This is really unique to the AI training clusters for reasons I'm not super clear on. Most other types of horizontally scaled workloads can sort of tolerate a slightly underperforming host, or hosts going bad every so often, with little P99/P99.9 impact. For some reason, AI training workloads really cannot.

I wonder if anybody's had similar experiences...

I've worked at two companies with RFC processes. At both companies, the processes were -- and I hate to use such a strong word -- a total sham.

I suppose the processes were crafted to look like some egalitarian meritocracy. In reality, it was just a test of who had the most political pull. If you had management on your side, your "RFC" was effectively law and dissenting voices were effectively career suicide.

Which, you know... fine. I understand that choices are made based on cliques and political capital rather than anything else. Cool. That's okay! Sucks sometimes, but that's how the world works. Hopefully your org structure itself is at least something of a meritocracy, so that edicts from on high are of a generally high-enough quality. If they get it right-ish often enough, your org will be okay... probably.

But what really rankled me was the fact that the RFC processes amounted to some kind of elaborate cosplay so that, I guess, folks could pretend that there was some sort of healthy collaborative process. If you're going to let a few "popular and management-blessed" engineers make all the decisions, fine, but don't add insult to injury by pretending otherwise.

I've heard similar things from others, elsewhere. I hope there are at least a few companies dedicated to being something better.