I apologize for the traffic. We clearly need to look at the “thundering herd” you are observing. That shouldn’t be happening at all.
Separately, the bug we fixed last time were about repeatedly cloning a repo in sequence even if it was unchanged, not about redundant parallel fetches. We fixed that only for Git, because Git makes it very easy to look at a branch or tag and get the tree hash, without downloading the full repo. This is exposed as the go command’s -reuse flag. The relevant Git code is at https://go.dev/src/cmd/go/internal/modfetch/codehost/git.go#... (CheckReuse).
What we need from any VCS to implement the reuse check is a cheap way to download a list of every tag and branch along with a cryptographic file tree checksum for each one, without doing a full repo clone. At the time, I convinced myself Mercurial did not support this. Perhaps I was wrong or perhaps it does now. If anyone can help us understand how that works, we could implement the -reuse flag for Mercurial too. (Any other VCSs would be great too, but Git is #1 by a very wide margin and I believe Mercurial is #2 also by a wide margin.)
Again, apologies for all the traffic, and thanks to Ted for the excellent analysis. We will look into both.
Anthropic learned this lesson. Google, Deepseek, Kimi, OpenAI and others keep repeating it. This feels like Gemini_2.5_final_FINAL_FINAL_v2.