The conclusion I came to is that humans have a few emotional needs in order to reach deep focus:
1. We need to not be distracted and we need to feel certainty that we will not be distracted. Before we are comfortable untaking the work to architect a giant mental castle, we need to feel confident that someone won't come around to kick it down.
2. At a more primitive, simian level, we need to feel that we're in a safe environment. It's hard to focus on code if you're worried that a tree is going to fall on you or a lion will drag you off into the jungle.
When it comes to sound, those can be opposing. Because we are a social species, I think one of the things that makes us feel most secure is the ambient present of a fellow tribe. To our early ancestors, dead silence meant you were alone, and being alone in the wilderness was often a death sentence. We instinctively feel safest when we hear the chatter and hub-bub of relaxed tribemates puttering around nearby. We know we're not alone, that there are others will also be alerted if something happens, and that they also feel safe.
But the presence of fellow people can also mean that at any moment one of them might wander over and start talking to us. So hearing that ambient chatter can make it really hard to focus.
This is, I think, part of why working in a coffeeshop can be so effective. You get the sense of safety in numbers, but since they are all strangers and at least in the US the social norms go against talking to strangers, you know the odds are slim that you'll actually be interrupted.