How do you balance that against the performance hit of working on-prem?
You can't just hand-wave away the unpaid labor and risk to your life that you undertake with every trip to and from the office. You can't ignore the drive-by conversations you get roped into when you're trying to do head-down work or the overall noise in an office. Yes, in ideal conditions your office might approach what you get working at home, but I've never experienced it, especially given that open plan offices are now an unquestioned default setting.
I suppose I can only speak from my experience. I can see definitely how many corp onsite environments are awful. There are productivity killers both onsite and remote and there are no absolutes. If we assume a 'good' remote vs 'good' onsite environment, collaboration is easier and faster onsite. Yes, you need to be very careful, and in my case we are super conscious of 'maker's time' and respect it with discipline.
I have been in plenty of organizations that actively cripple their remote workers' performance, and others where I have been my most productive. Remote work productivity depends directly on the culture surrounding it.
If you believe a remote team will be less productive, then it becomes a self-fulfilling prophecy. Instead, set clear expectations that your team can excel, and then ensure the culture enables them to realize that potential.
Sure, this philosophy can go only so far, but -- without that foundation -- the results will only fall short of your expectations.