The difference is in the thermal mass of the building and the surface area exposed to the sun.
Anyways, if you write out all of the heat balance equations, you get a few W/m2 of flux on the inside wall of the home and a few W/m2 of flux on the inside faces of the fridge, assuming a typical wood frame construction in summer time and steady states all around.
So yes, of course multiplying the flux through the home’s wall by the surface area of the home results in a massive heat gain value compared to the heat gain conducted through the surface of the refrigerator, but that’s arguably precisely because of the two different volume requirements.
The insulation is actually solvable, and for heating can basically remove the power requirements: a house heated and using heat exchange on air leaving vs entering can be heated a lot just by having people inside it, let alone the other energy they use for other purposes. It's just more expensive to build this way, and with cheap energy it can a long time to pay back. Cooling you can't push down past the heat generated inside the house divided by the COP of your cooler, though.