Being cold weakens your immune system. Draft air increases heat loss. There is nothing complex to understand. Outside you would wear a scarf or other appropriate clothing to not feel cold.
That‘s one of the biggest health myths around. Cold weather does NOT weaken your immune system AT ALL (except if you‘re actually hypothermic, which is very different from just feeling uncomfortable). It’s the CONDITIONS that RESULT from cold weather that actually cause those infections to ramp up in winter (think more people staying inside in enclosed spaces).