Talking politics == my free time, and I don't want to exchange my leisure hours for talking politics with my co-workers. The entire rest of your life can be talking about politics with your friends, your life shouldn't just be work.
It seems the real problem is too much of people's lives is work and so they think they have no time to talk politics.
I should also add I don't think unions should not be involved in politics; I think they are used as crutch for democratic representation. The real solution is removing corporate lobbying.