This still leaves a case where someone manages to get the final key out of memory but you're pretty hosed at that point anyway. I'd prefer a system where the yubikey itself is doing the final credential decryption instead of the CPU, unfortunately most people aren't that paranoid though.
Also, if you're struggling with getting things done because you're always dopamine seeking, get checked for ADHD. There are some extremely effective treatments that can make your life way easier if that turns out to be the problem.