I think what Ilya is trying to get at here is more like: someone very smart can seem "unpredictable" to someone who is not smart, because the latter can't easily reason at the same speed or quality as the former. It's not that reason itself is unpredictable, it's that if you can reason quickly enough you might reach conclusions nobody saw coming in advance, even if they make sense.
I think it's important for us to all understand that if we build a machine to do valuable reasoning, we cannot know a priori what it will tell us or what it will do.
Just look at inductive reasoning. Each step builds from a previous step using established facts and basic heuristics to reach a conclusion.
Such a mechanistic process allows for a great deal of "predictability" at each step or estimating likelihood that a solution is overall correct.
In fact I'd go further and posit that perfect reasoning is 100% deterministic and systematic, and instead it's creativity that is unpredictable.
[1] https://survey.stackoverflow.co/2025/technology