I think algorithm design is hard. Most students/people can't break down a problem into small enough steps for the computer to do them.
Are you familiar with the concept of "chunking" in memory research? Well, computers don't chunk but humans do. Humans tend to think about their algorithms in terms of the chunks they already know, but for computers sometimes each chunk has to be broken down into much smaller sub-steps.
And that's non-intuitive and hard. When explaining stuff to a human you get immediate non-verbal or sometimes verbal feedback if they don't understand the chunks, but computers just give a syntax error.
SECONDLY. Students have trouble making an accurate mental model of what the computer is doing at each step, so they can't trace through the code, much less create new code.
Those are the two biggies, in my experience.
It sounds like what you're describing is not figuring out at a high level what to do, but formalizing that into low-level instructions for a computer. This goes exactly to my point. What I meant by "algorithm design" is strategy selection, which I want to be careful to separate from strategy formalization for a computer. It is the latter that I think laypeople struggle with the most, however, the fascination with algorithms often drives attention to the former. For example, it's easy to be impressed by the cleverness of binary search, but it's actually relatively straightforward to understand as a high-level concept. The harder thing for a layperson is to formalize binary search into a working set of computer steps. I think there is a certain cognitive bias at play, where the formality of fine detail seems lowly and menial, so we want to skip over it, in spite of the fact that it's actually the biggest hurdle for most people to overcome.
A trivial example (your editor can do this for you):