I've noticed over and over again at various jobs that people underestimate the benefit of speed, because they imagine doing the same workflow faster rather than doing a different workflow.
For example, if you're running experiments in one big batch overnight, making that faster doesn't seem very helpful. But with a big enough improvement, you can now run several batches of experiments during the day, which is much more productive.
Whenever we make our code faster the users just run bigger models :P.
Although, reasoning about performance can be hard anyway.
For the uninitiated, most high-performance CPUs of recent years:
- Are massively out-of-order. It will run any operation that has all inputs satisfied in the next slot of the right type available.
- Have multiple functional units. A recent apple CPU can and will run 5+ different integer ops, 3+ load/stores and 3+ floating point ops per cycle if it can feed them all. And it may well do zero-cost register renames on the fly for "free".
- Functional units are pipelined, you can throw 1 op in the front end of the pipe each cycle, but the result sometimes isn't available for consumption until maybe 3-20 cycles later (latency depends on the type of the op and if it can bypass into the next op executed).
- They will speculate on branch results and if they get them wrong it needs to flush the pipeline and do the right thing.
- Assorted hazards may give +/- on the timing you might get in a different situation.