During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user?
I assume the former has massive overhead, but maybe it is worthwhile to keep responsiveness up for everyone.
In addition to stuff like that they also handle it with rate limits, that message that Claude would throw almost all the time when they were like "demand is high so you have automatically switched to concise mode", making batch inference cheaper for API customers to convince them to use that instead of real time replies. The site erroring out during a period of high demand also works, prioritizing business customers during a rollout, the service degrading. It's not like any provider has a track record for effortlessly keeping responsiveness super high. Usually it's more the opposite.
And it's ~$1000 to build a PC with a similar CPU, somewhat larger form factor, and fans. Unless the AI processor is actually useful for AI, and you need that, this is silly.
Framework desktop dimensions are 20.6 x 9.7 x 22.6 LWH. My IM01 case is 37.2 x 18.5 x 28.7. It won't be going in my bag, but it fits nicely on a desktop.
Pre-builts are so expensive these days...