I think that more than physics the bottleneck for this is political (at least in the US). All of the local large projects around me are expensive because of massive amounts of red tape (environmental studies, zoning, planning), and political patronage systems. After the kick backs, political donations, promises to only work 8 hours a day, only use union labor, hire x police officers for y hours in overtime security positions a month, use xyz contractor etc. a small cost seems to be the actual labor and materials. Hell these robots if they work will be made illlegal.

Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.

I'm relieved to see that its not written in rust

Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".

Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and then use that initialized tokenizer to initialize a new tokendagger and test identity of results.