What happens when something is put on the scale while it's sampling? Does the curve depend on properties of the scale, or just properties of the object and the manner in which it was put on the scale?
It's the latter. The scale is meant for real-time monitoring of rapidly varying force (the primary application is about monitoring the force derivative and repeatable max force logging). It uses an aluminum load cell if you're familiar with those, there's a tad of a multi-kHz resonance that is typically overshadowed by the object properties.
The tokens themselves are a form of compression. Lets say we have the word "WaffleHouse", character level this would be 11 tokens, but with an embedder this would be perhaps 2 or 3 tokens (I didn't actually run through the tokenizer but we could verify precisely). This matters a lot for on device processing especially.
So while we could get more intelligence out of the model by bumping up the "knowledge" parameters, the device would need to process more input and output tokens.
Another advantage on small devices is the embeddings are just a lookup table which requires little to no computation. Its the rest of the parameters that have the expensive matrix multplications, so if we increased those we'd also be increasing the number of FLOPs needed for a forward pass.
This blog post explains it well. https://www.adamcasson.com/posts/transformer-flops
So all this to say is there are definite tradeoffs between model size, performance on evals, and compute cost. We ran many internal experiments with different choices to see could work well, and then picked what we believed work will best for the open community.