Readit NewsSatya saved Microsoft by doubling down on Azure and Cloud. Something that Balmer failed to do with Mobile.
karterk commented on I fixed the strawberry problem because OpenAI couldn't xeiaso.net/blog/2024/stra... · Posted by u/xena
karterk commented on I fixed the strawberry problem because OpenAI couldn't xeiaso.net/blog/2024/stra... · Posted by u/xena
Solving the strawberry problem will probably require a model that just works with bytes of text. There have been a few attempts at building this [1] but it just does not work as well as models that consume pre-tokenized strings.
karterk commented on Eagle 7B: Soaring past Transformers blog.rwkv.com/p/eagle-7b-... · Posted by u/guybedo
It's interesting how all focus is now primarily on decoder-only next-token-prediction models. Encoders (BERT, encoder of T5) are still useful for generating embedding for tasks like retrieval or classification. While there is a lot of work on fine-tuning BERT and T5 for such tasks, it would be nice to see more research on better pre-training architectures for embedding use cases.
karterk commented on What are farm animals thinking? science.org/content/artic... · Posted by u/mooreds
> If people spent time with meat animals, there would be a lot more vegetarians.
Having grown up on a ranch, yes I know cows are smart... chickens not so much. It does not change my views on eating beef at all.
Cows are treated very well in all the ranches I've been on. One because it makes them taste better, another because we about our livestock.
Your alternative is them not existing at all, they cannot survive in nature.
The cow does not know it's going to be slaughtered.
It lives a nice happy life roaming the fields, on the last few days it goes to processing to eat a massive last great meal.
It's the nicest form of us being eating our prey.
> Cows are treated very well in all the ranches I've been on.
Unfortunately the entire premise of your argument is based on a personal anecdote that's a gross generalization of vast number of cattle farms.
karterk commented on JAX – NumPy on the CPU, GPU, and TPU jax.readthedocs.io/en/lat... · Posted by u/peter_d_sherman
Does it support arrays of variable lengths now? Last time I looked, I think this was not supported. So it means, for every variable dimension, you need to use an upper bound, and then use masking properly, and hope that it would not waste computation too much on the unused part (e.g. when running a loop over it).
I'm working with sequences, e.g. speech recognition, machine translation, language modeling. This is a quite fundamental property for this type of models, that we have variable lengths sequences.
In those cases, for some example code, I have seen that training also used only fixed size dimensions. And at inference time, they had some non-JAX code for the loop over the sequence around the JAX code with fixed-size dimensions.
This seems like a quite fundamental issue to me? I wonder a bit that this is not an issue for others.
For JIT-ing you need to know the sizes upfront. There was an experimental branch for introducing jagged tensors, but as far as I know, it has been abandoned.
That is the current issue with most C++ codebases nowadays, I only see modern C++ on conference slides, when I look into codebases even from ISO C++ members, it is always C++ full of C idioms no matter what.
I bet most candidates to C++ job offers end up discovering the hardly reality of existing code.
Can you give some examples of problematic C idioms in C++?
Also Amazon is in another capital intensive business. Retail. Spending billions on dubious AWS moonshots vs just buying more widgets and placing them across the houses of US customers for even faster deliveries does not make sense.
This is not really true. Google has all the compute but in many dimensions they lag behind GPT-5 class (catching up, but it has not been a given).
Amazon itself did try to train a model (so did Meta) and had limited success.