Readit News_ntka commented on How do I get started with Jax on TPU VMs github.com/shawwn/website... · Posted by u/yeesian
The current SOTA models (GPT4, DALL-E, Sora) were trained on GPUs. The next one (GPT5) will be, too. And the one after that. Besides, only very few people train models that need more than a few hundred H100s at a time, and PyTorch works well at that scale. And when you train large scale stuff the scaling problems are demonstrably surmountable, unlike, say, capacity problems which you will run into if you need a ton of modern TPU quota, because Google itself is pretty compute starved at the moment. Also, gone are the days when TPUs were significantly faster. GPUs have “TPUs” inside them, too, nowadays
No, I am saying, with JAX you train on G.P.U., with a G, and your training runs are >2x faster, so your training costs are 2x lower, which matters whether your training spend is $1k or $100M. You're not interested in that? That's ok, but most people are.
_ntka commented on How do I get started with Jax on TPU VMs github.com/shawwn/website... · Posted by u/yeesian
_ntka commented on How do I get started with Jax on TPU VMs github.com/shawwn/website... · Posted by u/yeesian
Are there any resources going into detail about why the big players prefer JAX? I've heard this before but have never seen explanations of why/how this happened.
It's all about cost and performance. If you can train a foundation model 2x faster with JAX on the same hardware, you are effectively slashing your training costs by 2x, which is significant for a multi-million dollar training run.
_ntka commented on How do I get started with Jax on TPU VMs github.com/shawwn/website... · Posted by u/yeesian
Hmm, 3% market share framework with barely any ecosystem and single vendor accelerators (Jax on TPU) vs a 60% market share framework with insanely rich ecosystem and ability to debug code on your own workstation (PyTorch on GPU)? In my informed opinion most people should use the latter unless they like wasting time on shiny things
JAX is used by almost every large genAI player (Anthropic, Cohere, DeepMind, Midjourney, Character.ai, XAi, Apple, etc.). Its actual market share in foundation models development is something like 80%.
Also JAX is not just for TPU. It's mainly for GPU. It's usually 2-3x faster than torch on GPU: https://keras.io/getting_started/benchmarks/
Far more industry users of JAX use it on GPU compared to TPU.
_ntka commented on High-performance image generation using Stable Diffusion in KerasCV keras.io/guides/keras_cv/... · Posted by u/tosh
Which GPU did you test on Colab? Are you comparing with one of the fp16 PyTorch versions? Their test shows little improvement on V100.
PyTorch is now quite a bit more popular than Keras in research-type code (except when it comes from Google) so I don't know if these enhancements will get ported. This port was done by people working on Keras which is kind of telling - there isn't a lot of outside interest.
This is not true, the initial Keras port of the model was done by Divam Gupta who is not affiliated with Keras or Google. He works at Meta.
The benchmark in the article uses mixed precision (and equivalent generation settings) for both implementations, it's a fair benchmark.
In the latest StackOverflow global developer survey, TensorFlow had 50% more users than PyTorch.
_ntka commented on Turning the Tide on Climate Change with Green Sand Beaches [pdf] projectvesta.org/wp-conte... · Posted by u/mrnobody_67
7 cubic km of olivine is a staggering amount. Makes you wonder whether the effect of deploying the olivine would even offset the energy cost and CO2 emissions of extracting and moving around all that rock.
The green glow used in pop culture has its origin in the widespread use of radium paint to achieve a glow-in-the-dark effect (e.g. on watch faces) in the early 20th century. I still own a radium watch. The paint was always fluorescent green. And it did glow.