Regardless if this or other paper is what you end up implementing, I wanted to say kudos for doing the right thing - having the right approach. Building stuff that's just now possible - in AI, from research papers - and productizing it, is how you get ahead of everyone else.
Thank you. I got something (but not the full method) working. However, I ended up abandoning this approach because I realized that the SDS process is too slow. It'd take at least 10 mins to generate a single SVG.
The special aspect of NNs (in the context of simulating worlds) is that NNs can mimic entire worlds from videos alone, without access to the source code (in the case of pokemon) or even without the source code having existed (as is the case for the real-world forest trail mimicked in this post). They mimic the entire interactive behavior of the world, not just the geometry (note e.g. the not-programmed-in autoexposure that appears when you look at the sky).
Although the neural world in the post is a toy project, and quite far from generating photorealistic frames with "trees that bend in the wind, lilypads that bob in the rain, birds that sing to each other", I think getting better results is mostly a matter of scale. See e.g. the GAIA-2 results (https://wayve.ai/wp-content/uploads/2025/03/generalisation_0..., https://wayve.ai/wp-content/uploads/2025/03/unsafe_ego_01_le...) for an example of what WMs can do without the realtime-rendering-in-a-browser constraints :)