What a great example of how a series of reasonable decisions can add up to something unreasonable when you scale up the inputs. This line is my favorite: “there was an open task for a few years to address it (T73412), and so I did it.”
This situation reminds me a little bit of trying to work with very large images in ImageMagick, like 60k x 60k resolution. A simple resize was taking 6 hours on my mac 10 years ago, due to IM trying to allocate the whole image at once and then swapping non-stop. And then I discovered that the Graphics Magick fork did streamed resizing, and did the task in a couple of minutes. It’s a small and relatively easy change, but someone had to prioritize handling large inputs first. This is one reason software is hard for me; every time I need to process large inputs I wish I had written a streaming mechanism, but every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Maybe it’s the right thing to do to avoid over-engineering, but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
> every time I start a project I decide to do the easy thing first and wait until I critically need streaming.
Absolutely reasonable decision making. Do what you need at the time you write the code, or at best, be slightly ahead of the curve. Don't forecast features you do not need. Humans are unreasonable about success of their own projects.
>Maybe it’s the right thing to do to avoid over-engineering
It is the only thing to do. You do not know the future use cases.
> [...] but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
If your software is subpar with regards to your needs, congrats, it is successful software!
Building the perfect software from the beginning is unreasonable because you might just never release or even use it.
Build imperfect things, and be reasonably frustrated BH them so that it gives you the energy to improve them!
>Building the perfect software from the beginning is unreasonable because you might just never release or even use it.
I built a software library for an external device from specification. We eventually got the device to test and it worked when plugged in, but not everything was right.. I was early in my career and disappointed. My boss said "If it works perfectly the first time, you spent too much time on it.".
Oh yes. I worked for year in medical imaging and we suffered greatly when dealing with sanned pathology slides, that easily have a long side of 30k to 200k pixels, and we had to deal with big volumes of them, go figure.
But then we found libvips (that's not only a library but a cli tool too) and it was like night and day in big images. Like 10x faster and 1/10 memory use. I really loved it!
I can second libvips, I had to work on a project that needed to resize GIFs and vips was probably 10 times faster than the ImageMagick or Pillow. I think the only thing that compared to libvips was ffmpeg.
Very happy that Aras is fighting the good fight, doing some great OSS work after his departure from Unity. The speed up numbers are quite impressive!
I remember when I was first getting into shader programming 10+ years ago, I asked a question on the Unity forums and Aras was one of the first to respond. Even then he was a titanic figure behind Unity’s graphics stack. I never forgot how kind he was to help a kid like me out, I’m sure he was quite busy!
This is why it is very difficult to justify optimization work to management. If there are 20 things to optimize that take 10 seconds each, the change isn't really noticeable until you're getting past half-way. And once your processing already takes a few minutes, what's the harm in adding another 10 seconds?
I once shipped a crud app a few weeks earlier than I wanted to because of pressure from managers. I warned them that I hadn't done any optimizations, and it would quickly become unusable with real world use. So naturally, as soon as it was deployed I was reassigned to another project. 3 weeks later the users were complaining it was slow, in 6 weeks it was taking 20 minutes to load. They finally agreed to let me optimize it, in an hour I had it to 5 minutes, in a week I had it loading in 10 seconds. I could get it faster, but I was reassigned again, and at this point it is plenty fast for the users in question.
If I had done the easy change to get it to 5 minute before deploying, it would likely still be that way today annoying my users, but not enough to justify changing.
There is a similar effect, but in reverse, when adding interdependent features.
Early ones don't have a big impact, but once you get to a certain point the inefficiencies add up and the program becomes bogged down.
Cache invalidations and memory swapping as you approach the limits are other examples.
Well it depends what you are doing. If the thing you are optimizing is the thing blocking everything else routinely, spending a lot of resources on optimization is a no-brainer. If the thing you are optimizing takes long, but that does not matter so much, it is not worth optimizing.
When you are creating VFX scenes for a still image you don't really care a lot about the render times as long as it is done the next day or so. If you are doing it for animation, you care a lot about render times, because anything times a thousand will be quite a duration.
The work of a programmer is always a factor in a multiplication. If the small change you make is used by thousand people a day, using it 100 times a day each second you shave off will save a collective time of ~28 hours. And this is only time. You could also think about electricity, about frustration, etc.
A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it.
This is of course very good advice and I follow it myself quite often, but there is something to say about quantitate changes leading to qualitative outcomes. Especially with changes with performance in a factor of 150 and the like.
Some workflows that were considered “too long to be worth it” suddenly become easy and routine. It’s kind of like “disrupting the market” in a sense.
And more times than I can count small changes in performance that were considered irrelevant led to very big cultural changes in a company.
For example we had some e2e tests running in our CI/CD pipeline that were taking ~ 15 minutes, people were stressed (or not very productive) as the usual excuse was “I’m waiting for the tests” with not enough time to do something productive but too long to wait patiently. I spend like a day optimizing it and brought them down to 4 mins, suddenly people began to accomplish more, they started writing a lot more tests.
So a day of investment led to happier devs (less turnover), more tests (stability improvement) and faster feature turnaround. And I had to fight tooth and nail to spare the time to actually do this.
> A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it.
Unfortunately this same fact also means that varrious kinds of important software are hard to get developed.
When someone writes code to blink a mildly annoying advert on youtube it effects a billion people and can make millions or hundreds of millions in revenue. But code needed to make a local car wash' robots more efficient? -- unlikely to get written unless someone thinks they can sell it to a chain: a large number of programmers get snatched up by places like google that deploy to billions of people. Even though in the past when there were far fewer qualified programmers it might have been easier to get the car wash software developed because there simply was nowhere that could deploy software to a billion people instead (much less highly profitably).
The enormous leverage of software is an undeniable force for good. But it also changes the incentive structures of the world in ways that have negative effects too. :(
This isn't limited to software either, improvements to mass production has made mass produced goods extremely inexpensive-- but by that same token custom work has become much more expensive. And the world around us has become much more homogenized and cookie cutter as a result. But the leverage that software potentially has is vastly greater than other things because of its zero marginal cost of production.
I might also be interested in how long did it take to reach such a massive improvement. I guess the OP has spent/waited (in a CI Job?) countless hours testing the import of large files until they finalized it.
I think the author was smarter than that. They used profiling tools to identify the bottlenecks, then used much smaller/faster test cases to fix the bottlenecks.
>The moral this time might be, try to not have functions that work on one object, and are O(N_objects) complexity. They absolutely might be fine when called on one object. But sooner or later, someone will need to do that operation on N objects at once, and whoops suddenly you are in quadratic complexity land.
I usually put it this way to my colleagues: on a long enough timescale, every function will be called from a `for` loop.
Excellent writeup and great to see the fixes upstreamed, all driven by a new use case that perhaps was beyond the magnitude of previous import efforts.
While impressive, this misses the point. The Moana scene is a test scene for /renderers/.
It is not meant to be used inside a DCC app, as-is.
If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong.
That said – I think it's very cool if you can import such heavy geometry and your DCC app doesn't crash.
But in 15 years of working in VFX I never dealt with heavy data inside a DCC directly. Indirectly yes, through proxies.
The three most interesting metrics for this scene are, if you're a renderer author (sorted by importance during lookdev/lighting):
1. Time to first pixel (is it seconds or hours?).
2. Time to completion (does it take days/hours/minutes?).
3. Memory footprint (does it 'fit' or does it go into swap?).
Two more are: subdivs and PTex.
I.e. can the renderer do true subdivision surfaces (vs just subdividing the geometry n-times leading to silhouette artifacts under certain viewing conditions)?
And: can the renderer ingest PTex textures or do you have to create UVs somehow and convert all the textures into UV-based ones before you can use the original Disney dataset?
> It is not meant to be used inside a DCC app, as-is.
> If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong.
are you sure you're not just saying this because years of software being slow made people cargo-cult workarounds around the issue which are now assumed to be "the way" in the field ?
> are you sure you're not just saying this because years of software being slow [...]
Yes. And I don't know where to even start explaining this. Don't get me wrong.
"Simply put": there is no use case for this. Moana e.g. has gazillions of vegetation geometry instances. Why would you have those inside a DCC?
Pebbles on the shore, etc. etc.
VFX pipelines are specialized. An animator does not care if there are 10 million hairs on their furry character while they do the animation. A lighter only cares to see the fur when it's rendering, not while operating the light rig in the DCC.
Modern pipelines have instant feedback/viewport rendering where proxy geometry is swapped out for "the real thing" by the renderer (or a custom plugin inside the renderer) on the fly.
The result, inside the renderer, is something like the Moana dataset. Hence its publication. It's a stress test for renderers, not DCCs.
The software is fast where it's needed - it makes the right trade-offs. There's little point in building IDEs capable of handling terabyte-sized source files, because just having a terabyte source file is ridiculous.
Scenes like Moana are not intended to be handled as a whole, they are assembled from little pieces - semantically independent and assignable to different people with different skill sets. It's not the matter of software performance, it's the matter of process organization.
What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC?
Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck.
I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile.
Just to be clear again: awesome they fixed the bugs that lead to crashes with this data.
Not sure if that large a dataset was required at all to do that though.
But the speed improvements? Do they matter for your average OBJ?
And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost.
There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
I disagree, the point was to show how object importing scales and he used an extreme example to make the point. He also gives an example of duplicating 10k cubes in Blender (something not out of the ordinary) also sees a massive speed up.
This kind of thing can help understand other bottlenecks in software like Blender as its users and developers get more ambitious as the software grows in capability.
Idealistically wouldn’t one like film makers to be able to work ‘in world’ decoupled from technical limitations?
I've seen much the same problems when trying to programatically construct a large number of objects using the (low-level) Python API. If this kind of optimization improves throughput of those APIs that would be a major enabler for work that I want to do.
Otherwise I'm side-tracked looking for relatively awkward SIMD-style programming (e.g. Geometry Nodes) as a workaround for "CRUD" operations being too slow. Can't be just me...
It doesn't miss the point at all. It's a large data set and they used it to improve a function of Blender. There's nothing that says it only has to be used to test renderers.
Readit News
This situation reminds me a little bit of trying to work with very large images in ImageMagick, like 60k x 60k resolution. A simple resize was taking 6 hours on my mac 10 years ago, due to IM trying to allocate the whole image at once and then swapping non-stop. And then I discovered that the Graphics Magick fork did streamed resizing, and did the task in a couple of minutes. It’s a small and relatively easy change, but someone had to prioritize handling large inputs first. This is one reason software is hard for me; every time I need to process large inputs I wish I had written a streaming mechanism, but every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Maybe it’s the right thing to do to avoid over-engineering, but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
Absolutely reasonable decision making. Do what you need at the time you write the code, or at best, be slightly ahead of the curve. Don't forecast features you do not need. Humans are unreasonable about success of their own projects.
>Maybe it’s the right thing to do to avoid over-engineering
It is the only thing to do. You do not know the future use cases.
> [...] but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
If your software is subpar with regards to your needs, congrats, it is successful software!
Building the perfect software from the beginning is unreasonable because you might just never release or even use it.
Build imperfect things, and be reasonably frustrated BH them so that it gives you the energy to improve them!
I built a software library for an external device from specification. We eventually got the device to test and it worked when plugged in, but not everything was right.. I was early in my career and disappointed. My boss said "If it works perfectly the first time, you spent too much time on it.".
But then we found libvips (that's not only a library but a cli tool too) and it was like night and day in big images. Like 10x faster and 1/10 memory use. I really loved it!
https://www.libvips.org/
I remember when I was first getting into shader programming 10+ years ago, I asked a question on the Unity forums and Aras was one of the first to respond. Even then he was a titanic figure behind Unity’s graphics stack. I never forgot how kind he was to help a kid like me out, I’m sure he was quite busy!
This is why it is very difficult to justify optimization work to management. If there are 20 things to optimize that take 10 seconds each, the change isn't really noticeable until you're getting past half-way. And once your processing already takes a few minutes, what's the harm in adding another 10 seconds?
If I had done the easy change to get it to 5 minute before deploying, it would likely still be that way today annoying my users, but not enough to justify changing.
Cache invalidations and memory swapping as you approach the limits are other examples.
When you are creating VFX scenes for a still image you don't really care a lot about the render times as long as it is done the next day or so. If you are doing it for animation, you care a lot about render times, because anything times a thousand will be quite a duration.
The work of a programmer is always a factor in a multiplication. If the small change you make is used by thousand people a day, using it 100 times a day each second you shave off will save a collective time of ~28 hours. And this is only time. You could also think about electricity, about frustration, etc.
A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it.
Some workflows that were considered “too long to be worth it” suddenly become easy and routine. It’s kind of like “disrupting the market” in a sense.
And more times than I can count small changes in performance that were considered irrelevant led to very big cultural changes in a company.
For example we had some e2e tests running in our CI/CD pipeline that were taking ~ 15 minutes, people were stressed (or not very productive) as the usual excuse was “I’m waiting for the tests” with not enough time to do something productive but too long to wait patiently. I spend like a day optimizing it and brought them down to 4 mins, suddenly people began to accomplish more, they started writing a lot more tests.
So a day of investment led to happier devs (less turnover), more tests (stability improvement) and faster feature turnaround. And I had to fight tooth and nail to spare the time to actually do this.
Unfortunately this same fact also means that varrious kinds of important software are hard to get developed.
When someone writes code to blink a mildly annoying advert on youtube it effects a billion people and can make millions or hundreds of millions in revenue. But code needed to make a local car wash' robots more efficient? -- unlikely to get written unless someone thinks they can sell it to a chain: a large number of programmers get snatched up by places like google that deploy to billions of people. Even though in the past when there were far fewer qualified programmers it might have been easier to get the car wash software developed because there simply was nowhere that could deploy software to a billion people instead (much less highly profitably).
The enormous leverage of software is an undeniable force for good. But it also changes the incentive structures of the world in ways that have negative effects too. :(
This isn't limited to software either, improvements to mass production has made mass produced goods extremely inexpensive-- but by that same token custom work has become much more expensive. And the world around us has become much more homogenized and cookie cutter as a result. But the leverage that software potentially has is vastly greater than other things because of its zero marginal cost of production.
I usually put it this way to my colleagues: on a long enough timescale, every function will be called from a `for` loop.
It is not meant to be used inside a DCC app, as-is.
If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong.
That said – I think it's very cool if you can import such heavy geometry and your DCC app doesn't crash. But in 15 years of working in VFX I never dealt with heavy data inside a DCC directly. Indirectly yes, through proxies.
The three most interesting metrics for this scene are, if you're a renderer author (sorted by importance during lookdev/lighting):
1. Time to first pixel (is it seconds or hours?).
2. Time to completion (does it take days/hours/minutes?).
3. Memory footprint (does it 'fit' or does it go into swap?).
Two more are: subdivs and PTex.
I.e. can the renderer do true subdivision surfaces (vs just subdividing the geometry n-times leading to silhouette artifacts under certain viewing conditions)?
And: can the renderer ingest PTex textures or do you have to create UVs somehow and convert all the textures into UV-based ones before you can use the original Disney dataset?
> If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong.
are you sure you're not just saying this because years of software being slow made people cargo-cult workarounds around the issue which are now assumed to be "the way" in the field ?
Yes. And I don't know where to even start explaining this. Don't get me wrong.
"Simply put": there is no use case for this. Moana e.g. has gazillions of vegetation geometry instances. Why would you have those inside a DCC? Pebbles on the shore, etc. etc.
VFX pipelines are specialized. An animator does not care if there are 10 million hairs on their furry character while they do the animation. A lighter only cares to see the fur when it's rendering, not while operating the light rig in the DCC. Modern pipelines have instant feedback/viewport rendering where proxy geometry is swapped out for "the real thing" by the renderer (or a custom plugin inside the renderer) on the fly.
The result, inside the renderer, is something like the Moana dataset. Hence its publication. It's a stress test for renderers, not DCCs.
Scenes like Moana are not intended to be handled as a whole, they are assembled from little pieces - semantically independent and assignable to different people with different skill sets. It's not the matter of software performance, it's the matter of process organization.
Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck.
I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile.
Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though.
But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost.
There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
This kind of thing can help understand other bottlenecks in software like Blender as its users and developers get more ambitious as the software grows in capability.
Idealistically wouldn’t one like film makers to be able to work ‘in world’ decoupled from technical limitations?
Otherwise I'm side-tracked looking for relatively awkward SIMD-style programming (e.g. Geometry Nodes) as a workaround for "CRUD" operations being too slow. Can't be just me...