A bit of a let down that the video demoing SDR->HDR conversion is itself only published in SDR. Makes as much sense as demoing a colorization tool in a grayscale video!
At this point, with any new model I think it makes sense to wait until you can run the model on your own input before making any assumptions based on cherry picked examples.
If they were serious about showing this tech off they should've provided a video file download. Also indicate that it's a HDR file and should only be viewed on a HDR display. Youtube is just making this look bad as people won't see a difference.
YouTube tends to post a downscaled SD version first, then they encode and post the higher-res versions when they get around to it. This can take days in some cases. Meanwhile the creator catches the flak...
You don't need high res for HDR on YouTube (144p HDR is a thing there oddly enough) and the 4k version had already processed when I posted that comment (with no change since in HDR availability). Usually media announcements/large channels pre-upload the video so it's ready when they want it to actually publish to avoid that kind of issue though.
4K processing takes just minutes, but HDR processing can take over a month to… never. There is no indication of this at all, no progress bar or eta. Just check manually every few days!
This is why everyone is giving up on HDR, it’s just too painful because the content distributors are all so bad at it, with Netflix being the sole exception.
It's more reliable then on linux though, and windows has been doing "auto HDR" for videos for years, so kinda hard to tell when something is HDR or not there.
At least as of a couple of years ago, HDR support on YouTube has been pretty bad[1]. I know they've been working to improve things since, but I kind of don't blame people for walking away from that mess.
I guess. There's a lot of details we don't know that would change the calculus on this.
To use a analogous workflow, it could be like saying, "It's pointless to shoot video in 10-bit log if it's going to be displayed on Rec.709 at 8-bits." It completely leaves out available transforms and manipulations in HDR that do have a noticeable impact even when SDR is the target.
Again, we can't know if it's important given the information that's available, but we can't know if it's pointless either.
I could see a future where this works really well. It doesn't seem to be the case right now though.
The "super resolution" showcased in the video seemed almost identical to adjusting the "sharpness" in any basic photo editing software. That is to say, perceived sharpness goes up, but actual conveyed details stays identical.
Allegedly the new one plus phone does this trick in real time as well as up sampling and interframe motion interpretation. Mrwhostheboss seems impressed, but I don't really trust his yet judgment on these things.
Whatever special sauce the Nvidia shield uses is honestly incredible. Real time upscaling of any stream, and not just optimized for low res source, its like a force multiplier on content that is already HD. Supposedly the windows drivers do it as well but the effect seems less noticeable to me in my tests
An HN search of ''Deep Space Nine'' and ''Topaz'' will show some great discussions here covering the dearth of such upscaling solutions, as well as some huge efforts before commonplace AI.
It's not exactly what you're after, as it's anime specific and you need to process the video yourself (eg disassemble to frames, run the upscaler, then assemble back to a movie file), but Real-ESRGAN is very good for cleaning up old, low resolution anime:
If you want to avoid manual processing, Anime4K runs in real time as a GLSL shader you can load into MPV (or Plex or something called IINA according to the readme) and still gives great results.
It depends on what do you mean by 'open-source', along with training materials and full setup? That will be hard to find. Upscaling was popular like 10 years back. That's why there is no much interest today. Training in old style isn't that hard. But artifacts are popping up in all videos I've seen.
The RTX video upscaling feature works really well, there's a bug in the Firefox implementation that allows you to switch between native and upscaled side by side and the difference is striking. I don't have an HDR monitor so I can't tell you how well this new HDR feature works.
I don't think making things up is the problem, it's if it's believable. If it's indistinguishable to a viewer, then who cares. I never would have thought the HDR of the clouds was "made up".
I recently had some old super8 films shot by my parents scanned into 1080p resolution in ProresHQ. Because of the poor optics of the original camera, imperfect focus when shooting, poor lightning conditions, and general deterioration of the film stock, most of the footage won't get anywhere near what 1080p could deliver.
What I'd like to try at some point is to let some AI/ML model process the frames, and instead of necessarily scaling it up to 4k etc., 'just' add (aka magic) missing detail into 1080p version and generally unblur it.
Is there anything out there, even in research phase that can take existing video stock, and then hallucinate into it detail that never was there to begin with? What NVidia is demoing here seem like steps to that direction...
I did test out Topaz Video and DaVinci's built-in super resolution feature, both of which gave me a 4k video with some changes to the original. But not the magic I am after.
I also restored some Super 8 footage recently and had great success. The biggest win I had wasn't resolution, but slowing down the speed to be correct in DaVinci, and interpolating frames to make it 60fps using the RIFE algorithm in FlowFrames. I then used Film9 to remove shake, colour-correct, sharpen and so on.
Correcting the speed and interpolating frames added an amazing amount of detail that wasn't perceptible to me in the originals (albeit it was there).
All of this processing does remove some of the charm of the medium, so I'll be keeping the original scans in any case.
An interesting thing about Super8: the resolution is generally very poor, but it can have quite the dynamic range. Also, with film in general (and video, but it's easier with film because you have global shutter) you can compensate motion blur and get more detail out which isn't visible when you look at the film frame by frame. And none of this needs AI.
Regarding hallucination, I agree with the sibling comment, the problem is that faces change. And with video, I'm not even sure the same person would have the same face in various parts of the video...
there is AI tech to do this already. it has a slight problem, though: it adds detail to faces (this is marketing speak for completely changes how people look).
The Shield is kind of an extreme outlier in today's environment. A device from 2015 that 9 years later is still one of the top tier choices in its (consumer) market is almost unheard of.
In fact it's reportedly the currently supported Android device out there with the longest support[0], it's crazy that mine still gets updates.
It really is awesome. I also enjoy the UI that allows you to side by side compare a stream and the difference is insane.
I have been meaning to see how well it handles streaming a desktop via moonlight to the shield to real time upscale a second monitor's content. I assume it's trained for video footage and not static UI components. The RTX windows drivers don't seem to upscale as well as the shield.
Making up information? The same can be said for most commonly used modern compressed video formats. Just low bitrate streams of data that gets interpolated and predicted into producing what looks like high resolution video. AV1 even has entire systems for synthesizing film grain.
The way i see it, if the ai generated HDR looks good, why not? It wouldn't be more fake or made up than the rest of the video.
Readit News
This is why everyone is giving up on HDR, it’s just too painful because the content distributors are all so bad at it, with Netflix being the sole exception.
It's more reliable then on linux though, and windows has been doing "auto HDR" for videos for years, so kinda hard to tell when something is HDR or not there.
Deleted Comment
1. https://www.youtube.com/watch?v=DwDWQyBF9II
To use a analogous workflow, it could be like saying, "It's pointless to shoot video in 10-bit log if it's going to be displayed on Rec.709 at 8-bits." It completely leaves out available transforms and manipulations in HDR that do have a noticeable impact even when SDR is the target.
Again, we can't know if it's important given the information that's available, but we can't know if it's pointless either.
Deleted Comment
The "super resolution" showcased in the video seemed almost identical to adjusting the "sharpness" in any basic photo editing software. That is to say, perceived sharpness goes up, but actual conveyed details stays identical.
https://www.youtube.com/watch?v=J9-9fP_pcEc&t=1107s
I looked back about a year ago, and it didn't seem like there were any good open-source solutions.
And that's by avoiding the word "topaz", where I see no story results with discussions and not much of comment results.
https://github.com/xinntao/Real-ESRGAN/
https://github.com/bloc97/Anime4K
What I'd like to try at some point is to let some AI/ML model process the frames, and instead of necessarily scaling it up to 4k etc., 'just' add (aka magic) missing detail into 1080p version and generally unblur it.
Is there anything out there, even in research phase that can take existing video stock, and then hallucinate into it detail that never was there to begin with? What NVidia is demoing here seem like steps to that direction...
I did test out Topaz Video and DaVinci's built-in super resolution feature, both of which gave me a 4k video with some changes to the original. But not the magic I am after.
Correcting the speed and interpolating frames added an amazing amount of detail that wasn't perceptible to me in the originals (albeit it was there).
All of this processing does remove some of the charm of the medium, so I'll be keeping the original scans in any case.
Regarding hallucination, I agree with the sibling comment, the problem is that faces change. And with video, I'm not even sure the same person would have the same face in various parts of the video...
- Is there any stand-alone live AI upscaling / 'enhance' alternative for android or any other platform?
In fact it's reportedly the currently supported Android device out there with the longest support[0], it's crazy that mine still gets updates.
[0]https://www.androidcentral.com/android-longest-support-life-...
I have been meaning to see how well it handles streaming a desktop via moonlight to the shield to real time upscale a second monitor's content. I assume it's trained for video footage and not static UI components. The RTX windows drivers don't seem to upscale as well as the shield.
It’s pricey, and being so old, I fear it will soon be obsoleted…
The way i see it, if the ai generated HDR looks good, why not? It wouldn't be more fake or made up than the rest of the video.