It's interesting that they are able to continue improving video compression. You'd think that it would have all been figured out by now.
Is this continued improvement related to the improvement of technology? Or just coincidental?
Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it?
Do we have algorithms today that can compress way better but would be too slow to encode/decode?
Video compression is a calculus of IO capacity, memory, and algorithmic complexity. Take the MPEG-1 codec for instance, it was new about 30 years ago. While today most people think of MPEG-1 videos as low quality the spec provides the ability to handle bit rates up to 100Mb/s and resolutions up to 4095x4095. That was way higher than the hardware of the time supported.
One of MPEG-1's design goals was to get VHS-quality video at a bitrate that could stream over T1/E1 lines or 1x CD-ROMs. The limit on bitrate led to increased algorithmic complexity. It was well into the Pentium/PowerPC era until desktop systems could play back VCD quality MPEG-1 video in software.
Later MPEG codecs increased their algorithmic complexity to squeeze better quality video into low bit rates. A lot of those features existed on paper 20-30 years ago but weren't practical on hardware of the time, even custom ASICs. Even within a spec features are bound to profiles so a file/stream can be handled by less capable decoders/hardware.
There's plenty of video codecs or settings for them that can choke modern hardware. It also depends on what you mean by "modern hardware". There's codecs/configurations a Threadripper with 64GB of RAM in a mains powered jet engine sounding desktop could handle in software that would kill a Snapdragon with 6GB of RAM in a phone. There's also codecs/configurations the Snapdragon in the phone could play using hardware acceleration that would choke a low powered Celeron or Atom decoding in software.
> Because H.266/VVC was developed with ultra-high-resolution video content in mind, the new standard is particularly beneficial when streaming 4K or 8K videos on a flat screen TV.
Compressing video is very different from gzipping a file. It's more about human perception than algorithms, really. The question is "what data can we delete without people noticing?", and it makes sense that answer is different for an 8k video than a 480p video.
So compressing a 1080p video with H266 will not result in similar file size/quality improvements as a 4k video? How much are we looking at for 1080p, 10%?
>Like, why couldn't have H.266 been invented 30 years ago?
It is all a matter of trade offs and engineering.
For MPEG / H.26x codec, the committees start the project by asking or defining the encoding and decoding complexities. And if you only read Reddit or HN, most of the comment's world view would be Video codec are only for Internet Video and completely disregard other video delivering platform. Which all have their own trade off and limitations. There is also cost in decoding silicon die size and power usage. If more video are being consumed on Mobile and battery is a limitation, can you expect hardware decoding energy usage to be within the previous codec? Does it scale with adding more transistors, are there Amdahl's law somewhere. etc It is easy to just say adding more transistor, but ultimately there is a cost to hardware vendors.
Vast majority of the Internet seems to think most people working on MPEG Video Codec are patents trolls and idiots and paid little to no respect to its engineering. When as a matter of fact Video Codec are thousands of small tools within the spec, and pretty much insane amount of trial and error. It may not be as complicated as 3GPP / 5G level of complexity, but it is still lot of work. Getting something to compress better while doing it efficiently is hard. And as Moore's Law is slowing down. No one can continue to just throwing transistors at the problem.
I don't know much about H.266, but some of the advances in H.265 depended on players having enough RAM to hold a bunch of previous decoded frames, so they could be referred to by later compressed data. Newer codecs tend to have a lot more options for the encoder to tune, so they need a combination of faster CPUs and smarter heuristics to explore the space of possible encodings quickly.
I think the number of previous frames for typical settings went from about 4 to about 6 as we went from H.264 to H.265. And the actual max in H.264 was 16. So that doesn't seem like a huge factor.
Computers wouldn't have been fast enough. Moore's law is a hell of a drug.
In the mid '90s, PCs often weren't fast enough to decode DVDs, which were typically 720x480 24FPS MPEG2. DVD drives were often shipped with accelerator cards that decoded MPEG2 in hardware. I had one. My netbook is many orders of magnitude faster than my old Pentium Pro. But it's not fast enough to decode 1080p 30fps H.265 or VP9 in software. It must decode VP9/H.265 on the GPU or not at all. MPEG2 is trivial to decode by comparison. I would expect a typical desktop PC of the mid '90s to take seconds to decode a frame of H.265, if it even had enough RAM to be able to do it at all.
It's an engineering tradeoff between compression efficiency of the codec and the price of the hardware which is required to execute it. If a chip which is capable of decoding the old standard costs $8, and a chip which is capable of decoding the new standard costs $9, sure, the new standard will get lots of adoption. But if a chip which is capable of decoding the new standard costs $90, lots of vendors will balk.
Indeed. The brand-new fancy Blue & White Power Mac G3's from early 1999 were the first Macs that shipped with a DVD drive, and they could play video DVD's but they had an obvious (and strange) additional decoder mezzanine card on the already unusual Rage128 PCI video card.
By the end of that year the G4 Power Macs were just barely fast enough to play DVD's with software decoding and assistance from the PCI or later AGP video card. And after a while (perhaps ~ 2002?), even the Blue G3's could do it in software even if you got a different video card, as long as you also upgraded to a G4 CPU (they were all in ZIF sockets).
It was very taxing on computers at y2k!
Later autumn 2000 G3 iMacs could also play DVD's but I think they needed special help from a video co-processor.
From what I've heard (would love to hear more expertise on this), it's incredibly hard to invent a new video compression algorithm without breaking an existing set of patents, and there's also no easy way to even know whether you're breaking anything as you develop the algo. Thus the situation we're in is not that it's too hard to develop better codecs, but that you've very disincentivized to do so.
Which then begs the question - why are video compression standards developed in the US at all? MPEG is obviously US based but Xiph is also a US nonprofit. The software patents should be hugely crippling the ability for Americans to develop competitive video codecs when every other nation doesn't have such nonsense. Why hasn't Europe invested in and developed better codecs that combine the techniques impossible to mix in the states?
Is it just basically the same mechanism that leads to so much drug development happening in the US despite how backwards its medical system is, because those regressive institutions create profit incentives not available elsewhere (to develop drugs or video codecs for profit) and thus the US already has capitalists throwing money at what could be profitable whereas everyone else would look at it as an investment cost for basically research infrastructure.
It's about the assumptions made during the standardization.
Compared to 30 years ago, we now have better knowledge and statistics about what low level primitives are useful in a codec.
E.g. jpeg operates on fixed 8×8 blocks independently, which makes it less efficient for very large images than a codec with variable block size. But variable block size adds overhead for very small images.
An other reason can be common hardware. As hardware evolves, different hardware accelerated encoding/decoding techniques become feasible that gets folded into the new standards.
Something that I learned about 10 years ago when bandwidth was still expensive is that you can make a very large version of an image, set the jpeg compression to a ridiculously high value, then scale the image down in the browser. The artifacts aren't as noticeable when the image is scaled down, and the file size is actually smaller than what it would be if you encoded a smaller image with less compression.
I still remember when my pc would take 24 hours to compress a dvd to a high quality h264 mkv, sure you could squeeze it down with fast presets in handbrake but the point was transparency. Now I'm sure for most normal pc's the time to compress at the same quality with h.265 is the same 24 hours, in 4k, even longer, I'm sure h266 would take more than twice as long easily.
Early pc's had separate and very expensive mpeg decode boards just to decode dvd, creative sold a set, the cpu simply couldn't even handle mpeg 2. I know its hard to believe but there was a time when playing back an mp3 was a big ask, all these algorithms could be made long ago, but they would have been impractical fantasy. Only now are we seeing real partial cheated resolution ray tracing in modern high end gaming hardware now which is a good comparison, ray tracing has been with us for a long time, only hardware advancement over decades has made it viable.
It amused me that they claimed 4k uhd h265 is now 10GB for a movie, that's garbage bitrate, they always ask too much of these codecs.
Good compression is quite complex and can go wrong in an unimaginable variety of ways. Remember when Xerox copiers would copy numbers incorrectly due to compression? The numbers would look clear, they just wouldn't always be the same numbers that you started with.
Xerox problem stemmed from simple replacement of "recognized" numbers with learned dictionary. Good implementation would use learned alphabet atlas as a supplement, encoding difference between guess and source image. That way even predicted 0 instead of 8 wouldnt be catastrophic with encoder filling in missing detail.
> Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it?
Here's something to consider:
In 1995, a typical video stream was 640 x 480 x 24fps. That's 7,372,800 pixels per second.
In 2020, we have some content that's 7680 x 4320 x 120fps. That's 3,981,312,000 pixels per second, or a 540 fold increase in 25 years.
The massive increase in image size actually makes it easier to use high compression ratios. I found this out the hard way recently, when I was trying to compress and email a powerpoint presentation that a coworker had presented on video. In a nutshell, the powerpoint doc with it's sharp edges and it's low resolution made it difficult to compress.
Increase framerate plays a factor too; due to decades of research on motion interpolation, algorithms have become quite good at guessing what content can be eliminated from a moving stream.
The way I understand it is that one way to compress a document would be to store a computer program and, at the decompression stage, interpret the program so that running it outputs the original data.
So suppose you have a program of some size that produces the correct output, and you want to know if a smaller-sized program can also. You examine one of the possible smaller-sized programs, and you observe that it is running a long time. Is it going to halt, or is it going to produce the desired output? To answer that (generally), you have to solve the halting problem.
(This applies to lossless compression, but maybe the idea could be extended to lossy as well.)
Ten-year software video compression engineer here:
TL;DR: it's partly because we're using higher video resolutions. A non-negligible part of the improvement stems from adapting existing algorithms to the now-doubled-resolution.
Almost all video compression standards split the input frame into fixed-size square blocks, aka "macroblocks". To put it simply, the macroblock is the coarsest granularity level at which compression happens.
- H.264 and MPEG-2 Video use 16x16 macroblocks (ignoring MBAFF).
- H.265 use configurable quad-tree-like macroblocks, with a frame-level configurable size up to 64x64.
- AV1 makes this block-size configurable up to 128x128.
Which means:
Compression to H.264 a SD video (720x576, used by DVDs) results in 1620 macroblocks/frame.
Compressing to H.265 a HD video (1920x1080) results in at least 506 macroblocks/frame.
Compressing to AV1 a 4K video (3840x2160) results in at least 506 macroblocks/frame.
But compressing to H.264 a 4K video (3840x2160) will result in 32400 macroblocks/frame.
The problem is, there are constant bitcosts per-macroblock ((mostly) regardless of the input picture). So using H.264 to compress 4K video will be inefficient.
When you take an old compression standard to encode recent-resolution content, you're using the compression standard outside of the resolution domain for which it was optimized.
> Is this continued improvement related to the improvement of technology? Or just coincidental?
Of course, there also "real" improvements (in the sense "qualitative improvements that would have benefited to compression old video resolutions, if only we had invented them sooner").
For example:
- the context-adaptive arithmetic coding from H.264, which is a net improvement over classic variable-length huffman coding used by MPEG-2 (and H.264 baseline profile).
- the entropy coding used by AV1, which is a net improvement over H.264's CABAC.
- integer DCT (introduced by H.264), which allow bit-accuracy checking and way lot easier and smaller hardware implementations (compared to floating point DCT that is used by MPEG2).
- loop filters: H.264 pioneered the idea of a normative post-processing step, whose output could be used to predict next frames. H.264 had 1 loop filter ("deblocking"). HEVC had 2 loop filters: "deblocking" and "SAO". AV1 has 4 loop filters.
All of these a real improvements, brought to us by time, and extremely clever and dedicated people. However, the compression gains of these improvements are nowhere near the "50% less bitrate" that is used to sell each new advanced-high-efficiency-versatile-nextgen video codec. Without increasing - a lot - the frame resolution, selling a new video compression standard will be a lot harder.
Besides, now that the resolutions seems to have settled up around 4K/8K (and that "high definition" has become the lowest resolution we might have to deal with :D), things are going to get interesting ... provided that we don't start playing the same game with framerates!
"You'd think that it would have all been figured out by now."
Would you? Video compression is one of the few things that we will work on for the next 1000 years and still be nowhere near finished. The best video compression would be to know the state of the universe at the big bang, have a timestamp of the beginning and end of your clip and spatial coordinates defining your viewport. Then some futuristic quantum computer would just simulate the content of your clip...
So yeah, sure we are done with video compression :). This is of course an extreme example of constant time compression that may or may not be ever feasible (if we live in a computer simulation of an alien race, then it is already happening).
But the gist is the same. Video compression is mostly about inferring the world and computing movement not by storing the content of the image.
For instance by taking a snapshot of the world, decomposing it into geometric shapes (pretty much the opposite of 3D rendering) and then computing the next frames by morphing these shapes + some diff data that snaps these approximations back in line with the actual data.
We are all but in the very infancy of video compression. What should surprise you is why it takes us so long to get anywhere.
The way I read the release was that it's not a lossless compression, it reads like it's downscaling 4k+ video to a lower format with 'no perceptible loss of quality.' Since this is also seemingly targeted at mobile, I'm guessing the lack of perceptible loss of quality is a direct function of screen size and pixel density on a smaller mobile devices.
For me, this is another pointless advance in video technology. 720p or 1080p is fantastic video resolution, especially on a mobile phone. Less than 1% of the population cares or wants higher resolution.
What new technologies are doing now is re-setting the patent expiration clock. As long as new video tech comes out every 5-10 years, HW manufacturers get to sell new chips, phone manufacturers get to sell new phones, TV manufacturers get to sell new TVs, rinse, repeat.
No, because the market is more than happy to pay a few cents or dollars per device to get better compression and lower transmission bandwidth. This observation has held true consistently in the 3 decades since compressed digital media was invented.
> No, because the market is more than happy to pay [...]
Is it? Because Google/YouTube, Amazon/Twitch, Netflix, Microsoft, Apple, Samsung, Facebook, Intel, AMD, ARM, Nvidia, Cisco, etc, are all part of AO Media:
That's the direction everyone is going, so I think h.266 is an attempt to give vendors pause before moving to av1.
AV1 will probably win in most circumstances (big tech) but is unlikely to win where there are big gains to be had by reducing file size (broadcasters with gigantic libraries).
Broadcasters are also used to paying a lot and not getting much.
Honestly, the first step in this is getting the ffmpeg av1 library to a good usable place. It's currently so slow as to be near unviable. I'd happily switch when it becomes a usable option.
Is there any indication that rav1e is so inefficient as to have substantive double digit percentage speedup left to be realized? Because even if you cut encode times in half, they already take hours per minute of video on a quad core.
AV1 is unlikely to ever be practical for "muggle" encode use, at least in this decade. It will only be worth committing that much compute workload to making a smaller file if the recipients will number in at least what, millions?
I'd be really curious what a hardware realtime AV1 encoder would even look like. How much silicon would that take? That kind of chip would have be be colossal even if it sacrifices huge amounts of efficiency to spit out frames at reasonable time (in the same way hardware hevc and vp9 encoders kind of suck).
There is no ffmpeg av1 library i.e. no native decoders or encoders. ffmpeg has wrappers for libaom and dav1d/rav1e. 3rd party scripts also add wrappers for svt-av1.
Just found av1 is about 20 to 30% more efficient than h265. I guess there was no reason to use patented algs, but h265 is now significantly more efficient than av1.
I wonder what would happen if ffmpeg made the choice to not implement a decoder for it. Or maybe just not a encoder?
Can we agree not to work on such projects? I feel that the lack of a good open-source encoder/decoder would spell the death of most codecs nowadays. That would also teach Fraunhoffer about it.
Of course, everyone is free to scratch their itch. And the bigger the void, the more itchy it gets. Luckily, we still have AV1.
They are compliant with MPEG patent licenses - they distribute source code you have to compile yourself and for non-commercial use.
If you build x264 or other open source implementation of h.264/h.265, and embed it for example in commercial video conferencing software/appliance, you have to pay patent licensing fees for that product.
It's also why Firefox downloads a blob from Cisco to handle MPEG-4 video - Cisco covers the licensing for distribution et al.
They don't get away with it. The are two separate issues: the software copyright license and the H.264 patent license. x264 itself is licensed under the GPL:
But if you use an H.264 encoder or decoder in a country that recognizes software patents then you need to buy a patent license if your usage comes under the terms of the license:
Naively hoped I'd read 'this will be released to the community under a GPL license' or similar. Instead found the words 'patent' and 'transparent licensing model'.
I appreciate that it costs money and time to develop these algorithms, but when you're backed by multi-billion dollar "partners from industry including Apple, Ericsson, Intel, Huawei, Microsoft, Qualcomm, and Sony" perhaps they could swallow the costs? It is 2020 after all.
> In my humble opinion, that license mess set back innovation in the portable audio space by a good 5 years.
Seeing all this, I'm convinced that copyright in general and patent system in particular does more harm than good by slowing down the technical progress of the humanity as a whole for the sake of some already rich people becoming a bit richer.
The initial idea behind patent system was sensible, but the way it's abused now... I mean, it could work in today's world as intended if patents lasted a year or two, not what is effectively eternity.
> In my humble opinion, that license mess set back innovation in the portable audio space by a good 5 years.
Or did it push it forward? If not for the licensing at the time, would it have been developed? Would it be allowed to be used by anyone just paying for the technology?
It's even worse than that. The main driving force behind VVC is AFAIK the Heinrich Hertz Institute, which used to be independent and has now been sucked into the Fraunhofer mothership, which is one of the big research organisations in Germany.
Fraunhofer has a budget of over 2 billion Euros, and 30% of their money comes from public funding. They run over 70 institutes, so they do much more than this.
The root problem here is that nowadays public research is funded with industry money, which means there has to be a return on investment, hence the patents. In fact, this has metastasized into universities being graded by their patent portfolio volume. So I would expect there to be patents even if 100% of the funding came from the tax payer.
It would have been possible to do the whole process just with public money and zero patents. In fact, I would love it if some research team collated all the patent tax payments across the population of Germany and compare the bottom line cost for the country.
I wager it would have been cheaper without patents, too.
Disclaimer: I work at a Fraunhofer institute, though not Fraunhofer HHI, which developed this codec, and I have no intimate knowledge on the financing of that institute or this project in particular. But some basic principles apply to all institutes the same.
Fraunhofer gets roughly 30% of its funding from public sources, the remainder is raised on a per-project basis. It's a fair assumption that those industry partners provided some funds towards the development here. Maybe they even covered all the payroll costs for the involved scientists for the duration of the project.
And yet more income means more money for other research projects. Maybe ones that are not as commercially interesting, or for which a partner decides to terminate a contract rather unexpectedly. While I am also a fan of OSS and would love for work like this to either have no patents or a liberal patent grant, I can also appreciate the desire to fund your research institute.
The dirty secret of video codecs is that you can't make a modern video codec that isn't patent encumbered, which in turn makes it so that even if they wanted to be open, they go for defensive patents, which in turn perpetuate the situation.
At least the patent licenses usually used with MPEG mean that private use of open source implementations is free.
> The dirty secret of video codecs is that you can't make a modern video codec that isn't patent encumbered
The existence of Theora, VP8, VP9, and now AV1 seems to contradict that theory.
You could argue that they infringe on some unknown patents, but that is also arguably true of patent cabals like MPEG (you just hope that the cabal is big enough that there aren't any patentholders lurking outside). The only difference is that with a patent cabal you have the fun of having to obey the restrictions of everyone who showed up with a possibly-related-in-some-way patent and joined the cabal.
Not to mention that it isn't necessary for a patent pool to be a cabal. AOMedia has a similar structure to a patent cabal except it doesn't act like a cabal (its patent pool is royalty-free in the style of the W3C). So even if the argument is that a patent pool is a good idea (and video codecs cannot be developed without them), there isn't a justification behind turning the patent pool into a cabal.
> At least the patent licenses usually used with MPEG mean that private use of open source implementations is free.
You say that, but there's a reason why some distributions (openSUSE for one) still can't ship x264 (even though the code itself is free software). Not to mention the need for Cisco's OpenH264 in Firefox (which you cannot recompile or modify otherwise you lose the patent rights to use it). The existence of the MPEG patent cabal isn't a good thing, and any minor concessions you get from them do not justify their actions.
Software patents aren't a thing in ~~Europe~~ a few European countries. Sure it's difficult to ignore the American market for a company, but an independent developer could specify a state of the art video codec without thinking about patents.
Edited because I didn't know that some European countries accept software patents.
Copyright != patents. A copyright license for a codec implementation or a spec text doesn't grant you patents for the ideas it contains.
A GPL implementation doesn't guarantee a patent grant. Even if you wrote the code yourself, even just for your personal use, your own work could still be illegal to use due to lacking a patent license from the original patent holders.
Be careful about using implementations of H.26x codecs, because in countries that recognize software patents the code may be illegal to use, regardless whether you've got a license for the code or not. Even when a FLOSS license says something about patent grants, it's still meaningless if the code author didn't own the patents to grant.
Of course, they can't do that, because the source technologies they've put together are themselves patent-encumbered. An AV-codec is a lot like a modern pop song: a piece of IP entirely made up of licensed samples of other people's IP.
I think a more subtle "open-sourcing" of this IP could still be possible, though. Maybe one that still requires that large corporate players that are going to sell their derivative products, acquire a license the traditional way (this is, after all, what the contributors to the codec's patent-pool and R&D efforts based their relative-R&D-labor-contribution negotiations around: that each contributor would end up paying for the devices of theirs that run the codec.)
Maybe there could be a foundation created under the stewardship of the patent-pool itself, which nominally pays the same per-seat/per-device licensing costs as every other member, but where this money doesn't come from revenue but rather is donated by those other members; and where this foundation then grants open-source projects an automatic but non-transferrable license to use the technology.
So, for example, a directly open-source project (e.g. ffmpeg) would be granted an automatic license (for its direct individual users); but that license wouldn't transfer to software that embeds it. Instead, other open-source software (e.g. Handbrake, youtube-dl, etc.) that embeds ffmpeg would acquire its own automatic license (and thus be its own line-item under the foundation); while closed-source software that embeds ffmpeg would be stuck needing a commercial license.
"or similar" - I mean an open-source free license to use, that is not encumbered by patents. GPL was the first license name to pop into my old-man brain.
Notice that the missing party here was 'Google'. These are the folks who are really competing with VP9, the royalty free codec that limited the uptake of H.265.
Does it mean money, or not? Because if not, then Fraunhofer does not exist.
But there is absolutely a problem here, because said 'mega businesses' actually should have a strategic imperative to want to make internet technologies more widespread.
Why on earth would MS want to limit their main line of business for a tiny big of IP related revenue?
It would seem to me, that G, MS, Huawei and all of the various patent holders should be trying their best to remove any and all barriers to adoption. There are enough bureaucratic hurdles in the way to worry about, let alone legal concerns.
Even if MS or whoever had to buy out some laggard IP owners who didn't want to play ball, it would probably still make sense for them.
Fraunhofer or anyone else are not in that situation, but the behemoths running vast surpluses are, it just seems shortsighted for them to hamstring any of this.
> will be released to the community under a GPL license' or similar
Both h264 and h265 have these implementations, I think FFMPEG library has both under the terms of GPLv2.
The decoders are almost completely useless. The video codec, at least the decoder, needs to be in the hardware, not in software.
Mobile devices just don’t have the resources to run the decoders on CPU. The code works on PC but consumes too much electricity and thermal budget. Even GPGPUs are not good enough for the job, couple generations ago AMD tried to use shader cores for video codecs, didn’t work well enough and they switched to dedicated silicon like the rest of them.
What you're looking for is the AV1 codec, which finalized a year and a half ago and will likely see much wider adoption, simply because none of the members want to pay royalties.
AV1 decoding has already been in Chrome and Firefox since at least a year ago. We're just waiting for hardware decoding and encoding support now, which should start appearing this year.
The next version of Chrome will also support the AV1-based AVIF image format this month:
Your comment was deaded. I vouched for it because I don't see anything that looks obviously inflammatory/incorrect, but I noticed that a large fraction of your comment history is also dead. You might want to look into that, since most of the dead comments also looked fine to me.
> perhaps they could swallow the costs? It is 2020 after all.
They should swallow so people again come back and say they are doing it to kill competition? Just received bill from doctor's visit for hundreds of dollar, I guess things are not gonna be free in 2020 after all.
Can anyone verify if this is a real number? It’s possible sometimes to make surprising claims (such as 50% lower size) by relying on unusual or unrealistic situations. I would rather if they use a standard set of test videos with different content and resolutions, and some objective measure of fidelity to the original, when quoting these percentages. But if the 50% number is real, then that is truly remarkable. I wonder how many more CPU instructions are required per second of decoded video compared to HEVC.
If one trusted numbers like this, and followed a chain of en vogue codecs back through history, you'd expect that a modern codec would produce files sizes like 3% of MPEG-2 on the same input data. It's all spin.
I'm sure it does better, but I'm equally sure it'll turn out to be an incremental benefit in practice.
> I wonder how many more CPU instructions are required per second of decoded video compared to HEVC.
CPU cycles are cheap. The real cost is the addition of yet another ?!@$!!#@ video codec block on every consumer SoC shipped over the coming decade.
Opinionated bile: video encoding is a Solved Problem in the modern world, no matter how much the experts want it to be exciting. The low hanging fruit has been picked, and we should just pick something and move on. JPEG-2000 and WebP failed too, but at least there it was only some extra forgotten software. Continuing to bang on the video problem is wasting an absolutely obscene amount of silicon.
MPEG-2 doesn't support video larger than 1920x1152, so it's hard to compare on 4k video, let alone 8k. But according to https://www.researchgate.net/publication/321412719_Subjectiv... H.265 can achieve similar visual quality with 10% of the bit rate of MPEG-2 even on SD video (832*480). [Edit: not 720p]
Video sizes are not shrinking. 8K is coming, with even bigger sizes on the way. That matters at scale.
As long as someone sees the benefit, people will keep pursuing it. Compression (as like all tech) is a moving target with the platforms regularly improving on many axes.
A measure that has consistently high agreement when evaluated by humans, repeatable across laboratories. Image quality tests can (and are) be conducted in a structured fashion.
Usually several test sets are created periodically to handle any type of situation in a variety of resolutions and depending on which part of the codec people are working on, this process is very complicated and take a lot of time to ensure that all the kind of content is representative. The metric is not optimal for video content, but gives an idea if they are going in the right direction. Ultimately, blind tests with people (experts and non experts) are done on the content to have subjective measures added to the objective ones
This has been going on since the birth of video codec. So for anyone who is new to this, the 50% are under the best case scenario. So they will compare it in 4K Res, where VVC has the highest advantage against HEVC, in measurement that fits those numbers. ( PSNR or SSIM ).
For lower res you will see lower percentage. You see the same claims from other codec as well.
PSNR is a terrible way to measure video quality. SSIM is better but not great. VMAF is good but it’s skewed towards the types of material netflix has.
Objective video quality is a tricky thing to do and you can easily come out with a codec that’s great at fine details in foreground people but terrible at high speed panning of water and trees. Depending on the material depends where you want the quality to go.
> A uniform and transparent licensing model based on the FRAND principle (i.e., fair, reasonable, and non-discriminatory) is planned to be established for the use of standard essential patents related to H.266/VVC.
Maybe. On the other hand, maybe not. Leoanardo Chiariglione, founder and chairman of MPEG, thinks MPEG has for all practical purposes ceased to be:
The disorganised and fractured licensing around HEVC contributed to that. And, so far, VVC's licensing looks like it's headed down the same path as HEVC.
Maybe AV1's simple, royalty-free licensing will motivate them to get their act together with VVC licensing.
Instead of a video file or stream, that would be more like shipping a program that recreates the video. It might be cool, but it's not really feasible to play back that kind of thing on normal TV hardware.
I'm not sure what you mean. There are already multiple research articles that show that deep neural network based video compression can be competitive, here's an example:
It's surprisingly difficult to guarantee a high quality on every kind of videos and formats using a neural network. Furthermore, the network should be able to handle all the corner cases (think about color profiles alone..)
The fact that the underground scene is still pumping 264 instead of 265 (I'd estimate 90/10 split optimistically) tells me the real world is not quite ready for 266.
So I guess it comes down to 266 hw support. Or powerful CPUs that can push sw decoding?
Readit News
Is this continued improvement related to the improvement of technology? Or just coincidental?
Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it?
Do we have algorithms today that can compress way better but would be too slow to encode/decode?
One of MPEG-1's design goals was to get VHS-quality video at a bitrate that could stream over T1/E1 lines or 1x CD-ROMs. The limit on bitrate led to increased algorithmic complexity. It was well into the Pentium/PowerPC era until desktop systems could play back VCD quality MPEG-1 video in software.
Later MPEG codecs increased their algorithmic complexity to squeeze better quality video into low bit rates. A lot of those features existed on paper 20-30 years ago but weren't practical on hardware of the time, even custom ASICs. Even within a spec features are bound to profiles so a file/stream can be handled by less capable decoders/hardware.
There's plenty of video codecs or settings for them that can choke modern hardware. It also depends on what you mean by "modern hardware". There's codecs/configurations a Threadripper with 64GB of RAM in a mains powered jet engine sounding desktop could handle in software that would kill a Snapdragon with 6GB of RAM in a phone. There's also codecs/configurations the Snapdragon in the phone could play using hardware acceleration that would choke a low powered Celeron or Atom decoding in software.
> Because H.266/VVC was developed with ultra-high-resolution video content in mind, the new standard is particularly beneficial when streaming 4K or 8K videos on a flat screen TV.
Compressing video is very different from gzipping a file. It's more about human perception than algorithms, really. The question is "what data can we delete without people noticing?", and it makes sense that answer is different for an 8k video than a 480p video.
It is all a matter of trade offs and engineering.
For MPEG / H.26x codec, the committees start the project by asking or defining the encoding and decoding complexities. And if you only read Reddit or HN, most of the comment's world view would be Video codec are only for Internet Video and completely disregard other video delivering platform. Which all have their own trade off and limitations. There is also cost in decoding silicon die size and power usage. If more video are being consumed on Mobile and battery is a limitation, can you expect hardware decoding energy usage to be within the previous codec? Does it scale with adding more transistors, are there Amdahl's law somewhere. etc It is easy to just say adding more transistor, but ultimately there is a cost to hardware vendors.
Vast majority of the Internet seems to think most people working on MPEG Video Codec are patents trolls and idiots and paid little to no respect to its engineering. When as a matter of fact Video Codec are thousands of small tools within the spec, and pretty much insane amount of trial and error. It may not be as complicated as 3GPP / 5G level of complexity, but it is still lot of work. Getting something to compress better while doing it efficiently is hard. And as Moore's Law is slowing down. No one can continue to just throwing transistors at the problem.
In the mid '90s, PCs often weren't fast enough to decode DVDs, which were typically 720x480 24FPS MPEG2. DVD drives were often shipped with accelerator cards that decoded MPEG2 in hardware. I had one. My netbook is many orders of magnitude faster than my old Pentium Pro. But it's not fast enough to decode 1080p 30fps H.265 or VP9 in software. It must decode VP9/H.265 on the GPU or not at all. MPEG2 is trivial to decode by comparison. I would expect a typical desktop PC of the mid '90s to take seconds to decode a frame of H.265, if it even had enough RAM to be able to do it at all.
It's an engineering tradeoff between compression efficiency of the codec and the price of the hardware which is required to execute it. If a chip which is capable of decoding the old standard costs $8, and a chip which is capable of decoding the new standard costs $9, sure, the new standard will get lots of adoption. But if a chip which is capable of decoding the new standard costs $90, lots of vendors will balk.
By the end of that year the G4 Power Macs were just barely fast enough to play DVD's with software decoding and assistance from the PCI or later AGP video card. And after a while (perhaps ~ 2002?), even the Blue G3's could do it in software even if you got a different video card, as long as you also upgraded to a G4 CPU (they were all in ZIF sockets).
It was very taxing on computers at y2k!
Later autumn 2000 G3 iMacs could also play DVD's but I think they needed special help from a video co-processor.
Is it just basically the same mechanism that leads to so much drug development happening in the US despite how backwards its medical system is, because those regressive institutions create profit incentives not available elsewhere (to develop drugs or video codecs for profit) and thus the US already has capitalists throwing money at what could be profitable whereas everyone else would look at it as an investment cost for basically research infrastructure.
Compared to 30 years ago, we now have better knowledge and statistics about what low level primitives are useful in a codec.
E.g. jpeg operates on fixed 8×8 blocks independently, which makes it less efficient for very large images than a codec with variable block size. But variable block size adds overhead for very small images.
An other reason can be common hardware. As hardware evolves, different hardware accelerated encoding/decoding techniques become feasible that gets folded into the new standards.
What kind of overhead?
The extra code shouldn't make a big difference if nothing is running it.
And space-wise, it should only cost a few bits to say that the entire image is using the smallest block size.
Is the worry just about complicating hardware decoders?
Early pc's had separate and very expensive mpeg decode boards just to decode dvd, creative sold a set, the cpu simply couldn't even handle mpeg 2. I know its hard to believe but there was a time when playing back an mp3 was a big ask, all these algorithms could be made long ago, but they would have been impractical fantasy. Only now are we seeing real partial cheated resolution ray tracing in modern high end gaming hardware now which is a good comparison, ray tracing has been with us for a long time, only hardware advancement over decades has made it viable.
It amused me that they claimed 4k uhd h265 is now 10GB for a movie, that's garbage bitrate, they always ask too much of these codecs.
can confirm. audio playback would stutter on my 486dx if one dared to multitask.
https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
Here's something to consider:
In 1995, a typical video stream was 640 x 480 x 24fps. That's 7,372,800 pixels per second.
In 2020, we have some content that's 7680 x 4320 x 120fps. That's 3,981,312,000 pixels per second, or a 540 fold increase in 25 years.
The massive increase in image size actually makes it easier to use high compression ratios. I found this out the hard way recently, when I was trying to compress and email a powerpoint presentation that a coworker had presented on video. In a nutshell, the powerpoint doc with it's sharp edges and it's low resolution made it difficult to compress.
Increase framerate plays a factor too; due to decades of research on motion interpolation, algorithms have become quite good at guessing what content can be eliminated from a moving stream.
The way I understand it is that one way to compress a document would be to store a computer program and, at the decompression stage, interpret the program so that running it outputs the original data.
So suppose you have a program of some size that produces the correct output, and you want to know if a smaller-sized program can also. You examine one of the possible smaller-sized programs, and you observe that it is running a long time. Is it going to halt, or is it going to produce the desired output? To answer that (generally), you have to solve the halting problem.
(This applies to lossless compression, but maybe the idea could be extended to lossy as well.)
Would you? AV1 was only officially released 2 years ago, h.265 7, h.264 14, …
TL;DR: it's partly because we're using higher video resolutions. A non-negligible part of the improvement stems from adapting existing algorithms to the now-doubled-resolution.
Almost all video compression standards split the input frame into fixed-size square blocks, aka "macroblocks". To put it simply, the macroblock is the coarsest granularity level at which compression happens.
- H.264 and MPEG-2 Video use 16x16 macroblocks (ignoring MBAFF).
- H.265 use configurable quad-tree-like macroblocks, with a frame-level configurable size up to 64x64.
- AV1 makes this block-size configurable up to 128x128.
Which means:
Compression to H.264 a SD video (720x576, used by DVDs) results in 1620 macroblocks/frame.
Compressing to H.265 a HD video (1920x1080) results in at least 506 macroblocks/frame.
Compressing to AV1 a 4K video (3840x2160) results in at least 506 macroblocks/frame.
But compressing to H.264 a 4K video (3840x2160) will result in 32400 macroblocks/frame.
The problem is, there are constant bitcosts per-macroblock ((mostly) regardless of the input picture). So using H.264 to compress 4K video will be inefficient.
When you take an old compression standard to encode recent-resolution content, you're using the compression standard outside of the resolution domain for which it was optimized.
> Is this continued improvement related to the improvement of technology? Or just coincidental?
Of course, there also "real" improvements (in the sense "qualitative improvements that would have benefited to compression old video resolutions, if only we had invented them sooner").
For example:
- the context-adaptive arithmetic coding from H.264, which is a net improvement over classic variable-length huffman coding used by MPEG-2 (and H.264 baseline profile).
- the entropy coding used by AV1, which is a net improvement over H.264's CABAC.
- integer DCT (introduced by H.264), which allow bit-accuracy checking and way lot easier and smaller hardware implementations (compared to floating point DCT that is used by MPEG2).
- loop filters: H.264 pioneered the idea of a normative post-processing step, whose output could be used to predict next frames. H.264 had 1 loop filter ("deblocking"). HEVC had 2 loop filters: "deblocking" and "SAO". AV1 has 4 loop filters.
All of these a real improvements, brought to us by time, and extremely clever and dedicated people. However, the compression gains of these improvements are nowhere near the "50% less bitrate" that is used to sell each new advanced-high-efficiency-versatile-nextgen video codec. Without increasing - a lot - the frame resolution, selling a new video compression standard will be a lot harder.
Besides, now that the resolutions seems to have settled up around 4K/8K (and that "high definition" has become the lowest resolution we might have to deal with :D), things are going to get interesting ... provided that we don't start playing the same game with framerates!
Would you? Video compression is one of the few things that we will work on for the next 1000 years and still be nowhere near finished. The best video compression would be to know the state of the universe at the big bang, have a timestamp of the beginning and end of your clip and spatial coordinates defining your viewport. Then some futuristic quantum computer would just simulate the content of your clip...
So yeah, sure we are done with video compression :). This is of course an extreme example of constant time compression that may or may not be ever feasible (if we live in a computer simulation of an alien race, then it is already happening).
But the gist is the same. Video compression is mostly about inferring the world and computing movement not by storing the content of the image.
For instance by taking a snapshot of the world, decomposing it into geometric shapes (pretty much the opposite of 3D rendering) and then computing the next frames by morphing these shapes + some diff data that snaps these approximations back in line with the actual data.
We are all but in the very infancy of video compression. What should surprise you is why it takes us so long to get anywhere.
For me, this is another pointless advance in video technology. 720p or 1080p is fantastic video resolution, especially on a mobile phone. Less than 1% of the population cares or wants higher resolution.
What new technologies are doing now is re-setting the patent expiration clock. As long as new video tech comes out every 5-10 years, HW manufacturers get to sell new chips, phone manufacturers get to sell new phones, TV manufacturers get to sell new TVs, rinse, repeat.
720p is far from fantastic. It's noticeably blurry, even on mobile.
1080p is minimally acceptable, and is now over 10 years old.
> Less than 1% of the population cares or wants higher resolution.
That's a very bold claim. Have any studies or polls to back that up?
Is it? Because Google/YouTube, Amazon/Twitch, Netflix, Microsoft, Apple, Samsung, Facebook, Intel, AMD, ARM, Nvidia, Cisco, etc, are all part of AO Media:
* https://aomedia.org/membership/members/
The main major tech player I don't see is Qualcomm.
AV1 will probably win in most circumstances (big tech) but is unlikely to win where there are big gains to be had by reducing file size (broadcasters with gigantic libraries).
Broadcasters are also used to paying a lot and not getting much.
AV1 is unlikely to ever be practical for "muggle" encode use, at least in this decade. It will only be worth committing that much compute workload to making a smaller file if the recipients will number in at least what, millions?
I'd be really curious what a hardware realtime AV1 encoder would even look like. How much silicon would that take? That kind of chip would have be be colossal even if it sacrifices huge amounts of efficiency to spit out frames at reasonable time (in the same way hardware hevc and vp9 encoders kind of suck).
I would still take freedom over patented software
> […] but h265 is now significantly more efficient than av1.
What did you mean?
Can we agree not to work on such projects? I feel that the lack of a good open-source encoder/decoder would spell the death of most codecs nowadays. That would also teach Fraunhoffer about it.
Of course, everyone is free to scratch their itch. And the bigger the void, the more itchy it gets. Luckily, we still have AV1.
If you build x264 or other open source implementation of h.264/h.265, and embed it for example in commercial video conferencing software/appliance, you have to pay patent licensing fees for that product.
It's also why Firefox downloads a blob from Cisco to handle MPEG-4 video - Cisco covers the licensing for distribution et al.
http://www.videolan.org/developers/x264.html
But if you use an H.264 encoder or decoder in a country that recognizes software patents then you need to buy a patent license if your usage comes under the terms of the license:
https://www.mpegla.com/programs/avc-h-264/
I appreciate that it costs money and time to develop these algorithms, but when you're backed by multi-billion dollar "partners from industry including Apple, Ericsson, Intel, Huawei, Microsoft, Qualcomm, and Sony" perhaps they could swallow the costs? It is 2020 after all.
In the early days of MP3, all MP3 rippers and players were built off of their implementation.
Hardware and software companies had to license in order to play MP3 files. As such there was not native support for MP3's for quite some time.
In the late 90's right around the explosion of MP3's on the internet, Fraunhofer was going after companies for doing so.
In my humble opinion, that license mess set back innovation in the portable audio space by a good 5 years.
Seeing all this, I'm convinced that copyright in general and patent system in particular does more harm than good by slowing down the technical progress of the humanity as a whole for the sake of some already rich people becoming a bit richer.
The initial idea behind patent system was sensible, but the way it's abused now... I mean, it could work in today's world as intended if patents lasted a year or two, not what is effectively eternity.
Or did it push it forward? If not for the licensing at the time, would it have been developed? Would it be allowed to be used by anyone just paying for the technology?
Dead Comment
Fraunhofer has a budget of over 2 billion Euros, and 30% of their money comes from public funding. They run over 70 institutes, so they do much more than this.
The root problem here is that nowadays public research is funded with industry money, which means there has to be a return on investment, hence the patents. In fact, this has metastasized into universities being graded by their patent portfolio volume. So I would expect there to be patents even if 100% of the funding came from the tax payer.
It would have been possible to do the whole process just with public money and zero patents. In fact, I would love it if some research team collated all the patent tax payments across the population of Germany and compare the bottom line cost for the country.
I wager it would have been cheaper without patents, too.
Fraunhofer gets roughly 30% of its funding from public sources, the remainder is raised on a per-project basis. It's a fair assumption that those industry partners provided some funds towards the development here. Maybe they even covered all the payroll costs for the involved scientists for the duration of the project.
And yet more income means more money for other research projects. Maybe ones that are not as commercially interesting, or for which a partner decides to terminate a contract rather unexpectedly. While I am also a fan of OSS and would love for work like this to either have no patents or a liberal patent grant, I can also appreciate the desire to fund your research institute.
Why would the taxpayer fund anything that isn’t open and free. Crazy.
At least the patent licenses usually used with MPEG mean that private use of open source implementations is free.
The existence of Theora, VP8, VP9, and now AV1 seems to contradict that theory.
You could argue that they infringe on some unknown patents, but that is also arguably true of patent cabals like MPEG (you just hope that the cabal is big enough that there aren't any patentholders lurking outside). The only difference is that with a patent cabal you have the fun of having to obey the restrictions of everyone who showed up with a possibly-related-in-some-way patent and joined the cabal.
Not to mention that it isn't necessary for a patent pool to be a cabal. AOMedia has a similar structure to a patent cabal except it doesn't act like a cabal (its patent pool is royalty-free in the style of the W3C). So even if the argument is that a patent pool is a good idea (and video codecs cannot be developed without them), there isn't a justification behind turning the patent pool into a cabal.
> At least the patent licenses usually used with MPEG mean that private use of open source implementations is free.
You say that, but there's a reason why some distributions (openSUSE for one) still can't ship x264 (even though the code itself is free software). Not to mention the need for Cisco's OpenH264 in Firefox (which you cannot recompile or modify otherwise you lose the patent rights to use it). The existence of the MPEG patent cabal isn't a good thing, and any minor concessions you get from them do not justify their actions.
Edited because I didn't know that some European countries accept software patents.
You can, but fraunhofer certainly isn't trying!
> At least the patent licenses usually used with MPEG mean that private use of open source implementations is free.
0_o that is not at all the truth.
Why is that?
A GPL implementation doesn't guarantee a patent grant. Even if you wrote the code yourself, even just for your personal use, your own work could still be illegal to use due to lacking a patent license from the original patent holders.
Be careful about using implementations of H.26x codecs, because in countries that recognize software patents the code may be illegal to use, regardless whether you've got a license for the code or not. Even when a FLOSS license says something about patent grants, it's still meaningless if the code author didn't own the patents to grant.
I think a more subtle "open-sourcing" of this IP could still be possible, though. Maybe one that still requires that large corporate players that are going to sell their derivative products, acquire a license the traditional way (this is, after all, what the contributors to the codec's patent-pool and R&D efforts based their relative-R&D-labor-contribution negotiations around: that each contributor would end up paying for the devices of theirs that run the codec.)
Maybe there could be a foundation created under the stewardship of the patent-pool itself, which nominally pays the same per-seat/per-device licensing costs as every other member, but where this money doesn't come from revenue but rather is donated by those other members; and where this foundation then grants open-source projects an automatic but non-transferrable license to use the technology.
So, for example, a directly open-source project (e.g. ffmpeg) would be granted an automatic license (for its direct individual users); but that license wouldn't transfer to software that embeds it. Instead, other open-source software (e.g. Handbrake, youtube-dl, etc.) that embeds ffmpeg would acquire its own automatic license (and thus be its own line-item under the foundation); while closed-source software that embeds ffmpeg would be stuck needing a commercial license.
Is there already a scheme that works like this?
Does it mean money, or not? Because if not, then Fraunhofer does not exist.
But there is absolutely a problem here, because said 'mega businesses' actually should have a strategic imperative to want to make internet technologies more widespread.
Why on earth would MS want to limit their main line of business for a tiny big of IP related revenue?
It would seem to me, that G, MS, Huawei and all of the various patent holders should be trying their best to remove any and all barriers to adoption. There are enough bureaucratic hurdles in the way to worry about, let alone legal concerns.
Even if MS or whoever had to buy out some laggard IP owners who didn't want to play ball, it would probably still make sense for them.
Fraunhofer or anyone else are not in that situation, but the behemoths running vast surpluses are, it just seems shortsighted for them to hamstring any of this.
Both h264 and h265 have these implementations, I think FFMPEG library has both under the terms of GPLv2.
The decoders are almost completely useless. The video codec, at least the decoder, needs to be in the hardware, not in software.
Mobile devices just don’t have the resources to run the decoders on CPU. The code works on PC but consumes too much electricity and thermal budget. Even GPGPUs are not good enough for the job, couple generations ago AMD tried to use shader cores for video codecs, didn’t work well enough and they switched to dedicated silicon like the rest of them.
You'd be surprised at how often these are used.
https://aomedia.org/av1/
https://aomedia.org/membership/members/
AV1 decoding has already been in Chrome and Firefox since at least a year ago. We're just waiting for hardware decoding and encoding support now, which should start appearing this year.
The next version of Chrome will also support the AV1-based AVIF image format this month:
https://chromium-review.googlesource.com/c/chromium/src/+/22...
YouTube, Netflix, and Amazon/Twitch are also likely to not support VVC (and some of them don't support h.265 either) for their streaming services.
FWIW, Firefox supports it already (behind the image.avif.enabled knob).
They should swallow so people again come back and say they are doing it to kill competition? Just received bill from doctor's visit for hundreds of dollar, I guess things are not gonna be free in 2020 after all.
I'm sure it does better, but I'm equally sure it'll turn out to be an incremental benefit in practice.
> I wonder how many more CPU instructions are required per second of decoded video compared to HEVC.
CPU cycles are cheap. The real cost is the addition of yet another ?!@$!!#@ video codec block on every consumer SoC shipped over the coming decade.
Opinionated bile: video encoding is a Solved Problem in the modern world, no matter how much the experts want it to be exciting. The low hanging fruit has been picked, and we should just pick something and move on. JPEG-2000 and WebP failed too, but at least there it was only some extra forgotten software. Continuing to bang on the video problem is wasting an absolutely obscene amount of silicon.
As long as someone sees the benefit, people will keep pursuing it. Compression (as like all tech) is a moving target with the platforms regularly improving on many axes.
This is part of the problem. What is an "objective" measure of perceptual fidelity to the original?
For lower res you will see lower percentage. You see the same claims from other codec as well.
For comparison, HEVC claimed 50 to 60% compared to AVC. You can compare with reality...
Objective video quality is a tricky thing to do and you can easily come out with a codec that’s great at fine details in foreground people but terrible at high speed panning of water and trees. Depending on the material depends where you want the quality to go.
https://en.wikipedia.org/wiki/Essential_Video_Coding
There are 3 video coding formats expected out of (former) MPEG this year:
https://www.streamingmedia.com/Articles/Editorial/Featured-A...
So this isn't necessarily the successor to HEVC (except that it is, in terms of development and licensing methods).
Maybe. On the other hand, maybe not. Leoanardo Chiariglione, founder and chairman of MPEG, thinks MPEG has for all practical purposes ceased to be:
https://blog.chiariglione.org/a-future-without-mpeg/
The disorganised and fractured licensing around HEVC contributed to that. And, so far, VVC's licensing looks like it's headed down the same path as HEVC.
Maybe AV1's simple, royalty-free licensing will motivate them to get their act together with VVC licensing.
NVIDIA's DLSS 2.0 supersampling is already moving into that direction.
https://papers.nips.cc/paper/9127-deep-generative-video-comp...
How do you know this?
TV hardware is on par with browsers. Anything is a program.
So I guess it comes down to 266 hw support. Or powerful CPUs that can push sw decoding?