FFmpeg is wonderful software. Growing up as a Windows user in the early 2000s, devices were far more picky than they are today about which video codecs they'd support. It was a non-trivial task as an 11yo trying to convert DivX .avis into an MP4 my old iPod Video could understand. Discovering ffmpeg and finding that someone was offering for free what I otherwise could only find under mountains of crappy shareware was a real watershed moment.
Lets write a new video compression algorithm that is super efficient - great
this lets us compress movies so they can fit on cheap CD's, instead of DVDs. - great
We can now give those CD's away with movies on them - great
And then every time someone puts one in a DivX player they can pay us to watch/rent it, instead of having to drive to blockbuster - wait, what?
Its easy, we'll just use a phone line that everyone has right near their entertainment center in their living room to phone home at night and send the data of what movies you watched and how many times. - what are they smoking?
I remember the default media player that shipped on Windows was absolutely terrible because it could only play a very limited number of file formats, none of which were actually used much by movie files found in the wild. If you wanted to actually play a video, you had to try your luck and choose among several 3p "codec packs" half of which were probably loaded with malware.
People who have always lived in a world with great software like VLC and MPV and ffmpeg underestimate how hard it was to actually play a video file on your computer back in 2000.
I've always said ffmpeg is one of the new wonders of the world. It powers so much, is so complex, so irreplaceable. It's crazy we get to enjoy it for free. I use it to encode my movies and tv shows to AV1/720p.
It's also a testament to the power of open source software. ffmpeg is in a significant amount of software touching audio/video in some way, and we are all enriched because everyone is free to use it.
I convert much to 720p PS3 compliant H264, for maximum device compatibility. I take an external drive with these files with me when I travel, and in 99% of the cases plugging it into hotel TVs just works.
We had an old software at work that would only take a specific file format, and of course the files came from multiple sources, which made it really painful for the users.
We transformed a relatively decent desktop into a ffmpeg transcoding machine, which would monitor files incoming from a samba share and it would output the converted file into another samba share.
It was just a bunch of scripts and cron jobs but it worked much better than I anticipated and it was mostly maintenance-free.
I remember Back in 2007, during a far away vacation, using ffmpeg on a windows netbook to convert star trek episodes into a format my little mp3 player could understand and play on its little screen (320x240?). It was amazing that even on the 600-900 MHz cpu, those videos transcoded in a matter of minutes.
I am always surprised how much it's taken over video processing tasks (and maybe how Apple lost the plot with Quicktime and such).
Would def be interesting if someone could write up a history of the project so far. I wonder how much industry input into the OSS commits there are (like MS/IBM into linux, postgres, etc)
The greatest addition to FFmpeg in the recent past was the addition of large language models translating my "ffmpeg command to mix audio file onto video file" into actually executable FFmpeg commands.
Being cheeky of course here. FFmpeg is great. An AI assistant was what I needed to execute my ~12 FFmpeg commands per year though, with ease and speed.
ChatGPT / LLMs really are great with ffmpeg. Not too long ago I wrote a blog post outlining an ffmpeg command I needed (mostly for my own future reference). It almost feels quaint now, given that you can get the same kind of explanation from GPT. I bet if I pasted the command in my article into ChatGPT I would get an explainer that's basically identical to what I wrote.
Seems risky to start running commands on your machine that a hallucinating AI spit out at you without understanding them, but I guess they can at least let you know what to look up so you can double check what the command will do.
Natural language syntaxes universally suck. A faux english syntax isn't easier to use if you don't know which english in particular will be accepted. A complex cli interface fundamentally can't be easy to use. A GUI can fix that by making things discoverable and by integrating the documentation into the UI, but the ffmpeg devs presumably see that as someone else job (and there have been people to step up).
1. ffmpeg exposes all of its options through the CLI, and there are a lot of options. So it's probably always going to be completely undiscoverable. It really needs a GUI to be usable, but that's a project in itself (I guess the project is Handbrake).
2. They probably didn't put a lot of work into the UX of the CLI since it's an open source project.
There's a low-hanging fruit that I think would make ffmpeg more helpful for regular people.
There's a million terrible websites that offer file conversion services. They're ad-ridden, with god-knows-what privacy/security postures. There's little reason for users to need to upload their files to a third-party when they can do it locally. But getting them to download fiddly technical software is tough - and they're right to mistrust it.
So, there's a WASM version of ffmpeg, already working and hosted at Netlify [1]. It downloads the WASM bundle to your browser and you can run conversions/transformations as you wish, in your browser. Sandboxed and pretty performant too!
If this tool a) was updated regularly b) had a nicer, non-CLI UI for everyday users and c) was available at an easily-Googlable domain name - it would solve all the problems I mentioned above.
Browsers are annoying. They constrict the designer, they constrict the user, they're overly complicated, slow, bloated... I don't know why people keep pushing them to do things they are bad at.
I wish 20 years ago we'd made a concerted effort to make Java suck less. We'd have the universal applications everyone wants but nobody wants to put effort into. But the web was new-ish, and people didn't realize that hypertext document viewers would become an entire application platform and mini OS.
What I'd really like to see is something like FlatPak, but for all platforms. Basically it would be containerized GUI apps, but one repository for one app that serves all platforms. On Android, MacOS, Windows, etc, you would run your "flatpak add https://some/repository/ my-app && flatpak pull my-app && flakpak run my-app" (but in a GUI, like an App Store you control). And that would pull the image for your platform and run it. Since it's containerized, you get all the dependencies, it's multi-arch, & you control how it executes in a sandbox. You could use the same programming language per platform, or different languages; same widgets, different widgets; it wouldn't matter because each platform just downloads and runs an image for that platform. This wouldn't stop us from having/making "a better Java", but it would make it easier to support all platforms, distribute applications securely, update them, run them in a sandbox, etc. Imagine being able to ship a single app to Windows and iOS users that's just a shell script and 'xdialog'. Or if you prefer, a single Go app. Or a sprawling Python or Node.js app. Whatever you want. The user gets a single way to install and run an app on any platform, and developers can support multiple platforms any way they want. No more "how do I develop for iOS vs Windows"; just write your app and push your container.
This proposal is to offer competition to web-based conversion websites. If users are willing and able to find Handbrake and download it, it can work for them. But everyday users are right to distrust software downloaded from the Internet.
Many users are in environments where its not possible to download new software (schools, work places, universities).
The browser has its disadvantages, but it is the most widely-deployed sandboxed execution environment providing incredibly easy distribution of software.
HandBrake suffers from the same problems as FFmpeg to a lesser extent. I have no idea what options I should be selecting to get the max quality, smallest file size for a conversion.
The presets are useful but when I'm converting an old WMV or some other ancient format I want to know that I'm not leaving anything behind.
If Java had won, we'd be complaining about Java instead of web technologies. It doesn't ultimately matter, when you have a platform as large as the web is, it's going to be complicated and bloated.
"Regular" people don't really need FFMPEG. Regular people need tools with GUIs that have a non-generic purpose. So stuff like https://kdenlive.org/en/ that are backed by ffmpeg are (imo) superior "regular" person tools.
FFMPEG isn't complicated (its as complicated as any other CLI tool), it's that video encoding/decoding specifically is a hard problem space that you have to explicitly learn to better understand what ffmpeg can do. I think if someone spent an hour learning about video codecs, bitrates, and container formats, they would immediately feel "better" at ffmpeg despite not learning more about the tool itself.
I mean, we have, what, 15+ years of StackOverflow posts to tell us what the common use cases are, e.g. "how do I make a GIF from this short screen cap?"; surely FFmpeg could offer prompt-based wizards for that kind of low hanging fruit.
≥ "If this tool a) was updated regularly b) had a nicer, non-CLI UI for everyday users and c) was available at an easily-Googlable domain name - it would solve all the problems I mentioned above."
No matter how nice you make it, it will probably still lose the SEO battle against the shitty ad-ladden sites fighting to win top place for google searches of "convert X to mp3"/etc.
That's probably true. In my ideal world, the OSS community comes together in a suite of offline-first, in-browser alternatives to common user needs. We need a site for document conversion (using WASM Pandoc?), PDF merging, image conversion, text utils (lower case, spell check). All these would live on one trusted site. Hopefully users can discover the site and be done searching, and spread its name by word of mouth.
I had to merge a bunch of PDFs for a rental application recently and it was painful. Having to upload very sensitive docs, every site being a funnel to their paid version, etc.
How is performance? Right now I use Handbrake to do video optimization. However, I'm not a video expert and all the options are pretty daunting to me. I don't know how to get the right mix of optimization without having the video encoding take forever. I would guess running the conversion in browser would just make everything slower. I wish there was something simpler for video, like ImageOptim. Just drag the video in and it compresses it using the best options based on compatibility needs.
"One of HandBrake’s strengths is its ability to open a wide variety of video formats. HandBrake uses FFmpeg under the hood and generally can open whatever FFmpeg will, in addition to disc-based formats like DVD and Blu-ray."
The options Handbrake exposes are essentially the ffmpeg flags. The built in presets in Handbrake are generally pretty sensible IMO and I've rarely had to deviate.
If it's 2-3x of native code, that's plenty good enough. Everyone uses Handbrake in an async mode - just set up the conversion and let it run overnight.
Of course, a website would be better for smaller conversion jobs (in case you have browser restarts or whatnot). Desktop apps can block computer restarts to a greater degree than websites.
I don't think users can distinguish a "local" website with a public one. So this software should just come with the OS as a "movie maker" package. As mentioned Handbrake UIs are a decent candidate.
I think OSs are generally over bundling helpful software, unless its a funnel to a paid version. The security risk is too high, and so is the legal risk of anti-monopoly action ("why are you killing independent movie editor businesses?").
So I'm trying to build ffmpeg via vcpkg today, and it turned out multiple of its dependencies are transitively depending on liblzma, but the downloading of liblzma source has been disabled by GitHub in light of the recent xz backdoor.
Lasse Collin pulled everything back to a standalone git repo, with all of the compromised code removed: https://git.tukaani.org/, which is now, I presume, the official distribution source for XZ.
Blocking downloads of liblzma seems to me to be an ill-advised decision. Now that the mechanism is known, the dangers are limited, but the educational value of being able to study what has been done is real.
While the dangers are limited, they certainly aren't zero. Even if the original attacker(s) have entirely gone to ground others may be scanning for hosts that managed to got compromised by following the bleeding edge and more could get compromised of downloads from primary sources are kept open.
Keeping the affected code visible somewhere could be useful for research purposes, but you don't want it where people or automations might unwittingly use it. If the official sources where the only place this could be found then it might be reasonable to expect them to put up a side copy for this reason, but given how many forks and other copies there will be out there I don't think this is necessary and they are better off working on removing known compromises (and attempting to verify there are no others that were slipped in) to return things to a good state.
Maybe someone needs a year to audit the history and find all the other backdoors. Who's going to work on it for a year for free or without being in on it, I don't know.
right now I'm sure it's a temporary measure, to limit the downloading of sources.
but I really worry that later this will become normalized first, after every exposed hack withrdraw source availability for a little bit aftewards, just while 'they' check for other attacks or whatever
later on, it'll take longer and longer to put the source back up. but let's hope this is merely my overactive paranoia and everything will be fine open source is still ok.
The obvious solution seems to be adding an extra hurdle, where it warns you the source may be compromised, so you can still get it, but aren't going to just grab it without knowing something happened.
There is value in making sure (potentially) compromised code doesn't just get used normally, but I agree that shouldn't mean totally blocking access to it in most cases.
I use it — because I am writing a rust program, and want to use ffmpeg functionality.
What’s the alternative? I could wrap the C API, and then try to make a nice rust interface from that, but then that’s exactly what this package does, so I don’t want to repeat the work.
Basically no one rewrites FFmpeg in recent years, in any language, at least not in the open source scene (and judging from the known usage of FFmpeg in world’s premier providers of multimedia content, probably not in the commercial scene either). It’s both too good and too daunting.
I used this wrapper to implement an opening and ending detection tool for “fun” [1].
However, it seems that many programs opt to instead shell out to the ffmpeg CLI. I think it’s usually simpler than linking against the library and to avoid licensing issues. But there are some cases where the CLI doesn’t cut it.
I have been using the xstack filter for several years now.
What I do is take several diverse short video segments, like 100, concatenate them into 4 segments (example 23+24+26+27 since they have diverse lengths) and then xstack them into a 2-by-2 mosaic video.
Before, I was doing it in a single stage, but now, after some advice, I do it in 5 stages: 4 concatenate stages and 1 xstack stage.
I have not profiled/timed it so see which is faster, but it works pretty well, although I often have a lot of different weird warnings.
> What I do is take several diverse short video segments, like 100, concatenate them into 4 segments (example 23+24+26+27 since they have diverse lengths) and then xstack them into a 2-by-2 mosaic video.
Just out of curiosity.. what use do you have for a 2-by-2 mosaic video?
Also, it seems there is currently one in progress to drop the "6" qualifier on the ffmpeg binaries <https://github.com/macports/macports-ports/pull/23315/files> so it'll be fascinating to see if any new ffmpeg7 then subsequently puts the "7" back, beginning the cycle again
ffmpeg updates their API very liberally, even in minor version bumps. Combine that with many packages depending on ffmpeg doesn't make it very easy to always have the latest ffmpeg version. They are working on it though: https://trac.macports.org/ticket/65623
(I'm not a MacPorts maintainer, but I've been burnt by ffmpeg API changes a couple times myself before).
I got frustrated having to install all of the runtime dependencies and just wanted an easy way to install the statically-linked version, so here it is.
ffmpeg is such a joy to use, once you make it over the very steep learning curve.
I'm making some youtube videos where I play through Demon's Souls flipping a coin to decide to equip items or not, and I wanted to have an onscreen coin flip animation and sound effect. With some effort, I created a transparent set of frames for the animation. Then with ffmpeg's filter_complex I was able to add the image sequence as a video stream, overlay it over the original video, and add a sound effect. That's on top of the existing subtitles, audio channel merging, and video resizing/compression. All in a single (long!) ffmpeg cli command.
and every other non-sadist would've done this in 2 seconds in <insert 500+ FOSS video editors>. not sure how wrangling a byzantine CLI is a joy to use.
Readit News
20 years later it's still a goto. Great tool.
Lets write a new video compression algorithm that is super efficient - great
this lets us compress movies so they can fit on cheap CD's, instead of DVDs. - great
We can now give those CD's away with movies on them - great And then every time someone puts one in a DivX player they can pay us to watch/rent it, instead of having to drive to blockbuster - wait, what?
Its easy, we'll just use a phone line that everyone has right near their entertainment center in their living room to phone home at night and send the data of what movies you watched and how many times. - what are they smoking?
https://en.wikipedia.org/wiki/DIVX
Afaict these aren't related.
https://en.m.wikipedia.org/wiki/DivX
People who have always lived in a world with great software like VLC and MPV and ffmpeg underestimate how hard it was to actually play a video file on your computer back in 2000.
I convert much to 720p PS3 compliant H264, for maximum device compatibility. I take an external drive with these files with me when I travel, and in 99% of the cases plugging it into hotel TVs just works.
We transformed a relatively decent desktop into a ffmpeg transcoding machine, which would monitor files incoming from a samba share and it would output the converted file into another samba share.
It was just a bunch of scripts and cron jobs but it worked much better than I anticipated and it was mostly maintenance-free.
I agree with you in general, but it still annoys the crap out of me that QuickTime Player on macOS still won't load completely valid videos.
My basic "QuickTime compliant" options are:
-tag:v hvc1 -c:v libx265 -crf 28 -c:a eac3
Of course for everything else there's VLC.
Would def be interesting if someone could write up a history of the project so far. I wonder how much industry input into the OSS commits there are (like MS/IBM into linux, postgres, etc)
Being cheeky of course here. FFmpeg is great. An AI assistant was what I needed to execute my ~12 FFmpeg commands per year though, with ease and speed.
https://epiccoleman.com/posts/2023-03-19-ffmpeg-mp4-webm
I mean I've only done it once with ffmpeg but it felt so good
1. ffmpeg exposes all of its options through the CLI, and there are a lot of options. So it's probably always going to be completely undiscoverable. It really needs a GUI to be usable, but that's a project in itself (I guess the project is Handbrake).
2. They probably didn't put a lot of work into the UX of the CLI since it's an open source project.
3. Backwards compatibility.
There's a million terrible websites that offer file conversion services. They're ad-ridden, with god-knows-what privacy/security postures. There's little reason for users to need to upload their files to a third-party when they can do it locally. But getting them to download fiddly technical software is tough - and they're right to mistrust it.
So, there's a WASM version of ffmpeg, already working and hosted at Netlify [1]. It downloads the WASM bundle to your browser and you can run conversions/transformations as you wish, in your browser. Sandboxed and pretty performant too!
If this tool a) was updated regularly b) had a nicer, non-CLI UI for everyday users and c) was available at an easily-Googlable domain name - it would solve all the problems I mentioned above.
[1]: https://ffmpegwasm.netlify.app/
Browsers are annoying. They constrict the designer, they constrict the user, they're overly complicated, slow, bloated... I don't know why people keep pushing them to do things they are bad at.
I wish 20 years ago we'd made a concerted effort to make Java suck less. We'd have the universal applications everyone wants but nobody wants to put effort into. But the web was new-ish, and people didn't realize that hypertext document viewers would become an entire application platform and mini OS.
What I'd really like to see is something like FlatPak, but for all platforms. Basically it would be containerized GUI apps, but one repository for one app that serves all platforms. On Android, MacOS, Windows, etc, you would run your "flatpak add https://some/repository/ my-app && flatpak pull my-app && flakpak run my-app" (but in a GUI, like an App Store you control). And that would pull the image for your platform and run it. Since it's containerized, you get all the dependencies, it's multi-arch, & you control how it executes in a sandbox. You could use the same programming language per platform, or different languages; same widgets, different widgets; it wouldn't matter because each platform just downloads and runs an image for that platform. This wouldn't stop us from having/making "a better Java", but it would make it easier to support all platforms, distribute applications securely, update them, run them in a sandbox, etc. Imagine being able to ship a single app to Windows and iOS users that's just a shell script and 'xdialog'. Or if you prefer, a single Go app. Or a sprawling Python or Node.js app. Whatever you want. The user gets a single way to install and run an app on any platform, and developers can support multiple platforms any way they want. No more "how do I develop for iOS vs Windows"; just write your app and push your container.
Many users are in environments where its not possible to download new software (schools, work places, universities).
The browser has its disadvantages, but it is the most widely-deployed sandboxed execution environment providing incredibly easy distribution of software.
The presets are useful but when I'm converting an old WMV or some other ancient format I want to know that I'm not leaving anything behind.
FFMPEG isn't complicated (its as complicated as any other CLI tool), it's that video encoding/decoding specifically is a hard problem space that you have to explicitly learn to better understand what ffmpeg can do. I think if someone spent an hour learning about video codecs, bitrates, and container formats, they would immediately feel "better" at ffmpeg despite not learning more about the tool itself.
No matter how nice you make it, it will probably still lose the SEO battle against the shitty ad-ladden sites fighting to win top place for google searches of "convert X to mp3"/etc.
I had to merge a bunch of PDFs for a rental application recently and it was painful. Having to upload very sensitive docs, every site being a funnel to their paid version, etc.
> https://handbrake.fr/docs/en/1.3.0/technical/source-formats....
"One of HandBrake’s strengths is its ability to open a wide variety of video formats. HandBrake uses FFmpeg under the hood and generally can open whatever FFmpeg will, in addition to disc-based formats like DVD and Blu-ray."
The options Handbrake exposes are essentially the ffmpeg flags. The built in presets in Handbrake are generally pretty sensible IMO and I've rarely had to deviate.
Of course, a website would be better for smaller conversion jobs (in case you have browser restarts or whatnot). Desktop apps can block computer restarts to a greater degree than websites.
Deleted Comment
Keeping the affected code visible somewhere could be useful for research purposes, but you don't want it where people or automations might unwittingly use it. If the official sources where the only place this could be found then it might be reasonable to expect them to put up a side copy for this reason, but given how many forks and other copies there will be out there I don't think this is necessary and they are better off working on removing known compromises (and attempting to verify there are no others that were slipped in) to return things to a good state.
A week or so is not a lot of time.
right now I'm sure it's a temporary measure, to limit the downloading of sources.
but I really worry that later this will become normalized first, after every exposed hack withrdraw source availability for a little bit aftewards, just while 'they' check for other attacks or whatever
later on, it'll take longer and longer to put the source back up. but let's hope this is merely my overactive paranoia and everything will be fine open source is still ok.
There is value in making sure (potentially) compromised code doesn't just get used normally, but I agree that shouldn't mean totally blocking access to it in most cases.
Why would one want a "Safe FFMpeg Wrapper"?
Edit: Got it, provides a rust API library to ffmpeg. Thanks @CJefferson.
What’s the alternative? I could wrap the C API, and then try to make a nice rust interface from that, but then that’s exactly what this package does, so I don’t want to repeat the work.
However, it seems that many programs opt to instead shell out to the ffmpeg CLI. I think it’s usually simpler than linking against the library and to avoid licensing issues. But there are some cases where the CLI doesn’t cut it.
[1] https://github.com/aksiksi/needle
What I do is take several diverse short video segments, like 100, concatenate them into 4 segments (example 23+24+26+27 since they have diverse lengths) and then xstack them into a 2-by-2 mosaic video.
Before, I was doing it in a single stage, but now, after some advice, I do it in 5 stages: 4 concatenate stages and 1 xstack stage.
I have not profiled/timed it so see which is faster, but it works pretty well, although I often have a lot of different weird warnings.
Just out of curiosity.. what use do you have for a 2-by-2 mosaic video?
[1]: https://ports.macports.org/port/ffmpeg/
[2]: https://ports.macports.org/port/ffmpeg6/
Also, it seems there is currently one in progress to drop the "6" qualifier on the ffmpeg binaries <https://github.com/macports/macports-ports/pull/23315/files> so it'll be fascinating to see if any new ffmpeg7 then subsequently puts the "7" back, beginning the cycle again
(I'm not a MacPorts maintainer, but I've been burnt by ffmpeg API changes a couple times myself before).
Not at all the same as running a 4.x release.
https://evermeet.cx/ffmpeg/
I'm making some youtube videos where I play through Demon's Souls flipping a coin to decide to equip items or not, and I wanted to have an onscreen coin flip animation and sound effect. With some effort, I created a transparent set of frames for the animation. Then with ffmpeg's filter_complex I was able to add the image sequence as a video stream, overlay it over the original video, and add a sound effect. That's on top of the existing subtitles, audio channel merging, and video resizing/compression. All in a single (long!) ffmpeg cli command.
ffmpeg is one of the true wonders of FOSS.
I'm not saying the CLI is easy to learn, but once you do learn it, you have a lot of power at your fingertips.