I've enjoyed using ffmpeg 1000% more since I was able to stop doing manually the tedious task of Googling for Stack Overflow answers and cobbling them into a command and got Chat GPT to write me commands instead.
I tried this (though with a different tool called aichat) for extremely simple stuff like just "convert this mov to mp4" and it generated overly complex commands that failed due to missing libraries. When I removed the "crap" from the commands, they worked.
So much like code assistance, they still need a fair amount of baby sitting. A good boost for experienced operators but might suck for beginners.
A while back I simply wrote my own bash function for this called `please`
as in
bash> please "use ffmpeg to extract audio from myfile.mov and save it as mp3"
It will then courteously show you the command it wants to run before you agree to do it.
Here is the whole thing, with its two dependent functions, so that people stop writing their own versions of this lol. All it needs is an OPENAI_API_KEY, feel free to modify for other LLMs
Suggestions welcome- for example I want to add a feature that either just copies it (for further modification) or prepopulates the command line with it somehow (possibly for further modification, or even for skipping the approval step)
>This will then be displayed in your terminal ready for you to edit it, or hit <enter> to execute the prompt. If the command doesnt't look right, hit Ctrl+C to cancel.
I appreciate the UI choice here. I have yet to do anything with AI (consciously and deliberately, anyway) but this sort of thing is exactly what I imagine as a proper use case.
While I love that that works, I still feel like just maybe ffmpeg needs a better interface. Not necessarily a GUI, just a better designed command line.
I think I’m finally sold on actually attempting to add some LLM to my toolbelt.
As a helper and not a replacement, this sounds grand. Like the most epic autocomplete. Because I hate how much time I waste trying to figure out the command line incantation when I already know precisely what I want to do. It’s the weakest part of the command line experience.
For the longest time I had ffmpeg in the same bucket as regex: "God I really need to learn this but I'm going to hate it so much." Then ChatGPT came along and solved both problems!
Interesting. Being able to use regexps for text processing through my career has probably saved me a few thousand hours of programming one-off solutions so far. It is one of those skills that really pays off to learn proper.
And speaking of ffmpeg, or tooling in general, I tend to make notes. After a while you end up with a pretty decent curated reference.
For me, it wasn’t so much learning ffmpeg, as it was understanding containers/codecs/encoders/streams/etc. Learning all of the intricacies there made ffmpeg make a lot more sense.
In addition to the many others mentioned, here's a script I just threw together that simplifies a lot of these chained commands - llmpeg: https://github.com/jjcm/llmpeg
If you have ffmpeg installed and an OpenAI env api key set, it should work out of the box.
learning how to use splits to do multiple things all in one command is a god send. the savings of only needed to read the source and convert to baseband video once is a great savings.
i started with avisynth, and it took time for my brain to switch to ffmpeg. i don't know how i could function without ffmpeg at this point
Truly, a net positive to my life. Just a few days ago I asked my AI buddy (Claude) to create a zsh script to organize my downloads folder according to the Johnny Decimal system. I’ve since modified it to move the files to a JD setup on my desktop.
The sense of elation I get when I wonder aloud to my digital friend and they generate what I thought was too much to expect. Well worth the subscription.
Basic syntax for re-encoding a video file did take me some time to memorize, but isn't in fact too hard:
ffmpeg <Input file(s)> <Codec(s)> <MAPping of streams> <Video Filters> output_file
- input file: -i, can be repeated for multiple input files, like so:
ffmpeg -i file1.mp4 -i file2.mkv
If there is more than one input file then some mapping is needed to decide what goes out in the output file.
- codec: -c:x where x is the type of codec (v: video, a: audio or s:subtitles), followed by its name, like so:
-c:v libx265
I usually never set the audio codec as the guesses made by ffmpeg, based on output file type, are always right (in my experience), but deciding the video codec is useful, and so is the subtitles codec, as not all containers (file formats) support all codecs; mkv is the most flexible for subtitles codecs.
- mapping of streams: -map <input_file>:<stream_type>:<order>, like so:
-map 0:v:0 -map 1:a:1 -map 1:a:0 -map 1:s:4
Map tells ffmpeg what stream from the input files to put in the output file. The first number is the position of the input file in the command, so if we're following the same example as above, '0' would be 'file1.mp4' and '1' would be 'file2.mkv'. The parameter in the middle is the stream type (v for video, a for audio, s for subtitles). The last number is the position of the stream IN THE INPUT FILE (NOT in the output file).
The position of the stream in the output file is determined by the position of the map command in the command line, so for example in the command above we are inverting the position of the audio streams (taken from 'file2.mkv'), as audio stream 1 will be in first position in the output file, and audio stream 0 (the first in the second input file) will be in second position in the output file.
This map thing is for me the most counter-intuitive because it's unusual for a CLI to be order-dependent. But, well, it is.
- video filters: -vf
Video filters can be extremely complex and I don't pretend to know how to use them by heart. But one simple video filter that I use often is 'scale', for resizing a video:
-vf scale=<width>:<height>
width and height can be exact values in pixels, or one of them can be '-1' and then ffmpeg computes it based on the current aspect ratio and the other provided value, like this for example:
-vf scale=320:-1
This doesn't always work because the computed value should be an even integer; if it's not, ffmpeg will raise an error and tell you why; then you can replace the -1 with the nearest even integer (I wonder why it can't do that by itself, but apparently, it can't).
And that's about it! ffmpeg options are immense, but this gets me through 90% of my video encoding needs, without looking at a manual or ask an LLM. (The only other options I use often are -ss and -t for start time and duration, to time-crop a video.)
> This doesn't always work because the computed value should be an even integer; if it's not, ffmpeg will raise an error and tell you why; then you can replace the -1 with the nearest even integer (I wonder why it can't do that by itself, but apparently, it can't).
It's not about integer, but some of the sizes need to be even. You can use `-vf scale=320:-2` to ensure that.
Use: sometimes I have a file with a lot of audio and or subtitle streams but only want one or two of each – here, 0:0 is the video, 0:5 is English audio, and 0:12 was the subtitle track I wanted. Setting the codecs to “copy” means nothing gets reencoded.
> then you can replace the -1 with the nearest even integer (I wonder why it can't do that by itself, but apparently, it can't).
Likely because the aspect ratio will no longer be the same. There will either be lost information (cropping), compression/stretching, or black bars, none of which should be default behaviour. Hence, the warning.
I think you're onto something. I've had hit or miss experiences with code from LLMs but it definitely makes the searching part different.
I had a problem I'd been thinking about for some time and I thought "Ill have some LLM give me an answer" and it did - it was wrong and didn't work but it got me to thinking about the problem in a slightly different way and my quacks after that got me an exact solution to this problem.
So I'm willing to give the AI more than partial credit.
It already does. It’s the cli flags. What you’re missing is the semantic which you can get with learning about containers, codecs, and other stuff. You don’t use grep and sed with no understanding of what a text file is.
ffmpeg and jq are 2 commands I've about given up trying to "use" with any facility and am more than happy to pawn that off to one of the Gippity's; chat, claude, etc.
This reminds me I need to publish my write up on how I've been converting digitized home video tapes into clips using scene detection, but in case anyone is googling for it, here's a gist I landed on that does a good job of it [0] but sometimes it's fooled by e.g. camera flashes or camera shake so I need to give it a start and end file and have ffmpeg concatenate them back together [1]
Weird thing is I got better performance without "-c:v h264_videotoolbox" on latest Mac update, maybe some performance regression in Sequoia? I don't know. The equivalent flag for my windows machine with Nvidia GPU is "-c:v h264_nvenc" . I wonder why ffmpeg doesn't just auto detect this? I get about 8x performance boost from this. Probably the one time I actually earned my salary at work was when we were about to pay out the nose for more cloud servers with GPU to process video when I noticed the version of ffmpeg that came installed on the machines was compiled without GPU acceleration !
> Probably the one time I actually earned my salary at work was when we were about to pay out the nose for more cloud servers with GPU to process video when I noticed the version of ffmpeg that came installed on the machines was compiled without GPU acceleration !
Issue with cloud CPU's is that they don't come with any of the consumer grade CPU built-in hardware video encoders so you'll have to go with the GPU machines that cost so much more. To be honest I haven't tried using HW accel in the cloud to have a proper price comparison, are you saying you did it and it was worth it?
Are the hardware encoders even good? I thought that unless you need something realtime, it's always better to spend the cpu cycles on a better encode with th software encoder. Or have things changed ?
We were a quick and dirty R&D team that had to do a lot of video processing quickly, we were not very cost sensitive and didn’t have anything other than AWS to work with, so I can’t speak to whether it was worth it :)
I used ffmpeg for empty scene detection- I have a camera pointed at the flight path for SFO, and stripped out all the frames that didn't have motion in them. You end up with a continuous movie of planes passing through, with none of the boring bits.
This is useful for batch encoding, when you're encoding a lot of different videos at once, because you can get better encoding throughput.
But in my limited experiments a while back, I found the output quality to be slightly worse than with libx264. I don't know if there's a way around it, but I'm not the only one who had that experience.
IIRC they have improved the hardware encoder over the generations of cards, but yes NVENC has worse quality than libx264. NVENC is really meant for running the compression in real-time with minimal performance impact to the system. Basically for recording/streaming games.
> I wonder why ffmpeg doesn't just auto detect this?
Hardware encoding is often less configurable and involves greater trade-offs than using sophisticated software codecs, and don't produce exactly equivalent results even with equivalent parameters. On top of that, systems often have multiple hardware APIs to choose from that often different features.
FFMpeg is a complex command-line tool intended for users who are willing to learn its intricacies, so I'm not sure it makes sense for it to set defaults based on assumptions.
In your snippets, you don't appear to be deinterlacing. If your pre-digitized clips are already deinterlaced, that's fine, but if they're not, you're encoding interlaced material as progressive, and mangling the quality. Try adding a bwdif filter so that your 30i content gets encoded as 60p (which will look more like the original videotapes).
I've gotten pretty good at various bits of ffmpeg over time. Its CLI has a certain logic to it... it's order dependent (not all unix CLIs are).
Lately, I've been playing around with more esoteric functionality. For example, storing raw video straight off a video camera on a fairly slow machine. I built a microscope and it reads frames off the camera at 120FPS in raw video format (YUYV 1280x720) which is voluminous if you save it directly to disk (gigs per minute). Disks are cheap but that seemed wasteful, so I was curious about various close-to-lossless techniques to store the exact images, but compressed quickly. I've noticed that RGB24 conversion in ffmpeg is extremely slow, so instead after playing around with the command line I ended up with:
This reads in raw video- because raw video doesn't have a container, it lacks metadata like "pixel format" and "image size", so I have to provide those. It's order dependent- everything before "-i test.raw" is for decoding the input, and everythign after is for writing the output. I do one tiny pixel format conversion (that ffmpeg can do really fast) and then write the data out in a very, very close to lossless format with a container (I've found .mkv to be the best container in most cases).
Because I hate command lines, I ended up using ffmpeg-python which composes the command line from this:
and then I literally write() my frames into the stdin of that process. I had to limit the number of threads because the machine has 12 cores and uses at least 2 at all times to run the microscope.
I'm still looking for better/faster lossless YUV encoding.
>Its CLI has a certain logic to it... it's order dependent (not all unix CLIs are).
Which is appropriate. A Unix pipeline is dependent on the order of the components, and complex FFMpeg invocations entail doing something analogous.
>I ended up using ffmpeg-python which composes the command line from this
A lot of people like this aesthetic, but doing "fluent" interfaces like this is often considered un-Pythonic. (My understanding is that ffmpeg-python is designed to mirror the command-line order closely.) The preference (reinforced by the design of the standard library and built-in types) is to have strong https://en.wikipedia.org/wiki/Command%E2%80%93query_separati... . By this principle, it would look something more like
where using a separate construction process for the input produces a different runtime type, which also cues the processing code that it needs to read from stdin.
To be honest what I really wanted is more like a programming API or config file than attempting to express complex pipelines and filters in a single command line.
As for what's unpythonic: don't care. My applications has code horrors that even Senior Fellows cannot unsee.
I spent some time with this on my data set, and in my hands I wasn't able to produce results that were convincingly better than libx264, but with slower encodes and larger output files. It's really hard to beat libx264.
This seems to be very forgotten tech. First time I used that was to load NetHack to ram instead of the slow diskette on my Atari. Now I still use it as webcache for work to not bother the database with so many requests.
When I set up the server, the ramdisk didn't have a way of shrinking when space wasn't needed so had to make sure it doesn't eat up all memory when growing unlimited. I bet it's smarter nowadays.
I thought this was going to be a website managed by an experienced user of FFmpeg sharing from their collection of accumulated knowledge, but then was immediately disappointed on the first example I clicked on.
Don’t call two extra tools to do string processing, that is insane. FFprobe is perfectly capable of giving you just the duration (or whatever) on its own:
> I think calling it "insane" is a bit of an exaggeration
Yes, I agree. It was decidedly the wrong word to use and the post would undoubtedly have been better without that part. Unfortunately, the edit window had already passed by the time I reread it.
FFmpeg is one of those tools I need to use so infrequently that he exact syntax never seems to stick. I've resorted to using an LLM to give me the command line I need. The only other tool that I ever had trouble with was 1990s-era MegaCLI from LSI Logic, also something I barely used from one year to the next (but one where you really need to get it right under pressure).
I've been using FFMPEG for 15+ years, and still can't remember almost any commands. LLMs have been amazing for using FFMPEG though. ChatGPT and Claude do wonders with "give me an ffmpeg command that will remux a video into mkv, include subtitle.srt in the file, and I only want it between 0:00:05 and 0:01:00." It produced this in case you were wondering: `ffmpeg -i input.mp4 -i subtitle.srt -ss 00:00:05 -to 00:01:00 -map 0 -map 1 -c copy -c:s mov_text output.mkv`
I wonder how small of an LLM you could develop if you only wanted to target creating ffmpeg commands. Perhaps it could be small enough to be hosted on a static webpage where it is run locally?
Perhaps small enough to include in ffmpeg itself so you can just write commands `ffmpeg do this thing I want`.
Now I say this, it seems like there should already be a shell that is also an LLM where you can mix bits of commands you vaguely remember and natural language a bit like Del Boy speaking French...
Yeah I commented the other day, tongue firmly in cheek, that it's probably worth burning down all the rainforests just so LLMs can tell me the right ffmpeg flags to do what I want.
Don't forget that Gstreamer exists and its command line and documentation make a little bit more sense than ffmpeg because GStreamer is pipeline based and the composition is a little bit more sane. I stopped using ffmpeg entirely and only use GStreamer for intense video work.
Gstreamer can give you more control and has friendlier API's if you're gonna make a pipeline programatically but for one off stuff ffmpeg seems much friendlier to me. For example it has sane x264 defaults while with gst-launch you have to really know what you're doing to get quality x264 encoding
I thought FFmpeg is pipeline based too; graph of filters. Am I missing something? You can set up a complex graph of source, sink and transform filters.
I wish there was some sort of local gui / tool to drag and drop nodes, connect them together, type-check the graph if possible, and it would only show the ffmpeg command to run which you could paste.
Anyone know or anything?
You're right, but gstreamer is a little bit more sane for many use cases. Maybe ffmpeg is more advanced; I am not sure. I find the pieces fit together better with gstreamer.
Readit News
So much like code assistance, they still need a fair amount of baby sitting. A good boost for experienced operators but might suck for beginners.
as in
It will then courteously show you the command it wants to run before you agree to do it.Here is the whole thing, with its two dependent functions, so that people stop writing their own versions of this lol. All it needs is an OPENAI_API_KEY, feel free to modify for other LLMs
EDIT: Moved to a gist: https://gist.github.com/pmarreck/9ce17f7996347dd532f3e20a2a3...
Suggestions welcome- for example I want to add a feature that either just copies it (for further modification) or prepopulates the command line with it somehow (possibly for further modification, or even for skipping the approval step)
I am sure that will never cause any problems.
(honestly, the work you share is very inspiring)
I appreciate the UI choice here. I have yet to do anything with AI (consciously and deliberately, anyway) but this sort of thing is exactly what I imagine as a proper use case.
As a helper and not a replacement, this sounds grand. Like the most epic autocomplete. Because I hate how much time I waste trying to figure out the command line incantation when I already know precisely what I want to do. It’s the weakest part of the command line experience.
And speaking of ffmpeg, or tooling in general, I tend to make notes. After a while you end up with a pretty decent curated reference.
If you have ffmpeg installed and an OpenAI env api key set, it should work out of the box.
Demo: https://image.non.io/1c7a92ef-0917-49ef-9460-6298c7a9116c.we...
i started with avisynth, and it took time for my brain to switch to ffmpeg. i don't know how i could function without ffmpeg at this point
The sense of elation I get when I wonder aloud to my digital friend and they generate what I thought was too much to expect. Well worth the subscription.
- codec: -c:x where x is the type of codec (v: video, a: audio or s:subtitles), followed by its name, like so:
I usually never set the audio codec as the guesses made by ffmpeg, based on output file type, are always right (in my experience), but deciding the video codec is useful, and so is the subtitles codec, as not all containers (file formats) support all codecs; mkv is the most flexible for subtitles codecs.- mapping of streams: -map <input_file>:<stream_type>:<order>, like so:
Map tells ffmpeg what stream from the input files to put in the output file. The first number is the position of the input file in the command, so if we're following the same example as above, '0' would be 'file1.mp4' and '1' would be 'file2.mkv'. The parameter in the middle is the stream type (v for video, a for audio, s for subtitles). The last number is the position of the stream IN THE INPUT FILE (NOT in the output file).The position of the stream in the output file is determined by the position of the map command in the command line, so for example in the command above we are inverting the position of the audio streams (taken from 'file2.mkv'), as audio stream 1 will be in first position in the output file, and audio stream 0 (the first in the second input file) will be in second position in the output file.
This map thing is for me the most counter-intuitive because it's unusual for a CLI to be order-dependent. But, well, it is.
- video filters: -vf
Video filters can be extremely complex and I don't pretend to know how to use them by heart. But one simple video filter that I use often is 'scale', for resizing a video:
width and height can be exact values in pixels, or one of them can be '-1' and then ffmpeg computes it based on the current aspect ratio and the other provided value, like this for example: This doesn't always work because the computed value should be an even integer; if it's not, ffmpeg will raise an error and tell you why; then you can replace the -1 with the nearest even integer (I wonder why it can't do that by itself, but apparently, it can't).And that's about it! ffmpeg options are immense, but this gets me through 90% of my video encoding needs, without looking at a manual or ask an LLM. (The only other options I use often are -ss and -t for start time and duration, to time-crop a video.)
It's not about integer, but some of the sizes need to be even. You can use `-vf scale=320:-2` to ensure that.
Likely because the aspect ratio will no longer be the same. There will either be lost information (cropping), compression/stretching, or black bars, none of which should be default behaviour. Hence, the warning.
I had a problem I'd been thinking about for some time and I thought "Ill have some LLM give me an answer" and it did - it was wrong and didn't work but it got me to thinking about the problem in a slightly different way and my quacks after that got me an exact solution to this problem.
So I'm willing to give the AI more than partial credit.
If the CLI is installed, you can do: gencmd -c ffmpeg extract first 1 minute of video
Or you can just search for the same in the browser page.
Deleted Comment
But really what ffmpeg is missing is an expressive language to describe its operation. Something well-structured, like what jq does for JSON.
Weird thing is I got better performance without "-c:v h264_videotoolbox" on latest Mac update, maybe some performance regression in Sequoia? I don't know. The equivalent flag for my windows machine with Nvidia GPU is "-c:v h264_nvenc" . I wonder why ffmpeg doesn't just auto detect this? I get about 8x performance boost from this. Probably the one time I actually earned my salary at work was when we were about to pay out the nose for more cloud servers with GPU to process video when I noticed the version of ffmpeg that came installed on the machines was compiled without GPU acceleration !
[0] https://gist.githubusercontent.com/nielsbom/c86c504fa5fd61ae...
[1] https://gist.githubusercontent.com/jazzyjackson/bf9282df0a40...
Issue with cloud CPU's is that they don't come with any of the consumer grade CPU built-in hardware video encoders so you'll have to go with the GPU machines that cost so much more. To be honest I haven't tried using HW accel in the cloud to have a proper price comparison, are you saying you did it and it was worth it?
But in my limited experiments a while back, I found the output quality to be slightly worse than with libx264. I don't know if there's a way around it, but I'm not the only one who had that experience.
But for multiple streams or speed requirements, nvenc is the only way to fly.
Hardware encoding is often less configurable and involves greater trade-offs than using sophisticated software codecs, and don't produce exactly equivalent results even with equivalent parameters. On top of that, systems often have multiple hardware APIs to choose from that often different features.
FFMpeg is a complex command-line tool intended for users who are willing to learn its intricacies, so I'm not sure it makes sense for it to set defaults based on assumptions.
Lately, I've been playing around with more esoteric functionality. For example, storing raw video straight off a video camera on a fairly slow machine. I built a microscope and it reads frames off the camera at 120FPS in raw video format (YUYV 1280x720) which is voluminous if you save it directly to disk (gigs per minute). Disks are cheap but that seemed wasteful, so I was curious about various close-to-lossless techniques to store the exact images, but compressed quickly. I've noticed that RGB24 conversion in ffmpeg is extremely slow, so instead after playing around with the command line I ended up with:
This reads in raw video- because raw video doesn't have a container, it lacks metadata like "pixel format" and "image size", so I have to provide those. It's order dependent- everything before "-i test.raw" is for decoding the input, and everythign after is for writing the output. I do one tiny pixel format conversion (that ffmpeg can do really fast) and then write the data out in a very, very close to lossless format with a container (I've found .mkv to be the best container in most cases).Because I hate command lines, I ended up using ffmpeg-python which composes the command line from this:
and then I literally write() my frames into the stdin of that process. I had to limit the number of threads because the machine has 12 cores and uses at least 2 at all times to run the microscope.I'm still looking for better/faster lossless YUV encoding.
Which is appropriate. A Unix pipeline is dependent on the order of the components, and complex FFMpeg invocations entail doing something analogous.
>I ended up using ffmpeg-python which composes the command line from this
A lot of people like this aesthetic, but doing "fluent" interfaces like this is often considered un-Pythonic. (My understanding is that ffmpeg-python is designed to mirror the command-line order closely.) The preference (reinforced by the design of the standard library and built-in types) is to have strong https://en.wikipedia.org/wiki/Command%E2%80%93query_separati... . By this principle, it would look something more like
where using a separate construction process for the input produces a different runtime type, which also cues the processing code that it needs to read from stdin.As for what's unpythonic: don't care. My applications has code horrors that even Senior Fellows cannot unsee.
[0] https://github.com/Ch00k/ffmpy
Look no further: https://trac.ffmpeg.org/wiki/Encode/FFV1
If you are doing processing with intermediate steps you do not want to keep? Ramdisks. Oh yeah. Oh yeah.
When I set up the server, the ramdisk didn't have a way of shrinking when space wasn't needed so had to make sure it doesn't eat up all memory when growing unlimited. I bet it's smarter nowadays.
e.g. When doing a simple copy, progress status messages upgrade to scientific notation.
https://www.ffmpegbyexample.com/examples/l1bilxyl/get_the_du...
Don’t call two extra tools to do string processing, that is insane. FFprobe is perfectly capable of giving you just the duration (or whatever) on its own:
Don’t simply stop at the first thing that works; once it does think to yourself if maybe there is a way to improve it.I like your solution better!
Yes, I agree. It was decidedly the wrong word to use and the post would undoubtedly have been better without that part. Unfortunately, the edit window had already passed by the time I reread it.
https://github.com/fastily/cheatsheet/blob/master/ffmpeg.md
I wonder how small of an LLM you could develop if you only wanted to target creating ffmpeg commands. Perhaps it could be small enough to be hosted on a static webpage where it is run locally?
Now I say this, it seems like there should already be a shell that is also an LLM where you can mix bits of commands you vaguely remember and natural language a bit like Del Boy speaking French...
Relevant XKCD https://xkcd.com/1168/
https://github.com/jdriselvato/FFmpeg-For-Beginners-Ebook