RealTime Adjustable FFmpeg Filters -- Part 1

FFmpeg allows dynamic filter adjustments, in past posts we've demonstrated how filters can be adjusted with respect to frame number and/or timestamp.  A more unconventional use allows real-time adjustments of filters by an independent process.  FFmpeg allows the ability of sending filter parameters by means of the 'zmqsend' utility.  This would allow modifying filter parameters during the processing pipeline.

Two things before we jump into the details; 1) only a select list of filters allow dynamic modifications, and 2) this is an non-standard feature which must be enabled as part of building FFmpeg from source.

Building with ZMQ Support

The zmqsend utility is used for sending filter parameters into the processing pipeline.  This utility utilizes ZMQ libraries, to enable you need to install the zmq libraries followed by enabling zmq support when building FFmpeg;

Install ZMQ Libraries

$ sudo apt-get install -y libzmq3 libzmq3-dev

Build FFmpeg

Building FFmpeg from source is done via the autotools process: configure, make, make install.  Typically I build utilizing the following flags, but the purposes of this post the '--enable-libzmq' serves the trick;

$ ./configure --enable-libx264 --enable-nonfree --enable-gpl --enable-libfreetype --enable-libmp3lame --enable-libzmq

Simple Example

FFmpeg documentation provides a simple example; https://ffmpeg.org/ffmpeg-filters.html#zmq_002c-azmq

This simple example does little more than demonstrate sending in parameters.  Let's focus on dynamically changing a crop filter instead.

Suppose you have a video presentation with a moving speaker and you wish to crop to the presenter while in motion.  Doing so can't readily be done by a time/frame-based equation and opens a door for using real-time parameter adjustments.

The following video snippet is that of a presentation I did some months ago.  The camera captures the 'stage'.

Let's say we want to crop a 640x480 region, then pad it to 720x480 we would specify a filter similar to this: -filter_complex "crop=640:480:0:0,pad=720:480:(ow-iw)/2:(oh-ih)/2"

The filter graph of this takes the form:

Of particular interest, the 'Parsed_crop_1' filter parameter name is needed for our purposes.  The x and y paramets specify the upper-left position of the crop window; refer https://ffmpeg.org/ffmpeg-filters.html#crop

So the idea is to begin the processing of the video file, then send in Parsed_crop_1 x and y parameter values.

$ ffmpeg -y -i input.mp4 -filter_complex "zmq,crop=640:480:0:0,pad=720:480:(ow-iw)/2:(oh-ih)/2" -acodec copy crop.mp4

The following command would update the x location to 100, albeit an jump motion;
$ echo Parsed_crop_1 x 100 | zmqsend

A slightly more complex motion would be to slowly adjust (x,y) positions in a linear manner;
$ cat mover.sh 
#!/bin/bash
for n in `seq 1 375`; do
  echo $n
  echo Parsed_crop_1 x $n | zmqsend
  echo Parsed_crop_1 y $n | zmqsend
  sleep 0.1
done;

The result would take the form of this:

This example again is not very useful, consider it a building block to a more sophisticated example which will be the subject of a future post.

Cheers.