FFMpeg Transitions -- Part 4

The past few posts have been focusing on dynamic filter transition effects.  This post will bookend this series.  Let's finish up.

Circular Transitions

Let's dust off our high school geometry notes; we can calculate the distance from position (x,y) from the center of the image (W/2, H/2) using the following:

If we compare this distance to a linearly increasing range, based on time we can apply a increasing/decreasing circular boundary.  In this case, we are utilizing max(W,H) to ensure the circular geometry finalizes with the dimensions of the video (width and/or height);

$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(gte(sqrt((X-W/2)*(X-W/2)+(H/2-Y)*(H/2-Y)),(T/1*max(W,H))),A,B)'" circleOpen.mp4

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(lte(sqrt((X-W/2)*(X-W/2)+(H/2-Y)*(H/2-Y)),(max(W,H)-(T/1*max(W,H)))),A,B)'" circleClose.mp4

Expanding Rectangular Window

Compounding on the horizontal and vertical opening effects we can get an expanding window effect;

$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(between(X,(W/2-T/1*W/2),(W/2+T/1*W/2))*between(Y,(H/2-T/1*H/2),(H/2+T/1*H/2)),B,A)'" expandingWindow.mp4

While there are countless other possible effects, this is a decent crack at a representative sampling of such effects.

Whelp, that's it; the knowledge bank is empty.  Hope you found this series of posts of some value, perhaps just a momentary read from your palace of solitude.

Cheers.

FFMpeg Transitions -- Part 3

Our third post concerning video transitions.  Be sure to read our previous posts, I suggest reading them once for knowledge, a second time purely for fun, and periodically thereafter for continued inspiration.

The previous posts focused primarily on the overlay filter.  This post will focus on the blend filter applied dynamically with respect to time or position.

Cross Fade Effect

The blend filter is most typically used in cross fading from one video to another.  Briefly discussed in the first post, the general idea is to provide the filter with two video frames and a fractional value to be applied to each.  For example, a 50/50 split will give an equal weight to each video.  These weights can dynamically change with respect to time.  Cross-fading takes the effect begins by applying a weight of 1.0 to the first video, 0.0 to the second video, then linearly decreasing the weight of the first video while simultaneously increasing the weight of the second.  Finally, ending with a weight of 0.0 for the first video, 1.0 for the second.  Easy, Peasy.

Let's take a look at the full filter example;

$ ffmpeg -i image01.mp4 -i image02.mp4  -filter_complex "[0:v][1:v]blend=all_expr='A*(1-min(T/1,1))+B*(min(T/1,1))'" blend.mp4

Note; like past posts, the denominator in the (T/1) implies that the transition will take 1 second.  Playing with that value will speed up or slow down the morphing.

Location-Based Blend

The previous blend filter is applied uniformly to the entire frame.  Using conditionals and (x,y) locations we can base the blending factors on position.  Consider a simple blend condition where the left-most half of the image uses the first video, the right-most half uses the second video.  Conceptually, the blend would look something like this;

'if(lte(X,H/2),A,B)'

When X is less than the middle of the video frame vertical center, apply A, otherwise B. 

This example, not particularly useful, but imagine applying a moving range rather than simply the middle of the vertical axis.  As that range moved from bottom-to-top the effect would emulate a rising curtain;

$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(lte(Y,(H-T/1*H)),A,B)'" curtainUp.mp4

Notice rather than (H/2), the range starts at H and progresses to 0 linearly during the 1 second duration.

Similarly, you can perform a curtain down effect like this;

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(gte(Y,(T/1*H)),A,B)'" curtainDown.mp4

Using the center point as a start or end point and mirroring the effect on each 1/2 of the frame can open the door to other effects, like;

$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(between(X,(W/2-T/1*W/2),(W/2+T/1*W/2)),B,A)'" verticalOpen.mp4

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]blend=all_expr='if(between(Y,(H/2-T/1*H/2),(H/2+T/1*H/2)),B,A)'" horizontalOpen.mp4

FFMpeg Transitions -- Part 2

This post continues on from last weeks post.  Be sure to read the past post to establish a foundation for the content here.

Our journey will continue to focus on creating scene transitions of the form;

In this post we will focus on wipe transitions, conducted by applying an overlay to a dynamic position.  

We are going to focus on creating wipe transitions; up, down, left, right and the diagonals.  

In general, these wipe transitions are done by applying an overlay on a time-based position.  We will begin by focusing on left/right/up/down which can then be applied to create the diagonals.

Wipe Right

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y=0[out]" -map "[out]" -y wipeRight.mp4

Wipe Left

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='max(0,W-(t/1)*W)':y=0[out]" -map "[out]" -y wipeLeft.mp4

Wipe Down

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='0':y='min(0,-H+(t/1)*H)'[out]" -map "[out]" -y wipeDown.mp4

Wipe Up

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='0':y='max(0,H-(t/1)*H)'[out]" -map "[out]" -y wipeUp.mp4

Diagonals

Now that we have the equations to manipulate the X or Y locations, the diagonals are simply created by applying position changes to X and Y.

$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y='min(0,-H+(t/1)*H)'[out]" -map "[out]" -y wipeRightDown.mp4
$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]overlay=x='max(0,W-(t/1)*W)':y='min(0,-H+(t/1)*H)'[out]" -map "[out]" -y wipeLeftDown.mp4
$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]overlay=x='max(0,W-(t/1)*W)':y='max(0,H-(t/1)*H)'[out]" -map "[out]" -y wipeLeftUp.mp4
$ ffmpeg -i image02.mp4 -i image01.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y='max(0,H-(t/1)*H)'[out]" -map "[out]" -y wipeRightUp.mp4

Next week we will spend some time on the blend filter.

Cheers

FFMpeg Transitions -- Part 1

As I look back at the content of this blog a good chunk of the posts revolve around FFMpeg, roughly 1/4 of them to-date.  I've been playing with FFMpeg for years, mostly out of curiosity as I lack the creativity or ambition to create any significant media content.  I think I'm drawn to this topic for a few reasons; 1) I had a need to use it out of necessity some years ago to transcode videos for image detection software, 2) it's extremely powerful and is a tribute to the sophisticated command-line utilities that Unix emphasizes, 3) despite it's power it's documentation is IMO lacking so I feel obligated to a degree to document what I discover.

This form of topic fits well into my goal of this blog as well, short consumable content that can be authored in spare hours of the evening.  I'm intending departing from this format temporarily however and author a multi-part series on video transition effects to give it sufficient attention and hopefully shed some light on the underlying mechanics to create these effects.

Let's skip to the end temporarily and look at the end result we will be shooting for.  We will use two input videos of the legendary duo and perform transitions between the two using various transition effects.  Despite these video being of a static image, the same effects can be used for dynamic videos.

So, that's our bogie, we will tackle each effect one-by-one in this and future posts.

Fundamentals

The effects all build on some FFMpeg and mathematical fundamentals, we will try to set the stage in this section.

Two key FFMpeg filters enable these transitions; overlay and blend.  

The overlay filter allows placing a video or image atop another and can look something like this;

We create this effect by instructing FFMpeg to first render the first image, then render the second image pinned at position (x,y).  Altering the start (x,y) position of the 2nd image can give the appearance of the image moving.

The blend filter allows blending two images or videos into a output video giving the appearance of one video melting and morphing into the other; looking something like this;

This effect is created by instructing FFMpeg to render each pixel as a composite of both input videos.  For example, we can tell FFMpeg to apply 50% of the first video and 50% of the second video to get something that looks like the above.  Similar in nature to the effect you'd get if you printed out the frames on transparencies (e.g. clear plastic), lined atop one another and held up to a strong light.

These two filters will be the basis of the future sections so we'll spend a bit more time with them here.

We need to start with two input videos, they need to be of the same dimensions and resolutions and helpful if they are of the same duration.

image01.mp4

image02.mp4

An example usage of the overlay filter takes the form:

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='W/4':y=0[out]" -map "[out]" -y example01.mp4

In short, the above command says; render a image01.mp4 frame, then overlay the subsequent image02.mp4 frame at (x=width/4, y=0).

An example of the blend filter takes the form:

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]blend=all_expr='A*(0.5)+B*(0.5)'" example02.mp4

In short, the above command says to blend 50% of the 1st video frame with 50% of the 2nd video frame; looking like this:

Future video transitions will make heavy use of these filters, but instead of using static values they will/may be based on time and/or position making for more sophisticated effects.  The FFMpeg filters can make use of internal filter variables and functions.  Let's consider ones that we may make use of:

Variables

W -- width of video

H -- height of video

X -- pixel x location

Y -- pixel y location

t/T -- time

Note: variables are filter specific; for example the time variable differs between the blend and overlay filter

Functions

between(i,min,max) -- return 1 if min <= i <= max

lte(i,j) -- less than equal to

gte(i,j) -- greater than equal to

sqrt(i) -- square root

Time-Based Expressions

Applying time-based expressions open a door to a number of sophisticated effects.  Suppose we use the following equation:

The T variable is an internally available variable, T0 is the duration of the video.  The result will be a fraction between 0.0 and 1.0.  This concept can be used to apply dynamic filters.  

For example, a filter x position can be applied to a filter using the following equation.  The result; x will begin at 0, ending at the width of the video, linearly traversing across the screen in T0 seconds.

Wipe Right Transition

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y=0[out]" -map "[out]" -y wipeRight.mp4

In the above example we are making use of the previous time-based location equation with a minor tweak; specifically the use of the min() function.

We start the overlay far left, outside the rendering dimensions; specifically -W, then slowly move the x start location right-wise.  The min() function is used to prevent the x overlay start position to exceed 0, snapping the final location directly over-top the lower image.  Also of note, T0 in this case is defined as 1 second.  This means the wipe effect completes within 1 sec.  Tweaking that value will speed up or slow down the motion.

Wipe Left Transition

Let's reverse the effect, moving the overlay from right-to-left;

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='max(0,W-(t/1)*W)':y=0[out]" -map "[out]" -y wipeLeft.mp4

Similarly natured equation as above, only starting at W and moving left.  The max() halts the position at its final location.

A reasonable stopping point, we will put a pin in it for now and follow on with additional effects in later posts.

Cheers.

FFMpeg Blending

FFMpeg can blend two input files which allows for a gradual transition from one to the other.  This post will demonstrate how this can be done with two image files or video files.  Let's get started.

Blending Images

Let's snag a couple images to show how blending can be used in transitioning from one to the other.  More specifically, we're grabbing one image and cropping them to form two images.

$ wget https://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/Bush-Arnie-morph.jpg/300px-Bush-Arnie-morph.jpg -O src.jpg


$ convert -crop 100x300+0+0 src.jpg s1.jpg

$ convert -crop 100x300+200+0 src.jpg s2.jpg

Next, let's generate a 5 second video using the two source images.  Notice the A/B coefficients are relative to time finally capping at 1 when there remains 1 second of video left.  The blending coefficients define how much of one or the other video frame is blended.

$ ffmpeg -loop 1 -i s1.jpg -loop 1 -i s2.jpg -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,$((5-1))),1,T/$((5-1))))+B*(1-(if(gte(T,$((5-1))),1,T/$((5-1)))))'" -t 5 foo.mp4

The result;

Blending Videos

We can provide a similar effect on videos.  Let's grab a source video and split it into two videos that we'll blend between.

$ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' -o src.mp4 https://www.youtube.com/watch?v=-sUXMzkh-jI
$ ffmpeg -i src.mp4 -ss 10 -t 5 -an vid1.mp4
$ ffmpeg -i src.mp4 -ss 60 -t 5 -an vid2.mp4

Let's blend the videos in the same manner as our images;

$ ffmpeg -i vid1.mp4 -i vid2.mp4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,$((5-1))),1,T/$((5-1))))+B*(1-(if(gte(T,$((5-1))),1,T/$((5-1)))))'" foo00.mp4

There you are, the most fun you can have while blending short of making margaritas.

Cheers.