Atypical Uses for Makefiles

Makefiles traditionally center themselves in a build process.  While they have taken a backseat to more modern build utilities, oftentimes they are simply that....in the backseat.  Eclipse, for example, autogenerated makefiles as part of a project build.  Why?  Because when properly done, makefiles are extraordinarily powerful.  Its dependency engine can parse large projects and selectively execute only what needs to be rebuilt.  Done poorly, you're at the mercy of 'make clean; make all' sequences to make sure everything is built up-to-date.

Aside from it's power as a build utility, make can be useful for other purposes as well.  Over the past couple years I've extended my use of make into a few additional areas where I find the dependency engine to prove to be very useful.  This post will touch on a few.

Quick Introduction to Makefiles

Whether you are familiar with makefiles or not, consider taking a peek at this past post.  While it touches on the general syntax and utility of make, in the later sections it shows some atypical uses.  As a means to demonstrate the dependency engine, it shows how Imagemagick can be used to transform images of varying file types.  A daisy-chained dependency chain can be created by converting an image through a series of steps JPG => PNG => GIF => JP2 => XWD.  Perhaps unpractical, but it shows an atypical use of make and it provides a simple and visible execution of the dependency engine.

Poor Man's Parallelism

I'm particularly proud of this usage, to my knowledge no one else has ever recommended this as a use of make.  Most, familiar with make, know that it can be multi-threaded and the dependency engine fires off the specified number of threads to speed up the process.  Given a large text file, make can provide an easy means to parallelize the process.  The trick; split the file into segments, process each segment, then join the results.  In ~30 lines of code, you can greatly improve the execution time for simple parallel processing, all enabled by the sophisticated dependency engine at the core of make.  Take a peek at the details here.

FFmpeg

While I've toyed to a good degree with bash scripts, python scripts and a variety of other shell utilities I've come home to make as the best tool for FFmpeg-based tasks.  Video conversions and/or modifications lend themselves nicely to makefiles.  Suppose you have a large directory full of images and you wish to create a slideshow....this post is a good starting point to showing you how makefiles can make that an easy process.  Want to take 6 hours of raw camera footage and transform it into something worth watchable....this post shows how that can be done.  Want to download a Youtube video and apply a series of transformations to create some completely new content.....this post can give you a head start.

Some of the beauties that were created by these posts;

Shell Script Replacements

Makefiles have become my adhoc replacements for simple shell scripts.  Certainly, they are limited to execute simplistic series of commands and are no substitute for sophisticated needs, but chances are if you have to execute a series of shell commands in a pre-defined order a makefile will scratch that itch.  Sophisticated recipes can become clumsy, and I've found myself authoring some ugly, ugly, ugly recipes, but recently I found a workaround.  Suppose you have a series of 5 tightly-bound commands and find it difficult to define as a recipe, a trick I've found useful is creating a shell script by means of a recipe, then using it in another target.

all: somethingCool.sh

        ${SH} ./$<

somethingCool.sh:

        ${SH} echo "echo 'doing something cool'" > $@

        ${SH} echo "sleep 1" >> $@

        ${SH} chmod a+x $@

clean:

        ${RM} somethingCool.sh

Notice the helper shell script is created by make as a dependency of the all target, used as needed and can be safely removed, recreated when later needed.  Certainly, this falls apart if the creation of the shell script by make is unnecessarily complex, but in many of my projects the only file under version control is the makefile, the rest are created as part of the project and removed by the clean target.  Nice and clean.  Rapid prototyping an alternative script can be done by a new target and once you get the hang of it can prove to be quite powerful.  The make, make run, make clean becomes the holy trinity of rapid prototyping.

I've found adhoc usage reports, grabbing debug logs, grepping for significant events, sorting and tallying event counts really align with raw, events, report targets and recipes.

Makefile Mystique

Greetings to my loyal Russian bots that frequent my blog, contributing to the dozens of weekly views....greetings my virtual comrades!  Let's jump on into it.

Originating in the work by Stuart Feldman at Bell Labs in 1976, the make utility has existed in a number of fashions since and is one of the most common build utilities for *nix based systems.  Despite that however, in my 16+ years of professional software development, authoring or maintaining makefiles takes on a classic game of 'not it!!' seemingly everywhere I work.

Few would argue that the utility lacks flexibility or power, the general complaint is the syntax/semantics are confusing and unmaintainable, one of the primary reasons that popular IDEs synthesize their own makefiles in an attempt to isolate the user from the pain and misery of doing it themselves.  The goal of this post isn't to complain about the utility, but instead to work through a few examples in an attempt to better understand it myself.  While authoring and maintaining a makefile may feel like a prostate exam, it's also likely as necessary as one.  In preparing for this post I referred the documentation here and I invite you to do the same.

Part of make's popularity and power is because of it's implicit rules.  Making use of these rules you'll find that like good liquor, a little goes a long way.  This is evident for example when your project utilizes C/C++.

With a simple source file and relying on implicit make rules the necessary makefile is simplistic;

$ cat main.c
#include <stdio.h>
int main()
{
printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}

A single rule comprises the makefile and provides a simplistic, minimalistic build system.

$ cat Makefile
main: main.o

Each makefile consists of 'rules' taking the form;

     target ... : prerequisites ...

     <tab> recipe

     <tab> ...

Examining the rule we find the target is defined as 'main' with a prerequisite of 'main.o'.  This simply means that in order to create 'main' the 'main.o' file must exist.  The absence of a recipe relies on the implicit rules.  This can be observed by looking at the output when running make as below;

$ make
cc -c -o main.o main.c
cc main.o -o main

The existence of implicit rules comes with some disadvantages, namely it's easy to not understand what is being done for you.  Conceptually, the implicit rule that generates the object files takes the form of the prefix rule below;

$ cat Makefile
main: main.o

.c.o:
    ${CC} ${CPPFLAGS} ${CFLAGS} -c $^

Understanding what is going on allows tailoring the behavior without explicitly defining a rule.  Note the usage of the CPPFLAGS and CFLAGS variables.  Tweaking the original makefile will allow us to add debugging info and specifying an optimization level 3 as below;

$ cat Makefile
CFLAGS += -g -o3
main: main.o

This results in a slight difference when we run make;

$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

The foundation of make is detecting changes to the prerequisites and determining when the targets need to be remade.  This can be observed by re-running make immediately after running make, the result is a notification that "'main' is up to date".  Affecting the main.c file timestamp by modifying the file or simply touching it will result  in the need for the rule to be applied once again.

$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

user@kaylee:~/make.blog/C$ make
make: `main' is up to date.

user@kaylee:~/make.blog/C$ touch main.c

user@kaylee:~/make.blog/C$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

Likely, you've seen this all before, but stay with me I assure you there's more interesting things to come.

Often, it's preferred to have a target that cleans up the directory and allows building from scratch.  The convention is to name such a target clean.  Below is a modified makefile that defines a clean target that simply deletes the executable and the object files.

CFLAGS += -g -o3
main: main.o
clean:
    ${RM} main main.o

Executing 'make clean' will result in deleting main and main.o files.  Adding a file to your project can be accomplished by adding the object file to the prerequisites for main and recipe for the clean target or we can make use of pattern.  We'll do this by explicitly defining each of the C source files in a variable, then perform a list replacement substituting the *.c with *.o extensions to get our object file list.  The object file list can then be used in the target prerequisites and in the clean target recipe.  Adding a file to the SRCS variable rather than duplication in multiple locations.

$ cat Makefile
CFLAGS += -g -o3
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
main: ${OBJS}
clean:
    ${RM} main ${OBJS}

Still however there is duplication, namely the multiple references of main, that can be addressed by a new variable definition.

$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}

clean:
    ${RM} ${PROGS} ${OBJS}

Definitely on the right path, but the addition of a file requires modification to the makefile.  The wildcard expansion demonstrated in the following makefile.  The addition or removal of a file with the .c extension in the current directory will take effect in the wildcard expansion.

$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=${wildcard *.c}
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}
clean:
    ${RM} ${PROGS} ${OBJS}

Let's look at some less typical usages of make which gives us a bit more insight into the creation of targets, prerequisites, and recipes.  Imagemagick is a common utility that we'll be making use in the following examples.

We'll build up the makefile as we go, incorporating what we've learned above.  We'll be satisfying the same objectives using two forms of makefiles; one that makes use of suffix rules, one that makes use of pattern rules.

Let's begin by defining our objectives.  Suppose our project requires taking in a list of JPG files and converting each into a series of other image file formats, namely PNG, GIF, JP2 and XWD files.

Using the suffix rule syntax, the makefile can begin taking the following form;

$ cat Makefile.suffix
.SUFFIXES:
.SUFFIXES: .jpg .png .gif .jp2 .xwd

all: image.xwd

.jpg.png:
    ${SH} convert $< $@

.png.gif:
    ${SH} convert $< $@

.gif.jp2:
    ${SH} convert $< $@

.jp2.xwd:
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd

The all target consists of the default target, the prerequisite of image.xwd.  In other words, make is complete when an up-to-date image.xwd file exists.  How it arrives at it is make magic, more precisely a series of suffix rules.  A series of prefix rules chaining is required to get to the final XWD file, each target we can kick off by explicitly specifying on the command line.  Specifying 'make -f Makefile.suffix image.png' results in firing of the .jpg.png suffix rule.  The suffix rules are chained as each must fire to arrive at the final XWD file.  Running 'make' performs this by stepping through a series of recipes; JPG => PNG => GIF => JP2 => XWD.

$ make -f Makefile.suffix
convert image.jpg image.png
convert image.png image.gif
convert image.gif image.jp2
convert image.jp2 image.xwd
rm image.jp2 image.gif image.png

Notice the final step removes intermediate files which can be preserved which can be prevented by adding ".PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd" line which tells make not to remove the intermediate files with the specified extensions.  The example is a bit fictional but done to demonstrate suffix rules and chaining.  Modifying the source file image.jpg followed by rerunning make will result in converting the new file to each of the alternative file formats.

As is, each prerequisite is generated via suffix rule chaining to completion before moving on to the next prerequisite.  In other words, if you specified image.xwd and image01.xwd the image.xwd would be generated to completion (ie. JPG => PNG => GIF => JP2 => XWD) before moving on to image01.xwd.

Meeting the same goals, let's utilize pattern rules rather than suffix rules which are somewhat dated in use.

$ cat Makefile.pattern
.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd
all: image.xwd

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd



The most noteworthy difference between the target/prerequisites.  The prefix rules define a target, the pattern rule defines a target and prerequisite making the illusion of the rules being reversed.

What if you want to convert each input images to Jpgs before moving on to Gifs before moving on to Jp2s before the Xwds.  This can be done by specifying a wildcard expansion for the source files and using the substitution expression for each of the formats then specifying each of the formats in as prerequisites for the all target, as follows;

$ cat Makefile.pattern

.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd

SRCS=${wildcard *.jpg}

PNGS=$(subst .jpg,.png,${SRCS})

GIFS=$(subst .jpg,.gif,${SRCS})

JP2S=$(subst .jpg,.jp2,${SRCS})

XWDS=$(subst .jpg,.xwd,${SRCS})

all: ${PNGS} ${GIFS} ${JP2S} ${XWDS}

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} ${PNGS} ${GIFS} ${JP2S} ${XWDS}

This way, each format is fully satisfied before moving on to the next.  Perhaps less necessary for image files, more applicable for generating source files.  For example, the Protobuf message compiler allows generation of header/c++ files which also allows interdependencies between message files.  This requires all the message files to be converted to C/H files before firing the compilation, otherwise a source file may reference a header file that hasn't been created yet.

Dasvidaniya my loyal Russian bot army.

Ffmpeg and Make -- A Match Made In Heaven

Although 'make' is most often used to compile, it's use far exceeds that.  'Make's greatest strength is that of it's dependency engine.  By specifying a set of rules, 'make' executes the rules in order to satisfy the dependency.  This is precisely why 'make' is a good match for ffmpeg, the remainder of this blog will hopefully demonstrate that.

Let's start with a simple rule; we need an input video file which can be satisfied by our first make rule:

$ cat Makefile 

input.mp4 :
 ${SH} youtube-dl https://www.youtube.com/watch?v=5xUFQKxdlxE -o $@

By issuing 'make', it will attempt to satisfy resolving the 'input.mp4' target by downloading the specified file from YouTube.

$ make
youtube-dl https://www.youtube.com/watch?v=5xUFQKxdlxE -o input.mp4
[youtube] 5xUFQKxdlxE: Downloading webpage
[youtube] 5xUFQKxdlxE: Downloading video info webpage
[youtube] 5xUFQKxdlxE: Extracting video information
WARNING: unable to extract uploader nickname
[youtube] 5xUFQKxdlxE: Downloading js player vflqFr_Sb
[download] Destination: input.f133.mp4
[download] 100% of 1.13MiB in 00:03
[download] Destination: input.mp4.f140
[download] 100% of 610.47KiB in 00:00
[ffmpeg] Merging formats into "input.mp4"
Deleting original file input.f133.mp4 (pass -k to keep)
Deleting original file input.mp4.f140 (pass -k to keep)

The format of each rule has 3 primary bits: a target, a prerequisite and a command.  In our rule, the target is 'input.mp4', there is no prerequisite, and the command is the YouTube download command.  Repeated execution of make will have no affect, since the file already exists there is no need to re-execute the rule command.

While simple, this rule doesn't really demonstrate the value of 'make', mostly because of the rule's lack of a prerequisite.  Let's look at another:

$ cat Makefile

input.mp4 :
 ${SH} youtube-dl https://www.youtube.com/watch?v=5xUFQKxdlxE -o $@

1x1.mp4: input.mp4
 ${SH} ffmpeg -i $< -vf scale=640:480 -acodec copy $@

Note the 2nd rule has a prerequisite.  A simple way to read the 2nd rule is "When I need the 1x1.mp4 file, I first need the 'input.mp4' file and once I have it use it in the rule command".  Suppose neither the 'input.mp4' nor the '1x1.mp4' file exists and you issue 'make 1x1.mp4';

• Make determines in order to create '1x1.mp4' it needs 'input.mp4' (it's prerequisite)
• Make then finds the 'input.mp4' rule and executes the rule command, downloading the file from YouTube and generates the target file (e.g. 'input.mp4')
• Now that the pre-requisite is resolved, make returns to the '1x1.mp4' rule and executes the rule command, scaling the video file and generating the target

This chaining of dependencies, when done correctly, can execute a complex series of commands in satisfying the final target.  Better yet, with make's intrinsic parallelism it can do so quicker than a sequential script.  That, my friend, is b-e-a-utiful.

Let's look at a complete makefile;

$ cat Makefile 
all: output.mp4

input.mp4 :
 ${SH} youtube-dl https://www.youtube.com/watch?v=5xUFQKxdlxE -o $@

1x1.mp4: input.mp4
 ${SH} ffmpeg -i $< -vf scale=640:480 -acodec copy $@

2x2.mp4: input.mp4
 ${SH} ffmpeg -i $< -i $< -i $< -i $< \
 -filter_complex " \
 nullsrc=size=640x480 [base]; \
 [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft]; \
 [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright]; \
 [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft]; \
 [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright]; \
 [base][upperleft] overlay=shortest=1 [tmp1]; \
 [tmp1][upperright] overlay=shortest=1:x=320 [tmp2]; \
 [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3]; \
 [tmp3][lowerright] overlay=shortest=1:x=320:y=240 \
 " -c:v libx264 -acodec copy $@

4x4.mp4: 2x2.mp4
 ${SH} ffmpeg -i $< -i $< -i $< -i $< \
 -filter_complex " \
 nullsrc=size=640x480 [base]; \
 [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft]; \
 [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright]; \
 [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft]; \
 [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright]; \
 [base][upperleft] overlay=shortest=1 [tmp1]; \
 [tmp1][upperright] overlay=shortest=1:x=320 [tmp2]; \
 [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3]; \
 [tmp3][lowerright] overlay=shortest=1:x=320:y=240 \
 " -c:v libx264 -acodec copy $@
8x8.mp4: 4x4.mp4
 ${SH} ffmpeg -i $< -i $< -i $< -i $< \
 -filter_complex " \
 nullsrc=size=640x480 [base]; \
 [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft]; \
 [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright]; \
 [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft]; \
 [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright]; \
 [base][upperleft] overlay=shortest=1 [tmp1]; \
 [tmp1][upperright] overlay=shortest=1:x=320 [tmp2]; \
 [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3]; \
 [tmp3][lowerright] overlay=shortest=1:x=320:y=240 \
 " -c:v libx264 -acodec copy $@

16x16.mp4: 8x8.mp4
 ${SH} ffmpeg -i $< -i $< -i $< -i $< \
 -filter_complex " \
 nullsrc=size=640x480 [base]; \
 [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft]; \
 [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright]; \
 [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft]; \
 [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright]; \
 [base][upperleft] overlay=shortest=1 [tmp1]; \
 [tmp1][upperright] overlay=shortest=1:x=320 [tmp2]; \
 [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3]; \
 [tmp3][lowerright] overlay=shortest=1:x=320:y=240 \
 " -c:v libx264 -acodec copy $@

clip01.mp4: 1x1.mp4
 ${SH} ffmpeg -i $< -ss 0 -t 5 -acodec copy $@

clip02.mp4: 2x2.mp4
 ${SH} ffmpeg -i $< -ss 5 -t 5 -acodec copy $@

clip03.mp4: 4x4.mp4
 ${SH} ffmpeg -i $< -ss 10 -t 5 -acodec copy $@

clip04.mp4: 8x8.mp4
 ${SH} ffmpeg -i $< -ss 15 -t 5 -acodec copy $@

clip05.mp4: 16x16.mp4
 ${SH} ffmpeg -i $< -ss 20 -t 5 -acodec copy $@

clip06.mp4: 4x4.mp4
 ${SH} ffmpeg -i $< -ss 25 -acodec copy $@

output.mp4: clip01.mp4 clip02.mp4 clip03.mp4 clip04.mp4 clip05.mp4 clip06.mp4
 ${RM} ./files.txt
 ${SH} for f in `echo $^`; do echo "file '$$f'" >> ./files.txt; done
 ${SH} ffmpeg -y -f concat -i ./files.txt -c copy $@
 ${RM} ./files.txt

clean:
 ${RM} *.mp4

The final target 'output.mp4' is defined as a pre-requisite of the first make rule (e.g. all).  Make attempts to execute the final rule, finds a series of pre-requisites (e.g. clip01.mp4...) attempting to satisfy each pre-requisite each of which has pre-requisites of their own.  Following the chain of dependencies you'll come to the YouTube download rule, with no pre-requisites.  Make then executes that rule, generating the input file and works its way back the dependency chain 'til it is capable of generating the final target.

The final video will start as a 1x1 frame, grow to a 2x2 mosaic, proceed to a 4x4 mosaic.....all the way to a 16x16 mosaic.

I've now used make and ffmpeg in a number of video projects and the more I use them, the more I love using them together.  The dependency engine prevents unnecessarily issuing ffmpeg commands when the target file already exists and easily allows incrementally building the series of files necessary for creating the final video.

Happy Encoding!