Flaunt Your Font Want

 

Last night, my wife and I rewatched Jurassic Park, later that night I drifted into a blissful slumber.  My unconscious mind wandered vastly into the future where a futuristic archaeologist chipping away an amber-encased USB device, having been perfectly preserved for 3000 years.  The contents, all my life's work; software projects, audio, video and including a copy of this blog post.  The archaeologist releasing the USB device from it's amber container goes deep into the cellar of cellars to retrieve a primitive device capable of reading this archaic media.  Amazingly, all my life's work comes to life once again, a group of interested historians eagerly scan the work only to soon show disappointment.  "Arial?  A dull, Arial font?  Clearly this work is of no substance." as they close the computer, remove the USB device and promptly toss it into the nearest waste basket.  A record of my life for the 'after people' only to be discarded like common trash do to lack of creativity in the font game.  No, no I say, we'll step up our font game now and forever for myself, for you as well as for the after people.

I've never really had much of a need for non-system fonts.  Most user interfaces I've developed were for engineers who cared little for cosmetics, have yet to ever work with a Ux designer nor a graphics designer in any amount of detail.  The default system fonts as a result have always been 'good enough' as a result.  In fact, font files have always been a mystery to me, never spending any time thinking about them at all.

But not today, today we're gonna spend a little time understanding what they are and if you're like me you may even be surprised.  After decades of not caring about fonts, why today?  Why now?  Whelp, as I've been creating images and video content it becomes valuable to provide some form of text overlay.  I've always used the system fonts, but suppose you are tweaking a military-style video you'd likely want to add a military-style font.  That opens the door to understanding where to find and install custom fonts to suit all your creative needs.

 

vs.

So the remainder of this post will touch on how to download and use custom fonts in images, but they can easily be used in video, slideshows,....

For years, font files were simply 'magic', but let's look at one briefly.  Smarter folks may already know this, but a font ttf file can be viewed as an image; behold, FreeeSerif.ttf

$ display /usr/share/fonts/truetype/freefont/FreeSerif.ttf

The image shows the alphabet, numerics and special characters.  It then demonstrates a variety of the fonts at differing font sizes.  This can give you a feeling as to whether the font strums your creative chord.

So, where does our search begin when seeking the elusive font?  Here is a good start, and the source we'll be using in the rest of the post: www.1001freefonts.com

Assuming that we will download, extract and install the fonts to a local directory, the process takes the form:

$ wget -P TEMP https://www.1001freefonts.com/d/4018/eraser-dust.zip

$ cd TEMP; 

$ unzip -o eraser-dust.zip; 

$ mv EraserDust.ttf ../EraserDust.ttf

$ rm -rf TEMP;

The result, a brand new shiny EraserDust.ttf font file just waiting for us to use.

Using ImageMagick, we can create an image using the new font with some meaningful text:

$ convert -background None -fill white -font ./EraserDust.ttf -pointsize 96 label:"EraserDust.ttf"  -rotate -15 foo.png

$ display foo.png

Now, you can make your own thumbnails, augment video, update documents with your newly acquired custom font.  Imagine how much more fun your engineering documentation can be using a whimsical font style :thumbsup

With our new found powers, let's download and install a series of fonts, use Imagemagick to create an image showcasing the new font, then slap them all into a video displaying each font for 1 second.  I find a makefile suites this purpose pretty well, the chained commands for the video mp4 target is a bit clumsy as a series of shell commands, but it's good enough to convey the steps.

$ cat -n Makefile 

     1 all: background.png video.mp4

     2

     3 video.mp4: FridayStroke.otf TopSecret.ttf BostonTraffic.ttf Camouflage.ttf DrippingMarker.ttf EraserDust.ttf SnackerComic.ttf

     4 ${SH} i=0; for f in `echo $^`; do \

     5 echo $$i; \

     6 echo $$f; convert -background None -fill white -font ./$$f -pointsize 96 label:"$$f"  -rotate -15 overlay.png; \

     7 composite -gravity center overlay.png background.png frame-$$i.png ; \

     8 i=$$((i+1)); \

     9 done

    10 ${SH} ffmpeg -r 1 -i frame-%d.png $@

    11

    12 background.webp:

    13 ${SH} wget -O $@ https://i.vimeocdn.com/video/521265093.webp?mw=1000

    14

    15 background.png: background.webp

    16 ${SH} ffmpeg -i $< -vframes 1 $@

    17

    18 SnackerComic.ttf:

    19 ${SH} wget https://www.1001freefonts.com/d/4631/snacker-comic.zip -P TEMP

    20 ${SH} cd TEMP; unzip -o *.zip; mv *.ttf ../$@

    21 ${RM} -rf TEMP

    22

    23 FridayStroke.otf:

    24 ${SH} wget https://www.1001freefonts.com/d/26677/the-friday-stroke.zip -P TEMP

    25 ${SH} cd TEMP; unzip -o the-friday-stroke.zip; mv The\ Friday\ Stroke\ Font\ by\ 7NTypes.otf ../$@

    26 ${RM} -rf TEMP

    27

    28 TopSecret.ttf:

    29 ${SH} wget https://www.1001freefonts.com/d/7052/top-secret.zip -P TEMP

    30 ${SH} cd TEMP; unzip -o top-secret.zip; mv Top\ Secret.ttf ../$@

    31 ${RM} -rf TEMP

    32

    33 BostonTraffic.ttf:

    34 ${SH} wget https://www.1001freefonts.com/d/3269/boston-traffic.zip -P TEMP

    35 ${SH} cd TEMP; unzip -o boston-traffic.zip; mv boston.ttf ../$@

    36 ${RM} -rf TEMP

    37

    38 Camouflage.ttf:

    39 ${SH} wget https://www.1001freefonts.com/d/12584/camouflage.zip -P TEMP

    40 ${SH} cd TEMP; unzip -o camouflage.zip; mv CamouflageW.ttf ../$@

    41 ${RM} -rf TEMP

    42

    43 DrippingMarker.ttf:

    44 ${SH} wget https://www.1001freefonts.com/d/14693/a-dripping-marker.zip -P TEMP

    45 ${SH} cd TEMP; unzip -o a-dripping-marker.zip; mv adrip1.ttf ../$@

    46 ${RM} -rf TEMP

    47

    48 EraserDust.ttf:

    49 ${SH} wget -P TEMP https://www.1001freefonts.com/d/4018/eraser-dust.zip

    50 ${SH} cd TEMP; unzip -o eraser-dust.zip; mv EraserDust.ttf ../$@

    51 ${RM} -rf TEMP

    52

    53 clean:

    54 ${RM} *.otf *.ttf *.png *.mp4 *.webp

    55

Lines 18-51 has a series of font targets, each downloading the font, extracting it in a disposable directory, moving the TTF file, then cleaning up.  

Lines 3-10 do the meat of the work, first defining each font file as a dependency (the make engine satisfying the dependency by executing the necessary recipe), then create a transparent overlay with the font name, slapping that over a background image and repeating for font.  

Finally, line 10 converts the images into a video slide show, 1 second per slide.

Now your amber-encased creative works will have some flair.

Makefile Mystique

Greetings to my loyal Russian bots that frequent my blog, contributing to the dozens of weekly views....greetings my virtual comrades!  Let's jump on into it.

Originating in the work by Stuart Feldman at Bell Labs in 1976, the make utility has existed in a number of fashions since and is one of the most common build utilities for *nix based systems.  Despite that however, in my 16+ years of professional software development, authoring or maintaining makefiles takes on a classic game of 'not it!!' seemingly everywhere I work.

Few would argue that the utility lacks flexibility or power, the general complaint is the syntax/semantics are confusing and unmaintainable, one of the primary reasons that popular IDEs synthesize their own makefiles in an attempt to isolate the user from the pain and misery of doing it themselves.  The goal of this post isn't to complain about the utility, but instead to work through a few examples in an attempt to better understand it myself.  While authoring and maintaining a makefile may feel like a prostate exam, it's also likely as necessary as one.  In preparing for this post I referred the documentation here and I invite you to do the same.

Part of make's popularity and power is because of it's implicit rules.  Making use of these rules you'll find that like good liquor, a little goes a long way.  This is evident for example when your project utilizes C/C++.

With a simple source file and relying on implicit make rules the necessary makefile is simplistic;

$ cat main.c
#include <stdio.h>
int main()
{
printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}

A single rule comprises the makefile and provides a simplistic, minimalistic build system.

$ cat Makefile
main: main.o

Each makefile consists of 'rules' taking the form;

     target ... : prerequisites ...

     <tab> recipe

     <tab> ...

Examining the rule we find the target is defined as 'main' with a prerequisite of 'main.o'.  This simply means that in order to create 'main' the 'main.o' file must exist.  The absence of a recipe relies on the implicit rules.  This can be observed by looking at the output when running make as below;

$ make
cc -c -o main.o main.c
cc main.o -o main

The existence of implicit rules comes with some disadvantages, namely it's easy to not understand what is being done for you.  Conceptually, the implicit rule that generates the object files takes the form of the prefix rule below;

$ cat Makefile
main: main.o

.c.o:
    ${CC} ${CPPFLAGS} ${CFLAGS} -c $^

Understanding what is going on allows tailoring the behavior without explicitly defining a rule.  Note the usage of the CPPFLAGS and CFLAGS variables.  Tweaking the original makefile will allow us to add debugging info and specifying an optimization level 3 as below;

$ cat Makefile
CFLAGS += -g -o3
main: main.o

This results in a slight difference when we run make;

$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

The foundation of make is detecting changes to the prerequisites and determining when the targets need to be remade.  This can be observed by re-running make immediately after running make, the result is a notification that "'main' is up to date".  Affecting the main.c file timestamp by modifying the file or simply touching it will result  in the need for the rule to be applied once again.

$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

user@kaylee:~/make.blog/C$ make
make: `main' is up to date.

user@kaylee:~/make.blog/C$ touch main.c

user@kaylee:~/make.blog/C$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

Likely, you've seen this all before, but stay with me I assure you there's more interesting things to come.

Often, it's preferred to have a target that cleans up the directory and allows building from scratch.  The convention is to name such a target clean.  Below is a modified makefile that defines a clean target that simply deletes the executable and the object files.

CFLAGS += -g -o3
main: main.o
clean:
    ${RM} main main.o

Executing 'make clean' will result in deleting main and main.o files.  Adding a file to your project can be accomplished by adding the object file to the prerequisites for main and recipe for the clean target or we can make use of pattern.  We'll do this by explicitly defining each of the C source files in a variable, then perform a list replacement substituting the *.c with *.o extensions to get our object file list.  The object file list can then be used in the target prerequisites and in the clean target recipe.  Adding a file to the SRCS variable rather than duplication in multiple locations.

$ cat Makefile
CFLAGS += -g -o3
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
main: ${OBJS}
clean:
    ${RM} main ${OBJS}

Still however there is duplication, namely the multiple references of main, that can be addressed by a new variable definition.

$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}

clean:
    ${RM} ${PROGS} ${OBJS}

Definitely on the right path, but the addition of a file requires modification to the makefile.  The wildcard expansion demonstrated in the following makefile.  The addition or removal of a file with the .c extension in the current directory will take effect in the wildcard expansion.

$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=${wildcard *.c}
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}
clean:
    ${RM} ${PROGS} ${OBJS}

Let's look at some less typical usages of make which gives us a bit more insight into the creation of targets, prerequisites, and recipes.  Imagemagick is a common utility that we'll be making use in the following examples.

We'll build up the makefile as we go, incorporating what we've learned above.  We'll be satisfying the same objectives using two forms of makefiles; one that makes use of suffix rules, one that makes use of pattern rules.

Let's begin by defining our objectives.  Suppose our project requires taking in a list of JPG files and converting each into a series of other image file formats, namely PNG, GIF, JP2 and XWD files.

Using the suffix rule syntax, the makefile can begin taking the following form;

$ cat Makefile.suffix
.SUFFIXES:
.SUFFIXES: .jpg .png .gif .jp2 .xwd

all: image.xwd

.jpg.png:
    ${SH} convert $< $@

.png.gif:
    ${SH} convert $< $@

.gif.jp2:
    ${SH} convert $< $@

.jp2.xwd:
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd

The all target consists of the default target, the prerequisite of image.xwd.  In other words, make is complete when an up-to-date image.xwd file exists.  How it arrives at it is make magic, more precisely a series of suffix rules.  A series of prefix rules chaining is required to get to the final XWD file, each target we can kick off by explicitly specifying on the command line.  Specifying 'make -f Makefile.suffix image.png' results in firing of the .jpg.png suffix rule.  The suffix rules are chained as each must fire to arrive at the final XWD file.  Running 'make' performs this by stepping through a series of recipes; JPG => PNG => GIF => JP2 => XWD.

$ make -f Makefile.suffix
convert image.jpg image.png
convert image.png image.gif
convert image.gif image.jp2
convert image.jp2 image.xwd
rm image.jp2 image.gif image.png

Notice the final step removes intermediate files which can be preserved which can be prevented by adding ".PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd" line which tells make not to remove the intermediate files with the specified extensions.  The example is a bit fictional but done to demonstrate suffix rules and chaining.  Modifying the source file image.jpg followed by rerunning make will result in converting the new file to each of the alternative file formats.

As is, each prerequisite is generated via suffix rule chaining to completion before moving on to the next prerequisite.  In other words, if you specified image.xwd and image01.xwd the image.xwd would be generated to completion (ie. JPG => PNG => GIF => JP2 => XWD) before moving on to image01.xwd.

Meeting the same goals, let's utilize pattern rules rather than suffix rules which are somewhat dated in use.

$ cat Makefile.pattern
.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd
all: image.xwd

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd



The most noteworthy difference between the target/prerequisites.  The prefix rules define a target, the pattern rule defines a target and prerequisite making the illusion of the rules being reversed.

What if you want to convert each input images to Jpgs before moving on to Gifs before moving on to Jp2s before the Xwds.  This can be done by specifying a wildcard expansion for the source files and using the substitution expression for each of the formats then specifying each of the formats in as prerequisites for the all target, as follows;

$ cat Makefile.pattern

.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd

SRCS=${wildcard *.jpg}

PNGS=$(subst .jpg,.png,${SRCS})

GIFS=$(subst .jpg,.gif,${SRCS})

JP2S=$(subst .jpg,.jp2,${SRCS})

XWDS=$(subst .jpg,.xwd,${SRCS})

all: ${PNGS} ${GIFS} ${JP2S} ${XWDS}

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} ${PNGS} ${GIFS} ${JP2S} ${XWDS}

This way, each format is fully satisfied before moving on to the next.  Perhaps less necessary for image files, more applicable for generating source files.  For example, the Protobuf message compiler allows generation of header/c++ files which also allows interdependencies between message files.  This requires all the message files to be converted to C/H files before firing the compilation, otherwise a source file may reference a header file that hasn't been created yet.

Dasvidaniya my loyal Russian bot army.

Video Transition Effects w/FFmpeg

Whether it be Instagram, Twitter, Snapchat, a vlog or YouTube, unless your uploading a uncut unedited video you're likely to incorporate some form of video transition(s). Scene changes, representing passage of time, representation of ambiguity can all be represented by common forms of video transitions. Investing a little effort in post production can transform a good video into a great one, a lengthy story into a shorter one and a long-winded story into a compelling quicker one.

FFMpeg can readily support some of the most common video transitions, and in this post we'll touch on some common transitions and how they can be done effectively with FFMpeg.

I hesitate referring to myself as an amateur videoographer, but over the years I've spent a good deal of time toying with videos. Recently, I've been researching the art/science to choosing the 'right' video transition, rather than simply arbitrarily choosing one from a hat. Recently I came upon this post which spends a bit of time explaining when/why to use particular video transitions. I'll touch on some of the key points and demonstrate simple examples on how to perform them with FFMpeg.

Fade

A fade-in consists of starting with a solid color (e.g. white, black) and smoothly morphing into the video shot.  Similarly, a fade-out smoothly morphs from the vido shot to a solid color.  Often, a fade-in and fade-out are coupled, fading out of the current scene followed by fading-in to the new scene.

This post focuses on providing overviews, more details for this form of transition can be found in this previous post.

Fade In

Often, a video will begin with a fade-in, much like most common movies.  This technique allows to slowly introduce the viewer into a new environment or scene.  Most often, fading in from black, which can be demonstrated by the following example;

Given a 24fps video, we can slowly fade in on the first 5 seconds like this:

$ ffmpeg -y -i video.mp4 -vf fade=in:0:120 -acodec copy fadeIn-5sec.mp4

The fade filter takes in frames, rather than time, so it requires knowledge of the video frame rate to convert time to frame range.

Fade Out

Fading out can signify the passing of time, a sense of completion, or represent ambiguity.  Let's look into an example.

Given a 24fps video, we can slowly fade out on the last 5 seconds of the video.  A 10 minute video at 24 fps (24*60*10 = 14400), 5 seconds (5*24=120) would be accomplished by the following;

$ ffmpeg -y -i video.mp4 -vf fade=out:14280:14400 -acodec copy fadeOut-5sec.mp4

Dissolve

Given two disjoint scenes, another common technique is to slowly transform directly from one scene into another.  This can represent the passage of time, bur more commonly represents moving from one location to another.  Consider an interview taking place in an office environment followed by continuing the interview in the street.  A rapid jump from office to street can be abrupt to the viewer, often remedied by dissolving from the first scene into the other.  Quick dissolves can represent a quick passage of time, while slower dissolves can represent the passage of months or years.

This effect is sometimes referred to as a crossfade, given two source videos we can dissolve or crossfade from the first to the second as follows:

$ ffmpeg -i image01.mp4 -i image02.mp4  -filter_complex "[0:v][1:v]blend=all_expr='A*(1-min(T/1,1))+B*(min(T/1,1))'" blend.mp4

A more comprehensive discussion with examples can be found here.

Wipes

Wipes can convey the existence of concurrent story lines, swapping one for another.  These can take on a variety of forms, left-to-right, right-to-left, upward, downward or any form of diagonals. 

An example of a wipe-right can take the form:

$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y=0[out]" -map "[out]" -y wipeRight.mp4

Like the fade-in, this filter doesn't use time, rather the units are pixels so authoring a 1 sec wipe will take some math, but if you're flexible in the wipe duration you can simply play with the filter arguments.

This past post describes a variety of wipes in more detail.

Zoom

Zooming in/out can communicate to the viewer 'pay attention to this', zooming in focuses the viewers attention to the subject of the zoom.  Zooming out can communicate attention from a particular subject out to the world at large, often done to represent an ending of the film....attention from a particular person out to the world a means of fading away from the subject.

Zooming is a bit more complicated than some of the other transitions, a simple example takes the form:

$ ffmpeg -y -i target.mp4 -vf "scale=iw*2.0:ih*2.0,zoompan=z='min(max(zoom,pzoom)+0.05,5.0)':d=1:x='560*2.0-(560*2.0/zoom)':y='400*2.0-(400*2.0/zoom)'" -an output.mp4

A more detailed explanation can be found here.

These are a few examples of common video transitions, I may spend some time on more sophisticated ones in the future. 

Now, go make your good videos great.

Limiting Network Bandwidth for Testing

Occasionally it's necessary to determine the affects of reduced network bandwidth. Authoring software that works for a networked system backed by Gigabit Ethernet as well as DSL requires testing on both forms of systems.  The alternative is to test on the most capable networking system and then test lower-bandwidth systems on the same network by artificially limiting the bandwidth.

Consulting the all-knowing Google provided a couple general options: trickle and wondershaper.  While there are other utilities, I mainly focused on these two.

Trickle is a 'lightweight user bandwidth shaper', it's particular value is the ability to limit processes independently and it doesn't require superuser privileges.

$ sudo apt-get install trickle

You can limit uploads and downloads to 100 KB/s for a command such as;

$ trickle -u 100 -d 100 wget http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_stereo.ogg

Independently, you could specify alternative limits for another command.  Trickle is limited to TCP sockets so if you need to limit bandwidth for UDP sockets you'll need to look for an alternative.  Applying system-wide bandwidth, TCP in this case, supposedly is supported via trickled daemon utility but after numerous hours I gave up on getting it to work.  Regardless, I abandoned it's use not only because I couldn't get it to work but also it's limitation to TCP and applying the daemon and client command required prepending 'trickle' to the command line.  This would disqualify applying the network bandwidth constraints to network services like sshd and such without modifying each service starting the process.

Seeking a system-wide bandwidth traffic shaper brought me back to Wondershaper once again.  My first experience with wondershaper I found it didn't always work, this round I tested the usage 1-10 Mbps and I quickly found that it limited the bandwidth like expected until reaching 10 Mbps where it jumped to realized bandwidth of 40 Mbps.  Head-scratcher; working fine for bandwidths under 10Mbps but broke at >= 10Mbps.  Consulting Google again brought me to a fix proposed by 'buzzy'; https://github.com/magnific0/wondershaper/issues/2.  Modifying the wondershaper script as 'buzzy' indicated resolved the issue and I could consistently specify bandwidths 1-40 Mbps and observe realized bandwidths accordingly.

$ cat go

#!/bin/bash



for i in `seq 1 50`; do

  echo "$i Mbps"

  K=`expr $i \* 1024`

  sudo wondershaper wlan0 $K $K

  iperf -t 10 -c 192.168.1.132 2> /dev/null | grep "Mbits/sec"

  sleep 2

done

sudo wondershaper wlan0 clear

$ ./go 

1 Mbps

[  3]  0.0-11.2 sec  1.38 MBytes  1.03 Mbits/sec

2 Mbps

[  3]  0.0-10.7 sec  2.62 MBytes  2.06 Mbits/sec

3 Mbps

[  3]  0.0-10.4 sec  3.75 MBytes  3.02 Mbits/sec

4 Mbps

[  3]  0.0-10.4 sec  5.00 MBytes  4.04 Mbits/sec

5 Mbps

[  3]  0.0-10.5 sec  5.12 MBytes  4.11 Mbits/sec

6 Mbps

[  3]  0.0-10.3 sec  7.38 MBytes  6.00 Mbits/sec

7 Mbps

[  3]  0.0-10.3 sec  8.50 MBytes  6.95 Mbits/sec

8 Mbps

[  3]  0.0-10.3 sec  9.75 MBytes  7.97 Mbits/sec

9 Mbps

[  3]  0.0-10.2 sec  10.6 MBytes  8.76 Mbits/sec

10 Mbps

[  3]  0.0-10.1 sec  12.0 MBytes  9.94 Mbits/sec

11 Mbps

[  3]  0.0-10.1 sec  13.2 MBytes  11.0 Mbits/sec

12 Mbps

[  3]  0.0-10.2 sec  14.5 MBytes  12.0 Mbits/sec

13 Mbps

[  3]  0.0-10.2 sec  15.5 MBytes  12.8 Mbits/sec

14 Mbps

[  3]  0.0-10.3 sec  16.1 MBytes  13.2 Mbits/sec

15 Mbps

[  3]  0.0-10.1 sec  16.4 MBytes  13.6 Mbits/sec

16 Mbps

[  3]  0.0-10.1 sec  19.0 MBytes  15.8 Mbits/sec

17 Mbps

[  3]  0.0-10.1 sec  19.8 MBytes  16.4 Mbits/sec

18 Mbps

[  3]  0.0-10.1 sec  21.1 MBytes  17.6 Mbits/sec

19 Mbps

[  3]  0.0-10.1 sec  22.1 MBytes  18.3 Mbits/sec

20 Mbps

[  3]  0.0-10.1 sec  23.1 MBytes  19.3 Mbits/sec

...

While the use of WonderShaper requires superuser privileges, it limits the system network bandwidth in it's entirety. Enjoy.

Performance Appraisals -- 'Compared to Who?'

Photo Credit: https://unsplash.com/@sernarial

Engineers often love to measure; we measure system performance, we measure clock drift, we measure all sorts of fascinating things, we even occasionally measure ourselves.  Even so, show me an engineer that didn't encounter anxiety when contributing to their first performance appraisal and I'll show you someone who's pulling your chain.

I recently stumbled upon a Reddit post for a junior'ish software engineer who was asking for advice with respect to his recent performance appraisal and I'd have to say, he was getting a good deal of 'bad advice' in my opinion.  A good deal of the contributors seemed jaded, perhaps bitter, and contributed tales as to the dire consequences of bad performance reviews or suggesting that performance reviews were completely useless.

I remember being tasked with my first performance review, an overwhelming anxiety filled me for weeks.  The feeling resembled being lined up in a middle-school gymnasium by height; 'will I be too short', 'stand up straight', 'should I stand on my tiptoes'?  This original poster (OP) was bombarded with conflicting advice; "pay no attention to them, they are meaningless", "rank yourself exceptional or you'll never get a promotion", "rank yourself honestly, it's just a conversation starter".

Personally, I think performance appraisals are useful and I have found them to help me progress personally as well as professionally.  That said, I've been fortunate to work with companies, managers, and leadership that used them online with how I felt they'd be used. The anxiety and conflicting advice to this poor junior engineer are probably due to advice givers getting burned in their own experiences.  As valuable as performance reviews can be, they tend to have some fundamental blindspots which could be addressed in training and/or simply communication to the teams.  Twenty-*mumble* years in the profession, formal training by a variety of employers and these questions tend never to be addressed, frankly.....because they can be hard questions and typically extend beyond the expertise and/or reach of human resources (who tend to facilitate and provide the training).

What is Average?

Nearly every performance appraisal form applies some form of a numeric scale; 1-need improvement, 5-exceptional or something of that nature.  Training will tend to say "rate yourself" without going into much detail on the scale, instead focusing on the skill definitions.  Most engineers come readily armed with a flurry of mathematical prowess, well-versed in the normal distribution bell curve,  and likely are self-aware enough to easily compare themselves to the average if they know what an 'average' is.  In the absence of a clear definition they tend to define our own criteria.  The result; wild deviations in personal ratings, not uncommon to observe a seasoned engineer proficient in X rating themselves average and an engineer less proficient rating themselves exceptional and you're left with the compelling question "what gives?".  Lack of guidance from the organization, manager and leadership contributes to such a quandary.

As an organization, you could/should provide guidance by defining the sample set, for example:

Average in relation to the team:

You're surrounded by your team, work with them every day, and likely know how you stack up with respect to a set of skills for each.  Perhaps you stand head and shoulders above your team in terms of communications, or perhaps you are new to a technology and below average in knowledge on the subject.  The value of this guidance, it is easily relatable and a bell-curve distribution would be expected from the team collective.  "I'm better at some things than most of the team, short in others".  A junior or new team member would likely expect to be below average for a period until they gain the same degree of proficiency as the rest of the team.  A below average wouldn't be treated as a red-flag or brand, instead it would be used to align opportunities to improve on the skill (if needed/desired) or better align assignments with team members.

Average in relation to your colleagues:

Some companies take exceptional pride in "we only hire exceptional people" and such phrases tend to add confusion.  "If I'm working here, and the company is comprised of only exceptional folks then I guess I should rate myself exceptional."  Without clear guidance, one could easily arrive at this on their own.  Say you worked on tech XXX for the past years, have become proficient in it to an expert level in the general population, then hired into a company that specializes in XXX.  Are you now average, or exceptional? 

Average in relation to your experience:

Consider the talent pipeline, junior engineers coming onto the team, transitioning into senior contributors over the course of their career.  Should they rate themselves low until they become as proficient with experience or should they rate themselves with respect to their years of experience?  Or should they rate themselves solely with respect to their peers sharing the same title/experience? 

How Will These Be Used?

The organization and leadership primarily control how performance appraisals are applied within the organization, but rarely formally communicate it.  A great deal of anxiety and fear with respect to performance appraisals are a consequence of this.  A fearing employee may feel that the organization may only promote those with exceptional ratings, use the ratings as a litmus test when downsizing or use the ratings for pay increases.  When you fear your ratings will be used against you you're more likely to elevate them artificially.  It's been my observation that peer reviews are often orchestrated and authored with a specific purpose, but later find their way into broader use.  Additionally, in an industry of lean development philosophies no one wants to dedicate significant time into something that could be streamlined or eliminated, time is too precious to waste.

Are They Scored Consistently?

As a manager, I strived to be consistent in rating my team.  Humans are easily influenced beings and it is important to self-manage external factors.  "Whelp, a miserable Monday, I have a ripping headache.....let's get back to John's performance appraisal" -- a recipe for disaster for you, John and the team.  It unfortunately doesn't end there, suppose the H/W manager and S/W manager used different definitions of 'average', one rating their team higher than the other.  Performance appraisals tend not to be contained to departments, so how will the VP of Engineering interpret this inconsistency?  Will they think one department is stronger than another?  Will HR?  The CEO?

These are but a few considerations to take into consideration, lack of guidance results in a team, or individuals simply 'filling in the gaps' which can be beneficial, but can also add complications and confusion down the line.

Performance appraisals can be extraordinarily useful, to an individual, to a team and to a company.  They can facilitate a self-directed review of who you are, what you're good at, and who you want to become.  They set a stage to have an honest dialog between individual contributors and leadership, how they see each other and how they affect one another, and can provide a popcorn trail tracking the progression of a fresh employee to one that progresses to an organizational giant.   Their sheer existence enforces a 'measure what you care about' philosophy and an organization committed to professional growth of their team really demands their use.  That said, with a little clarification of definitions they could reduce a good deal of anxiety, confusion and inconsistencies.

Software System Forensics

While a good deal of my career has revolved around green-field development projects, as of recent I've been more heavily involved in existing systems.  Since then, I've acquired a new set of skills that help in acquiring knowledge about how the system is operating.  As a recent contractor, I've found myself frequently given assignments of the form "we need to understand this component of the system....go research that".  More often than not, all subject-matter experts are long gone so you're primarily on your own.  Luckily, there are a series of *nix tools that can assist in this discovery process.  Let's hit on a few;

While source code is the life-blood of software systems, outside the developer community most team members are more familiar with system components and likely couldn't point you to the source code.  Could be a shell script, Perl, Python, Java,.....and while many managers/team-leads likely couldn't aim you toward the source tree of any given system component, they likely can refer you to a process name (or snippet) and that alone can get you rolling on your investigation.

Let's say your big-bossman points you to a data processing feed called spooler1 that is currently running on the system;

Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie

%Cpu(s):  5.7 us,  2.0 sy,  0.5 ni, 91.4 id,  0.4 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem :  8167120 total,   781680 free,  1680440 used,  5705000 buff/cache

KiB Swap:  8385532 total,  8371072 free,    14460 used.  5928716 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                   

29905 user  20   0   14008   4400   2708 R  20.0  0.1   0:00.22 spooler1

  Not a lot to go on, but you may be surprised what you can gleen from a running process.

lsof -- list open files

Let's start with the lsof utility in our investigational journey.  This utility does precisely what the name says....it lists all open files that the process has running.

$ lsof -p 29905

lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing

      Output information may be incomplete.

COMMAND    PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME

logMaker 12600 user  cwd    DIR   8,33     4096 70975491 /home/user/blog/SystemForensics

logMaker 12600 user  rtd    DIR    8,1     4096        2 /

logMaker 12600 user  txt    REG    8,1  1037528 41418771 /bin/bash

logMaker 12600 user  mem    REG    8,1  2981280 20186616 /usr/lib/locale/locale-archive

logMaker 12600 user  mem    REG    8,1  1868984 45355547 /lib/x86_64-linux-gnu/libc-2.23.so

logMaker 12600 user  mem    REG    8,1    14608 45355429 /lib/x86_64-linux-gnu/libdl-2.23.so

logMaker 12600 user  mem    REG    8,1   167240 45355621 /lib/x86_64-linux-gnu/libtinfo.so.5.9

logMaker 12600 user  mem    REG    8,1   162632 45355450 /lib/x86_64-linux-gnu/ld-2.23.so

logMaker 12600 user  mem    REG    8,1    26258 20449837 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache

logMaker 12600 user    0u   CHR 136,20      0t0       23 /dev/pts/20

logMaker 12600 user    1w   REG   8,33    46450 53624834 /home/user/blog/SystemForensics/spooler.log

logMaker 12600 user    2u   CHR 136,20      0t0       23 /dev/pts/20

logMaker 12600 user    3r  FIFO   0,12      0t0 76487063 pipe

logMaker 12600 user  255r   REG   8,33      857 53624833 /home/user/blog/SystemForensics/spooler1

There is a host of information available from this command, but for now, let's focus on two key things (highlighted in blue); the current working directory and the log file.  While not relevant to our above example, the lsof utility will provide evidence of relevant libraries and/or network ports.  For example, if you observe open network ports and the process linking in Corba libraries....you can get a gauge of what you're in for.  A good starting point when trying to understand a system element is understanding the 'goes-intas, and the goes-outtas', so library-dependencies input and output files can set the stage for investigation tasks.

In our example, we now know where this utility is run from and that it generates a log file.

While not every process generates a log file, if a log file exists you've likely stumbled upon a plethora of information.  No amount of source code browsing will ever compare to spending time with a quality log file in gaining knowledge of what's going on.  Likely you're looking into a system component because it isn't operating as expected, and hopefully the log file will provide evidence of that and give you a hint at where you should focus your next steps.

Simple Log File Analysis

If you're fortunate enough to have access to sophisticated log file analysis tools by all means....use them.  For many of us we're stuck with the general system utilities and in many cases you can go far with these simple utilities.  I've applied these techniques on some pretty sophisticated log files on existing systems but many companies would consider the logs as proprietary, so we'll use a sample log file format for our purposes.

If we peek at the first few lines of our log file we can get a feel for what we can do with it;

$ head spooler.log 

2020-06-10 15:04:02.85|processing Table-E record

2020-06-10 15:04:04.45|processing Table-C record

2020-06-10 15:04:05.15|processing Table-D record

2020-06-10 15:04:05.75|processing Table-C record

2020-06-10 15:04:06.96|processing Table-D record

2020-06-10 15:04:07.66|processing Table-E record

2020-06-10 15:04:08.96|processing Table-D record

2020-06-10 15:04:09.86|Spooler::init() Connection established with peer pub.server.org port 9000

2020-06-10 15:04:11.06|processing Table-B record

2020-06-10 15:04:12.46|processing Table-E record

A well-planned log file will have a well-defined format, hopefully with a time-stamp and a series of events.  In our example, we have 6 categories of events {connection requests, and 5-table record processing events}, but only a fraction are evident from the head of the log file.  Let's see how we can get a comprehensive list of types of events.

With a focus on discovering a comprehensive list of event types we first see that the timestamp is ever-changing, so we need to ignore the timestamp and extract the rest of the line and see what we have.

$ cat spooler.log | cut -f 2- -d '|' | head -10

processing Table-E record

processing Table-C record

processing Table-D record

processing Table-C record

processing Table-D record

processing Table-E record

processing Table-D record

Spooler::init() Connection established with peer pub.server.org port 9000

processing Table-B record

processing Table-E record

Whelp, that shows promise.  Let's sort uniquely the result and see if that gives us what we want;

$ cat spooler.log | cut -f 2- -d '|' | sort -u

processing Table-A record

processing Table-B record

processing Table-C record

processing Table-D record

processing Table-E record

Spooler::init() Connection established with peer pub.server.org port 9000

BAM!  Looks good.  Now what?  The frequency of an event can provide you with some insight into how the system is behaving.  For example, repeated 'Connection established...' instances may hint at an unreliable network connection or unstable companion process.  Let's look at how many instances for each type of event.  We can do this by sorting the events followed by counting the instances of each type.  The sort+uniq utilities will do the trick;

$ cat spooler.log | cut -f 2- -d '|' | sort | uniq -c

   1422 processing Table-A record

   3593 processing Table-B record

   1427 processing Table-C record

   3572 processing Table-D record

   3625 processing Table-E record

    761 Spooler::init() Connection established with peer pub.server.org port 9000

From this we can see we have B,D,E records providing the highest volume of entries, a smaller number of A,C records and ~700 instances of connection retries.  

While this provides a nice summary of the types of events coming in as well as the quantities, often time-based summaries are essential.  Perhaps we're expecting burst-transfers in the off-hours and a smaller volume during the day.  We can bring the timestamps back into the equation for a look.

Data analysis is a discovery process, you often find a need to repeatedly refine your search criteria from your starting point.  

Let's breakdown a log entry so we can see how we can perform some time-based categorization;

2020-06-10 19:04:23.10|processing Table-B record

date -- char [1..10]

time -- char[12..22]

event -- char[24..]

The cut utility allows us to extract specific fields from each log line.  Let's say we are interested in the number of events organized by date, we'd want to preserve the data + event type like this:

$ cat spooler.log | cut -b 12-14,24-  

15:processing Table-E record

15:processing Table-C record

...

A bit ugly, but by extracting the hour+':'+event type, you've got the beginnings of what we want.  I purposely left the HH:MM colon delimiter in place to separate the hour from the event type;

$ cat spooler.log | cut -b 12-14,24-  | sort | uniq -c

    326 15:processing Table-A record

    847 15:processing Table-B record

    340 15:processing Table-C record

    831 15:processing Table-D record

    829 15:processing Table-E record

    180 15:Spooler::init() Connection established with peer pub.server.org port 9000

    347 16:processing Table-A record

    914 16:processing Table-B record

    332 16:processing Table-C record

    896 16:processing Table-D record

    932 16:processing Table-E record

    174 16:Spooler::init() Connection established with peer pub.server.org port 9000

    377 17:processing Table-A record

    895 17:processing Table-B record

    350 17:processing Table-C record

    876 17:processing Table-D record

    893 17:processing Table-E record

    202 17:Spooler::init() Connection established with peer pub.server.org port 9000

    348 18:processing Table-A record

    868 18:processing Table-B record

    373 18:processing Table-C record

    904 18:processing Table-D record

    912 18:processing Table-E record

    190 18:Spooler::init() Connection established with peer pub.server.org port 9000

     24 19:processing Table-A record

     69 19:processing Table-B record

     32 19:processing Table-C record

     65 19:processing Table-D record

     59 19:processing Table-E record

     15 19:Spooler::init() Connection established with peer pub.server.org port 9000

We can now see that how event counts change over the course of the hours.  Bursty processing loads would be evident by spending some time with this raw data, or better yet, plotting quantities over time can provide you with real insight into your system.  High volume of atypical events can give you a clue to problems; for example, say you observed a high volume of retries/restarts in the wee hours of the morning may imply controlled restarts or potential problems.

Let's prepare to power down on this post.  With a handful of lesser-known utilities we can start with a process id, locate an available log file and perform some high-level metrics on it in relatively short order.  I've found these utilities particularly useful in these past few months and have acquired a newly-found proficiency using them out of pure necessity.  Hopefully it is useful to you fellow readers. 

Cheers.

Mandelbrot Set with Python

Data can be beautiful.  Visualizing data is a worthwhile skill to acquire and it's relatively simple with Python.  Let's explore how to do some data visualization as an exercise.

The Mandelbrot set is often regarded as an example of art meeting science.  It's generated by evaluating the behavior of complex numbers and generating the results.  The end effect is a infinite and beautiful depiction of pure mathematics, an acid-trip of color and structure as you continuously zoom into the graph.

One of the best descriptions of the Mandelbrot set can be found here, I recommend you spend a few minutes to appreciate the concept before we begin graphing it with a simple Python snippet.

In about 30 lines of code we can create our own colorized visualization of the mandelbrot set.

Let's look at the source, then step into some of the details;
$ cat -n mandelbrot 
     1 #!/usr/bin/python
     2 import matplotlib.pyplot as plt;
     3 import sys;
     4
     5 def colorize(n):
     6   h="#%06x"%(int(n*2**23));
     7   return h;
     8
     9 xRange=[-2,1];
    10 yRange=[-1.5,1.5];
    11 incr=0.005;
    12
    13 x=xRange[0];
    14 while(x < xRange[1]):
    15   y=yRange[0];
    16   while(y < yRange[1]):
    17     c=x+y*1j;
    18     z=0;
    19     try:
    20       for k in range(50):
    21         z=z**2+c;
    22       if(abs(z) < 2):
    23         rgb=colorize(abs(z));
    24         plt.plot(x,y,'.',color=rgb);
    25     except:
    26       pass;
    27     y += incr;
    28   x += incr;
    29
    30 plt.xlim(xRange[0],xRange[1]);
    31 plt.ylim(yRange[0],yRange[1]);
    32 plt.savefig(sys.argv[1]);

Let's look over the non-Mandelbrot stuff first.  We're using the matplotlib library for our simple plotting example, line 9-10 we define our plot x and y ranges and enforce them in lines 30-31.  Finally in line 31 we save the plot to a figure rather than display to the screen.  We pass in the figure filename as a command line argument so ./mandelbrot foo.png would generate a foo.png file.  Lines 23-24 calculates a color for the pixel and plots it at (x,y).  Without going into details, the nested loop (lines 13-28) steps through the floating point 2D range defined by the x and y ranges, stepping by 0.005 (defined in line 11).  Each iteration, we selectively plot, or don't plot, a colorized pixel.  We plot a pixel if it's position is part of the Mandelbrot set.

The rest of the details are specific to calculating the Mandelbrot set.  In particular, lines 17,20-26.  The referenced video explores how and why this is done, but let's revisit some of the particulars.

Line 17 consists of the assignment of the complex number for each (x,y) position in the range.  Note 1j is the Python representation of complex number i.   This assignment was covered in the video, specifically, but easily overlooked;

The inclusion in the mandelbrot set for this position is characterized by how this complex number behaves under the influence of iteration of the function starting at 0.  This means, z starts as 0 and we repeatedly inject the f(z)=z^2+c in our loop, the end result will either blow up implying it's not in the mandelbrot set, or it'll remain bounded (e.g. <= 2) which means it's part of the set.  Lastly, we use the magnitude of the result to determine the color of the pixel.  This is optional, but it adds a level of beauty. We can represent the looping of (x,y) range and determination of it's inclusion/exclusion in the mandelbrot set by the clip from the reference video;

The end result is our visualization;

No go do something cool!