Sunday, August 30, 2020

How To Quit



Some time back a colleague, who recently joined our profession, found himself unchallenged in his current role and threw his resume into the ring.  Shortly thereafter he was offered a new job more fitting of his  interests and asked me for advice on how to proceed with 'giving notice'.  This post will revolve around that advice, expanding on it a bit.

Before we get into it, I find myself struggling to recall how I ever came to these opinions.  School certainly didn't touch on such topics, and I don't recall ever engaging colleagues or mentors about the matter.  Google wasn't nearly as popular as a junior engineer, nor Reddit, Twitter,....so I can only surmise that I came to these conclusions by means of sidebar conversations or personal conclusions.  Being long-in-the-tooth I often take for granted such topics, but have no recollection as to how/when I came to such opinions.  There is an entire world full of junior folks entering this, and every, profession that are faced with such questions, trying their best to do the right thing, and a general lack of good advice IMHO.  Reddit subreddits on such matters are plagued with near-toxic advice partly because much of the community is under a lot of stress.  So I'd encourage you, as professionals, to be approachable for the younger folks, offer them your support, encouragement and knowledge readily, patiently and promptly.  Now, let's get into the matter at hand.

Assumptions

While every situation has unique qualities, I'm going to gear my advise to a graceful self-initiated departure with a goal of leaving on good terms.  Sometimes deciding to leave a company can be easy, specifically when you hate your tasks, your boss or your teammates.  Far more often I find it can be a difficult decision, I spent a little time on that topic in a previous post Intellectual Wrestling When Contemplating Leaving a Company if you have some time to burn.

General Guidance

I recommend some general advice, not necessarily hard-n-fast rules that can't ever be broken, but stuff I try to apply myself.

Keep Your Communication Positive, Professional and Honest

    People change, organizations change, and while you may never imagine yourself ever working for this particular company again you may quickly find that the tech community is smaller than you initially think. I routinely find my old teammates working for different companies and while I don't particularly believe in the "don't burn any bridges or you'll regret it" advice, I have observed that people can be a product of their work environment. I've worked with folks that were terrible to work with at one organization but were completely different at another; people are capable of change. I've also known many-a-folk that have worked for the same organization on multiple occasions and the work environment has been drastically different between the instances; companies can change. Toxic elements can be replaced with better ones, bad managers can retire, quit, or be terminated for better ones. New management who are ill-equipped for the role may grow into becoming a great leader.  Organizations can be organic, changing for the better, or sadly for the worse.
    I personally feel that you should be positive and professional because you want to, not because you're expected to or forced to.  Additionally, it's important to stay true to yourself.  If you've been anxiously waiting 3+ months to tell your boss or teammate to 'shove it' and you'll regret not saying it, say it rather than live with the regret.  
    When leaving, you get to set the stage as to how much info you are willing to share.  You can be as vague or as detailed as you wish, but I always encourage being honest.  You can be honest and vague -- "I'm leaving for an opportunity that is better suited to my personal interests", or as detailed as you wish -- "I'm leaving because I don't have confidence in the financial outlook of the company.".  One big factor, in my opinion, as to the appropriate level of detail really depends on whether you believe the organization/company/team will act on your feedback.  I truly believe that companies want to be better and recognize that they need to be better.  When companies lose customers and team members they should want to know why they are leaving so that they can take those factors into consideration and determine if they should or need to change.  Imagine being a company with a revolving door of talented folks coming and leaving and not knowing why?  That, my friend, is a recipe for disaster.  With your upcoming resignation, you have a new-found sense of freedom that others may not share.  You may know that the majority of your team is miserable for the same reasons that drove you to look elsewhere.  You have an ability to speak on behalf of the team, and a good organization/team wants to know where they can improve (or what's driving folks away).
     

Deliver Your Resignation In Writing

Personally, I prefer providing a copy of my resignation either via e-mail or physical print.  Often, a copy of your resignation will be placed in your HR folder, kinda book-ending your employment record.  Additionally, I feel that exclusively verbal exchanges tend to lack recall of details, like when your last day will be.  

I tend to follow something of the recipe: 

    • announcement of resignation, 
    • thank them for opportunities, 
    • acknowledge talent of team, 
    • identify final working day  
    • optionally provide contact information
For example;

It is with regret that I hereby tender my resignation from [companyX].  I appreciate the opportunities this position has offered me over the past years and have thoroughly enjoyed working with such a talented team.

My resignation is effective today with my final working day of [last day date] unless it is felt an earlier separation date is more appropriate.

I wish you, my team and everyone at [companyX] all the very best for your continued success.

Short, direct and to the point.  The purpose of this exchange is pretty limited, a graceful announcement of your resignation and final working date.  The audience, your manager and HR typically.  Team announcements and follow-up conversations tend to take place independently and later. 

Preserve the Chain-of-Command

The term 'chain of command' has some pretty dated and perhaps negative connotations but it's worth preserving for a number of reasons.  In a world of transparency and open-communication, the responsibility goes both ways.  I've known folks that openly shared their job search details and upcoming resignation with seemingly everyone but their manager and personally I feel that's a bit unprofessional for a few reasons.
Your current and upcoming tasks need to find a new home, specifically someone to do them.  Your manager will be responsible for doing just that, perhaps hiring your replacement, offloading your tasks to existing team members or re-prioritizing tasks to account for the change in the team.  It's common professional courtesy to give them some time to get their ducks in a row before letting everyone else know.  Give them time to prepare for the question "With Bob leaving at the end of the month, how will his tasks be handled?".  Remember our goal should be a 'graceful transition' and giving leadership some additional time will assist in precisely that.  Unless you truly hate the company, your team and everyone else for that matter, this additional time will reduce the stress on all those folks affected; if you like your team, give your leadership some time to come up with a transition plan, its more for your team than preserving appearances.

Process

I feel that the typical flow of events take the following form:

Establish a Final Work Date

A pretty typical convention is to provide your current employer with 2-weeks notice.  Often, this is a professional courtesy rather than legal obligation.  That said, it's of my opinion that it's best to preserve a 2-week notice if possible for a few reasons:

    • some company policies require it
    • some contracts (e.g. contractor agreement) legally require it
    • give time for a graceful transition, knowledge transfer, task hand-off

Defining this final work date is typically done by getting a new formal job offer, planning a time to notify your manager and tagging on 2 weeks from that day.  Job offer arrives Friday afternoon, plan is to notify your manager on Monday morning, last day is second Friday to follow.  

Author a Resignation Letter/Email

Armed with your final day, you can author your resignation letter calling out your final working date to avoid any confusion.

Deliver Verbal Resignation

I've changed jobs a number of times in my career, to this day this step continues to come hand-in-hand with anxiety and discomfort, but you press through it.

Ideally, I prefer this be done in-person as I feel it is shows more respect.  Managers can be overly busy, so I try to arrange it with pre-established private 1-on-1 meetings, or try to catch them when they have a free moment, requesting a private conversation and delivery the message.

Despite all the best intentions, sometimes this plan falls flat.  Your new employer is expecting you on date X, you need to deliver your resignation on X-14 days and something can always go wrong.  Your manager scheduled work travel and is across the country when you didn't expect it, a sick child resulted in him/her going home to take care of them, their schedule is packed with end-to-end meetings throughout the day.  The best laid plans can often go off the tracks, this is where you may find a need to deliver the message off-plan; e-mail, to another party, later than planned...

Hopefully, if you are considered a valued member of the team you'll likely be asked a couple things; 1) why you are leaving, and 2) is there anything that would make you stay.   It's worth putting in some time in thinking about how you would respond to such questions beforehand.

One final topic I always ask in this chat is "how would you prefer to communicate to the team"?  Two things you're looking for: 1) when should the team be notified, and 2) by whom.  Depending on your manager and organization, your manager may prefer to make the announcement, especially if you have customer and/or intra-departmental relationships.  Otherwise, they may prefer you make the announcement directly to your team.  That covers the 'how', it's also important to get a 'when'.  Your manager may want a day or so before you tell your team, they may want to begin the process of hiring someone, they may want to re-prioritize activities, they may want to just spend some time on how to address your leaving.  Be prepared to give them a bit of time to get their plan in play.

Deliver Formal Letter/Email

Immediately, or shortly after the verbal exchange, deliver the printout/e-mail to the your manager.  Your manager will likely provide a copy to HR for your employee file and will use the last working date for notifying the affected parties (e.g. HR, leadership,...).  Additionally, they may initiate a hiring process by authoring a job posting and coordinating it with HR.

Announce to Team

Initiated by an official announcement from your manager, or a side-note you offer via Slack channel, e-mail or during a daily standup.  Typically, this is a short exchange, positive, professional with the direct goal of making everyone aware.  Most often, this is short announcement to the group and throughout the day the team will reach out to you individually to share their opinions and feelings about your departure.

Handoff/Knowledge Transfer

With the clock running, you've acquired a great deal of knowledge and responsibilities that now need to find a new home.  Typically, the team will be begin a desperate flurry of handing off your existing tasks and performing knowledge transfers.  Having been responsible for a number of tasks, ones which you are counted on and accomplish well.....now your team is faced with how will they get done when you're gone?  This tends to take the form of: you training someone directly, or documenting how you do it.

Exit Interview

Optionally, well-established companies will have an exit interview for departing employees.  Good companies understand the value in retaining talent, so they want to understand why folks elect to leave.  A big ol' pile of money goes into hiring someone new and training them to become an effective contributor, so people leaving is the equivalent of cash walking out of the door.

Often, Human Resources will conduct an exit interview with you, with the purpose of determining why you choose to leave.  Those details are tallied and perhaps one-day applied to reduce employee turn-over.  For example; If 80% of folks are leaving due to compensation, then the company can gather the information, establish the trend, and make changes to address the situation.  Equally, they can choose to not act on the findings as well.

You are in full control of how much detail you wish to share, be as vague as you wish, or as detailed as you wish.  

Final Day

The big day; you cleaned out your desk, preserved all your work, exchanged personal goodbyes to your teammates, and as a last act you will author your departing e-mail.

Normally, this last act is an e-mail calling out your final day, a form of gratitude to having the opportunity to work with the team, an acknowledgement of you learning a lot and a desire to keep in touch often with your personal contact information (e.g. phone, e-mail).  This is sent shortly before handing over your equipment, badge, parking pass and someone accommodating you to the door.  A firm handshake, a thank you, and you're off to another adventure.


 

Friday, August 28, 2020

Software Consulting -- What The Heck Does '...as an additional insured...' Contract Clause Mean?




While software contracting comes with a great deal of perks, it also comes with some exhaustively dull tasks.  Since our company prefers corp-to-corp contracts, every new contract comes with an arduous task of reviewing the contract specifics.  Contracts, authored by lawyers, speaking lawyer-speak can be mentally taxing for anyone and some folks will simply sign whatever is placed in front of them just to avoid reading pages of mumbo-jumbo.

The whole idea of just signing whatever gives me the hives, but I can understand the reluctance of reading these agreements with diligence.  Contracts are binding legal documents, they need to be taken seriously, but unfortunately it sometimes requires more offline research than any sane person wants to perform.  Lately, there is a trend in agreements that absolutely scared the crap out of me:

YourCompany shall add OurCompany as an additional insured to the Comprehensive General Liability policy. OurCompany’s insurance shall be primary, and any insurance maintained by OurCompany shall be excess to and not contribute to YourCompany's insurance.

Please excuse the phrasing; OurCompany == them, YourCompany == me as typically the agreement is provided from them.

My layman's interpretation of this statement implied that I'd add this new company onto our CGL policy and in the event that they were sued it would be covered by our policy!  The whole idea seemed completely absurd!  I could only compare it to inviting some stranger off the street, giving them my car and legally signing a document that I was legally responsible for them driving through the Mall of America.  They act unprofessionally or careless, we foot the bill.  The whole thing seemed ridiculous at its core and the first time I had seen this clause it seemed unique to this one contract that was authored by a massive media company inclined to bully subcontractors into whatever the hell they wanted.  Fearing I was over-reacting, I asked for clarification on the statement and their response was "*shrug* it was asked for by our legal department".  Unconvinced, I spent the next couple nights researching via Google and it continued to appear that my concern with the clause was justified.  In that particular instance, I chose not to sign that contract partly because this clause would not be removed from the agreement.  That was 2'ish years ago.

Fast-forward to a couple weeks ago, a new contract, similar clause, and an appearance of it trending in newer contracts.  I sought out the wisdom from insurance professionals on Reddit, but was met with crickets.  While I probably could have negotiated it's removal, this time I skipped Google and went directly to an authority on the subject, namely our CGL insurance provider.  The policy agent didn't shed much light on the topic, but put me in touch with one of their actuaries (Quentin) who provided insight into it's meaning.  Here's what I learned.

Quentin stated that I was applying a 'broader definition to it than it means', the clause is limited to the services that you perform for them.  The purpose of me holding a CPL is to cover any claims against work that I perform.  OurCompany has a similar CPL  to cover claims against work that they perform.  This clause essentially provides that separation as a provision for the court system.  Without the clause, a claim can be brought against any company for any work, this statement essentially says 'if there is a claim against them, you need to file a claim against them', work that they perform is covered by their policy, work that you perform is covered by your policy.  Claims are limited to work that you perform for them, not a universal catch-all of coverage which is what my layman's interpretation of the clause was.

The phrasing to this day still gives me the willies, but the clincher that set my mind at ease was when Quentin pointed out that insurance companies by nature are risk-averse, they aren't going to do something that puts a lot of additional liability on them.  "This additional insurer, we literally give it out for free any time your client requires it", "if we thought it could generate claims against us, we would charge for it".  So, if someone who daily performs statistical analysis of risk isn't concerned by this I guess it shouldn't concern us, it's part of the blanket policy.

Please, perform your own research on the topic, consult your own insurance provider.  Given that there is a great deal of confusion on the topic and it caused me a great deal of anxiety and time I thought I'd share it with those that may find it useful.

Cheers.





  

Tuesday, August 25, 2020

Software System Forensics -- Auto Generated Message Trace Diagrams


Understanding an existing software system can be a daunting task.  Diving head-first into a source code repository with the objective of gaining a system understanding can be particularly challenging.  Taking the high-dive into source code often results in crawling down a variety of rabbit holes that may or may not be of particular relevance.  It's not uncommon for software to have edge cases and/or 'dead code' that while are compiled into the release are rarely (or ever) executed due to run-time constraints.  But, really, what are the alternatives?

Whelp friends, what if you could execute a software system, gather method calls, w/caller and callees, and create a visual representation of the process flow?  That will be the topic of this particular blog post.

Let's introduce our team:

Our power forward; the hustle with the muscle, the beta with aaaalllllll the data.....GDB.


At point guard; the mate that will translate, the teammate that will update....your buddy and mine...Python.

And rounding out the crew, a battering ram of a diagram....WebSequenceDiagram.  


That's our roster; GDB to collect caller/callee information, Python to convert GDB output into something that can be used to generate a visual diagram, and WebSequenceDiagram to create the diagram.  This particular team has proven to be quite beneficial when I've been tossed into the deep end of the pool without my water-wings.  Let's work through a simple example;

Behold, an overly simple software system source file:
$ cat -n main.cpp 
     1 #include <stdio.h>
     2
     3 class C
     4 {
     5   public:
     6     C() { }
     7     void beak();
     8     void flap();
     9     void shake();
    10     void clap();
    11 };
    12
    13 void C::beak() {}
    14 void C::flap() {}
    15 void C::shake() {}
    16 void C::clap() {}
    17
    18 class B
    19 {
    20   public:
    21     B():c_() { }
    22     void stepOnce();
    23   private:
    24     C c_;
    25 };
    26
    27 void B::stepOnce() { c_.beak(); c_.flap(); c_.shake(); c_.clap(); }
    28
    29 class A
    30 {
    31   private:
    32     B b_;
    33   public:
    34     A():b_() { }
    35     void run();
    36 };
    37 void A::run() { for(int i=0; i<10; ++i) b_.stepOnce(); }
    38
    39 int main()
    40 {
    41   printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
    42   A obj;
    43   obj.run();
    44   printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
    45 }

Even the most modest of software engineers can peek at this code and understand it without the need for any advanced tools, but this process of capturing debug info and transforming it into a sequence diagram works for far more complicated systems, frankly it's saved me hours and hours of tracing through source code.  Fred R. Barnard may have not been a software engineer, but he just as well could have been when he coined the phrase "a picture is worth a thousand words".  

So, that's our system, let's turn our attention to GDB.  We'll author a GDB command script which will perform all the heavy lifting; we'll enable logging, write gdb info to a gdb.log file, set up breakpoints in methods we are particularly interested in (e.g. class A, B, C), the breakpoints will print the backtrace and release the process to continue.  The backtraces saved in the gdb log file will be used to extract the caller/callee methods for our diagram.
$ cat -n gdb.cmd 
     1 set pagination off
     2 set logging file ./gdb.log
     3 set logging overwrite on
     4 set logging on
     5
     6 define MyTrace
     7   bt 2
     8   cont
     9 end
    10
    11 break main
    12 commands
    13   rbreak ^A::
    14     commands
    15       MyTrace
    16   end
    17   
    18   rbreak ^B::
    19     commands
    20       MyTrace
    21   end
    22   
    23   rbreak ^C::
    24     commands
    25       MyTrace
    26   end
    27   
    28   cont
    29 end
    30
    31 run
    32 quit

Armed with the gdb command script, we simply run our main process under gdb as follows:
$ gdb --batch -x ./gdb.cmd ./main 2> /dev/null

When the process terminates, we have a gdb.log file that takes the form:
$ more gdb.log 
Breakpoint 1 at 0x40063c: file main.cpp, line 40.

Breakpoint 1, main () at main.cpp:40
40 {
Breakpoint 2 at 0x4006e4: file main.cpp, line 34.
void A::A();
...
Breakpoint 2, A::A (this=0x7fffffffdc77) at main.cpp:34
34     A():b_() { }
#0  A::A (this=0x7fffffffdc77) at main.cpp:34
#1  0x0000000000400670 in main () at main.cpp:42

Breakpoint 4, B::B (this=0x7fffffffdc77) at main.cpp:21
21     B():c_() { }
#0  B::B (this=0x7fffffffdc77) at main.cpp:21
#1  0x00000000004006f0 in A::A (this=0x7fffffffdc77) at main.cpp:34

Breakpoint 6, C::C (this=0x7fffffffdc77) at main.cpp:6
6     C() { }
#0  C::C (this=0x7fffffffdc77) at main.cpp:6
#1  0x00000000004006d4 in B::B (this=0x7fffffffdc77) at main.cpp:21

Breakpoint 3, A::run (this=0x7fffffffdc77) at main.cpp:37
37 void A::run() { for(int i=0; i<10; ++i) b_.stepOnce(); }
#0  A::run (this=0x7fffffffdc77) at main.cpp:37
#1  0x000000000040067c in main () at main.cpp:43

Since we created breakpoints for all our class A,B,C methods, hitting one will produce a backtrace depth of 2, the caller(#1) and the callee(#0).  Since the stack trace has the class name and method, we have sufficient info to create a sequence diagram, we just have to parse the gdb log file and extract the info.

Python is an amazing tool for file processing/parsing and the one we'll be using.  We will use some regex magic and string commands to transform the gdb raw output into a text file similar to this: 
$ cat -n mtd.txt
     1 main -> A:A()
     2 A -> B:B()
     3 B -> C:C()
     4 main -> A:run()
     5 A -> B:stepOnce()
     6 B -> C:beak()
     7 B -> C:flap()
     8 B -> C:shake()
     9 B -> C:clap()
This string format, <object> -> <class>:<method>(), is compliant with Web Sequence Diagram, simply copy-n-pasting in the contents into the web-app will produce magic.  More on that later, let's turn our head toward the necessary Python script.
$ cat -n mkMtd 
     1 #!/usr/bin/python
     2 import re;
     3 import sys;
     4
     5 # https://www.websequencediagrams.com/
     6
     7 def methodName(S):
     8   retVal="";
     9   m1=re.search(".+ (.+)::(.+)\((.+)\)",S);
    10   if m1:
    11     retVal="%s:%s()"%(str(m1.group(1).strip()),str(m1.group(2).strip()));
    12   else:
    13     m2=re.search(".+ in (.+)\(.*\) (.+)",S);
    14     if m2:
    15       cName=' '.join(m2.group(2).split(' ')[1:]).split('.')[0];
    16       retVal="%s:%s"%(cName, str(m2.group(1)));
    17     else:
    18       m2=re.search(".+ (.+)\(.*\) at (.+)",S);
    19       cName=m2.group(2).split(".")[0];
    20       retVal="%s:%s"%(cName, str(m2.group(1)));
    21   return retVal;
    22
    23 def parseDebugOutput(fileName):
    24   with open(fileName, 'r') as fp:
    25     C=fp.read();
    26   lastLine=(None,None);
    27   noDupCallMap=dict();
    28   for line in C.split('\n'):
    29     callerX=re.search("#0 .*",line);
    30     if callerX:
    31       m1=methodName(line);
    32     calledX=re.search("#1 .*",line);
    33     if calledX:
    34       m2=methodName(line);
    35       mtdLine="%s -> %s"%(m2.split(':')[0],m1);
    36       print mtdLine;
    37
    38 inFile=sys.argv[1];
    39 parseDebugOutput(inFile);

You run this delicious little bastard as follows:

$ ./mkMtd ./gdb.log

And it spits out Web Sequence Diagram compliant input commands;

Export the results into a PNG and you can include it in your design documentation;

With a bit of additional work, the diagram creation could be also automated by using the code from a previous post: https://dragonquest64.blogspot.com/2020/05/python-generated-sequence-diagrams.html

It's worth noting that while this method have time-and-time again proven useful to me, it presents a specific challenge;
You're likely to use this on a sophisticated system, one with dozens of classes, hundreds of methods and setting a breakpoint in each of them is technically possible, your diagram will quickly become an eye-sore.  The challenge is carving out the uninteresting methods from the breakpoints or the gdb log file and that process can be time-consuming.  I'd argue, not as time-consuming as spending dozens of hours browsing source code, but it will take a time investment of trial-n-error.  So, be prepared to spend some time on that.

I've used this technique in multi-process systems (capturing and displaying message entry/exit points), investigated in-memory DB accesses (during system initialization) and executed this capture/analysis on specific user scenarios.  It's an incredibly useful technique, produces valuable information, but takes some fine-tuning to find the right balance in breakpoint/method captures.

Cheers.


Monday, August 17, 2020

Flaunt Your Font Want

 

Last night, my wife and I rewatched Jurassic Park, later that night I drifted into a blissful slumber.  My unconscious mind wandered vastly into the future where a futuristic archaeologist chipping away an amber-encased USB device, having been perfectly preserved for 3000 years.  The contents, all my life's work; software projects, audio, video and including a copy of this blog post.  The archaeologist releasing the USB device from it's amber container goes deep into the cellar of cellars to retrieve a primitive device capable of reading this archaic media.  Amazingly, all my life's work comes to life once again, a group of interested historians eagerly scan the work only to soon show disappointment.  "Arial?  A dull, Arial font?  Clearly this work is of no substance." as they close the computer, remove the USB device and promptly toss it into the nearest waste basket.  A record of my life for the 'after people' only to be discarded like common trash do to lack of creativity in the font game.  No, no I say, we'll step up our font game now and forever for myself, for you as well as for the after people.

I've never really had much of a need for non-system fonts.  Most user interfaces I've developed were for engineers who cared little for cosmetics, have yet to ever work with a Ux designer nor a graphics designer in any amount of detail.  The default system fonts as a result have always been 'good enough' as a result.  In fact, font files have always been a mystery to me, never spending any time thinking about them at all.

But not today, today we're gonna spend a little time understanding what they are and if you're like me you may even be surprised.  After decades of not caring about fonts, why today?  Why now?  Whelp, as I've been creating images and video content it becomes valuable to provide some form of text overlay.  I've always used the system fonts, but suppose you are tweaking a military-style video you'd likely want to add a military-style font.  That opens the door to understanding where to find and install custom fonts to suit all your creative needs.

 
vs.

So the remainder of this post will touch on how to download and use custom fonts in images, but they can easily be used in video, slideshows,....

For years, font files were simply 'magic', but let's look at one briefly.  Smarter folks may already know this, but a font ttf file can be viewed as an image; behold, FreeeSerif.ttf

$ display /usr/share/fonts/truetype/freefont/FreeSerif.ttf


The image shows the alphabet, numerics and special characters.  It then demonstrates a variety of the fonts at differing font sizes.  This can give you a feeling as to whether the font strums your creative chord.

So, where does our search begin when seeking the elusive font?  Here is a good start, and the source we'll be using in the rest of the post: www.1001freefonts.com

Assuming that we will download, extract and install the fonts to a local directory, the process takes the form:

$ wget -P TEMP https://www.1001freefonts.com/d/4018/eraser-dust.zip

$ cd TEMP; 

$ unzip -o eraser-dust.zip; 

$ mv EraserDust.ttf ../EraserDust.ttf

$ rm -rf TEMP;

The result, a brand new shiny EraserDust.ttf font file just waiting for us to use.


Using ImageMagick, we can create an image using the new font with some meaningful text:

$ convert -background None -fill white -font ./EraserDust.ttf -pointsize 96 label:"EraserDust.ttf"  -rotate -15 foo.png

$ display foo.png


Now, you can make your own thumbnails, augment video, update documents with your newly acquired custom font.  Imagine how much more fun your engineering documentation can be using a whimsical font style :thumbsup


With our new found powers, let's download and install a series of fonts, use Imagemagick to create an image showcasing the new font, then slap them all into a video displaying each font for 1 second.  I find a makefile suites this purpose pretty well, the chained commands for the video mp4 target is a bit clumsy as a series of shell commands, but it's good enough to convey the steps.

$ cat -n Makefile 
     1 all: background.png video.mp4
     2
     3 video.mp4: FridayStroke.otf TopSecret.ttf BostonTraffic.ttf Camouflage.ttf DrippingMarker.ttf EraserDust.ttf SnackerComic.ttf
     4 ${SH} i=0; for f in `echo $^`; do \
     5 echo $$i; \
     6 echo $$f; convert -background None -fill white -font ./$$f -pointsize 96 label:"$$f"  -rotate -15 overlay.png; \
     7 composite -gravity center overlay.png background.png frame-$$i.png ; \
     8 i=$$((i+1)); \
     9 done
    10 ${SH} ffmpeg -r 1 -i frame-%d.png $@
    11
    12 background.webp:
    13 ${SH} wget -O $@ https://i.vimeocdn.com/video/521265093.webp?mw=1000
    14
    15 background.png: background.webp
    16 ${SH} ffmpeg -i $< -vframes 1 $@
    17
    18 SnackerComic.ttf:
    19 ${SH} wget https://www.1001freefonts.com/d/4631/snacker-comic.zip -P TEMP
    20 ${SH} cd TEMP; unzip -o *.zip; mv *.ttf ../$@
    21 ${RM} -rf TEMP
    22
    23 FridayStroke.otf:
    24 ${SH} wget https://www.1001freefonts.com/d/26677/the-friday-stroke.zip -P TEMP
    25 ${SH} cd TEMP; unzip -o the-friday-stroke.zip; mv The\ Friday\ Stroke\ Font\ by\ 7NTypes.otf ../$@
    26 ${RM} -rf TEMP
    27
    28 TopSecret.ttf:
    29 ${SH} wget https://www.1001freefonts.com/d/7052/top-secret.zip -P TEMP
    30 ${SH} cd TEMP; unzip -o top-secret.zip; mv Top\ Secret.ttf ../$@
    31 ${RM} -rf TEMP
    32
    33 BostonTraffic.ttf:
    34 ${SH} wget https://www.1001freefonts.com/d/3269/boston-traffic.zip -P TEMP
    35 ${SH} cd TEMP; unzip -o boston-traffic.zip; mv boston.ttf ../$@
    36 ${RM} -rf TEMP
    37
    38 Camouflage.ttf:
    39 ${SH} wget https://www.1001freefonts.com/d/12584/camouflage.zip -P TEMP
    40 ${SH} cd TEMP; unzip -o camouflage.zip; mv CamouflageW.ttf ../$@
    41 ${RM} -rf TEMP
    42
    43 DrippingMarker.ttf:
    44 ${SH} wget https://www.1001freefonts.com/d/14693/a-dripping-marker.zip -P TEMP
    45 ${SH} cd TEMP; unzip -o a-dripping-marker.zip; mv adrip1.ttf ../$@
    46 ${RM} -rf TEMP
    47
    48 EraserDust.ttf:
    49 ${SH} wget -P TEMP https://www.1001freefonts.com/d/4018/eraser-dust.zip
    50 ${SH} cd TEMP; unzip -o eraser-dust.zip; mv EraserDust.ttf ../$@
    51 ${RM} -rf TEMP
    52
    53 clean:
    54 ${RM} *.otf *.ttf *.png *.mp4 *.webp
    55

Lines 18-51 has a series of font targets, each downloading the font, extracting it in a disposable directory, moving the TTF file, then cleaning up.  

Lines 3-10 do the meat of the work, first defining each font file as a dependency (the make engine satisfying the dependency by executing the necessary recipe), then create a transparent overlay with the font name, slapping that over a background image and repeating for font.  
Finally, line 10 converts the images into a video slide show, 1 second per slide.



Now your amber-encased creative works will have some flair.


Sunday, August 9, 2020

Makefile Mystique



Greetings to my loyal Russian bots that frequent my blog, contributing to the dozens of weekly views....greetings my virtual comrades!  Let's jump on into it.



Originating in the work by Stuart Feldman at Bell Labs in 1976, the make utility has existed in a number of fashions since and is one of the most common build utilities for *nix based systems.  Despite that however, in my 16+ years of professional software development, authoring or maintaining makefiles takes on a classic game of 'not it!!' seemingly everywhere I work.


Few would argue that the utility lacks flexibility or power, the general complaint is the syntax/semantics are confusing and unmaintainable, one of the primary reasons that popular IDEs synthesize their own makefiles in an attempt to isolate the user from the pain and misery of doing it themselves.  The goal of this post isn't to complain about the utility, but instead to work through a few examples in an attempt to better understand it myself.  While authoring and maintaining a makefile may feel like a prostate exam, it's also likely as necessary as one.  In preparing for this post I referred the documentation here and I invite you to do the same.


Part of make's popularity and power is because of it's implicit rules.  Making use of these rules you'll find that like good liquor, a little goes a long way.  This is evident for example when your project utilizes C/C++.


With a simple source file and relying on implicit make rules the necessary makefile is simplistic;



$ cat main.c
#include <stdio.h>
int main()
{
printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}


A single rule comprises the makefile and provides a simplistic, minimalistic build system.


$ cat Makefile
main: main.o


Each makefile consists of 'rules' taking the form;

     target ... : prerequisites ...

     <tab> recipe

     <tab> ...


Examining the rule we find the target is defined as 'main' with a prerequisite of 'main.o'.  This simply means that in order to create 'main' the 'main.o' file must exist.  The absence of a recipe relies on the implicit rules.  This can be observed by looking at the output when running make as below;



$ make
cc -c -o main.o main.c
cc main.o -o main


The existence of implicit rules comes with some disadvantages, namely it's easy to not understand what is being done for you.  Conceptually, the implicit rule that generates the object files takes the form of the prefix rule below;



$ cat Makefile
main: main.o

.c.o:
    ${CC} ${CPPFLAGS} ${CFLAGS} -c $^

Understanding what is going on allows tailoring the behavior without explicitly defining a rule.  Note the usage of the CPPFLAGS and CFLAGS variables.  Tweaking the original makefile will allow us to add debugging info and specifying an optimization level 3 as below;



$ cat Makefile
CFLAGS += -g -o3
main: main.o

This results in a slight difference when we run make;



$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

The foundation of make is detecting changes to the prerequisites and determining when the targets need to be remade.  This can be observed by re-running make immediately after running make, the result is a notification that "'main' is up to date".  Affecting the main.c file timestamp by modifying the file or simply touching it will result  in the need for the rule to be applied once again.




$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main

user@kaylee:~/make.blog/C$ make
make: `main' is up to date.

user@kaylee:~/make.blog/C$ touch main.c

user@kaylee:~/make.blog/C$ make
cc -g -o3 -c -o main.o main.c
cc main.o -o main


Likely, you've seen this all before, but stay with me I assure you there's more interesting things to come.


Often, it's preferred to have a target that cleans up the directory and allows building from scratch.  The convention is to name such a target clean.  Below is a modified makefile that defines a clean target that simply deletes the executable and the object files.



CFLAGS += -g -o3
main: main.o
clean:
    ${RM} main main.o


Executing 'make clean' will result in deleting main and main.o files.  Adding a file to your project can be accomplished by adding the object file to the prerequisites for main and recipe for the clean target or we can make use of pattern.  We'll do this by explicitly defining each of the C source files in a variable, then perform a list replacement substituting the *.c with *.o extensions to get our object file list.  The object file list can then be used in the target prerequisites and in the clean target recipe.  Adding a file to the SRCS variable rather than duplication in multiple locations.



$ cat Makefile
CFLAGS += -g -o3
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
main: ${OBJS}
clean:
    ${RM} main ${OBJS}



Still however there is duplication, namely the multiple references of main, that can be addressed by a new variable definition.



$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=main.c
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}

clean:
    ${RM} ${PROGS} ${OBJS}


Definitely on the right path, but the addition of a file requires modification to the makefile.  The wildcard expansion demonstrated in the following makefile.  The addition or removal of a file with the .c extension in the current directory will take effect in the wildcard expansion.




$ cat Makefile
CFLAGS += -g -o3
PROGS=main
SRCS=${wildcard *.c}
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}
clean:
    ${RM} ${PROGS} ${OBJS}




Let's look at some less typical usages of make which gives us a bit more insight into the creation of targets, prerequisites, and recipes.  Imagemagick is a common utility that we'll be making use in the following examples.


We'll build up the makefile as we go, incorporating what we've learned above.  We'll be satisfying the same objectives using two forms of makefiles; one that makes use of suffix rules, one that makes use of pattern rules.


Let's begin by defining our objectives.  Suppose our project requires taking in a list of JPG files and converting each into a series of other image file formats, namely PNG, GIF, JP2 and XWD files.


Using the suffix rule syntax, the makefile can begin taking the following form;




$ cat Makefile.suffix
.SUFFIXES:
.SUFFIXES: .jpg .png .gif .jp2 .xwd

all: image.xwd

.jpg.png:
    ${SH} convert $< $@

.png.gif:
    ${SH} convert $< $@

.gif.jp2:
    ${SH} convert $< $@

.jp2.xwd:
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd


The all target consists of the default target, the prerequisite of image.xwd.  In other words, make is complete when an up-to-date image.xwd file exists.  How it arrives at it is make magic, more precisely a series of suffix rules.  A series of prefix rules chaining is required to get to the final XWD file, each target we can kick off by explicitly specifying on the command line.  Specifying 'make -f Makefile.suffix image.png' results in firing of the .jpg.png suffix rule.  The suffix rules are chained as each must fire to arrive at the final XWD file.  Running 'make' performs this by stepping through a series of recipes; JPG => PNG => GIF => JP2 => XWD.



$ make -f Makefile.suffix
convert image.jpg image.png
convert image.png image.gif
convert image.gif image.jp2
convert image.jp2 image.xwd
rm image.jp2 image.gif image.png


Notice the final step removes intermediate files which can be preserved which can be prevented by adding ".PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd" line which tells make not to remove the intermediate files with the specified extensions.  The example is a bit fictional but done to demonstrate suffix rules and chaining.  Modifying the source file image.jpg followed by rerunning make will result in converting the new file to each of the alternative file formats.


As is, each prerequisite is generated via suffix rule chaining to completion before moving on to the next prerequisite.  In other words, if you specified image.xwd and image01.xwd the image.xwd would be generated to completion (ie. JPG => PNG => GIF => JP2 => XWD) before moving on to image01.xwd.


Meeting the same goals, let's utilize pattern rules rather than suffix rules which are somewhat dated in use.




$ cat Makefile.pattern
.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd
all: image.xwd

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} *.gif *.jp2 *.xwd



The most noteworthy difference between the target/prerequisites.  The prefix rules define a target, the pattern rule defines a target and prerequisite making the illusion of the rules being reversed.


What if you want to convert each input images to Jpgs before moving on to Gifs before moving on to Jp2s before the Xwds.  This can be done by specifying a wildcard expansion for the source files and using the substitution expression for each of the formats then specifying each of the formats in as prerequisites for the all target, as follows;



$ cat Makefile.pattern

.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd

SRCS=${wildcard *.jpg}

PNGS=$(subst .jpg,.png,${SRCS})

GIFS=$(subst .jpg,.gif,${SRCS})

JP2S=$(subst .jpg,.jp2,${SRCS})

XWDS=$(subst .jpg,.xwd,${SRCS})

all: ${PNGS} ${GIFS} ${JP2S} ${XWDS}

%.png:%.jpg
    ${SH} convert $< $@

%.gif:%.png
    ${SH} convert $< $@

%.jp2:%.gif
    ${SH} convert $< $@

%.xwd:%.jp2
    ${SH} convert $< $@

clean:
    ${RM} ${PNGS} ${GIFS} ${JP2S} ${XWDS}


This way, each format is fully satisfied before moving on to the next.  Perhaps less necessary for image files, more applicable for generating source files.  For example, the Protobuf message compiler allows generation of header/c++ files which also allows interdependencies between message files.  This requires all the message files to be converted to C/H files before firing the compilation, otherwise a source file may reference a header file that hasn't been created yet.



Dasvidaniya my loyal Russian bot army.

Tuesday, August 4, 2020

Video Transition Effects w/FFmpeg




Whether it be Instagram, Twitter, Snapchat, a vlog or YouTube, unless your uploading a uncut unedited video you're likely to incorporate some form of video transition(s). Scene changes, representing passage of time, representation of ambiguity can all be represented by common forms of video transitions. Investing a little effort in post production can transform a good video into a great one, a lengthy story into a shorter one and a long-winded story into a compelling quicker one.

FFMpeg can readily support some of the most common video transitions, and in this post we'll touch on some common transitions and how they can be done effectively with FFMpeg.

I hesitate referring to myself as an amateur videoographer, but over the years I've spent a good deal of time toying with videos. Recently, I've been researching the art/science to choosing the 'right' video transition, rather than simply arbitrarily choosing one from a hat. Recently I came upon this post which spends a bit of time explaining when/why to use particular video transitions. I'll touch on some of the key points and demonstrate simple examples on how to perform them with FFMpeg.

Fade

A fade-in consists of starting with a solid color (e.g. white, black) and smoothly morphing into the video shot.  Similarly, a fade-out smoothly morphs from the vido shot to a solid color.  Often, a fade-in and fade-out are coupled, fading out of the current scene followed by fading-in to the new scene.

This post focuses on providing overviews, more details for this form of transition can be found in this previous post.

Fade In

Often, a video will begin with a fade-in, much like most common movies.  This technique allows to slowly introduce the viewer into a new environment or scene.  Most often, fading in from black, which can be demonstrated by the following example;

Given a 24fps video, we can slowly fade in on the first 5 seconds like this:

$ ffmpeg -y -i video.mp4 -vf fade=in:0:120 -acodec copy fadeIn-5sec.mp4

The fade filter takes in frames, rather than time, so it requires knowledge of the video frame rate to convert time to frame range.

Fade Out

Fading out can signify the passing of time, a sense of completion, or represent ambiguity.  Let's look into an example.

Given a 24fps video, we can slowly fade out on the last 5 seconds of the video.  A 10 minute video at 24 fps (24*60*10 = 14400), 5 seconds (5*24=120) would be accomplished by the following;
$ ffmpeg -y -i video.mp4 -vf fade=out:14280:14400 -acodec copy fadeOut-5sec.mp4

Dissolve

Given two disjoint scenes, another common technique is to slowly transform directly from one scene into another.  This can represent the passage of time, bur more commonly represents moving from one location to another.  Consider an interview taking place in an office environment followed by continuing the interview in the street.  A rapid jump from office to street can be abrupt to the viewer, often remedied by dissolving from the first scene into the other.  Quick dissolves can represent a quick passage of time, while slower dissolves can represent the passage of months or years.

This effect is sometimes referred to as a crossfade, given two source videos we can dissolve or crossfade from the first to the second as follows:

$ ffmpeg -i image01.mp4 -i image02.mp4  -filter_complex "[0:v][1:v]blend=all_expr='A*(1-min(T/1,1))+B*(min(T/1,1))'" blend.mp4

A more comprehensive discussion with examples can be found here.



Wipes

Wipes can convey the existence of concurrent story lines, swapping one for another.  These can take on a variety of forms, left-to-right, right-to-left, upward, downward or any form of diagonals. 

An example of a wipe-right can take the form:
$ ffmpeg -i image01.mp4 -i image02.mp4 -filter_complex "[0:v][1:v]overlay=x='min(0,-W+(t/1)*W)':y=0[out]" -map "[out]" -y wipeRight.mp4

Like the fade-in, this filter doesn't use time, rather the units are pixels so authoring a 1 sec wipe will take some math, but if you're flexible in the wipe duration you can simply play with the filter arguments.

This past post describes a variety of wipes in more detail.

Zoom

Zooming in/out can communicate to the viewer 'pay attention to this', zooming in focuses the viewers attention to the subject of the zoom.  Zooming out can communicate attention from a particular subject out to the world at large, often done to represent an ending of the film....attention from a particular person out to the world a means of fading away from the subject.

Zooming is a bit more complicated than some of the other transitions, a simple example takes the form:
$ ffmpeg -y -i target.mp4 -vf "scale=iw*2.0:ih*2.0,zoompan=z='min(max(zoom,pzoom)+0.05,5.0)':d=1:x='560*2.0-(560*2.0/zoom)':y='400*2.0-(400*2.0/zoom)'" -an output.mp4

A more detailed explanation can be found here.



These are a few examples of common video transitions, I may spend some time on more sophisticated ones in the future. 

Now, go make your good videos great.