Will The WFH Trend Result In An Increase In Outsourcing?

Photo by Oleg Magni from Pexels

In general, I desperately try to see the good in things; every cloud has a silver lining style stuff. Typically, if you look hard enough you can find good in most things, but you tend to have to dedicate a real effort to find it.
This dumpster fire of a year has required a herculean effort from the world to find the good in a historically crappy year. Remember when 46 million acres of Australia burned to ash and we were all thinking that would be the disaster of the year? Devastating yes, but not nearly the 'disaster of the year nominee' we may have thought at the time. Fast forward to the world's pandemic, a worldwide unemployment rate of 8.3%, businesses encountering challenges of a lifetime, young and old affected and the mental health of the world just teetering on the razors edge between partial insanity and full-blown mental breakdown. Finding good, any modest level of good, is like an epic game of Where's Waldo.

In search of good some, I'd even argue many, have found a glimmer of a positive in the form of a world-wide 'working from home (WFH)' policy change. Personally, I have found solace in the fact that I enjoy the WFH environment much more than I had expected. I was always a 'go into the office guy', preferring the separation, and as a life-long participant in classical conditioning leaving home to go into the office was a means of removing distractions and turning on 'work mode'. Work would certainly follow me home, but the majority of work was conducted at the office. Enter economic and health survival mode; those that can perform work from home are encouraged to. A nation, a world, thrust a massive population into the WFH deep end, they with a doggy paddle, progressing to treading water and on our way to finding proficiency in the 'new norm'. Most of my colleagues and friends feel the WFH policy, full or part-time, is a big win for many and anticipating the policy to continue post-pandemic. Trading the frustration, cost and time wasted in traffic for added personal time can be a big win for the current and future's workforce. 

So, if you're like me, you set your eye on this glimmer of good on the horizon when you can anticipate the benefits of working from home when the world has reclaimed some form of normal. You look to a future to a better world where your commute remains 12 feet from your bed, society has returned to some form of normal and you continue to have the ability to use your lunch break to take your dog for a walk or have lunch with your spouse/partner.

Desperate to find the negative in this sliver of a perk, the dark shadows of my unconscious push a thought into my head; "if the workforce transitions to primarily remote, will outsourcing become the new norm?". A well-delivered shin-kick to my fragile emotional state delivered with precision and purpose, your future job security is now on the table....well done dark forces....well done.

Personally and professionally I have some very strong opinions on outsourcing and as a courtesy to any dedicated readers that have made it this far I'll refrain detailing my opinions but suggest that collaborating teams require timely communications with one another and conflicting timezones are the supervillain in such matters. Perhaps in time we will find a way to work effectively in highly remote settings, or corporate culture will change to accommodate. My current team has folks in Florida, the midwest, and California, a 3 hour difference at the extremes. If our culture continues to be flexible in remote worker hours of 9-5 (locally) that means folks lose 3 hours of shared communication time, or are forced to adjust. As a software engineer, it's not uncommon for managers to be even more accommodating, allowing team members to start later or earlier which can potentially compound the issue. 

Perhaps as a nationwide workforce we fail to work effectively and a WFH policy becomes a failed experiment only to return to localized teams in cube farms. Alternatively, maybe we evolve into a workforce and find ways to address such issues, timezones become irrelevant and our workforce comes from a worldwide pool of talent.

As an industry we should recognize the possibilities solving remote and WFH policies present and strive to become as effective (or more effective) than colocated teams. As a workforce we should recognize the complications we can add, the benefits it comes with and the risks to our profession that may be on our horizon. 

Only time will tell.

Generating Multi-Plot Real-Time Plots with Python

In an earlier post the real-time plotting capabilities were demonstrated, we're extending on this by showing how to generate multiple plots simultaneously.  A couple noteworthy observations, in the past post the X and Y scaling was automatically scaled after each element addition.  While you can still do this, typically for multiplots we would prefer maintaining a shared X range.  While somewhat unnecessary, I've elected to maintain a uniform Y range.

#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test02():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(311);
  ax2=fig.add_subplot(312);
  ax3=fig.add_subplot(313);
  l1,=ax1.plot(100,100,'r-');
  l2,=ax2.plot(100,100,'r-');
  l3,=ax3.plot(100,100,'r-');
  time.sleep(3);

  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i),cos(i),cos(i*2)));
    T1=[x[0] for x in D];
    L1=[x[1] for x in D];
    L2=[x[2] for x in D];
    L3=[x[3] for x in D];

    l1.set_xdata(T1);
    l1.set_ydata(L1);

    l2.set_xdata(T1);
    l2.set_ydata(L2);

    l3.set_xdata(T1);
    l3.set_ydata(L3);

    ax1.set_xlim([0,50]);
    ax2.set_xlim([0,50]);
    ax3.set_xlim([0,50]);
    ax1.set_ylim([-1.5,1.5]);
    ax2.set_ylim([-1.5,1.5]);
    ax3.set_ylim([-1.5,1.5]);

    plt.draw();
    i+=0.10;
  show(block=True);

#---main---
log("main process initializing");
test02();
log("main process terminating");

Easy Peasy;

Atypical Uses for Makefiles

Makefiles traditionally center themselves in a build process.  While they have taken a backseat to more modern build utilities, oftentimes they are simply that....in the backseat.  Eclipse, for example, autogenerated makefiles as part of a project build.  Why?  Because when properly done, makefiles are extraordinarily powerful.  Its dependency engine can parse large projects and selectively execute only what needs to be rebuilt.  Done poorly, you're at the mercy of 'make clean; make all' sequences to make sure everything is built up-to-date.

Aside from it's power as a build utility, make can be useful for other purposes as well.  Over the past couple years I've extended my use of make into a few additional areas where I find the dependency engine to prove to be very useful.  This post will touch on a few.

Quick Introduction to Makefiles

Whether you are familiar with makefiles or not, consider taking a peek at this past post.  While it touches on the general syntax and utility of make, in the later sections it shows some atypical uses.  As a means to demonstrate the dependency engine, it shows how Imagemagick can be used to transform images of varying file types.  A daisy-chained dependency chain can be created by converting an image through a series of steps JPG => PNG => GIF => JP2 => XWD.  Perhaps unpractical, but it shows an atypical use of make and it provides a simple and visible execution of the dependency engine.

Poor Man's Parallelism

I'm particularly proud of this usage, to my knowledge no one else has ever recommended this as a use of make.  Most, familiar with make, know that it can be multi-threaded and the dependency engine fires off the specified number of threads to speed up the process.  Given a large text file, make can provide an easy means to parallelize the process.  The trick; split the file into segments, process each segment, then join the results.  In ~30 lines of code, you can greatly improve the execution time for simple parallel processing, all enabled by the sophisticated dependency engine at the core of make.  Take a peek at the details here.

FFmpeg

While I've toyed to a good degree with bash scripts, python scripts and a variety of other shell utilities I've come home to make as the best tool for FFmpeg-based tasks.  Video conversions and/or modifications lend themselves nicely to makefiles.  Suppose you have a large directory full of images and you wish to create a slideshow....this post is a good starting point to showing you how makefiles can make that an easy process.  Want to take 6 hours of raw camera footage and transform it into something worth watchable....this post shows how that can be done.  Want to download a Youtube video and apply a series of transformations to create some completely new content.....this post can give you a head start.

Some of the beauties that were created by these posts;

Shell Script Replacements

Makefiles have become my adhoc replacements for simple shell scripts.  Certainly, they are limited to execute simplistic series of commands and are no substitute for sophisticated needs, but chances are if you have to execute a series of shell commands in a pre-defined order a makefile will scratch that itch.  Sophisticated recipes can become clumsy, and I've found myself authoring some ugly, ugly, ugly recipes, but recently I found a workaround.  Suppose you have a series of 5 tightly-bound commands and find it difficult to define as a recipe, a trick I've found useful is creating a shell script by means of a recipe, then using it in another target.

all: somethingCool.sh

        ${SH} ./$<

somethingCool.sh:

        ${SH} echo "echo 'doing something cool'" > $@

        ${SH} echo "sleep 1" >> $@

        ${SH} chmod a+x $@

clean:

        ${RM} somethingCool.sh

Notice the helper shell script is created by make as a dependency of the all target, used as needed and can be safely removed, recreated when later needed.  Certainly, this falls apart if the creation of the shell script by make is unnecessarily complex, but in many of my projects the only file under version control is the makefile, the rest are created as part of the project and removed by the clean target.  Nice and clean.  Rapid prototyping an alternative script can be done by a new target and once you get the hang of it can prove to be quite powerful.  The make, make run, make clean becomes the holy trinity of rapid prototyping.

I've found adhoc usage reports, grabbing debug logs, grepping for significant events, sorting and tallying event counts really align with raw, events, report targets and recipes.

Cartoonize Video with FFmpeg -- Take 2

 Some time back, I posted a rough attempt to convert a video into a cartoon-style equivalent; here.  That technique converted the video into a series of still images and ran each through ImageMagick sketch filter.  The result looked something like this:

Since then, I play with this time and time again, revising it to something like this:

Again, recently I thought of another technique.  Often, cartoon coloring schemes look 'flattened', meaning red'ish elements are colored red, blue'ish colored blue, so on and so forth.  The 'gex' filter can perform such a conversion pixel-by-pixel and seems promising.  It would require applying a series of filters, for each color so I thought I'd give it a try on simple grayscale first...if it showed promise then expand on a colorized equivalent.  Here is what I got, seems promising:

The technique, in a nutshell, is to convert the video to grayscale, then perform a selective filter on each pixel.  Pixels within the [0,50] luminosity range get assigned 0, pixels within the (50,100) range get assigned 50;

[0,50] = 0

(50,100] = 50

(100,150] = 100

(150,200] = 150

(200,255] = 200

The filter can be a mouthful, taking the form:

$ ffmpeg -i input.mp4 -filter_complex "format=gray,geq=lum_expr='if(lte(lum(X,Y),50),0,if(lte(lum(X,Y),100),50,if(lte(lum(X,Y),150),100,if(lte(lum(X,Y),200),150,if(lte(lum(X,Y),255),200,0)))))'" -acodec copy cartoonX.mp4

Notice, it's a chain of less-than-equal-to conditions of the form if(condition,X,Y) which means if condition set pixel to X, otherwise Y.  To execute the series of else conditions, a nesting of alternative conditions are replaced for Y.  Ugly, but effective and given it doesn't require converting a video to frames and back again a ton faster; still slow'ish, but way faster than the alternative.

Seems to me that this shows a good deal of promise.  The result is dark'ish, but that could be addressed by migrating the pixel assignments to the ceiling of the range, or perhaps applying lightening filter pre-filtering could also address it.  Seems plausible for converting a video to a grayscale cartoon'ish equivalent, using the same technique for color seems like a good next step.

Deep Thinking / How To Be A Better Software Engineer

Over the years you get to understand what 'works' for you and oftentimes you'll only later find that what works for 'you' often works for 'others' and is also known by 'others'.  And sometimes, decades after you learn these tricks and techniques you discover that they are readily known and even recorded in print.  Here are a few techniques that can improve your thought process, translating into better software products and contributions.

Thinking In Motion

My first job in software engineering was at a 40-acre engineering facility, a series of buildings covered by a massive superstructure.  Rain, sleet, or snow you had 40 acres of walking available, uncompromised by environmental conditions 24/7.  The typical ritual was a lap around the facility after lunch seemed to evolve into a routine whenever you hit a mental block.  Struggling on a design or implementation often meant figuratively, or sometimes literally, banging your head against the wall.

So there you are, trying to work out a troubling problem, frustrated and in time the frustration turns to a bit of anger.  Surrounded by your office mates, your workspace is no place to lose your cool so you decide by happen-stance to take a walk, mostly just to relocate to a place you can grit your teeth and maybe cuss a bit under your breath.  Your focus, still on the problem at hand, but now you're out and about thinking, venting while in motion.  Like the sun pushing through the dark and stormy clouds a solution begins to emerge, first as a glimmer of possibility evolving into a full-blown solution.  Now, armed with a solution you double-time it back to your desk reinvigorated and supercharged to prove it out.  

Weeks pass, you find yourself in a similar situation; puzzled and frustrated, you take a walk, arrive back at your desk with a solution and implement it on your return.  Again, and again, this proves to be an effective process for you, and as far as you're concerned it works exclusively for you.  Later, you find your co-workers have arrived at the same process.  Later in life, you learn this exercise-based thought process is well-documented by the likes of Scientific America and such.

In general, even mild exercise increases blood flow which energizes the old cerebrum and hippocampus and pull you out of your mental funk.

Thinking in a Free Space

You spend a good portion of the day trying to solve something, but it continues to evade you. Tattered and defeated you pack up for the day, make your way to your car and begin your commute home.
You can get stuck on a small number of associations that aren't working when consciously trying to remember something. When you relax, your mind more freely and randomly associates, which gives it a chance to find a different association to the information then the fruitless ones you were consciously pushing on. A guilty pleasure comes on the radio and you proceed to car dance your way home to a Hungry Man's dinner and a cold beer. Mid-song you're hit square betweeen the eyes with a solution to your day's problem......BAM!! Like your 'take a walk' process, this happens again and again, and you're left with "why the heck does this work?".


Some say that the reason this works is that your brain is an association solving machine. Sometimes you get stuck on a small number of associations on the conscious level that won't solve your problem, but you continue to try to make them work and continue to fail. Then, your mind set free by a ridiculous radio song it is free to more freely and randomly find associations and possibly landing on a completely new thought that will solve your problem. Sometimes, the best way to solve a problem is to set it aside for a bit and 'background process it'.

Thinking in Speech

You find yourself stumbling with a problem and under the prodding of your manager you reluctantly take a walk over to your coworker for an assist.  You're not sure how to solve it and begin to explain the problem to your teammate, you describe the problem, you describe what you've tried and mid-sentence you stop, a new solution comes to mind and you return to your desk.  The act of verbalizing the problem and your thought process opened new doors and didn't even require a contribution from your teammate.  Sometimes simply talking the problem out outloud allows you to arrive at new solutions.  A newly found thought trick, goes right into your personal toolbelt.  In time, you later find others have arrived at the same tactic and it is often known as the 'rubber ducking' principle as described in the 'The Pragmatic Programmer'.  The audience of your verbal mind-dumping can be a teammate, your bartender, a reluctant spouse/partner/date, your pet or your given choice of any given inanimate object.

Thinking in Print

I've never known a software engineer that likes authoring documentation, perhaps they exist and are but elusive, or perhaps they simply don't exist in this universe.  Despite a general distaste of writing documentation, we tend to recognize its value and necessity to quality software and will do so reluctantly.  I find, despite not liking doing it, that the sheer act of writing stuff down betters your design.  You are forced to describe your architecture, your classes, their responsibilities, their interactions and you find yourself internally challenging your design 'why did i do this', 'what if i had done that', 'what are this particular classes responsibilities'.  In doing so, you leave with a list of refactoring opportunities and/or changes to the design and make the time to incorporate them, ending with a more robust and sophisticated product.  I find this likely a simple extension on verbalization, as it gives you the same benefits.

These are likely but a few tips and tricks for supercharging your thought process.  To my knowledge these tricks appear to be pretty well-known, some well-established and documented, some passed along via developer palaver over drinks.  My gift to you.