Tuesday, July 7, 2020

Mandelbrot Set with Python



Data can be beautiful.  Visualizing data is a worthwhile skill to acquire and it's relatively simple with Python.  Let's explore how to do some data visualization as an exercise.

The Mandelbrot set is often regarded as an example of art meeting science.  It's generated by evaluating the behavior of complex numbers and generating the results.  The end effect is a infinite and beautiful depiction of pure mathematics, an acid-trip of color and structure as you continuously zoom into the graph.

One of the best descriptions of the Mandelbrot set can be found here, I recommend you spend a few minutes to appreciate the concept before we begin graphing it with a simple Python snippet.

In about 30 lines of code we can create our own colorized visualization of the mandelbrot set.

Let's look at the source, then step into some of the details;
$ cat -n mandelbrot 
     1 #!/usr/bin/python
     2 import matplotlib.pyplot as plt;
     3 import sys;
     4
     5 def colorize(n):
     6   h="#%06x"%(int(n*2**23));
     7   return h;
     8
     9 xRange=[-2,1];
    10 yRange=[-1.5,1.5];
    11 incr=0.005;
    12
    13 x=xRange[0];
    14 while(x < xRange[1]):
    15   y=yRange[0];
    16   while(y < yRange[1]):
    17     c=x+y*1j;
    18     z=0;
    19     try:
    20       for k in range(50):
    21         z=z**2+c;
    22       if(abs(z) < 2):
    23         rgb=colorize(abs(z));
    24         plt.plot(x,y,'.',color=rgb);
    25     except:
    26       pass;
    27     y += incr;
    28   x += incr;
    29
    30 plt.xlim(xRange[0],xRange[1]);
    31 plt.ylim(yRange[0],yRange[1]);
    32 plt.savefig(sys.argv[1]);


Let's look over the non-Mandelbrot stuff first.  We're using the matplotlib library for our simple plotting example, line 9-10 we define our plot x and y ranges and enforce them in lines 30-31.  Finally in line 31 we save the plot to a figure rather than display to the screen.  We pass in the figure filename as a command line argument so ./mandelbrot foo.png would generate a foo.png file.  Lines 23-24 calculates a color for the pixel and plots it at (x,y).  Without going into details, the nested loop (lines 13-28) steps through the floating point 2D range defined by the x and y ranges, stepping by 0.005 (defined in line 11).  Each iteration, we selectively plot, or don't plot, a colorized pixel.  We plot a pixel if it's position is part of the Mandelbrot set.

The rest of the details are specific to calculating the Mandelbrot set.  In particular, lines 17,20-26.  The referenced video explores how and why this is done, but let's revisit some of the particulars.

Line 17 consists of the assignment of the complex number for each (x,y) position in the range.  Note 1j is the Python representation of complex number i.   This assignment was covered in the video, specifically, but easily overlooked;

The inclusion in the mandelbrot set for this position is characterized by how this complex number behaves under the influence of iteration of the function starting at 0.  This means, z starts as 0 and we repeatedly inject the f(z)=z^2+c in our loop, the end result will either blow up implying it's not in the mandelbrot set, or it'll remain bounded (e.g. <= 2) which means it's part of the set.  Lastly, we use the magnitude of the result to determine the color of the pixel.  This is optional, but it adds a level of beauty. We can represent the looping of (x,y) range and determination of it's inclusion/exclusion in the mandelbrot set by the clip from the reference video;

The end result is our visualization;
No go do something cool!

Tuesday, June 30, 2020

Visualization of Sorting Algorithms

Some time back a trending video made its way into my life, it provided a graphical representation of common sorting algorithms.

This simple video demonstrated the value of visualizing complex algorithms, hopefully something that is incorporated into CS coursework.  The value of divide-n-conquer techniques can be more easily understood with a side-by-side comparison.

We can generate an equivalent video by monitoring the list of numbers, generate a graph at each step and render the sequence of graphs into a video.

In order to generate each graph, we could modify the sorting algorithm to create and store a graph image.  This approach however would require modification to the source code.  An alternative, use a debugger to monitor the list of numbers and print out the list of numbers whenever it is modified.  Then, post-run, we could use the debugger output to create individual graphs and stitch them together.

So, our approach will take the form:
1) create C++ sorting algorithm
2) run w/debugger and monitor the state of the input array
3) parse the debugger log, creating a graphical representation of the input array at each step
4) stitch each frame into a video

1 - Sorting Algorithm
Let's create a simple C++ example with a variety of sorting algorithms.  We'll create a randomized list of integers and prepare to send it through a sorting routine.
$ cat -n main.cpp 
     1 #include <stdio.h>
     2 #include <stdlib.h>
     3 #include <vector>
     4
     5 #ifndef N
     6 #define N 100
     7 #endif
     8
     9 void swap(int& a, int& b)
    10 {
    11   const int tmp=a;
    12   a=b;
    13   b=tmp;
    14 }
    15
    16 void display(int list[], int size)
    17 {
    18   for(int i=0; i<size; i++)
    19   {
    20     printf("%d\n",list[i]);
    21   }
    22 }
    23
    24 //============Selection Sort============
    25 void selectionSort(int list[], int size)
    26 {
    27   for(int i=0; i<size-1; i++)
    28   {
    29     for(int j=i+1; j<size; j++)
    30     {
    31       if (list[j] < list[i])
    32       {
    33         swap(list[j],list[i]);
    34       }
    35     }
    36   }
    37 }
    38 //======================================
    39
    40 //==============Quick Sort==============
    41 int partition(int *a,int start,int end)
    42 {
    43   int pivot=a[end];
    44   //P-index indicates the pivot value index
    45
    46   int P_index=start;
    47   int i,t; //t is temporary variable
    48
    49   //Here we will check if array value is
    50   //less than pivot
    51   //then we will place it at left side
    52   //by swapping
    53
    54   for(i=start;i<end;i++)
    55   {
    56       if(a[i]<=pivot)
    57       {
    58           t=a[i];
    59           a[i]=a[P_index];
    60           a[P_index]=t;
    61           P_index++;
    62       }
    63    }
    64    //Now exchanging value of
    65    //pivot and P-index
    66     t=a[end];
    67     a[end]=a[P_index];
    68     a[P_index]=t;
    69
    70    //at last returning the pivot value index
    71    return P_index;
    72 }
    73
    74 void Quicksort(int *a,int start,int end)
    75 {
    76   if(start<end)
    77   {
    78     int P_index=partition(a,start,end);
    79     Quicksort(a,start,P_index-1);
    80     Quicksort(a,P_index+1,end);
    81   }
    82 }
    83
    84 void quickSort(int list[], int size)
    85 {
    86   Quicksort(list,0,size);
    87 }
    88
    89 void insertionSort(int list[], int size) {
    90
    91   int valueToInsert;
    92   int holePosition;
    93   int i;
    94  
    95   // loop through all numbers 
    96   for(i = 1; i < size; i++) { 
    97
    98      // select a value to be inserted. 
    99      valueToInsert = list[i];
   100
   101      // select the hole position where number is to be inserted 
   102      holePosition = i;
   103
   104      // check if previous no. is larger than value to be inserted 
   105      while (holePosition > 0 && list[holePosition-1] > valueToInsert) {
   106         list[holePosition] = list[holePosition-1];
   107         holePosition--;
   108         printf(" item moved : %d\n" , list[holePosition]);
   109      }
   110
   111      if(holePosition != i) {
   112         printf(" item inserted : %d, at position : %d\n" , valueToInsert,holePosition);
   113         // insert the number at current hole
   114         list[holePosition] = valueToInsert;
   115      }
   116
   117      printf("Iteration %d#:",i);
   118   } 
   119 }
   120
   121 void sort(int list[], int size)
   122 {
   123 //selectionSort(list,size);
   124 //quickSort(list,size);
   125   insertionSort(list,size);
   126 }
   127
   128 int main(int argc, char* argv[])
   129 {
   130   printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
   131   std::vector<int> L;
   132
   133   for(int i=0; i<N; ++i)
   134   {
   135     L.push_back(rand()%100);
   136   }
   137   sort(L.data(), L.size());
   138  
   139   display(L.data(), L.size());
   140
   141   printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
   142 }
 
On line 5-8, we declare N which represents the size of the array size, then on lines 131-136 we allocate a list of integers and populate them with randomized numbers.  Line 137 we provide the list as an argument to the sort() method.  

2 - Run w/Debugger
Macros are evil so let me justify the use of N in the C++ source file.  Typically, such a macro would be replaced with a static constant allowing more type-safe implementation.  In this example though the size of the array needs to be known by the GDB commands as well.  The makefile assigns the value of N, passing it in to the compiler as a pre-compiler directive and creates the gdb command file which also needs to know the value of N.
$ cat -n Makefile 
     1 CC=g++
     2 MAIN=main
     3 N=100
     4
     5 all: video.mp4
     6
     7 ${MAIN}: ${MAIN}.cpp
     8 ${CC} -D N=${N} -g $< -o $@
     9
    10 run: ${MAIN}
    11 ${SH} ./${MAIN}
    12
    13 plot: gdb.log
    14 ${SH} ./plotter $<
    15
    16 gdb.cmd:
    17 ${SH} echo "set pagination off" > $@
    18 ${SH} echo "break sort" >> $@
    19 ${SH} echo "commands" >> $@
    20 ${SH} echo "  watch *(int(*)[${N}])(list)" >> $@
    21 ${SH} echo "  commands" >> $@
    22 ${SH} echo "    cont" >> $@
    23 ${SH} echo "  end" >> $@
    24 ${SH} echo "  cont" >> $@
    25 ${SH} echo "end" >> $@
    26 ${SH} echo "run" >> $@
    27 ${SH} echo "quit" >> $@
    28
    29 gdb.log: gdb.cmd ${MAIN}
    30 ${SH} gdb -x gdb.cmd ${MAIN} > $@
    31
    32 video.mp4: plot
    33 ${SH} ffmpeg -i frame%05d.png $@
    34
    35 clean:
    36 ${RM} ${MAIN} *.o gdb.log gdb.cmd
    37 ${RM} *.png *.mp4

Notice on line 3 we define our array size as 100.  On line 8 we pass in the value when compiling the C++ source.  On line 20 we create a gdb command file that we will use to run with the debugger.  I've recently found it useful for makefiles to create such input files as part of the build process rather than author manually and preserve in version control.  The gdb.cmd file is generated as part of the build process and in this case allows the makefile to preserve a consistent value for N in the gdb command file and the C++ source file.

Notice on line 29, we define a  gdb.log target, which runs the executable with the necessary gdb command script and generates a gdb log file.  Line 20 is the key to the gdb script, it attaches a watchpoint to array, when the array changes it spits out the old/new value of the array.  

3 - Parse the Debugger Log
The gdb log file takes the form of:
$ cat -n gdb.log | head -50
     1 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
     2 Copyright (C) 2014 Free Software Foundation, Inc.
     3 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
     4 This is free software: you are free to change and redistribute it.
     5 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
     6 and "show warranty" for details.
     7 This GDB was configured as "x86_64-linux-gnu".
     8 Type "show configuration" for configuration details.
     9 For bug reporting instructions, please see:
    10 <http://www.gnu.org/software/gdb/bugs/>.
    11 Find the GDB manual and other documentation resources online at:
    12 <http://www.gnu.org/software/gdb/documentation/>.
    13 For help, type "help".
    14 Type "apropos word" to search for commands related to "word"...
    15 Reading symbols from main...done.
    16 Breakpoint 1 at 0x400dcc: file main.cpp, line 125.
    17
    18 Breakpoint 1, sort (list=0x604270, size=100) at main.cpp:125
    19 125   insertionSort(list,size);
    20 Watchpoint 2: *(int(*)[100])(list)
    21 Watchpoint 2: *(int(*)[100])(list)
    22
    23 Old value = {83, 86, 77, 15, 93, 35, 86, 92, 49, 21, 62, 27, 90, 59, 63, 26, 40, 26, 72, 36, 11, 68, 67, 29, 82, 30, 62, 23, 67, 35, 29, 2, 22, 58, 69, 67, 93, 56, 11, 42, 29, 73, 21, 19, 84, 37, 98, 24, 15, 70, 13, 26, 91, 80, 56, 73, 62, 70, 96, 81, 5, 25, 84, 27, 36, 5, 46, 29, 13, 57, 24, 95, 82, 45, 14, 67, 34, 64, 43, 50, 87, 8, 76, 78, 88, 84, 3, 51, 54, 99, 32, 60, 76, 68, 39, 12, 26, 86, 94, 39}
    24 New value = {83, 86, 86, 15, 93, 35, 86, 92, 49, 21, 62, 27, 90, 59, 63, 26, 40, 26, 72, 36, 11, 68, 67, 29, 82, 30, 62, 23, 67, 35, 29, 2, 22, 58, 69, 67, 93, 56, 11, 42, 29, 73, 21, 19, 84, 37, 98, 24, 15, 70, 13, 26, 91, 80, 56, 73, 62, 70, 96, 81, 5, 25, 84, 27, 36, 5, 46, 29, 13, 57, 24, 95, 82, 45, 14, 67, 34, 64, 43, 50, 87, 8, 76, 78, 88, 84, 3, 51, 54, 99, 32, 60, 76, 68, 39, 12, 26, 86, 94, 39}

When the list array is modified the watchpoint breakpoint is tripped which results in lines 23-24, outputting the old/new values.  

4 - Stitch Frames into Video
Last bit, we need a simple Python script that is capable of parsing the gdb log file and generate a image representative of the state of the array.
$ cat -n plotter 
     1 #!/usr/bin/python
     2 import sys;
     3 import re;
     4 import matplotlib.pyplot as plt;
     5
     6 def plot(fileName):
     7   print "plotting %s"%(fileName);
     8   with open(fileName,'r') as fp:
     9     C=fp.read();
    10
    11   i=0;
    12   for line in C.split('\n'):
    13     m=re.match('New value = {(.*)}.*',line);
    14     if m:
    15       L=[int(el) for el in m.group(1).split(',')];
    16       print L;
    17       plt.ylim(0,100);
    18       plt.bar(range(len(L)),L,label='');
    19       plt.savefig('frame%05d.png'%(i));
    20       plt.close();
    21       i+=1;
    22   pass;
    23
    24 #---main---
    25 plot(sys.argv[1]);

This script will parse the 'New value =' lines and generate a bar graph that represents the order of the array.  For example, an initial value of a 10 element array could look like this;
Generating a series of images, during each step of the sorting routine and stitching them together into a video using FFmpeg.

Final results, i give you;

Selection Sort

Insertion Sort

Quick Sort



Sunday, June 7, 2020

Pay Now or Pay Later

These past months have been unprecedented, having placed an unusual amount of load on unsuspecting systems.  These are the days that reveal the advantage of robust systems and the consequences of fragile systems.  Robust system engineering behave as a young willow tree when the winds of demand blow; 'bend, don't break'.  Resilient systems will survive the storm and in some cases thrive in it.  Properly designed systems prevail and demonstrate their true value in times like these. 

Like a high performance engine, or a professional athlete, properly engineered computer systems require an investment in building (time and money) as well as continuous maintenance.  Lack of maintaining these systems can have dramatic consequences; like this:

https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-programmers-to-fix-unemployment-system.html

Unemployment systems built in the 1960's provide the digital heart to the social programs for entire states.  Departments who financed the building of such systems fall prey to complacency in maintaining them and are publically pants'ed and humiliated when the consequences of their actions hit the headlines. 

No bones about it, maintaining systems is expensive it's simply a manner of how you will pay.  The cost of tech debt, or system maintenance can be viewed like that of a mortgage payment....pay it monthly, or pay it in one solid balloon payment at the end of the term; either way, you'll be paying it.  New Jersey backed into a balloon payment after years of neglect.

There are no real easy answers to these decisions, modernizing a 60 year old software-intensive system is a difficult and complex task and it's easy to be seduced to think the right call is to simply stay-fast with the existing system and I can only suspect that's what NJ did. 

The decision to not modernize isn't necessarily the lower cost option either.  Mainframes are dinosaurs in this technology era and the maintenance of these systems likely fell to single-source high-cost specialty engineering houses like IBM billing out at upwards of $200/hr; pay now and pay later.

If NJ could roll back the clock and play out the ideal narrative the hardware system would be replaced to more modern architected systems and the near-mummified COBOL would be replaced with a more modern language.  Then, single-sourced high-cost hardware would be replaced with common hardware and engineering teams well-versed in modern programming languages would open the field to all kinds of talented personnel.   This is frankly the only course of action to preserve a future of this (and other like) system.  Changes like this don't happen overnight, it's a significant time and financial investment but it requires being done, postponement costs only add to the overall cost of system replacement.


Sunday, May 31, 2020

World On Fire -- Bear Lake Day Trip

As a resident of Minneapolis sadly we're in a time where literally our city is on fire.  I considered skipping posting this week but rather decided to instead apply some of my FFmpeg knowledge to a number of day-trip media files and create a video of hiking in one of the most beautiful parks in the region.  Hoping to share some beauty in these trying times.

Took 139 pictures/videos, generated a number of video clips and panning crop filters on static images and stitched them all together using the following Makefile.

Enjoy.



$ cat -n Makefile 
     1 FPS=30
     2 JPGSECS=6
     3 Images=${wildcard */*.jpg} 
     4 ScaledImageClips=${filter-out %-scaled-scaled.mp4, ${subst .jpg,.mp4,${subst .jpg,-scaled.jpg,${Images}}}}
     5 Vids=${wildcard */*.mp4}
     6 ScaledVidClips=${filter-out %-scaled-scaled.mp4, ${subst .mp4,-scaled.mp4,${Vids}}}
     7
     8 all: final.mp4
     9
    10 final.mp4: video.mp4 audio.mp3
    11 # ${SH} ffmpeg -i video.mp4 -i audio.mp3 -acodec copy -shortest $@
    12 ${SH} ffmpeg -y -i video.mp4 -i audio.mp3 -af 'afade=t=out:st=193:d=3' -strict -2 -shortest $@
    13
    14 video.mp4: clip01.mp4 clip01b.mp4 clip02.mp4 clip03.mp4 clip04.mp4 clip05.mp4 clip06.mp4 clip07.mp4 clip07a.mp4 clip08.mp4 clip09.mp4 clip10.mp4 clip11.mp4 clip11a.mp4 clip12.mp4 clip13.mp4 clip13a.mp4 clip13b.mp4 clip14.mp4 clip15.mp4 clip16.mp4 clip17.mp4 clip18.mp4 clip19.mp4 clip19b.mp4 clip20.mp4 clip21.mp4 clip21a.mp4 clip21b.mp4 clip21c.mp4 clip22.mp4 clip23.mp4 clip24.mp4 clip25.mp4 clip26.mp4
    15 ${SH} rm concat.txt | true
    16 ${SH} for f in $^; do echo "file '$$f'" >> concat.txt; done
    17 ${SH} ffmpeg -y -f concat -i concat.txt -an $@
    18
    19 all2: ${ScaledImageClips} ${ScaledVidClips}
    20
    21 sortByTs:
    22 ${SH} echo "#!/usr/bin/python" > $@
    23 ${SH} echo 'import sys;' >> $@
    24 ${SH} echo 'import re;' >> $@
    25 ${SH} echo '' >> $@
    26 ${SH} echo 'inFile=sys.argv[1];' >> $@
    27 ${SH} echo '' >> $@
    28 ${SH} echo 'with open(inFile, "r") as fp:' >> $@
    29 ${SH} echo '  C=fp.read();' >> $@
    30 ${SH} echo '  L=C.split("\\n");' >> $@
    31 ${SH} echo '' >> $@
    32 ${SH} echo 'D=dict();' >> $@
    33 ${SH} echo 'for e in L:' >> $@
    34 ${SH} echo '  m=re.match(r"\D+(\d+_\d+)(\D+)",e);' >> $@
    35 ${SH} echo '  if m:' >> $@
    36 ${SH} echo '    D[m.group(1)]=e;' >> $@
    37 ${SH} echo '' >> $@
    38 ${SH} echo 'for k in sorted(D.keys()):' >> $@
    39 ${SH} echo '  print D[k];' >> $@
    40 ${SH} echo '' >> $@
    41 ${SH} chmod +x $@
    42
    43 storyboard.txt: files.list sortByTs 
    44 ${SH} ./sortByTs $< > $@
    45
    46 files.list:
    47 ${SH} for f in $(shell echo ${ScaledImageClips} ${ScaledVidClips} ); do echo $$f >> $@ ; done
    48
    49 storyboard.mp4: ${ScaledImageClips} ${ScaledVidClips} storyboard.txt
    50 ${SH} for f in `cat storyboard.txt`; do echo "file '$$f'" >> concat.txt; done
    51 ${SH} ffmpeg -y -f concat -i concat.txt -strict -2 $@
    52 ${SH} rm concat.txt
    53
    54 %-scaled.mp4: %.jpg 
    55 ${SH} mogrify -auto-orient $<
    56 ${SH} ffmpeg -y -loop 1 -i $< -f lavfi -i aevalsrc=0 -shortest -filter_complex "scale=1920:-2:force_original_aspect_ratio=decrease,scale=-2:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,drawtext=fontcolor=white:fontsize=36:fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf:text=$^" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -strict -2 $@
    57
    58 %-scaled.mp4: %.mp4 
    59 ${SH} ffmpeg -y -i $< -filter_complex "scale=1920:-2,drawtext=fontcolor=white:fontsize=36:fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf:text=$^,scale=-2:1080,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -r 30/1 -acodec copy $@
    60
    61 background.jpg: 
    62 ${SH} convert -size 1920x1080 xc:black $@
    63
    64 clip01.mp4: Motorola/IMG_20200515_091545370_HDR.jpg
    65 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1000:1300-(5*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
    66
    67 clip01b.mp4: Samsung/20200515_101726.mp4
    68 ${SH} ffmpeg -i $< -acodec copy -ss 2 -t 6 -vcodec libx264 $@
    69
    70 clip02.mp4: Motorola/IMG_20200515_100605080.jpg
    71 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1000+(2*n):1300-(6*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
    72
    73 clip03.mp4: Samsung/20200515_101151.mp4
    74 ${SH} ffmpeg -i $< -vf "crop=1920/2:1080/2:0:240,scale=1920:1080" -pix_fmt yuv420p -acodec copy -ss 21 -t 8 -vcodec libx264 $@
    75
    76 clip04.mp4: Samsung/20200515_101523.jpg
    77 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1920+(5*n):1280-(2*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
    78
    79 clip05.mp4: Samsung/20200515_102050.mp4
    80 ${SH} ffmpeg -i $< -vf "crop=1920:1080:0:0" -acodec copy -ss 11 -t 4 -vcodec libx264 $@
    81
    82 #--here
    83 clip06.mp4: Samsung/20200515_102948.mp4
    84 ${SH} ffmpeg -i $< -acodec copy -ss 5 -t 6 -vcodec libx264 $@
    85
    86 clip07.mp4: Motorola/VID_20200515_103223863.mp4
    87 ${SH} ffmpeg -i $< -acodec copy -ss 18 -t 3 -vcodec libx264 $@
    88
    89 clip07a.mp4: Samsung/20200515_104151.mp4
    90 ${SH} ffmpeg -i $< -acodec copy -ss 0 -t 6 -vcodec libx264 $@
    91
    92 clip08.mp4: Motorola/VID_20200515_104921374.mp4
    93 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 18 -t 3 -vcodec libx264 $@
    94
    95 clip09.mp4: Motorola/VID_20200515_104921374.mp4
    96 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 32 -t 6 -vcodec libx264 $@
    97
    98 clip10.mp4: Samsung/20200515_105549.mp4
    99 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 25 -t 6 -vcodec libx264 $@
   100
   101 clip11.mp4: Motorola/VID_20200515_105953712.mp4
   102 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 6 -vcodec libx264 $@
   103
   104 clip11a.mp4: Motorola/VID_20200515_112402173.mp4
   105 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 6 -t 6 -vcodec libx264 $@
   106
   107 clip12.mp4: Samsung/20200515_113100.mp4
   108 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 19 -t 6 -vcodec libx264 $@
   109
   110 clip13.mp4: Motorola/VID_20200515_113612595.mp4
   111 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 6 -vcodec libx264 $@
   112
   113 clip13a.mp4: Motorola/VID_20200515_123906857.mp4
   114 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 25 -t 6 -vcodec libx264 $@
   115
   116 clip13b.mp4: Motorola/IMG_20200515_124841138.jpg
   117 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1920/2+(5*n):1280-(2*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
   118
   119 clip14.mp4: Motorola/VID_20200515_114433396.mp4
   120 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 10 -vcodec libx264 $@
   121
   122 clip15.mp4: Samsung/20200515_115303.mp4
   123 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 86 -t 6 -vcodec libx264 $@
   124
   125 clip16.mp4: Motorola/VID_20200515_121029775.mp4
   126 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 4 -vcodec libx264 $@
   127
   128 clip17.mp4: Motorola/VID_20200515_121448017.mp4
   129 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 2 -r 30 -vcodec libx264 $@
   130
   131 clip18.mp4: Motorola/VID_20200515_122255180.mp4
   132 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 14 -t 3 -vcodec libx264 $@
   133
   134 clip19.mp4: Motorola/VID_20200515_130750462.mp4
   135 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 6 -vcodec libx264 $@
   136
   137 clip19b.mp4: Motorola/IMG_20200515_131917817.jpg
   138 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1920/2+(5*n):960-(2*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
   139
   140 clip20.mp4: Motorola/VID_20200515_132127514.mp4
   141 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 2 -t 3 -r 30 -vcodec libx264 $@
   142
   143 clip21.mp4: Samsung/20200515_133613.mp4
   144 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 9 -t 6 -vcodec libx264 $@
   145
   146 clip21a.mp4: Samsung/20200515_134350.mp4
   147 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 9 -t 6 -vcodec libx264 $@
   148
   149 clip21b.mp4: Motorola/VID_20200515_140305984.mp4
   150 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 0 -t 6 -vcodec libx264 $@
   151
   152 clip21c.mp4: Samsung/20200515_140926.mp4
   153 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 3 -t 6 -vcodec libx264 $@
   154
   155 clip22.mp4: Motorola/IMG_20200515_144241756_HDR.jpg
   156 ${SH} ffmpeg -y -loop 1 -i $< -vf "crop=1920:1080:1920/2+(5*n):1280-(2*n)" -r ${FPS} -vframes $(shell echo $(FPS)*$(JPGSECS) | bc) -vcodec libx264 $@
   157
   158 clip23.mp4: Samsung/20200515_145819.mp4
   159 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 28 -t 6 -vcodec libx264 $@
   160
   161 clip24.mp4: Samsung/20200515_145925.mp4
   162 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 7 -t 6 -vcodec libx264 $@
   163
   164 clip24a.mp4: Samsung/20200515_150020.mp4
   165 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 3 -t 6 -vcodec libx264 $@
   166
   167 clip25.mp4: Samsung/20200515_150020.mp4
   168 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 46 -t 6 -vcodec libx264 $@
   169
   170 clip26.mp4: Samsung/20200515_150837.mp4
   171 ${SH} ffmpeg -i $< -vcodec copy -acodec copy -ss 32 -t 6 -vcodec libx264 $@
   172
   173 audio.mp3: audio.mp4
   174 ${SH} ffmpeg -i $< -vn $@
   175
   176 audio.mp4: 
   177 # ${SH} youtube-dl https://www.youtube.com/watch?v=BLQaGEI5D2Q --format mp4 -o $@
   178 ${SH} youtube-dl https://www.youtube.com/watch?v=qSZGt_vQMcs --format mp4 -o $@
   179
   180 clean:
   181 ${RM} ${ScaledImageClips} ${ScaledVidClips}
   182 ${SH} svn revert $(shell svn status | grep "^M" | grep -v Makefile | cut -b 2-) | true
   183 ${RM} files.list files.txt
$ 

Sunday, May 24, 2020

Our 100th Blog Post


Armed with a 6th grade vocabulary and an appalling lack of proper use of grammar, I'll never consider myself a writer, but I do seem to enjoy writing at times.

My sister-in-law happened upon a clipping from my childhood local newspaper.  While I have absolutely no recollection of the event, right there in black-n-white apparently I had a taste for writing as a lad.  This 'young Grant', with his slim belt, bowl-cut hair and youthful interest in writing grew into a fat, middle-aged engineer.....with a bowl-cut haircut.  Somewhere along the way, his interest in writing withered and like most engineers he grew to dislike it.  Late in his 40's, this old lad repurposed his keyboard from it's purpose of coding and instead began writing about the one topic he truly, deeply and passionately loved....Computer Science.  

For years I was a 'bad contributor', a taker of knowledge, rarely contributing.  Routinely I'd encounter a question, or a compiler error, and hit up ye olde Stackoverflow to readily find a solution, a parasitic tick on the Internet knowledge vein.  Years of this going on, with the intent to one day add my contributions.

For many of those years I found myself learning something, not applying it immediately and learning it again when it became relevant to current work.  Like a good carpenter, software engineers acquire a surplus of general-purpose and specialized tools in our tool-shed and draw upon them when the proper time arrives.  Unfortunate for this alcohol-fueled engineer, there is limited quantity of retention in ye' olde brainpan, yet infinitely-available virtual space just waiting to store content.  So, one day I made the decision to document stuff as I learned it; that way, it was readily available for my own use when I was ready for it and it would be available for anyone else that was interested.  An on-line engineering workbook of sorts and a way to make a contribution back to the Internet community.

Early on, for months, perhaps a full year, my only motivation was writing for myself and making it available to anyone else that holds an interest.  Google-search would serve to provide an audience.  I didn't pay attention to traffic, simply writing what I want, when I wanted.  I would occasionally run out of content and not post for a few weeks, later only to restart as inspiration presented itself, but I always felt a sense of embarrassment when I failed to post.  Eventually, I became more dedicated to posting weekly without interruption and would spend considerable time authoring a queue of available content in the can, awaiting publishing.  My selected content was always: software-related, technical, a broad range of topics and 5-minute reads.  I'm inherently an impatient person, so writing for myself (or those like me) would have to fit that criteria.

In time I learned there is a great number of indirect benefits from writing.  Likely best described by the 'Feynman Technique', the best way to learn a topic is to focus your efforts in teaching it to someone.  Translate that to writing about it and you have a strong motivation for writing.  An article, report, paper or blog post should tell a story and while your university notebook may be comprised of clippings and snippets of information it lacks structure.  Writing an article, even if it is never read by anyone else, forces you to think through the topic and I guarantee you'll understand the material better after writing about it.  That's what I've found and a strong motivation for writing about things I'm interested in.

Along the journey however my motivations changed slightly, the more work you put into something the more you eventually want out of it.  In time, I began monitoring traffic to see if I was finding an audience, and sadly I found little evidence of one.  I began struggling to make the time to write something that would likely never be read.  I've struggled with that for the past few months, it's honestly difficult to consistently think of a topic, research it sufficiently, author some content and finally write about it.  My 'in the can' queue once in the dozen's have now dwittled to single-handed digits.  While there are countless topics available, to do them justice would require a significant time-investment, tough to invest the time knowing it would have little benefit/interest.  

I have one loyal reader; a champion of men, a leader of leaders, the guru of greatness; and to he I express the warmest of thanks (@MM).  

So, as of late I struggle with motivation.  I entered this journey writing for 'someone like me', thinking that if a topic is of interest to me there's gotta be others who find it interesting.  As the gas tank of inspiration runs dry I'm left with the overwhelming question of 'why am I spending time on this'?  I've dabbled in opinion-based topics, a good deal of FFMpeg stuff, and a spackling of various topics but struggled to find an audience. I've had some heavily viewed stuff, primarily driven by submitting posts to Hacker News;

There is always uncertainty with these statistics, an unknown quantity of the traffic is surely 'bots'.

In an effort to reclaim self-worth from this effort I'm gonna spend some time focusing on that promoting (e.g. marketing *shudder*).  In a world full of click-baiting, pleads for views, and evil techniques like spam it will be interesting to see how I can get more viewership while maintaining professional integrity and comfort zone.  

I'd like to express my thanks to those of you that made the time to read this.  I'd welcome any suggestions or recommendations you may have that would make this blog more useful and enjoyable.  Thank you, and stay safe.