Data can be beautiful. Visualizing data is a worthwhile skill to acquire and it's relatively simple with Python. Let's explore how to do some data visualization as an exercise.
The Mandelbrot set is often regarded as an example of art meeting science. It's generated by evaluating the behavior of complex numbers and generating the results. The end effect is a infinite and beautiful depiction of pure mathematics, an acid-trip of color and structure as you continuously zoom into the graph.
One of the best descriptions of the Mandelbrot set can be found here, I recommend you spend a few minutes to appreciate the concept before we begin graphing it with a simple Python snippet.
In about 30 lines of code we can create our own colorized visualization of the mandelbrot set.
Let's look at the source, then step into some of the details; $ cat -n mandelbrot 1#!/usr/bin/python 2import matplotlib.pyplot as plt; 3import sys; 4 5def colorize(n): 6 h="#%06x"%(int(n*2**23)); 7 return h; 8 9xRange=[-2,1]; 10yRange=[-1.5,1.5]; 11incr=0.005; 12 13x=xRange[0]; 14while(x < xRange[1]): 15 y=yRange[0]; 16 while(y < yRange[1]): 17 c=x+y*1j; 18 z=0; 19 try: 20 for k in range(50): 21 z=z**2+c; 22 if(abs(z) < 2): 23 rgb=colorize(abs(z)); 24 plt.plot(x,y,'.',color=rgb); 25 except: 26 pass; 27 y += incr; 28 x += incr; 29 30plt.xlim(xRange[0],xRange[1]); 31plt.ylim(yRange[0],yRange[1]); 32plt.savefig(sys.argv[1]);
Let's look over the non-Mandelbrot stuff first. We're using the matplotlib library for our simple plotting example, line 9-10 we define our plot x and y ranges and enforce them in lines 30-31. Finally in line 31 we save the plot to a figure rather than display to the screen. We pass in the figure filename as a command line argument so ./mandelbrot foo.png would generate a foo.png file. Lines 23-24 calculates a color for the pixel and plots it at (x,y). Without going into details, the nested loop (lines 13-28) steps through the floating point 2D range defined by the x and y ranges, stepping by 0.005 (defined in line 11). Each iteration, we selectively plot, or don't plot, a colorized pixel. We plot a pixel if it's position is part of the Mandelbrot set.
The rest of the details are specific to calculating the Mandelbrot set. In particular, lines 17,20-26. The referenced video explores how and why this is done, but let's revisit some of the particulars.
Line 17 consists of the assignment of the complex number for each (x,y) position in the range. Note 1j is the Python representation of complex number i. This assignment was covered in the video, specifically, but easily overlooked;
The inclusion in the mandelbrot set for this position is characterized by how this complex number behaves under the influence of iteration of the function starting at 0. This means, z starts as 0 and we repeatedly inject the f(z)=z^2+c in our loop, the end result will either blow up implying it's not in the mandelbrot set, or it'll remain bounded (e.g. <= 2) which means it's part of the set. Lastly, we use the magnitude of the result to determine the color of the pixel. This is optional, but it adds a level of beauty. We can represent the looping of (x,y) range and determination of it's inclusion/exclusion in the mandelbrot set by the clip from the reference video;
Some time back a trending video made its way into my life, it provided a graphical representation of common sorting algorithms.
This simple video demonstrated the value of visualizing complex algorithms, hopefully something that is incorporated into CS coursework. The value of divide-n-conquer techniques can be more easily understood with a side-by-side comparison.
We can generate an equivalent video by monitoring the list of numbers, generate a graph at each step and render the sequence of graphs into a video.
In order to generate each graph, we could modify the sorting algorithm to create and store a graph image. This approach however would require modification to the source code. An alternative, use a debugger to monitor the list of numbers and print out the list of numbers whenever it is modified. Then, post-run, we could use the debugger output to create individual graphs and stitch them together.
So, our approach will take the form:
1) create C++ sorting algorithm
2) run w/debugger and monitor the state of the input array
3) parse the debugger log, creating a graphical representation of the input array at each step
4) stitch each frame into a video
1 - Sorting Algorithm
Let's create a simple C++ example with a variety of sorting algorithms. We'll create a randomized list of integers and prepare to send it through a sorting routine.
$ cat -n main.cpp
1#include <stdio.h>
2#include <stdlib.h>
3#include <vector>
4
5#ifndef N
6#define N 100
7#endif
8
9void swap(int& a, int& b)
10{
11 const int tmp=a;
12 a=b;
13 b=tmp;
14}
15
16void display(int list[], int size)
17{
18 for(int i=0; i<size; i++)
19 {
20 printf("%d\n",list[i]);
21 }
22}
23
24//============Selection Sort============
25void selectionSort(int list[], int size)
26{
27 for(int i=0; i<size-1; i++)
28 {
29 for(int j=i+1; j<size; j++)
30 {
31 if (list[j] < list[i])
32 {
33 swap(list[j],list[i]);
34 }
35 }
36 }
37}
38//======================================
39
40//==============Quick Sort==============
41int partition(int *a,int start,int end)
42{
43 int pivot=a[end];
44 //P-index indicates the pivot value index
45
46 int P_index=start;
47 int i,t; //t is temporary variable
48
49 //Here we will check if array value is
50 //less than pivot
51 //then we will place it at left side
52 //by swapping
53
54 for(i=start;i<end;i++)
55 {
56 if(a[i]<=pivot)
57 {
58 t=a[i];
59 a[i]=a[P_index];
60 a[P_index]=t;
61 P_index++;
62 }
63 }
64 //Now exchanging value of
65 //pivot and P-index
66 t=a[end];
67 a[end]=a[P_index];
68 a[P_index]=t;
69
70 //at last returning the pivot value index
71 return P_index;
72}
73
74void Quicksort(int *a,int start,int end)
75{
76 if(start<end)
77 {
78 int P_index=partition(a,start,end);
79 Quicksort(a,start,P_index-1);
80 Quicksort(a,P_index+1,end);
81 }
82}
83
84void quickSort(int list[], int size)
85{
86 Quicksort(list,0,size);
87}
88
89void insertionSort(int list[], int size) {
90
91 int valueToInsert;
92 int holePosition;
93 int i;
94
95 // loop through all numbers
96 for(i = 1; i < size; i++) {
97
98 // select a value to be inserted.
99 valueToInsert = list[i];
100
101 // select the hole position where number is to be inserted
102 holePosition = i;
103
104 // check if previous no. is larger than value to be inserted
105 while (holePosition > 0 && list[holePosition-1] > valueToInsert) {
106 list[holePosition] = list[holePosition-1];
107 holePosition--;
108 printf(" item moved : %d\n" , list[holePosition]);
109 }
110
111 if(holePosition != i) {
112 printf(" item inserted : %d, at position : %d\n" , valueToInsert,holePosition);
113 // insert the number at current hole
114 list[holePosition] = valueToInsert;
115 }
116
117 printf("Iteration %d#:",i);
118 }
119}
120
121void sort(int list[], int size)
122{
123//selectionSort(list,size);
124//quickSort(list,size);
125 insertionSort(list,size);
126}
127
128int main(int argc, char* argv[])
129{
130 printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
131 std::vector<int> L;
132
133 for(int i=0; i<N; ++i)
134 {
135 L.push_back(rand()%100);
136 }
137 sort(L.data(), L.size());
138
139 display(L.data(), L.size());
140
141 printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
142}
On line 5-8, we declare N which represents the size of the array size, then on lines 131-136 we allocate a list of integers and populate them with randomized numbers. Line 137 we provide the list as an argument to the sort() method.
2 - Run w/Debugger
Macros are evil so let me justify the use of N in the C++ source file. Typically, such a macro would be replaced with a static constant allowing more type-safe implementation. In this example though the size of the array needs to be known by the GDB commands as well. The makefile assigns the value of N, passing it in to the compiler as a pre-compiler directive and creates the gdb command file which also needs to know the value of N.
$ cat -n Makefile
1CC=g++
2MAIN=main
3N=100
4
5all: video.mp4
6
7${MAIN}: ${MAIN}.cpp
8${CC} -D N=${N} -g $< -o $@
9
10run: ${MAIN}
11${SH} ./${MAIN}
12
13plot: gdb.log
14${SH} ./plotter $<
15
16gdb.cmd:
17${SH} echo "set pagination off" > $@
18${SH} echo "break sort" >> $@
19${SH} echo "commands" >> $@
20${SH} echo " watch *(int(*)[${N}])(list)" >> $@
21${SH} echo " commands" >> $@
22${SH} echo " cont" >> $@
23${SH} echo " end" >> $@
24${SH} echo " cont" >> $@
25${SH} echo "end" >> $@
26${SH} echo "run" >> $@
27${SH} echo "quit" >> $@
28
29gdb.log: gdb.cmd ${MAIN}
30${SH} gdb -x gdb.cmd ${MAIN} > $@
31
32video.mp4: plot
33${SH} ffmpeg -i frame%05d.png $@
34
35clean:
36${RM} ${MAIN} *.o gdb.log gdb.cmd
37${RM} *.png *.mp4
Notice on line 3 we define our array size as 100. On line 8 we pass in the value when compiling the C++ source. On line 20 we create a gdb command file that we will use to run with the debugger. I've recently found it useful for makefiles to create such input files as part of the build process rather than author manually and preserve in version control. The gdb.cmd file is generated as part of the build process and in this case allows the makefile to preserve a consistent value for N in the gdb command file and the C++ source file.
Notice on line 29, we define a gdb.log target, which runs the executable with the necessary gdb command script and generates a gdb log file. Line 20 is the key to the gdb script, it attaches a watchpoint to array, when the array changes it spits out the old/new value of the array.
3 - Parse the Debugger Log
The gdb log file takes the form of:
$ cat -n gdb.log | head -50
1GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
2Copyright (C) 2014 Free Software Foundation, Inc.
3License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
4This is free software: you are free to change and redistribute it.
5There is NO WARRANTY, to the extent permitted by law. Type "show copying"
6and "show warranty" for details.
7This GDB was configured as "x86_64-linux-gnu".
8Type "show configuration" for configuration details.
9For bug reporting instructions, please see:
10<http://www.gnu.org/software/gdb/bugs/>.
11Find the GDB manual and other documentation resources online at:
When the list array is modified the watchpoint breakpoint is tripped which results in lines 23-24, outputting the old/new values.
4 - Stitch Frames into Video
Last bit, we need a simple Python script that is capable of parsing the gdb log file and generate a image representative of the state of the array.
$ cat -n plotter
1#!/usr/bin/python
2import sys;
3import re;
4import matplotlib.pyplot as plt;
5
6def plot(fileName):
7 print "plotting %s"%(fileName);
8 with open(fileName,'r') as fp:
9 C=fp.read();
10
11 i=0;
12 for line in C.split('\n'):
13 m=re.match('New value = {(.*)}.*',line);
14 if m:
15 L=[int(el) for el in m.group(1).split(',')];
16 print L;
17 plt.ylim(0,100);
18 plt.bar(range(len(L)),L,label='');
19 plt.savefig('frame%05d.png'%(i));
20 plt.close();
21 i+=1;
22 pass;
23
24#---main---
25plot(sys.argv[1]);
This script will parse the 'New value =' lines and generate a bar graph that represents the order of the array. For example, an initial value of a 10 element array could look like this;
Generating a series of images, during each step of the sorting routine and stitching them together into a video using FFmpeg.
These past months have been unprecedented, having placed an unusual amount of load on unsuspecting systems. These are the days that reveal the advantage of robust systems and the consequences of fragile systems. Robust system engineering behave as a young willow tree when the winds of demand blow; 'bend, don't break'. Resilient systems will survive the storm and in some cases thrive in it. Properly designed systems prevail and demonstrate their true value in times like these.
Like a high performance engine, or a professional athlete, properly engineered computer systems require an investment in building (time and money) as well as continuous maintenance. Lack of maintaining these systems can have dramatic consequences; like this:
Unemployment systems built in the 1960's provide the digital heart to the social programs for entire states. Departments who financed the building of such systems fall prey to complacency in maintaining them and are publically pants'ed and humiliated when the consequences of their actions hit the headlines.
No bones about it, maintaining systems is expensive it's simply a manner of how you will pay. The cost of tech debt, or system maintenance can be viewed like that of a mortgage payment....pay it monthly, or pay it in one solid balloon payment at the end of the term; either way, you'll be paying it. New Jersey backed into a balloon payment after years of neglect.
There are no real easy answers to these decisions, modernizing a 60 year old software-intensive system is a difficult and complex task and it's easy to be seduced to think the right call is to simply stay-fast with the existing system and I can only suspect that's what NJ did.
The decision to not modernize isn't necessarily the lower cost option either. Mainframes are dinosaurs in this technology era and the maintenance of these systems likely fell to single-source high-cost specialty engineering houses like IBM billing out at upwards of $200/hr; pay now and pay later.
If NJ could roll back the clock and play out the ideal narrative the hardware system would be replaced to more modern architected systems and the near-mummified COBOL would be replaced with a more modern language. Then, single-sourced high-cost hardware would be replaced with common hardware and engineering teams well-versed in modern programming languages would open the field to all kinds of talented personnel. This is frankly the only course of action to preserve a future of this (and other like) system. Changes like this don't happen overnight, it's a significant time and financial investment but it requires being done, postponement costs only add to the overall cost of system replacement.
As a resident of Minneapolis sadly we're in a time where literally our city is on fire. I considered skipping posting this week but rather decided to instead apply some of my FFmpeg knowledge to a number of day-trip media files and create a video of hiking in one of the most beautiful parks in the region. Hoping to share some beauty in these trying times.
Took 139 pictures/videos, generated a number of video clips and panning crop filters on static images and stitched them all together using the following Makefile.
Armed with a 6th grade vocabulary and an appalling lack of proper use of grammar, I'll never consider myself a writer, but I do seem to enjoy writing at times.
My sister-in-law happened upon a clipping from my childhood local newspaper. While I have absolutely no recollection of the event, right there in black-n-white apparently I had a taste for writing as a lad. This 'young Grant', with his slim belt, bowl-cut hair and youthful interest in writing grew into a fat, middle-aged engineer.....with a bowl-cut haircut. Somewhere along the way, his interest in writing withered and like most engineers he grew to dislike it. Late in his 40's, this old lad repurposed his keyboard from it's purpose of coding and instead began writing about the one topic he truly, deeply and passionately loved....Computer Science.
For years I was a 'bad contributor', a taker of knowledge, rarely contributing. Routinely I'd encounter a question, or a compiler error, and hit up ye olde Stackoverflow to readily find a solution, a parasitic tick on the Internet knowledge vein. Years of this going on, with the intent to one day add my contributions.
For many of those years I found myself learning something, not applying it immediately and learning it again when it became relevant to current work. Like a good carpenter, software engineers acquire a surplus of general-purpose and specialized tools in our tool-shed and draw upon them when the proper time arrives. Unfortunate for this alcohol-fueled engineer, there is limited quantity of retention in ye' olde brainpan, yet infinitely-available virtual space just waiting to store content. So, one day I made the decision to document stuff as I learned it; that way, it was readily available for my own use when I was ready for it and it would be available for anyone else that was interested. An on-line engineering workbook of sorts and a way to make a contribution back to the Internet community.
Early on, for months, perhaps a full year, my only motivation was writing for myself and making it available to anyone else that holds an interest. Google-search would serve to provide an audience. I didn't pay attention to traffic, simply writing what I want, when I wanted. I would occasionally run out of content and not post for a few weeks, later only to restart as inspiration presented itself, but I always felt a sense of embarrassment when I failed to post. Eventually, I became more dedicated to posting weekly without interruption and would spend considerable time authoring a queue of available content in the can, awaiting publishing. My selected content was always: software-related, technical, a broad range of topics and 5-minute reads. I'm inherently an impatient person, so writing for myself (or those like me) would have to fit that criteria.
In time I learned there is a great number of indirect benefits from writing. Likely best described by the 'Feynman Technique', the best way to learn a topic is to focus your efforts in teaching it to someone. Translate that to writing about it and you have a strong motivation for writing. An article, report, paper or blog post should tell a story and while your university notebook may be comprised of clippings and snippets of information it lacks structure. Writing an article, even if it is never read by anyone else, forces you to think through the topic and I guarantee you'll understand the material better after writing about it. That's what I've found and a strong motivation for writing about things I'm interested in.
Along the journey however my motivations changed slightly, the more work you put into something the more you eventually want out of it. In time, I began monitoring traffic to see if I was finding an audience, and sadly I found little evidence of one. I began struggling to make the time to write something that would likely never be read. I've struggled with that for the past few months, it's honestly difficult to consistently think of a topic, research it sufficiently, author some content and finally write about it. My 'in the can' queue once in the dozen's have now dwittled to single-handed digits. While there are countless topics available, to do them justice would require a significant time-investment, tough to invest the time knowing it would have little benefit/interest.
I have one loyal reader; a champion of men, a leader of leaders, the guru of greatness; and to he I express the warmest of thanks (@MM).
So, as of late I struggle with motivation. I entered this journey writing for 'someone like me', thinking that if a topic is of interest to me there's gotta be others who find it interesting. As the gas tank of inspiration runs dry I'm left with the overwhelming question of 'why am I spending time on this'? I've dabbled in opinion-based topics, a good deal of FFMpeg stuff, and a spackling of various topics but struggled to find an audience. I've had some heavily viewed stuff, primarily driven by submitting posts to Hacker News;
There is always uncertainty with these statistics, an unknown quantity of the traffic is surely 'bots'.
In an effort to reclaim self-worth from this effort I'm gonna spend some time focusing on that promoting (e.g. marketing *shudder*). In a world full of click-baiting, pleads for views, and evil techniques like spam it will be interesting to see how I can get more viewership while maintaining professional integrity and comfort zone.
I'd like to express my thanks to those of you that made the time to read this. I'd welcome any suggestions or recommendations you may have that would make this blog more useful and enjoyable. Thank you, and stay safe.