Executing Commands Repeatedly

While I've been using Linux pretty regularly since '98, I only recently became acquainted with the 'watch' command.  The primary purpose of this utility is to repeatedly run a command, at a specified frequency, and display the results.

Suppose you're writing to a file and wish to monitor the file size as it goes along.  Whelp, you could issue 'ls -l somefile' repeatedly, write a simple script or make use of the watch command.

If you wish to issue an 'ls -l somefile' every 5 seconds, the syntax is:

$ watch -n 5 'ls -l somefile'

If find this particularly of use if I am monitoring the disk usage of a file or file system.

$ watch -n 5 'du -sh .'

$ watch -n 5 'df -h /'

I find this quite useful on any number of occasions, especially when you wish the output displayed.


If the output isn't particularly interesting, perhaps you only need to run an identical command X times to perform run-time comparisons then in-line bash scripting is your friend;

$ for i in $(seq 1 5); do ls -l somefile; done

Anything worth doing once is worth doing N times.

Adventures In Field Testing

I consider myself very fortunate, perhaps not as fortunate as others, but fortunate the same.  I genuinely love what I do, again perhaps not every day, but on the whole I’m lucky to work in a career I thoroughly, completely and deeply enjoy.  I believe that’s to a large degree because I’ve had the pleasure of working with some extraordinary people on exciting products.  When I reflect on some of my most fulfilling moments my mind gravitates to times of field testing.

Field testing likely has numerous meanings to others, to me it means leaving an office environment and going out into the ‘field’.  To me, this means outside, away from the luxuries of air-conditioning, away from comfy office chairs and into the outdoors.  This may include working in an urban environment, alongside an intersection testing camera-based vehicle detection systems, in a 50x40’ Quonset building on a military base or even in a farmer’s bean field.

I’m not sure specifically why I smile when I reflect on these adventures, they certainly aren’t all fun-n-games, nor without frustration and often comes with a great deal more stress.  Partly, it’s simply a change in pace from the typical office setting and antics.  Partly it’s narrowing down the team to its critical components; non-critical members often remain at the office and the key contributors make the field testing journey.  Partly, I feel it’s an albeit short return to my blue-collar roots; humping equipment in and out of the truck, setting up and tearing down the system, packing lunches in a budget motel with gas station bread and bologna packs.  Perhaps I think fondly on those moments because they were times when I could better relate to our end users.  Many of these systems were military systems and setting up and tearing down the systems out in the elements allowed relating to how these soldiers would need to utilize these systems.

Crusader

Fresh out of school, first job; I really didn’t have any idea of what ‘field testing’ was.  Working with the team of controls engineers who were involved with conducting system testing in Yuma, AZ firing the system from a 10x40’ mobile office trailer with a massive berm separating the trailer from the 155mm howitzer.  The berm protected the crew from direct shrapnel in the event of propellant round touching off.  “But, nothing provides you protection from a 300lb blast panel from crashing through the ceiling”, a conversation I recall from one of the controls engineers who spent a good deal of time at the firing range.  The thought of being near the system we were developing, seeing it first-hand how it was performing, seeing what only a handful of folks would see….the whole idea was enticing.  Alas, as my schedule for testing neared the program budgeting folks scaled back and my ride to observe was erases with the stroke of a pen.  My enthusiasm was unwavering, my time would come.

MDUGS

With the cancellation of Crusader, a marriage on deck, and a need for a house I found myself looking for another employer/project.  I had a taste for military systems and I found I liked it.  Something sexy about helping develop military assets, it likely helped fill a regret of not serving and perhaps a dated mantra but ‘boys and guns, guns and boys’.  There were about a million other reasons that I felt drawn to these projects but I don’t want to get too off topic.

MDUGS was a contract to develop a large scale cluster of sensors that would detect and track personnel and vehicles travelling through the cluster field.  The MD stood for ‘massively deployable’, envisioning a day when a cluster field would consist of hundreds of sensors either air-dropped or deployed by mortar/howitzer in an unstructured layout.

The development phase for such a system took a pretty standard cycle.  A set of sensors were created, these sensors were taking into the field for individual testing (e.g. characterizing the sensor readings at variety of distances), bringing that info back to the lab and utilizing it to author sensor modeling to be used in the field.  Given the impracticality of having a massive lab to simulate field factors a good chunk of the development was done with simulated sensors, a whole ‘nother topic in itself.  After the system was sufficiently matured, the whole system was packed up, taken to a remote location and tested using the real sensors; field testing.
This program really set the stage for what I consider successful field testing.  Unlike deep-pocketed projects, MDUGS budget was limited which drove innovation and creativity.  The majority of the testing was conducted in an adhoc workspace housed in the back of a Budget Moving Truck, 3x8' sheets atop saw horses.  Our 'field testing' equipment looked more like the contents of a handyman's shed than that of what most would expect of a government military program.

High-speed data networks were created by housing a 12v motorcycle battery powering Linksys Ethernet switches all contained within a $5 plastic toolbox.  We drug 100' CAT5 cables between these boxes.

After months of testing and need for larger cluster deployments this setup was transformed into adhoc wireless networking via Linksys routers and wireless access points mounted atop 2" PVC piping providing line-of-sight connectivity on rolling fields.  Innovation at it's finest.

Limited time, a dedicated team, minimal budget, unpredictable weather....ice fishing styled software development.

JLTV 
The Joint Light Tactical Vehicle is the military's successor to the long-in-the-tooth Humvee.  The Army conducted a technical demonstrator phase where multiple teams developed concept vehicles with a decided winner of the production contract evaluated later.  The field testing was conducted in Yuma, AZ in a 40'x60' garage facility with the vehicle maintained between testing.  Unlike previous testing, the days consisted of 'hurry up' and 'wait' cycles.  Conducted in a secure location, photos were frowned upon.  A maintenance shop with the outer perimeter housing folding tables, laptops and computers with the 'brides of the ball', the vehicles, taking center stage in the building.  Still, valuable experience given the opportunity of meeting and relating to the soldiers use and needs of the vehicle.

DOT Products
Field testing can take the form of urban environments as well.  At Image Sensing Systems, field testing computer vision-based detection takes place at intersections.  Some luxuries over remote locations, dining facilities, mobile and wireless connectivity but still some of the same complications as remote locations......weather, laptops balanced on a knee...and some new challenges like rubber-necking spectators.

As I was penning this post, and now that it's nearly ended I ask myself....what's the point of the post?  Honestly, I'm not entirely sure.  There is no specific bullet points, specific lessons to take-away besides an appreciation of the challenges and advantages of participating in field testing.  Systems testers often have ownership of these activities and rarely do the developmental engineers participate.  Some of my fondest memories in my career revolve around the weeks of field testing over my 21 years in the profession.  Must be meaningful enough to make a significant impression.

A Post Of Many Colors - Metapixel

We've all seen them in some form or another, a mosaic portrait that is formed by a tiled set of smaller images.  A patient creative person with a good deal of time on their hands and a big 'ol wall and a collection of images could piece together a masterpiece.

But what if.....you're not creative.....have little patience but still have a boat-load of images and a desire to make a collage?

Enter the metapixel utility;
To make your very own 'wall hanger' you need but 3 things:
1) the metapixel utility
2) a supply of images which will be resized into tiles to generate the target image
3) the target image (the image you want to create from organizing the tiles)

Installing Metapixel

Assuming you're running a Debian-based install, installing the metapixel utility is as simple as:

$ sudo apt-get install metapixel
Reading package lists... Done
Building dependency tree      
Reading state information... Done
metapixel is already the newest version (1.0.2-7.4).
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.

Supply of Images

I invite you to use your very own images, but for now let's use a collection of images we can both get our hands on to ease repeatability.

$ wget http://www.vision.caltech.edu/Image_Datasets/Caltech256/256_ObjectCategories.tar -O db.tar

Let's extract these files to a known directory.

$ mkdir imageDb
$ tar -xvf db.tar -C ./imageDb

Next, you need to let metapixel generate a series of tiles on these images.  In order to do so, you specify the input directory.  The utility doesn't recurse the directories so you'll need to submit each subdirectory to the utility.  Easiest way to do so is to use the find command as follows:

$ mkdir ./ws/
$ find imageDb -type d -exec metapixel-prepare {} ./ws/ \;

Whelp, that gives us a database of source tile images that we can then use to generate our target image.

Target Image

Let's use the following as our target image (e.g. input.jpg).

The final step is to request the generation of the mosaic:
$ metapixel --metapixel input.jpg output.jpg --library ws/

The result, this beauty:

As you can see, upon closer look you'll find the image is generated out of a series of small tiles:

Go forth and generate wall-sized cat portraits from the plethora of Internet cat photos.

Python List Comprehension

One of the more Python'esk features of the language is list comprehension.  List comprehension is a means of transforming one list into another.  It's worth knowing, any list comprehension can readily be implemented by a for-loop but list comprehension is often considered more Python'ic.

So, why not simply use a for-loop?  Whelp, probably a couple reasons: 1) done right, a list comprehension can be more understandable, 2) it eliminates the need for locally scoped temporary variables.

Let's look at an example; taking a list of random numbers and returning a sublist of those that are even.

$ cat -n b
     1 #!/usr/bin/python
     2 import random
     3 
     4 L = random.sample(xrange(100), 10)
     5 print L;
     6 
     7 L0=[];
     8 for el in L:
     9   if el % 2 == 0:
    10     L0.append(el);
    11 print L0;
    12 
    13 L1=[el for el in L if el % 2 == 0];
    14 print L1;

Line 4 establishes a list of random numbers, 10 deep.
Lines 7-10 establishes L0, a sublist of all even numbers, preserving the order.
Line 13 performs the same effect, using list comprehension.
List comprehension is arguably much more readable.

Unlike the for-loop, the list comprehension doesn't suffer from a variable hanger-on, namely variable 'el'.  Often not an issue, but you happen to have this embedded in a series of statements you may quickly find a side-effect of overloading the variable name and the consequences that come along with it.