Detect Command Timeouts using Bash

 A common, and often overlooked, need for scripting is addressing hanging, or unresponsive commands.  What should your script do if one of it's ste

ps takes much, much, much longer than expected?  A hanging, or unresponsive, subprocess will result in a hanging job which in turn can prevent oth

er jobs from running.  Let's spend a bit of time how to address such an need.

 

The 'timeout' command is provided by the coreutils package, often readily available on default installation for most distributions.

 

$ sleep 10

$ echo $?

0

 

Execution of subprocesses generally set a return code, zero or non-zero, often zero indicating successful execution of the command, non-zero return codes if it failed for some reason.  The above command sequence will execute a sleep command (for 10 seconds), the second command echoing the return code.  Assignment of $? to zero indicates the sleep command executed successfully.

 

So, what if we wanted to prevent a hanging command to go undetected and/or prevent the rest of a script from running.  Let's say we want to enforce a command takes no longer than 5 seconds.
 

$ timeout 5 sleep 10

$ echo $?

124

 

After 5 seconds, the command terminates, the return code 124 indicates the command timed out.  Usage of this allows setting hard constraints on how long a command is allowed to run before terminating or being terminated.  The return code allows determining if the command timed-out.
 

Cheers.

Synchronizing Completion of Concurrent Commands

 

Running concurrent commands in a script generally is done by executing processes in the background.  Numerous commands can be spawned concurrently then synchronized by waiting for some, or all, of them to complete before proceeding or terminating the parent script.

     1  #!/bin/bash

     2

     3  sleep 10 &

     4  pId1=$!

     5

     6  sleep 2 &

     7  pId2=$!

     8

     9  sleep 3 &

    10  pId3=$!

    11

    12  echo "waiting for jobs to complete"

    13  #wait ; #--wait for all jobs to complete

    14  wait $pId1 $pId2 $pId3 ; #--wait for specific jobs to complete

 

In the above script, a sequence of sleeping {10, 2, and 3} seconds are executed in sequence, each in the background.  The longest running command  (e.g. 10 seconds in this example) should take 10 seconds, the other two commands concluding earlier.

We can test the proper exection, the script completing in ~10 seconds by timing the execution as follows:

$ time ./foo

waiting for jobs to complete

 

real    0m10.008s

user    0m0.003s

sys     0m0.004s

$

bash-4.1$

Python Distribution Error (TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero')

 Last year I cut my teeth at creating a Python package; Dividere

Recently, building the package was encountering an error:

 

 

python3 setup.py sdist bdist_wheel
running sdist
running egg_info
creating dividere.egg-info
writing dividere.egg-info/PKG-INFO
writing dependency_links to dividere.egg-info/dependency_links.txt
writing requirements to dividere.egg-info/requires.txt
writing top-level names to dividere.egg-info/top_level.txt
writing manifest file 'dividere.egg-info/SOURCES.txt'
reading manifest file 'dividere.egg-info/SOURCES.txt'
adding license file 'LICENSE'
Traceback (most recent call last):
  File "/home/lipeltgm/dividere/setup.py", line 15, in 
    setup(
  File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 186, in setup
    return run_commands(dist)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 202, in run_commands
    dist.run_commands()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 983, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/sdist.py", line 59, in run
    self.run_command('egg_info')
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 339, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 999, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 1002, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 312, in run
    self.find_sources()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 320, in find_sources
    mm.run()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 548, in run
    self.prune_file_list()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/sdist.py", line 162, in prune_file_list
    super().prune_file_list()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 380, in prune_file_list
    base_dir = self.distribution.get_fullname()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 272, in get_fullname
    return _distribution_fullname(self.get_name(), self.get_version())
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 290, in _distribution_fullname
    canonicalize_version(version, strip_trailing_zero=False),
TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
make: *** [Makefile:26: buildPipPackage] Error 1

Apparently, there was an error introduced in a recent version of setuptools.  A work-around is to downgrade to an older version to temporarily resolve the issue.

 

~/dividere$ sudo pip3 install setuptools==70.0
 
  

 

 

 

 

Extending Video Introduction

 Sometimes it's useful to slow, or extend the introduction of a video.  For example, suppose you're utilizing a slow video transition (like a fade out/in effect) but don't want to miss the beginning frames of the second video.  By extracting the first frame of the video, elongating it to X seconds we can preserve the transition effect without losing video content.

Let's take a peek at how to accomplish this;

$ cat -n Makefile
     1    all: postVideo.mp4
     2   
     3    video.mp4: BigBuckBunny.mp4
     4        ${SH} ffmpeg -i $< -codec copy -strict -2 -t 10 $@
     5   
     6    image.jpg: video.mp4
     7        ${SH} ffmpeg -i $< -vf "select=eq(n\,0)" -q:v 3 $@
     8        ${SH} display $@
     9   
    10    preVid.mp4: image.jpg
    11        ${SH} ffmpeg -loop 1 -i $< -f lavfi -i aevalsrc=0 -t 3 $@
    12   
    13    postVideo.mp4: preVid.mp4 video.mp4
    14    #    ${SH} ffmpeg -i preVid.mp4 -i video.mp4 -filter_complex "[0:v] [0:a] [1:v] [1:a] concat=n=2:v=1:a=1 [vv] [aa]" -map "[vv]" -map "[aa]" $@
    15        ${SH} ffmpeg -i preVid.mp4 -i video.mp4 -filter_complex "[0:v] [0:a] [1:v] [1:a] concat=n=2:v=1:a=1 [vo] [ao]" -map "[vo]" -map "[ao]" $@
    16   
    17    clean:
    18        ${RM} *.jpg
    19        ${SH} find . -name "*.mp4" -not -name "BigBuckBunny.mp4" -delete

The input video (video.mp4) is generated by grabbing the first 10 seconds of BigBuckBunny.mp4; refer lines 3-4

Then, the first frame of the input video (video.mp4) is extracted and saved as image.jpg (ref lines 6-8)

Then, a new introduction video clip is generated by converting the image into a video clip of X seconds; ref lines 10-11).  Note, a null audio track is created to preserve the audio in the outgoing video.

Lastly, a new video is created by concatenating the intro clip with the original clip, the result as a video with extended first frame. 

One other trick worth mentioning with this makefile, I frequently want a clean target to clobber any video files created along the way.  However, you don't want to delete the original source file, this can be accomplished by utilizing a find+not condition; ref line 19.  This proves useful for many projects.

And, just like that, you've got a video with an elongated intro.

Youtube-Dl Broke/Fixing

 'Some folks' use youtube-dl on a regular basis, to pull raw video and create new content.  Early in 2023 it appeared to stop working.  

$ youtube-dl https://www.youtube.com/watch?v=8QWjCzULyNA
[youtube] 8QWjCzULyNA: Downloading webpage
ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Forums seemed to imply the issue was known and a fix was in play, so I put my feet up and figured I'd check back in a few weeks.  A month, or two, goes by and reinstallation/retrying didn't seem to resolve.  So I revisited the fix posts and found it referenced a different package/utility so I tried that out.

$ sudo pip3 uninstall youtube-dl

$ sudo pip3 install yt-dlp

$ yt-dlp https://www.youtube.com/watch?v=8QWjCzULyNA

Hope this helps someone else.

Cheers

Yolo AutoCropping Presentation Videos

 

As camera resolutions continue to improve the feasibility of capturing a full scene of a classroom, lecture, presentation hall, or the such and autonomously focusing attention on the presenter becomes more practical.  Generally, a camera operator pans and zooms in on the presenter as they make their way around the stage to draw the audience attention to the intended target.  Professionally filmed videos draw the audiences attention to the speaker and their production quality contributes to a more informative presentation.

Wide-angle, static camera positions are an alternative for capturing presentations but generally fail to draw the audience attention to the speaker.  With robust object-detection, the position of the presenter can be automated and thru the use of auto-cropping the presenter can offer a budget-friendly alternative to more professional video production facilities.

YOLO (You Only Look Once) takes a different approach from classic computer vision by utilizing a classifier as a detector.  Authored by Joseph Redmon at the University of Washington, YOLO sub-samples and image into regions, assumes each region has an object and executes a classifier on each region, then merges the classifier groups into a list of final objects.  

 

Below is a proof-of-concept utilizing YOLO in an auto-cropping manner.  The wide-angle source video is used as input, object-detection is focused on the front of the room, detects the presenter and auto-cropped around the presenter.  Once the presenter location is available, we use a variety of means to 'pan the camera', the first by snapping to the presenter location, the second by smoothing the camera motion by incorporating a 2-dimensional shaper, the third using the shaper but only moving the camera when the presenter nears the edges of the current crop window.

Each mechanism is a rough implementation, focused on rapid proof-of-concept rather than optimal results, but you get the idea.  

The source video was found on here; Minnebar7

 

 

Published My First Python Package

 

 

I started dabbling with Python back in 2012'ish, using it pretty regularly over the years but generally keeping my projects close to home.  Recently, I dipped my toe into publishing a Python package, out to the known universe.

Back in the late 90's, the Precambrian Digital Age, I took a couple courses that continue to pique my interest time and time again.  Parallel processing was primarily constrained to supercomputers like the Cray-1 that was homed in a nearby lab on campus, on full display behind a full-glass wall.  A workhorse which eagerly awaited computationally intensive parallelized programs. 

A customized version of Fortran, its vocabulary, the Computer Science department rarely used it, aerospace and atmospheric sciences most heavily used the system.

The second course, Distributed Operating Systems, taken a bit later seemed to pair well with this budding interest in high performance computing.  Beowulf clusters, commodity-grade networked computers running Linux, could be created from RadioShack-provided equipment fueled by inspiration.   Cloud computing, virtual machines and even network-intensive applications hadn't breached the digital horizon, but small-cluster networked labs provided inspiration that one day multitudes of computing assets would one day join hands in forming highly networked, parallel, distributed systems that can be considered common today.

While robust and reliable distributed systems are highly sought after, engineering them is plagued with challenges.  Failed requests could be due to loss of the sent message, the loss of the response, the destination service abruptly terminating, relocation of the service, a over-tasked memory/cpu that slows the response,....or any number of other factors.  Python and ZeroMQ pair well to allow the creation of a distributed system framework which inspired my budding project.

dividere UG

The public project repository is located at:

https://github.com/lipeltgm/dividere

This is my first cut at publishing a python package, I tried to apply good design, test and documentation principles along the way.  One particular challenge I encountered is that the package dependencies require a version of Protobuf that isn't currently available via 'normal channels'.  I'm hoping in time that complication will self-correct when compliant versions become the default.

Until then, it likely will require manual installation of protobuff-v3.19 (or later) before installing via pip3 from pypi:

https://pypi.org/project/dividere/ 

$ pip3 install dividere

With the foundation in place, I'm intending on extending the framework to support more reliable messaging, database components, robust failover detection and recovery.  

More to come in the future, fingers-crossed.