Create a Bootable USB stick

For decades, Linux installations was done using CDs, later DVDs.  As the installation sizes increased and the popularity of CD/DVDs dropped the more recent releases are done via USB media.

With a source iso image, and an unmounted USB on /dev/sd1, you can create a bootable USB disk using the following command;

$ sudo dd bs=4M if=~/Downloads/ubuntu-24.04.1-desktop-amd64.iso of=/dev/sd1> conv=fdatasync  status=progress

 Take particular care to insure you've specified the right /dev/sd1 or your day may get much, much, much crappier.

 

Cheers.

 

Chaining Find Commands

For the better part of my career I used 'find' incorrectly, or at the very least very naively.  The '-exec' trickery eluded me for over a decade;

I often relied on running child processes or _worse_ saving results to a file to run subsequent command on.

For instance, if I wanted to examine log files that had 'error' in it, I'd fashion a command to open VI with the arguments of the result from a gr

ep command;

e.g. $ vi `grep -li error *.log`

 The command line would first execute the grep command, the results formed the arguments to the vi command.  Alternatively, I'd very occasionally
use xargs as an alternative.  Alternatively, to the alternative, I'd run the equivalent of the grep command and redirect the output to a file, then 'cat' the file contents as arguments to whatever command I wanted.

e.g.

$ grep -li error *.log > files.txt

$ vi `cat files.txt`

Clumsy, clumsier and often times clumsiest.  But, lacking an understanding of the '-exec' option, it was what I knew, got me where I needed to be, but was a source of embarrasement for someone who used Linux professionally for years.

 Then, about 8 years ago, I spent some time familiarizing myself with '-exec' and everyday tasks got a bit easier.  These past days I've been spending time on chaining '-exec' commands, something I wasn't familar with until recently.

Let's say you are asked to search a server for scripts that use 'ftp', not 'sftp', as a security audit.  A whole lotta scripts exist that use neither, a subset of them using ftp/sftp, and a subset of that subset exclusively use ftp (our desired file list).  An approach would be to first find all scripts that reference 'ftp', then filter out the ones that use 'sftp'.

$ find /src/scripts/ -type f -exec grep -l 'sftp' {} \;  #-- finds scripts referencing sftp

$ grep -L 'sftp' [filelist]; #--lists files w/o sftp expression

 You want to feed the results of the first command as arguments to the second command.  A clumsy approach (the one I used for years) would be:

$ grep -L 'sftp' `find /src/scripts/ -type f -exec grep -l 'ftp' {} \;`

 But, the equivalent can be accomplished by chaining '-exec' commands in a single command.

$ find /src/scripts/ -type f -exec grep -l 'ftp' {} \; -exec grep -L sftp {} \;

 Chained exec commands are executed left-to-right, each executed as child commands.  Subsequent '-exec' commands are only executed on files/lines that satisfy the previous '-exec' command.  So, 'grep -L sftp...' is only run on the results of the previous command (e.g. "grep -l 'ftp'...).  The end results is first finding all files containing 'ftp', then finding files that don't contain 'sftp'.  The output of both '-exec' commands, each on seperate lines, which can be confusing.  We can suppress the output from the first command by specifying '-quiet (or -q)' to the first grep command, as follows:

 $ find /src/scripts/ -type f -exec grep -lq 'ftp' {} \; -exec grep -L sftp {} \;

 Now, the resulting file list from the first exec is still passed to the subsequent exec and the output to stdout is representative of only the second command.

Chaining Find Exec Commands for Quick Data Analysis

 Quick review of basic find command before getting to the content.

$ find . -name "*.txt" -exec grep -l dog {} \;

Executing commands via exec must end with \; and often use {} as a placeholder for each file that the find commands locates.  In the above example, imagine the command first locates the desired files, in this case files in the current and child directories that end with .txt.  These X filenames are then inserted/replacing the curly brackets.  Then, grep searches for 'dog' in each of those files and returns the filename of each file that matches.

Searching for a regex1 or regex2 in one pass can readily be accomplished by a single regex with one-pass file traversal.

$ find . -name "*.txt" -exec egrep -lo "reg1|reg2" {} \;

But, what if you're looking for matching files that include both regex1 and regex2?

 

Let's set up some simple files to play with, 3 files; cats.txt, dogs.txt, catsanddogs.txt

~/AdvancedFind$ more *.txt
::::::::::::::
catsanddogs.txt
::::::::::::::
cat
dog
::::::::::::::
cats.txt
::::::::::::::
cat
::::::::::::::
dogs.txt
::::::::::::::
dog

Finding files that contain 'cat' can be done by:

~/AdvancedFind$ find . -name "*.txt" -exec grep -li cat {} \;
./cats.txt
./catsanddogs.txt

Similarly, finding files that contain 'dog' can be done by:

~/AdvancedFind$ find . -name "*.txt" -exec grep -li dog {} \;
./catsanddogs.txt
./dogs.txt

You can 'chain' exec commands to satisfy finding files with 'cat' and 'dog' references.

~/AdvancedFind$ find . -name "*.txt" -exec grep -q cat {} \; -exec grep -l dog {} \;
./catsanddogs.txt

Files matching the first '-exec' command are passed to the second '-exec' command.  Notice the '-q' suppresses the first grep output, otherwise you'll get the output from both exec commands.

Similarly, you can find files that contain 'dog' that don't contain 'cat' can be done by:

~/AdvancedFind$ find . -name "*.txt" -exec grep -q cat {} \; -exec grep -L dog {} \;
./cats.txt
The '-L' means 'files without match', resulting in the first exec command identifying files that contain 'cat', the second exec command matching files that don't have a 'dog' reference.

 

Chaining exec commands can help perform quick data analysis using commonly available utilities.

Cheers.

Conditional Subsequent Command Calls

 

Occasionally there is a need in bash scripting to execute a series of commands in sequence conditionally.  Command2 needs to be run after command1 if, and only if, the command1 succeeded (or failed).

One means of performing this is to execute the command, assign the return value to a variable, and evaluate the return value executing the subsequent command based on the value.

 $ cat -n foo

     1  #!/bin/bash

     2

     3  sleep 1

     4  #sleep -1

     5  rc=$?

     6

     7  echo $rc

     8  if [ $rc == 0 ]; then

     9    echo "run something else"

    10  fi

    11

Note, the above the script executes a sleep command, preserves the return code, and executes the subsequent command (e.g. echo) based on the return value.  Note; we can force a failed return code by specifying a negative sleep duration.

An alternative shortened syntax is to specify the command sequence on a single line using a '&&' or '||' seperator, the seperator determing if the subsequent command should be exectued on success/failure of the preceeding command.

              command1 && command2

 This syntax issues command2 if, and only if, command1 returns an exit status of zero.

        An OR list has the form

               command1 ││ command2

Now, command2  is  executed  if  and only if command1 returns a non-zero exit status.  This can be used to readily define recovery, or notification, behaviors for a failed command. 

The return status of AND and OR lists is the exit status of the last command executed in the list.

      1  #!/bin/bash

     2

     3  sleep 1 && echo "run something else"; #--execute command2 if command1 was successful

     4  sleep -1 || echo "run something else"; #--execute command2 if command1 failed

     5  sleep -1 && echo "run something else"; #--execute command2 if command1 was successful

     6

Read-Only Variables in Bash

 

Bash has a little known concept of read-only variables.  When specifying 'readonly' during a variable assignment prevents the variable from being reassigned.

 For example;

 1  #!/bin/bash

2

3  readonly x=0

4  echo $x

5  x=1

6  echo $x

Line 3 specifies a read-only assignment of variable x to 0, an attempt to reassign on line 5 will be disregarded and produce an error message.  The error doesn't prevent continuation of the script however.

 

::::::::::::::

Detect Command Timeouts using Bash

 A common, and often overlooked, need for scripting is addressing hanging, or unresponsive commands.  What should your script do if one of it's ste

ps takes much, much, much longer than expected?  A hanging, or unresponsive, subprocess will result in a hanging job which in turn can prevent oth

er jobs from running.  Let's spend a bit of time how to address such an need.

 

The 'timeout' command is provided by the coreutils package, often readily available on default installation for most distributions.

 

$ sleep 10

$ echo $?

0

 

Execution of subprocesses generally set a return code, zero or non-zero, often zero indicating successful execution of the command, non-zero return codes if it failed for some reason.  The above command sequence will execute a sleep command (for 10 seconds), the second command echoing the return code.  Assignment of $? to zero indicates the sleep command executed successfully.

 

So, what if we wanted to prevent a hanging command to go undetected and/or prevent the rest of a script from running.  Let's say we want to enforce a command takes no longer than 5 seconds.
 

$ timeout 5 sleep 10

$ echo $?

124

 

After 5 seconds, the command terminates, the return code 124 indicates the command timed out.  Usage of this allows setting hard constraints on how long a command is allowed to run before terminating or being terminated.  The return code allows determining if the command timed-out.
 

Cheers.

Synchronizing Completion of Concurrent Commands

 

Running concurrent commands in a script generally is done by executing processes in the background.  Numerous commands can be spawned concurrently then synchronized by waiting for some, or all, of them to complete before proceeding or terminating the parent script.

     1  #!/bin/bash

     2

     3  sleep 10 &

     4  pId1=$!

     5

     6  sleep 2 &

     7  pId2=$!

     8

     9  sleep 3 &

    10  pId3=$!

    11

    12  echo "waiting for jobs to complete"

    13  #wait ; #--wait for all jobs to complete

    14  wait $pId1 $pId2 $pId3 ; #--wait for specific jobs to complete

 

In the above script, a sequence of sleeping {10, 2, and 3} seconds are executed in sequence, each in the background.  The longest running command  (e.g. 10 seconds in this example) should take 10 seconds, the other two commands concluding earlier.

We can test the proper exection, the script completing in ~10 seconds by timing the execution as follows:

$ time ./foo

waiting for jobs to complete

 

real    0m10.008s

user    0m0.003s

sys     0m0.004s

$

bash-4.1$