missing_class

Shell scripting

menu_book

Table of Contents

Based on https://missing.csail.mit.edu/2020/shell-tools/

Assignment and string delimiters

Functions

bash allows you to define functions

mcd () {
    mkdir -p "$1"
    cd "$1"
}

Special Variables

bash has many special variables to refer to arguments, error codes etc. Reference list.

Streams

Stream redirection

Return codes and conditional execution

$ false || echo "Oops, fail"
Oops, fail

As false returns 1, echo is executed

$ true || echo "Will not be printed"
$

As true returns 0, echo will not be executed

$ true && echo "Things went well"
 Things went well

The second command executes as the first runs without errors

$ false && echo "Will not be printed"
$

The second command doesn’t execute as the first returns an error state

$ false ; echo "This will always run"
This will always run

Command and process substitution

$ foo=$(pwd)
$ echo $foo
/home/user/

Here we are getting the output of command pwd and storing it as a variable.

Here’s an example that iterates through provided arguments greps for string foobar and appends to the file as a comment if it isn’t found

#!/bin/bash

echo "Starting program at $(date)" # Date will be substituted

echo "Running program $0 with $# arguments with pid $$"

for file in $@; do
    grep foobar $file > /dev/null 2> /dev/null
    # When pattern is not found, grep has exit status 1
    # We redirect STDOUT and STDERR to a null register since we do not care about them
    if [[ $? -ne 0 ]]; then     # if return code of last command is not equal to 0
        echo "File $file does not have any foobar, adding one"
        echo "# foobar" >> "$file"
    fi
done

Shell globbing

Shell globbing: expanding expression e.g. filename expansion

More examples:

This creates files foo/a, foo/b, … foo/h, bar/a, bar/b, … bar/h:

$ mkdir foo bar
$ touch {foo,bar}/{a..j}
$ touch foo/x bar/y

Show differences between files in foo and bar

$ diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

Shebang

e.g. at start of script you will see line #!/bin/bash

Shell functions and scripts: the differences

Shell Tools

Finding how to use commands

Given a command how do you go about finding out what it does and its different options?

  1. Run with -h or --help flags
  2. Look at man page for more detail
  3. Interactive tools (e.g. ncurses): :help or ?
  4. TLDR pages focuses on giving example use cases of a command so you can quickly figure out which options to use.

Finding files

find

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '**/test/**/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

Find can also perform actions on the stuff it finds, which helps simplify monotonous tasks.

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {.}.jpg \;

fd

find has tricky syntax: e.g. to find files that match some pattern PATTERN you have to execute find -name '*PATTERN*' (or -iname if you want the pattern matching to be case insensitive).

locate

locate

Finding code

But grep -R can be improved in many ways, such as ignoring .git folders, using multi CPU support, etc. Various alternatives, but pretty much cover same use case:

Some examples:

# Find all python files where I used the requests library
rg -t py 'import requests'
# Find all files (including hidden files) without a shebang line
rg -u --files-without-match "^#!"
# Find all matches of foo and print the following 5 lines
rg foo -A 5
# Print statistics of matches (# of matched lines and files )
rg --stats PATTERN

History: Finding shell commands

History-based autosuggestions

Directory Navigation

More complex tools to get an overview of a directory structure

Alternative Shells

zsh: Z shell is an extended version of the Bourne Shell (sh), based on bash zsh cheatsheet Pros: ref

Installation guide Install Oh My Zsh To get fonts working correctly using Ubuntu via WSL, had to install powerline fonts on Windows by cloning the repo and installing (ref):

>git clone https://github.com/powerline/fonts.git
>cd fonts
>.\install.ps1

Manually select a powerline font on the Ubuntu window for special characters to work. Change directory colours (select from here)

# using dircolors.ansi-dark
curl https://raw.githubusercontent.com/seebi/dircolors-solarized/master/dircolors.ansi-dark --output ~/.dircolors
## set colors for LS_COLORS
eval `dircolors ~/.dircolors`

NB configuration menu accessed by

$ autoload -Uz zsh-newuser-install
$ zsh-newuser-install -f

Exercises

  1. Read man ls and write an ls command that lists files in the following manner

    • Includes all files, including hidden files
    • Sizes are listed in human readable format (e.g. 454M instead of 454279954)
    • Files are ordered by recency
    • Output is colorized

    A sample output would look like this

     -rw-r--r--   1 user group 1.1M Jan 14 09:53 baz
 drwxr-xr-x   5 user group  160 Jan 14 09:53 .
 -rw-r--r--   1 user group  514 Jan 14 06:42 bar
 -rw-r--r--   1 user group 106M Jan 13 12:12 foo
 drwx------+ 47 user group 1.5K Jan 12 18:08 ..

  2. Write bash functions marco and polo that do the following. Whenever you execute marco the current working directory should be saved in some manner, then when you execute polo, no matter what directory you are in, polo should cd you back to the directory where you executed marco. For ease of debugging you can write the code in a file marco.sh and (re)load the definitions to your shell by executing source marco.sh.

  3. Say you have a command that fails rarely. In order to debug it you need to capture its output but it can be time consuming to get a failure run. Write a bash script that runs the following script until it fails and captures its standard output and error streams to files and prints everything at the end. Bonus points if you can also report how many runs it took for the script to fail.

#!/usr/bin/env bash

n=$(( RANDOM % 100 ))

if [[ n -eq 42 ]]; then
   echo "Something went wrong"
   >&2 echo "The error was using magic numbers"
   exit 1
fi

echo "Everything went according to plan"

  1. As we covered in lecture find’s -exec can be very powerful for performing operations over the files we are searching for. However, what if we want to do something with all the files, like creating a zip file? As you have seen so far commands will take input from both arguments and STDIN. When piping commands, we are connecting STDOUT to STDIN, but some commands like tar take inputs from arguments. To bridge this disconnect there’s the xargs command which will execute a command using STDIN as arguments. For example ls | xargs rm will delete the files in the current directory.

    Your task is to write a command that recursively finds all HTML files in the folder and makes a zip with them. Note that your command should work even if the files have spaces (hint: check -d flag for xargs)

     find . -type f -name "*.html" | xargs -d '\n'  tar -cvzf archive.tar.gz

  2. (Advanced) Write a command or script to recursively find the most recently modified file in a directory. More generally, can you list all files by recency?

Solutions

  1. $ ls -laht --color
  2.  marco() {
     export MARCO=$(pwd)
 }

 polo() {
     cd "$MARCO"
 }

  3.  #!/usr/bin/env bash

 n=$(( RANDOM % 100 ))

 if [[ n -eq 42 ]]; then
   echo "Something went wrong"
   >&2 echo "The error was using magic numbers"
   exit 1
 fi

 echo "Everything went according to plan"
  4. First create some html files 
    $ mkdir foo bar
$ touch {foo,bar}/{a..g}.html
$ touch {a..g}.html
$ find . -path "*.html"
./a.html
./b.html
./bar/a.html
./bar/b.html
./bar/c.html
./bar/d.html
./bar/e.html
./bar/f.html
./bar/g.html
./c.html
./d.html
./e.html
./f.html
./foo/a.html
./foo/b.html
./foo/c.html
./foo/d.html
./foo/e.html
./foo/f.html
./foo/g.html
./g.html
$ find . -name "*.html" | xargs -d '\n' tar -cvzf test.tar.gz

Edit this page.