Tutorial

A 20 minute tutorial describing how to use stubbs to create a new rerun module. You will learn how to:

• create a new rerun module
• create commands and options
• create tests
• auto generate documentation
• package the module for distribution

• You will be running this tutorial in a bash shell on a *nix host.
• See the Installation guide to learn how to install rerun.
• Read What is a module? to learn about what a module is.
• The documentation part of the tutorial requires markdown and pygmentize.

Imagine we have a utility script that continually pings a host until it is reached. Here is the source for our script, "waitfor-ping.sh".

File listing: waitfor-ping.sh

#!/bin/bash
HOST=$1
INTERVAL=$2
until ( ping -c 1 $HOST | grep -q ^64 )
do
   sleep $INTERVAL
   echo Pinging $HOST...
done
 
echo "OK: $HOST is pingable."

It's a simple straightforward shell script. A command pipeline made up of ping and grep are used as a test executed by the until command. Command line arguments specify the host and the number of seconds between checks. Here's how you can run it:

$ waitfor-ping.sh myserver 30
OK: myserver is pingable.

It's easy to take this kind of script and make it a rerun command. As a rerun command, it gets built-in option processing, packaging, documentation and an easy way to test it.

Let's stop and think about design a little bit. Besides the "waitfor-ping.sh" script, imagine we have a set of scripts useful to check and verify things. For example, we often want to know:

• when is a host reachable via ping?
• when does a file contain a particular text string?
• when does a URL respond with the expected output?

For this tutorial, we'll create a module called "waitfor" because we imagine there is a class of these kinds of checks. A module is a good way to organize each kind of check. Each check would become a command in the module. If we have an existing script that does this check, it can be modified to fit in the waitfor module.

Each check looks at the state of something. For these kinds of checks it is sometimes required to wait for the thing to be reached so the ability to retry up to a maximum number of attempts is important, too. These parameters can be defined as module "options" that can be passed to a command.

During the tutorial you will also learn the general development cycle for a rerun module:

• Create a module.
• Add a command and any options.
• Edit and test the command.
• Release a module.

Rerun has a simple syntax.

$ rerun module:command [options]

The "module" is the name of the module. The "command" is the name of the command in the module you want to execute. The "[options]" are any command line options the command supports.

We'll be using the stubbs commands to help create the waitfor module.

The first step is to create a new module structure. Use stubbs:add-module to create the "waitfor" module.

rerun stubbs:add-module --module waitfor --description "utility commands that wait for a condition."

The stubbs:add-module command doesn't do to much other than create the direcotry structure and initial files.

The diagram below shows off the minimal directory structure created for the new module.

modules/waitfor/
|-- commands
|-- lib
|   `-- functions.sh
`-- metadata

• The commands directory will contain subdirectories for each command.
• The lib directory contains any common code. The functions.sh file provides general functions to all the commands in the module.
• The metadata file contains properties in a key=value format. This is where the module's name and description are stored.

Stubbs writes the files relative to the $RERUN_MODULES directory. By default, $RERUN_MODULES is relative to your rerun executable. You can set this to another abitrary location if need be.

The initial module structure is essentially empty. To make the waitfor module useful, add a command.

Let's create a command called "ping" that uses the "waitfor-ping.sh" code to check the ping status of a host.

Use stubbs:add-command to create a new command for the waitfor module.

rerun stubbs:add-command --module waitfor --command ping --description "wait for ping response from host"

You can see stubbs:add-command produced more output. Two files to note are "script" and "ping-1-test.sh" files.

Wrote command script: /Users/alexh/rerun/modules/waitfor/commands/ping/script
Wrote test script: /Users/alexh/rerun/modules/waitfor/tests/ping-1-test.sh

The "script" file is what will be executed by rerun anytime somebody calls rerun waitfor:ping. The "ping-1-test.sh" file contains the test plan for the waitfor:ping command.

The diagram below shows the module directory structure after adding the new "ping" command:

modules/waitfor/
|-- commands
|   `-- ping
|       |-- metadata
|       `-- script
|-- lib
|   `-- functions.sh
|-- metadata
`-- tests
    |-- functions.sh
    `-- ping-1-test.sh

The command is ready to run. Try the new waitfor:ping command:

$ rerun waitfor:ping

Nothing happens. Well that's to be expected. Stubbs doesn't do all the work. We've basically got an empty command script.

The "waitfor:ping" command will need to take parameters to specify which server to reach or how long to wait between checks.

• host: the server to reach
• interval: the seconds between each check

Each declared option becomes a command line flag "--$OPTION". Options that take an argument are specified as: "--$OPTION arg". Options can also declare a default value used if the user does not specify it. Option values become accessible to the shell script as a variable.

Create a new "--host" option so users can specify which host to reach.

rerun stubbs:add-option --option host --description "the host to reach" \
  --module waitfor --command ping \
  --required true --export false --default '""'

The "stubbs:add-option" command produces output showing new module files. They declare the option and assign the option to the command.

Wrote option metadata: /Users/alexh/rerun/modules/blar/options/host/metadata
Updated command metadata:  /Users/alexh/rerun/modules/blar/commands/ping/metadata
Updated command script header: /Users/alexh/rerun/modules/blar/commands/ping/script

Next, create an "--interval" option specifying how long to wait between checks.

rerun stubbs:add-option --option interval --description "seconds between checks" \
  --module waitfor --command ping \
  --required false --export false --default 30

The rerun listing mode will display the command usage:

$ rerun waitfor
ping: "wait for ping response from host"
    --host <>: "the host to reach"
    [ --interval <30>]: "seconds between checks"

The listing shows two options have been defined for the ping command. The usage output format shows that both options take an argument ("<>") and that the interval option has a default value ("< 30 >").

The option values specified by the user will be accessible as shell variables to the command script. In this case, those variables will be HOST and INTERVAL.

It's time to make the command do something by adding some useful code. Each rerun command has a corresponding script which implements it. The rerun convention dictates the script location will be:

$RERUN_MODULES/$MODULE/commands/$COMMAND/script

In the case of the waitfor:ping command the path would be:

$RERUN_MODULES/waitfor/commands/ping/script

For convenience, the "stubbs:edit" command opens the script in your $EDITOR.

rerun stubbs:edit --module waitfor --command ping

Opening the generated script file will show something similar to the content below (some of the comment lines have been left out).

#!/usr/bin/env bash
 
#/ command: waitfor:ping: "wait for ping response from host"
#/ usage: rerun waitfor:ping [options]
 
. $RERUN_MODULE_DIR/lib/functions.sh ping || { 
  echo >&2 "Failed loading function library." ; exit 1 ; 
}
 
trap 'rerun_die $? "*** command failed: waitfor:ping. ***"' ERR
set -o nounset -o pipefail
 
#/ rerun-variables: RERUN, RERUN_VERSION, RERUN_MODULES, RERUN_MODULE_DIR
#/ option-variables: HOST INTERVAL
 
rerun_options_parse "$@"
 
# Command implementation
# ----------------------
 
# - - -
# Put the command implementation here.
# - - -
 
# Done. Exit with last command exit status.
 
exit $?

For the most part, you will typically focus below the “Command implementation” part of the file. Above the Command implementation section is boiler plate code that takes care of sourcing the module function library, sets up error handling and parses the command line options.

To create the implementation, let's use the code from the original script "waitfor-ping.sh".

# Command implementation
# ----------------------
 
# - - -
until ( ping -c 1 $HOST | grep -q ^64 )
do
   sleep $INTERVAL
   echo Pinging $HOST...
done
 
echo "OK: $HOST is pingable."
# - - -
 
exit $?

The HOST and INTERVAL variables are declared by the command options: --host and --interval. When the command runs, the rerun_options_parse function reads the command input and assigns those variables.

Give the command a try. Specify localhost since we know that will be reachable:

$ rerun waitfor:ping --host localhost --interval 1
OK: localhost is pingable.

As expected, localhost is reachable with ping.

How do you know your command will always work? Write a test to find out and then run the test regularly! Stubbs makes this easy by including the simple test framework called roundup. A test written in roundup is just another simple shell script following a few simple conventions.

Whenever a command is created by stubbs:add-command, a roundup test plan is generated for the command. Each test plan can have multiple tests.

To run a test use stubbs:test and specify your module.

$ rerun stubbs:test --module waitfor
========================================================
 TESTING MODULE: waitfor
=========================================================
ping
  it_fails_without_a_real_test:                    [FAIL]
=========================================================
Tests:    1 | Passed:   0 | Failed:   1

Each test run can result with either a "PASS" or "FAIL" status. In the example above, the test called "it_fails_without_a_real_test" failed. Of course, the test is designed to fail so you will be prompted to replace it with a useful one.

If you have more than one command in your module you'll end up having multiple test plans. To execute just a single plan use the --plan flag. The following runs just the test plan for the waitfor:ping command:

rerun stubbs:test --module waitfor --plan ping

Generally, each plan corresponds to a command in the module. This is just a naming convention though. You can add test plans and name them according to roundup's file glob pattern:

*-test.sh

Writing tests for your module is easy. You only need to know shell scripting and roundup's simple conventions.

All tests are located in a "tests" subdirectory of the module:

$RERUN_MODULES/$MODULE/tests

Below you see the boilerplate roundup script for the waitfor:ping command: $RERUN_MODULES/waitfor/tests/ping-1-test.sh.

The test file is roughly divided into two sections. The first section defines any helper functions and describes the plan. The describe function is just for logging purposes. Roundup tests are written as shell functions prefixed with "it_". Any command that exits non-zero inside a test function will cause the test to fail.

Let's look at the "it_fails_without_a_real_test" test function. Firstly, its name is descriptive. Imagine if you replaced the underscore characters with spaces. It would then read: "it fails without a real test". The function name should express the intent of the test.

The body of the test function executes the exit 1 command. This will cause the test function to fail. Any command that exits with a non-zero exit code will cause the test to fail.

#!/usr/bin/env roundup
#
#/ usage:  rerun stubbs:test -m waitfor -p ping [--answers <>]
#
 
# Helpers
# -------
[[ -f ./functions.sh ]] && . ./functions.sh
 
# The Plan
# --------
describe "ping"
 
# ------------------------------
# Replace this test. 
it_fails_without_a_real_test() {
    exit 1
}
# ------------------------------     

Let's replace the test with a useful one. The waitfor:ping command takes a required option, "--host". Let's be sure it causes an error if the user does not specify it. We'll name the function "it fails without required options" to express the intent of the test.

it_fails_without_required_options() {
    OUT=$(mktemp /tmp/waitfor:ping-XXXX)
    ! rerun waitfor:ping 2> $OUT
    grep 'missing required option: --host' $OUT
    rm $OUT
}

The test implementation is simple.

• A temporary file is created by mktemp to store the command output.
• The "rerun waitfor:ping" command is executed. Adding the ! before the rerun command negates the error caused by the missing option.
• Then grep is used to check the output file for the expected string.
• The temporary file is deleted by rm to clean up.

Let's add a functional test. This time let's confirm localhost can be reached.

it_reaches_localhost() {
    OUT=$(mktemp /tmp/waitfor:ping-XXXX)
    rerun waitfor:ping --host localhost > $OUT
    grep 'OK: localhost is pingable.' $OUT
    rm $OUT
}

Again, the test implementation is straightforward and follows the pattern from our first test.

Run the tests again.

$ rerun stubbs: test --module waitfor --plan ping
=========================================================
 TESTING MODULE: waitfor 
=========================================================
ping
  it_fails_without_required_options:               [PASS]
  it_reaches_localhost:                            [PASS]
=========================================================
Tests:    2 | Passed:   2 | Failed:   0

Successful tests! From here on you can work in a "test-driven" style to be sure you own your code and not the other way around!

For any other commands you want to add to your module, you follow the same cycle:

• Create command.
• Add any options the command needs.
• Edit and test your command.

Eventually, you'll finish adding and implementing commands and want to put your module into operation. Releasing your module brings up a few concerns.

• How to document the module so you don't have to be around to answer questions?
• What's the best way to distribute it?
• How to handle dependencies your module might have?

You won't always be around to answer questions about how to use or modify your module. This is where documentation comes into play. Writing docs is tedious work and takes time to make it something easy to use and easy for you to keep up. To make documentation easier, stubbs:docs will generate documentation pages from your module source. The stubbs:docs command uses markdown and Ryan Tomayko's excellent shocco doc generator that produces documentation from your shell scripts.

Here's how to generate documentation for the waitfor module:

$ rerun stubbs:docs --module waitfor
Unix manual: /Users/alexh/rerun/modules/blar/docs/blar.1
HTML site: /Users/alexh/rerun/modules/blar/docs/index.html

Two kinds of output are produced. Nroff is generated for a Unix manual page. You can use the nroff command to render it:

$ nroff -man /Users/alexh/rerun/modules/blar/docs/blar.1 | more

An HTML site is created which you can browse from the index.

The stubbs:docs command generates a lot of documentation just from the source of your module. Here's what comes for free:

• Module index page: Describes basic module usage. Links to pages describing commands, options, tests and function libraries.
• Per- command page: Gives a command synopsys, lists any tests and provides links to see HTML renderings of the command script sources and test plans.
• Per option page: Gives option synopsys, lists any commands that use the option.

Screenshots

You'll notice throughout the HTML documentation there is an empty section called "README". The stubbs:docs command looks for README.md files in the module source tree and inserts their text into the corresponding documentation page.

Here are the locations stubbs:docs looks for README.md files:

• RERUN_MODULE_DIR/README.md: Add general usage information about the module.
• RERUN_MODULE_DIR/commands/*/README.md: Add use case examples for each command.
• RERUN_MODULE_DIR/options/*/README.md: Add use case examples for each option.

Your rerun module can run where you developed it. You can copy the rerun modules directory (eg, $RERUN_MODULES) to other hosts but that's messy and doesn't make handoffs easy.

The stubbs:archive command can create a distribution of your modules. There are two kinds of archive formats: bin and rpm. The --format option lets you specify what kind of archive ("bin" is default).

bin format

The "bin" format is a self-extracting shell script. It will include any modules you specify and the invoking rerun itself.

$ rerun stubbs:archive --modules "waitfor"
Wrote self extracting archive script: /Users/alexh/rerun/rerun.bin

You execute the rerun.bin file like you would execute rerun. Here's a listing of the archived modules:

$ ./rerun.bin
  waitfor: "utility commands that wait for a condition." - 1.0.0

Running a command is just the same as running it via rerun:

$ ./rerun.bin waitfor:ping --host localhost

rpm format

If you are on a Centos/Redhat machine you can use stubbs:archive to generate an RPM. The "rpm" format works a bit differently than the "bin" format in that it packages up single modules into an RPM.

Here's how to create an RPM for the waitfor module (assumes you are on a Centos/Redhat host):

$ rerun stubbs:archive --modules waitfor --format rpm
Wrote waitfor module rpm : rerun-waitfor-1.0.0-1.noarch.rpm

The stubbs:archive command reads the module metadata to look for version and licence information. The metadata also contains properties to manage dependencies.

• REQUIRES: In case your module needs commands that come from other modules, specify their names in the REQUIRES.
• EXTERNALS: If your module has dependencies on external tools and they are accessible as RPMs use the EXTERNALS property.

The example below shows the waitfor RPM module depends on the stubbs RPM (REQUIRES=stubbs). It also depends on the wget package (EXTERNALS=wget).

$ cat modules/waitfor/metadata
# generated by stubbs:add-module
# Wed Dec 26 14:38:37 PST 2012
NAME=waitfor
DESCRIPTION="utility commands that wait for a condition."
SHELL="bash"
VERSION=1.0.0
REQUIRES=stubbs
EXTERNALS=wget
LICENSE=

Stubbs:archive uses the metadata to generate the appropriate RPM specification details to support package dependencies. Yum's ability to download dependencies makes it easier to maintain your modules and their supporting tools.