Summary and Schedule

Welcome to our mini-tutorial!

We will do a sequence of hands-on tasks on a toy research project to demonstrate a selection of Software Tools for Reproducible Research. The essential tools are Python, Make, Git, and LaTeX. Here is a longer list of tools we recommend.

We used software from the carpentries (without their endorsement) to build today’s tutorial. The tools are connected to a literature about software practices.

Quick Start

Without reading further, you can see some of the tools in action with these shell commands

BASH 

$ git clone git@gitlab.com:fraserphysics/zipf.git work_dir
$ cd work_dir
$ git checkout 06-documentation
$ make

If that is successful, the last line of output will be

OUTPUT 

Latexmk: All targets (build/report.pdf) are up-to-date

and you can view the result in build/report.pdf.

If it didn’t work, go to the setup page.

Prerequisite

Prerequisites

We will assume that you are somewhat familiar with the following tools (in order of familiarity)

bash
It’s probably OK if you use a different shell, but you should be comfortable with command line tools.
A text editor of your choice
Software Carpentry insists that instructors use Nano to avoid frightening the learners. We will use Emacs or Vim so that we can cover more material.
Python
Most of the code we will discuss is written in Python. If you are not comfortable with Python, you should not sign up for this mini-tutorial.
git
We use git to keep code up to date for each learner as we progress from one section to the next throughout the mini-tutorial. While we will provide you with the necessary git commands, it will make more sense to you if you have used the tool before.
LaTeX
We use LaTeX to format results. As with git, it will make more sense to you if you have used the tool before.

A Real Project

While we use a toy project in this tutorial to introduce a collection of tools, Andy Fraser has used the same tools to build the second edition of “Hidden Markov Models and Dynamical Systems” which SIAM will publish in 2027. Here you can read a brief description of that project.

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Setup

For this tutorial you will use some files that represent a toy research project. Some of those files are data and some are code. You should fetch those project files using the git commands that we specify below.

You will also need some software in an environment on your computer. We assume that your computer provides a UNIX-like environment. How you install the software will depend on your operating system and your environment.

Please clone the git repository and install the required software before coming to the tutorial.

Project Files via Git

You need to clone a git repository to follow this tutorial. We’ve made a separate branch in the repository for each episode so that as we progress from one episode to the next you can save your work on the old branch and check-out the next branch to get files that are identical to the ones the instructor is using. Here are the steps:

  1. Open a Bash shell window.

  2. (Optional) Navigate to a convenient directory.

  3. Use the following commands to fetch the project and list the branches:

BASH 

$ git clone git@gitlab.com:fraserphysics/zipf.git work_dir
$ cd work_dir
$ git branch -a

The result of the last command should be something like

OUTPUT 

* 06-documentation
  remotes/origin/01-command-line
  remotes/origin/02-first-makefile
  remotes/origin/03-no-hands
  remotes/origin/04-testing
  remotes/origin/05-standards
  remotes/origin/06-documentation
  remotes/origin/HEAD -> origin/06-documentation

Minimum Software

You will need to have the following software installed on your computer to follow the first episodes of this tutorial:

Make
We use Gnu Make to specify the whole sequence of operations from raw data to a pdf document. Other versions of make may work, but we have not tested them. We occasionally use features that only the Gnu version of make provides.
Python
We use Python for calculations, and we segregate all plotting into scripts that use Matplotlib.
Git
We will use git branches to keep episodes using various tools separate.
Sed
Sed is a command line stream editor that we use to extract documentation from makefiles in response to “make help”.
LaTeX
The tutorial uses LaTeX for formatting.

The following commands exercise this minimal set of software tools:

BASH 

$ git checkout 04-testing
$ make help

should yield

OUTPUT 

 help                 : Holy mackerel!  What is this?

and

$ make

should yield several screens of output ending with

Latexmk: All targets (build/report.pdf) are up-to-date

More Software

The later episodes use the following additional tools:

yapf
We use (Yet Another Python Formatter) to coerce code to follow Google’s Python style.
mypy
Mypy is a static type checker for Python.
pylint
Analyzes and scores python code against a user defined style guide, pylintrc.
sphinx
We use Sphinx to format documentation. We use the autodoc extension that imports the modules we are documenting. It pulls in documentation from docstrings in a semi-automatic way.
pytest
We use pytest to write small, readable tests.
pdb
pdb is Python’s built-in interactive source code debugger.

You can verify that these tools are installed with the following commands

BASH 

$ yapf --version
$ mypy --version
$ pylint --version
$ sphinx-build --version
$ pytest --version
$ python -m pdb --help