All Collections
Tutorials
Tutorial: how to write a postInstall script that downloads and installs research software
Tutorial: how to write a postInstall script that downloads and installs research software

With reference to translating a GitHub repo's readme into a working script.

Shahar Zaks avatar
Written by Shahar Zaks
Updated over a week ago

Background knowledge:

  • If you look at a Code Ocean capsule's base environment, you will almost always see that it is built on Ubuntu 18.04 or 20.04, two popular variants of the Linux operating system.

  • General-purpose libraries for Linux are typically distributed through APT, the 'Advanced Package Tool,' which is available on Code Ocean as the apt-get package manager in each capsule's environment

  • For software libraries not available through a package manager, we recommend that you write a postInstall script.

This article is about how to write such a script, and is aimed at readers without a lot of Linux or bash scripting experience.

Translating a GitHub repo's readme and releases into a postInstall:

Consider ICARUS Verilog, written by Stephen Williams and available at https://github.com/steveicarus/iverilog; it is "intended to compile ALL of the Verilog HDL as described in the IEEE-1364 standard."

(Note: you can install this via the apt-get package manager. Add iverilog  via apt-get; as of this writing, that will install iverilog version 10.1 in an Ubuntu:18.04 capsule. But you might need versions 10.2 or beyond, in which case, you will write a custom script.)

First, identify all dependencies. These are often listed as either 'dependencies' or 'prerequisites'. In this repo's readme.txt , in 2.1 Compile Time Prerequisites, we see that 8 are listed:


- GNU Make
...
- ISO C++ Compiler
...
- bison and flex
...

- gperf 3.0 or later
...
- readline 4.2 or later
...
- termcap
...
- autoconf 2.53 or later

These are the names of the libraries in general, but not necessarily the names of the packages you will install on Code Ocean. To find the name of a package on your particular operating system, you may need to search https://launchpad.net/ubuntu/bionic, https://launchpad.net/ubuntu/xenial, StackOverflow, or a search engine.
Two general notes:

  • If you are building packages from source, it is often easiest to add build-essential , which includes both gcc  and make and many other useful packages.

  • You can test whether you have the name of a package right by launching a terminal on a Code Ocean capsule, without any dependencies installed, and running apt-get update; apt-get install -y [your-guess-at-the-package-name] ; repeat the second step as necessary.

For ICARUS Verilog, the needed packages are: autoconf build-essential bison flex gperf libncurses5-dev libreadline-dev

Second, install these dependencies. If you will not need these dependencies during runtime,¹ we recommend the following syntax:

buildDeps="autoconf build-essential bison flex gperf libncurses5-dev libreadline-dev"
apt-get update && apt-get install -y $buildDeps

# ... install your code here

apt-get purge -y --autoremove $buildDeps
rm -rf /var/lib/apt/lists/*

Here, buildDeps is assigned to refer to the accompanying text; $buildDeps (note the $ sign) tells the system to insert that list into the specified command; and apt-get update && apt-get install -y tells the system to: 

  • Download the full list of available packages from http://archive.ubuntu.com/ubuntu

  • Install the needed packages (the -y tells the system to automatically answer any yes/no prompts with a 'yes').

Finally, apt-get purge -y --autoremove $buildDeps  removes those dependencies, and rm -rf /var/lib/apt/lists/*  clears the cached lists of apt-get packages. These steps create a smaller capsule, which will also load faster.²

Third, download the library. To help ensure long-term reproducibility, we recommend looking for a section called 'releases.' Looking at https://github.com/steveicarus/iverilog/releases, we see 229 separate releases,³ each available as a .zip or .tar.gz file:

  • Right/alt click the tar.gz text and copy the link to your clipboard.

  • Now we need to curl and untar⁴ the needed library, which you can do here as: curl -L https://github.com/steveicarus/iverilog/archive/v10_3.tar.gz | tar xz.

  • This downloads the release and unpacks it into a folder called iverilog-10_3.⁵

  • If there are no releases available, you can instead git clone the repo, e.g. git clone https://github.com/steveicarus/iverilog

Fourth, follow the instructions to build the software. This library's readme suggests that you:

  • run  sh autoconf.sh if you are building from Git, which we are (as we have taken code from the GitHub release);

  • use the commands: ./configure and make  to compile the software;

  • run make install  to install the software (you can add make check  to run some checks, but we'll skip that for now).

So you will add to your shell script:

cd iverilog-10_3
sh autoconf.sh
./configure
make
make install

 The final script will be:

#!/usr/bin/env bash
set -e

buildDeps="autoconf build-essential bison flex gperf libncurses5-dev libreadline-dev"
apt-get update && apt-get install -y $buildDeps

curl -L https://github.com/steveicarus/iverilog/archive/v10_3.tar.gz | tar xz
cd iverilog-10_3
sh autoconf.sh
./configure
make
make install

apt-get purge -y --autoremove $buildDeps
rm -rf /var/lib/apt/lists/*

Footnotes:

  1. If you will need the software's dependencies during runtime, instead of adding them to a buildDeps line, you can install them via the apt-get package manager in each capsule's environment.

  2. For more on best practices when configuring containers, see https://docs.docker.com/develop/develop-images/dockerfile_best-practices/.

  3. If you are unsure of which release to download, choose the latest.

  4. For a quick overview of these two commands, and piping them together, see https://www.tecmint.com/download-and-extract-tar-files-with-one-command/.

  5. To see what folder the software will be unpacked to, if you are using a Unix-based system (such as OS X), you can run the above commands on your local computer.

Did this answer your question?