Environment modules on the Nevis particle-physics Linux cluster

Environment modules are a way of setting up a Linux shell environment. On the cluster, we use the module command to set up software packages available on the applications server.

Quick start guide

To set up ROOT:

module load root

To set up Geant4:

module load geant4

To see what packages and versions are available:

module avail

If you just want to see the versions for a particular package, e.g., ROOT:

module avail root

To get a brief description of a given package, e.g., Geant4.9.6:

module whatis geant/4.9.6

To get a longer description:

module help geant/4.9.6

What are environment modules?

The command-line environment you use to issue commands in Linux is called the shell. Your shell has an "environment": a collection of variables, aliases, and definitions to make the shell easier to use. A software package that is not built-in to Linux (such as ROOT or Geant4) needs the shell environment to be set up in various ways: defining shell variables such as $ROOTSYS, adding binary executables to the shell's search path variable $PATH, adding library locations to the shell's library search variable $LD_LIBRARY_PATH, etc.

Environment modules are a way taking care of these details for you. Instead of having to type something like:

export ROOTSYS=/usr/nevis/ROOT-05.34.23
export PATH=$ROOTSYS/bin:$PATH
export LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH

every time you use ROOT, you can just type this instead:

module load root

Another advantage of environment modules is that they are shell-independent. For example, the above export commands are valid in bash and zsh, but in csh you must use:

setenv ROOTSYS /usr/nevis/ROOT-05.34.23
setenv PATH $ROOTSYS/bin:$PATH
setenv LD_LIBRARY_PATH $ROOTSYS/lib:$LD_LIBRARY_PATH

Even in csh, the environment modules command is still module load root.

Another advantage of environment modules: the obscure bits of package set ups are handled for you. For example, neither of the above ROOT examples demonstrate how to set the value of $PYTHONPATH in case you want to use PyROOT. The Nevis ROOT environment modules take care of this.

A benefit for more advanced users is that environment modules can be used in some scripting languages such as perl and python; see the man page for the module command for details.

Using environment modules

Typically everything you need to know is in the "Quick Start" section above. There are more details in the man page for the module command. Here are a few additional tips.

To see what module you've loaded:

module list

To remove a module's effects on your shell environment; e.g., any version of ROOT you've loaded:

module unload root

Python

If you try module avail, you will see more packages than just ROOT. In particular, if you're working with python, it's worth looking at the version in the Nevis environment modules.

module load python

This will load the latest version of Python (3.10 as of May-2022) from the Linux cluster software libraries (use module avail python to see all the available versions). You can see which packages I've installed as part of our Python distribution with:

pip freeze

Take a look. You might be surprised by what I've included. As of May-2022, the list includes:

jupyter jupyterlab iminuit numpy scipy matplotlib pandas sympy terminado urllib3 pycurl tables rootpy rootkernel uproot scikit-learn tensorflow keras torch torchvision scikit-hep h5py astropy gammapy fitsio healpy astropy-healpix cython numba numba_stats

All the packages you see on that list are also available via our notebook server.

If you want a package that's not on the list, you can install it in your home directory with:

pip install --user --upgrade <package-name>

If a package is of sufficient interest that other users may want it, let WilliamSeligman know. He'll install it on the both the library server and the notebook server.

This approach has some advantages:

• It saves disk space. If every user starts installing entire software suites in their home directories, there's less space available for other work.

• The library server directory /usr/nevis is available to every system in the Linux cluster, including the batch nodes. It might simplify the task of making necessary software packages available for your condor jobs.

Note that module load root will also load a compatible version of python, with at least the packages listed above.

Do not use the module switch command at Nevis

The reason is that most of the modules are "nested" and call sub-modules; e.g., the command module load geant4 will call module load common/geant4, which will call module load clhep, which will call module load common/clhep, which will call module load gcc, which will call module load common/gcc. This poses no problem if you use module unload AAA/xxx followed by module load AAA/yyy, but module switch AAA/xxx AAA/yyy will confuse the environment modules mechanism.

Be careful of compiler versions

If you type this command, you'll see the brief help message for every module at Nevis:

module whatis

As you'll see, there's a tangled history of different versions of ROOT, Geant4, and the versions of GCC that were used to compile them.

The following basic setup command will always work here, as the default versions of ROOT and Geant4 will always be compiled with the same compiler version:

module load root geant4

If you want to know the default ROOT and Geant4 versions:

module avail root
module avail geant4

It becomes complicated when you're working with a program that uses both Geant4 and ROOT, and you want to use versions other than the defaults. Then you'll want to check that the versions were compiled with the same or compatible versions of the compiler.

To see the compiler used to compile ROOT is simple. If you want to check ROOT v5.30.00:

module help root/05.30.00

To see which compiler was to used to compile a specific version of Geant4, e.g., Geant4.9.1:

module help geant4/4.9.1

This will tell you which version of CLHEP was used to compile Geant4; in this case it's 2.0.3.1. Then you'll have to look at the details of that version of CLHEP:

module help clhep/2.0.3.1

What if you want to use versions of ROOT and Geant4 simultaneously that were compiled with incompatible compilers? You can contact WilliamSeligman to create/recompile the software versions for you. You can also be creative and compile your own versions and use your own environment modules to set them up. This takes us to the next section...

Writing your own environment modules

Why would you want to do this?

• You may want to use a different shells for your interactive work than for work that you submit to the Nevis Condor batch farm. For example, the default Nevis shell is zsh, while the default shell for batch jobs is bash.

• Different places may use different shells. For example, at Fermilab bash is the preferred shell, while at CERN they prefer zsh, except for electronics design for which scripts are typically written in tcsh.

• You may want to install your own version of ROOT or Geant4, or even a software package specific to your group, and make use of existing Nevis modules.

Writing your own environment modules is fairly easy. The documentation is in the modulefile man page. You can see how WilliamSeligman set up his environment by looking at his initialization files:

less ~seligman/.myprofile
less ~seligman/modules/my-login-init

You can also pick up tips by looking at the Nevis module files in /usr/nevis/modules.

The scripting language used in module files is Tcl. You don't have to know Tcl to do anything basic with environment modules, but if you want to do anything fancy (e.g., load a package version based on system architecture) you'll find plenty of documentation is just a web search away.

How are environment modules set up at Nevis?

The initial set-up is in /usr/nevis/adm/nevis-init.sh (for sh-style shells) and /usr/nevis/adm/nevis-init.csh (for csh-style shells). These scripts are invoked by the default shell initialization scripts in your home directory: .bashrc, .zprofile, and .cshrc for bash, zsh, and csh respectively.

The Nevis-specific module files are in /usr/nevis/modules. There are some example/utility files that part of the default Linux environment modules package; these are located in /usr/share/Modules/modulefiles and /etc/modulefiles.

Transition: From setup to environment modules

Why?

Prior to Feb-2015, software packages at Nevis were set up using a custom-written setup command. Why make a change?

• The setup script only works on the Nevis Linux cluster. Environment modules are an open-source project and are available on almost every flavor of Linux, Mac OS, and Windows.

• When WilliamSeligman wrote the initial setup script in 1997, it didn't conflict with any existing command. Since then, a setup command has been added to Redhat Linux. Also, the Fermilab UPS tool has its own setup command, which is used in products such as LArSoft. These command conflicts have sometimes been confusing to work around.

• WilliamSeligman maintained the setup command by revising two different scripts, /usr/nevis/adm/setup.sh and /usr/nevis/adm/setup.csh, for sh-style shells and csh-style shells respectively. Keeping these two scripts synchronized was tricky. It was not uncommon for there to be an error in one of them that was not in another.

• If you look at the two above scripts, you'll see that they consist of lots of "if-then" conditional statements that grew ever more baroque as more package versions were installed at Nevis. The Nevis environment modules in /usr/nevis/modules have their own level of complexity (especially for Geant4), but at least the peculiar properties of each package's version can be isolated and customized separately.

• The "default" versions of Geant4 and ROOT for setup were compiled with different version of the GCC compiler, and so were inconsistent with each other; this means the simple command setup geant4 root might cause problems. Going forward, the default versions of ROOT and Geant4 as maintained by the Nevis environment modules will use the same compilers, so you can type module load geant4 root and not get compiler or library errors.

How?

For the most part, you don't have to do anything. The setup command is not going away any time soon. It simply won't be maintained anymore for any new package versions.

Over time, if you have something like setup root in your scripts, you can replace it with module load root. This may have unexpected results (the default versions of ROOT and Geant4 are different for the setup and module commands).

If you're loading a specific version of ROOT or Geant4, it's typically obvious how to update them, with some guidance from the module avail command. For example, setup root532 becomes module load root/05.32.00.