JupyterROOTContainers < Main

Main Web>Computing>IPython>JupyterROOTContainers (2021-09-19, WilliamSeligman) (raw view)
---+!! Jupyter/ROOT Containers

<div style="float: right; background-color: #ebeef0; margin: 0 0 20px 20px; padding: 0 10px 0 10px;">
%TOC{depth="3" title="On this page:"}%
</div>

Users of the Nevis particle-physics [[Linux cluster]] have access to a [[IPython][notebook server]] that lets them execute Python and ROOT C++ scripts on a web browser. This page describes alternative methods for installing and using the Jupyter/ROOT/Python/C++ combination with (hopefully) a minimum of installation overhead. For example, this may be helpful for:

   * Nevis researchers who want to be able to run [[https://root.cern.ch/][ROOT]] on their laptops when they don't have internet access (e.g., on an airplane trip);

   * Folks who'd like to go through the [[https://www.nevis.columbia.edu/~seligman/root-class/][ROOT tutorial]] but who don't have computer accounts with a Nevis particle-physics group. 

---++ Anaconda

Anaconda is a software manager that allows you download and execute packages in your home directory. 

If you use the Nevis Linux cluster, then you should consider using [[environment modules]] over Anaconda, if for no other reason than to save disk space (see below). But if you're on a different system, or the Nevis environment modules don't offer the package or version you're looking for, Anaconda is a better choice. All the Nevis Linux cluster systems running !CentOS 7 have conda installed. 

If you're not at Nevis, you can also separately [[https://docs.anaconda.com/anaconda/install/][install Anaconda]]. 

Once you've installed Anaconda, you’ll probably have to include [[https://conda-forge.org/][conda-forge]]. The following commands set it up:
<verbatim>
conda config --add channels conda-forge
conda config --set channel_priority strict
</verbatim>

To install Jupyter/ROOT:
<verbatim>
conda create --name jupyter-pyroot jupyter python root
</verbatim>

Note that the name <tt>jupyter-pyroot</tt> is arbitrary; you can use any name for the [[https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html][conda environment]] that you wish. 

There are additional standard packages that you’ll probably want to include in your conda environment (note that [[https://docs.conda.io/projects/conda/en/latest/commands/install.html][<tt>conda install</tt>]] adds more packages to an environment you've already [[https://docs.conda.io/projects/conda/en/latest/commands/create.html][created]]):
<verbatim>
conda install --name jupyter-pyroot jupyterlab numpy scipy matplotlib 
</verbatim>

Your working group may use additional packages. For example, the VERITAS group at Nevis might want to use (in addition to the above):
<verbatim>
conda install --name jupyter-pyroot astropy gammapy 
</verbatim>

You only have to go through the above steps once to define an environment (e.g., jupyter-pyroot). Afterwards, once per login session, it’s necessary to activate it:
<verbatim>
conda activate jupyter-pyroot
</verbatim>

Once activated, you should be able to run ROOT by simply typing:
<verbatim>
root
</verbatim>

You can run jupyter with:
<verbatim>
jupyter notebook
</verbatim>
or to enable remote access (read all the instructions at the bottom of the  [[IPython][Jupyter]] page):
<verbatim>
jupyter notebook --no-browser --port=XXXX
</verbatim>

---+++ Warnings

<ul>

<li>If you’re installing packages that have been pre-compiled by the maintainers of conda and conda-forge, then this process usually works well. However, if you’re using a system that does not have a pre-compiled version, conda will compile them on your computer. Not only can this take some time, but the results can be unreliable; I’ve seen a computer pause for many hours to re-compile ROOT only to finish with a program that issued errors every time I tried to run it.</li> 

<li>A conda environment can take up a lot of disk space, since it not only installs the packages you list, but any other packages that they depend on. On my desktop computer, the minimal Jupyter/ROOT container described above takes up 3G. This may not seem like much, but summer students at Nevis have a disk quota of 10G. It’s possible to run into disk-space problems, especially if you’re sharing disk space with other users, if you add more packages to your environment, or begin to define multiple environments for different projects. </li>
 
<li>Anaconda changes your shell’s execution environment. It may be incompatible with other environment setups (such as !MicroBooNE’s !LArSoft, ATLAS’ Athena, or Nevis’ [[environment modules][module load]] command).</li>

<li>Your [[shell]]’s prompt will be changed by conda. Even when you’re not using conda, the text <tt>(base)</tt> will appear at the beginning of the prompt. If this doesn’t bother you, then ignore it. If it does, you can try:
<verbatim>
conda config --set auto_activate_base false
</verbatim>
You’ll have to log off then log in again to see the change. If you don’t want conda to alter your prompt even when you’re using an environment, this command will suppress conda’s prompt changes:
<verbatim>
conda config --set changeps1 False
</verbatim>
</li>
</ul>

---++ Docker

While Anaconda is an environment-level container, [[https://www.docker.com/][Docker]] (and Singularity below) are OS-level containers.  Docker is probably the best method of running Jupyter+pyroot without having to worry about issues like package compilation. Its disadvantage is that it requires administrative access to the host computer system (e.g., your laptop), both to install Docker and to run the Docker container. 

The first step is to install Docker. For Mac and Windows systems, use [[https://www.docker.com/products/docker-desktop][Docker Desktop]]; there's a different procedure needed for [[https://runnable.com/docker/install-docker-on-linux][Linux systems]]. 

Once Docker is installed and running, you'll be able to download and run a Docker container:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

(Windows users will probably need to use =%CD%= instead of =$PWD=.)

The first time you run this command, it will download a ~2.5GB container. Give it time. 

Finally you'll see some output. Look at that output carefully, as it will tell you how to access Jupyter via a web browser. For example, assume the output contains something like this:
<verbatim>
    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
    Or copy and paste one of these URLs:
        http://649d0c4b4dc1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
     or http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

Then start up a web browser and visit =http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c=. You'll see the standard Jupyter home page. 

---+++ Changing the port

Consider the command:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

That first =8080= is the port to use on your local computer. If you want to use a different port on your computer (for example, you're already using port 8080 for something else), change that first =8080= to a different port. Note that if you change the port, you'll also have to change the port in the URL in the output; e.g.,
<verbatim>
sudo docker run -p 7000:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

means you'll have to change:
<verbatim>
http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

to:
<verbatim>
http://127.0.0.1:7000/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

---+++ Changing the directory

Again, consider:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

That =$PWD= (=%CD%= in WIndows) just means "the current directory." The execution environment within the container uses =/work= for its files; the =-v= option in the command means "map =/work= to the current directory in the terminal." If you'd like to use a different directory on your computer as the work directory for the Docker container, just substitute that directory for =$PWD=. For example:
<verbatim>
sudo docker run -p 8080:8080 -v ~jsmith/root-class:/work wgseligman/jupyter-pyroot
</verbatim>

---+++ Changing the container

You can use =New -> Terminal= within Jupyter to get a shell. Within that shell, you can modify anything within the container you want; for example, you can use [[https://pip.pypa.io/en/stable/user_guide/][pip3]] to install new Python packages or [[http://yum.baseurl.org/wiki/YumCommands.html][yum]] to install new Linux packages. (If you install something that might be of general interest, let WilliamSeligman know so he can add it to the main jupyter-pyroot container.)

However, any changes you make to the Docker container won't be automatically saved when you quit the container. When you next start the container, it will start "fresh". If you want to save your changes, you'll have to [[https://docs.docker.com/engine/reference/commandline/commit/][commit]] them. 

For example, assume that you've made some changes to your copy of the jupyter-pyroot container. Look up the ID of the container as assigned by your local docker process:
<verbatim>
sudo docker container ls
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
1105371318e8        wgseligman/jupyter-pyroot   "jupyter notebook ..."   13 minutes ago      Up 13 minutes       0.0.0.0:7000->8080/tcp   cranky_albattani
</verbatim>

Your output will be different; you'll have different =CONTAINER ID= and =NAMES=. Commit a revised container using your own image name:
<verbatim>
sudo docker commit 1105371318e8 $USER/jupyter-pyroot
</verbatim>

You'll can see your new image with the [[https://docs.docker.com/engine/reference/commandline/images/][docker images]] command. For example, if =$USER= is "jsmith":
<verbatim>
sudo docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
jsmith/jupyter-pyroot               latest              97ca601cbf9c        7 seconds ago       2.66 GB
docker.io/wgseligman/jupyter-pyroot   latest              16c3bbdc8144        17 hours ago        2.66 GB
</verbatim>

From that point forward, you'll probably want to run your new container with your changes:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work jsmith/jupyter-pyroot 
</verbatim>

---+++ Docker container notes

WilliamSeligman prepared the container wgseligman/jupyter-pyroot to be similar to the environment of the [[IPython][notebook server]]; for example, it runs the same version of the OS and of ROOT (as of Sep-2021, that's !CentOS 7 and ROOT 6.24.04). 

A little bit web searching will show there are other ROOT containers available. For example:
<verbatim>
sudo docker run -p 3000:8080 pedwink/pyroot-notebook
</verbatim>

That particular container uses Fedora 28 and ROOT 6.14, and it also offers Python 2 versions of its notebook kernels (wgseligman/jupyter-pyroot only offers Python 3).  

So if you can't find the feature you want in wgseligman/jupyter-pyroot, hunt around a bit. It's probably out there.

---++ Singularity

If you don't have admin access to your local computer, or you simply prefer it, you can use [[https://sylabs.io/guides/3.5/user-guide/][Singularity]] instead. You still need admin access to install Singularity, or a willing sysadmin to do it for you. (Singularity is installed on all the systems in the Nevis [[Linux cluster]].)

To download the container and convert it to Singularity's =.sif= format:
<verbatim>
singularity pull docker://wgseligman/jupyter-pyroot
</verbatim>

After some processing, you'll have the image file =jupyter-pyroot_latest.sif=. Then you can run Singularity on that container:
<verbatim>
singularity run --bind=$PWD:/work jupyter-pyroot_latest.sif
</verbatim>

Note that while you can change the mapping of the =/work= directory within the container (see above), you can't change Jupyter's binding to port 8080. This might be a problem if you're running on a shared computer system and more than one user wants to run this container at the same time. 

---++ The hard way

If all other methods fail, you can embark on the adventure of compiling these packages on your own. You can install [[https://wiki.python.org/moin/BeginnersGuide/Download][Python]], [[https://root.cern.ch/downloading-root][ROOT]], and [[http://jupyter.readthedocs.io/en/latest/install.html][Jupyter]] on your laptop. In fact, Jupyter is meant to be a laptop tool; the container installations described above are to save you time. If you want to try your own installation:

   * These are _not_ applications that you can just double-click to install. The process requires some knowledge of the command shell. 
   * You'll need to read the documentation for the package installations and use some thought and initiative. The links in the previous paragraph point to the installation documentation.
   * The [[https://github.com/wgseligman/docker-jupyter-pyroot/blob/master/Dockerfile][Dockerfile]] used to create wgseligman/jupyter-pyroot may provide a clue to what you can do to create your own installation.
Topic revision: r12 - 2021-09-19 - WilliamSeligman
Main