(r8) JupyterROOTContainers < Main

Main Web>Computing>IPython>JupyterROOTContainers (revision 8) (raw view)~~EditAttach~~
---+!! Jupyter/ROOT Containers

<div style="float: right; background-color: #ebeef0; margin: 0 0 20px 20px; padding: 0 10px 0 10px;">
%TOC{depth="3" title="On this page:"}%
</div>

Users of the Nevis particle-physics [[Linux cluster]] have access to a [[IPython][notebook server]] that lets them execute Python and ROOT C++ scripts on a web browser. This page describes alternative methods for installing and using the Jupyter/ROOT/Python/C++ combination with (hopefully) a minimum of installation overhead. For example, this may be helpful for:

   * Nevis researchers who want to be able to run [[https://root.cern.ch/][ROOT]] on their laptops when they don't have internet access (e.g., on an airplane trip);

   * Folks who'd like to go through the [[https://www.nevis.columbia.edu/~seligman/root-class/][ROOT tutorial]] but who don't have computer accounts with a Nevis particle-physics group. 

---++ Docker

[[https://www.docker.com/][Docker]] is probably the best method of running Jupyter+pyroot with a minimum of fuss. Its disadvantage is that it requires administrative access to the host computer system (e.g., your laptop), both to install Docker and to run the Docker container. 

The first step is to install Docker. For Mac and Windows systems, use [[https://hub.docker.com/?overlay=onboarding][Docker Desktop]]; there's a different procedure needed for [[https://runnable.com/docker/install-docker-on-linux][Linux systems]]. 

Once Docker is installed and running, you'll be able to download and run a Docker container:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

(Windows users will probably need to use =%CD%= instead of =$PWD=.)

The first time you run this command, it will download a ~2.5GB container. Give it time. 

Finally you'll see some output. Look at that output carefully, as it will tell you how to access Jupyter via a web browser. For example, assume the output contains something like this:
<verbatim>
    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
    Or copy and paste one of these URLs:
        http://649d0c4b4dc1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
     or http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

Then start up a web browser and visit =http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c=. You'll see the standard Jupyter home page. 

---+++ Changing the port

Consider the command:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

That first =8080= is the port to use on your local computer. If you want to use a different port on your computer (for example, you're already using port 8080 for something else), change that first =8080= to a different port. Note that if you change the port, you'll also have to change the port in the URL in the output; e.g.,
<verbatim>
sudo docker run -p 7000:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

means you'll have to change:
<verbatim>
http://127.0.0.1:8080/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

to:
<verbatim>
http://127.0.0.1:7000/?token=97d7242fc79734f1429bc425c627ccc4f586675c01ecdd9c
</verbatim>

---+++ Changing the directory

Again, consider:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work wgseligman/jupyter-pyroot
</verbatim>

That =$PWD= (=%CD%= in WIndows) just means "the current directory." The execution environment within the container uses =/work= for its files; the =-v= option in the command means "map =/work= to the current directory in the terminal." If you'd like to use a different directory on your computer as the work directory for the Docker container, just substitute that directory for =$PWD=. For example:
<verbatim>
sudo docker run -p 8080:8080 -v ~jsmith/root-class:/work wgseligman/jupyter-pyroot
</verbatim>

---+++ Changing the container

You can use =New -> Terminal= within Jupyter to get a shell. Within that shell, you can modify anything within the container you want; for example, you can use [[https://pip.pypa.io/en/stable/user_guide/][pip3]] to install new Python packages or [[http://yum.baseurl.org/wiki/YumCommands.html][yum]] to install new Linux packages. (If you install something that might be of general interest, let WilliamSeligman know so he can add it to the main jupyter-pyroot container.)

However, any changes you make to the Docker container won't be automatically saved when you quit the container. When you next start the container, it will start "fresh". If you want to save your changes, you'll have to [[https://docs.docker.com/engine/reference/commandline/commit/][commit]] them. 

For example, assume that you've made some changes to your copy of the jupyter-pyroot container. Look up the ID of the container as assigned by your local docker process:
<verbatim>
sudo docker container ls
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
1105371318e8        wgseligman/jupyter-pyroot   "jupyter notebook ..."   13 minutes ago      Up 13 minutes       0.0.0.0:7000->8080/tcp   cranky_albattani
</verbatim>

Your output will be different; you'll have different =CONTAINER ID= and =NAMES=. Commit a revised container using your own image name:
<verbatim>
sudo docker commit 1105371318e8 $USER/jupyter-pyroot
</verbatim>

You'll can see your new image with the [[https://docs.docker.com/engine/reference/commandline/images/][docker images]] command. For example, if =$USER= is "jsmith":
<verbatim>
sudo docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
jsmith/jupyter-pyroot               latest              97ca601cbf9c        7 seconds ago       2.66 GB
docker.io/wgseligman/jupyter-pyroot   latest              16c3bbdc8144        17 hours ago        2.66 GB
</verbatim>

From that point forward, you'll probably want to run your new container with your changes:
<verbatim>
sudo docker run -p 8080:8080 -v $PWD:/work jsmith/jupyter-pyroot 
</verbatim>

---+++ Docker container notes

WilliamSeligman prepared the container wgseligman/jupyter-pyroot to be similar to the environment of the [[IPython][notebook server]]; for example, it runs the same version of the OS and of ROOT (as of Aug-2020, that's !CentOS 7 and ROOT 6.22.00). 

A little bit web searching will show there are other ROOT containers available. For example:
<verbatim>
sudo docker run -p 3000:8080 pedwink/pyroot-notebook
</verbatim>

That particular container uses Fedora 28 and ROOT 6.14, and it also offers Python 2 versions of its notebook kernels (wgseligman/jupyter-pyroot only offers Python 3).  

So if you can't find the feature you want in wgseligman/jupyter-pyroot, hunt around a bit. It's probably out there.

---++ Singularity

If you don't have admin access to your local computer, or you simply prefer it, you can use [[https://sylabs.io/guides/3.5/user-guide/][Singularity]] instead. You still need admin access to install Singularity, or a willing sysadmin to do it for you. (Singularity is installed on all the systems in the Nevis [[Linux cluster]].)

To download the container and convert it to Singularity's =.sif= format:
<verbatim>
singularity pull docker://wgseligman/jupyter-pyroot
</verbatim>

After some processing, you'll have the image file =jupyter-pyroot_latest.sif=. Then you can run Singularity on that container:
<verbatim>
singularity run --bind=$PWD:/work jupyter-pyroot_latest.sif
</verbatim>

Note that while you can change the mapping of the =/work= directory within the container (see above), you can't change Jupyter's binding to port 8080. This might be a problem if you're running on a shared computer system and more than one user wants to run this container at the same time. 

---++ Anaconda

Docker and Singularity are OS-level containers (in contrast to emulators like VMware, which are machine-level containers). [[https://www.anaconda.com/distribution/][Anaconda]] is an environment-level container; it doesn't change the operating system, but it allows you download and execute packages in your home directory.

If you use the Nevis Linux cluster, then you should consider using [[environment modules]] over Anaconda. But if you're on a different system, or the Nevis environment modules don't offer the package or version you're looking for, Anaconda is a better choice. All the Nevis Linux cluster systems running !CentOS 7 have conda installed. You can also separately [[https://docs.anaconda.com/anaconda/install/][install Anaconda in your home directory]] if you wish. 

To install Jupyter/ROOT via conda, this command is supposed to work:
<verbatim>
conda create --name jupyter-pyroot jupyter python root
</verbatim>

However, I've found this method to be unreliable. Typically there's no problem with jupyter or python, but installing ROOT via Anaconda is hit-or-miss.

Assuming you succeed, you can run the =jupyter= command as described near the bottom of the [[IPython][notebook server]] page; e.g.,:
<verbatim>
jupyter notebook
</verbatim>
or to enable remote access (again, read all the instructions at the bottom of the  [[IPython][Jupyter]] page):
<verbatim>
jupyter notebook --no-browser --port=XXXX
</verbatim>

---++ The hard way

If all other methods fail, you can embark on the adventure of compiling these packages on your own. You can install [[https://wiki.python.org/moin/BeginnersGuide/Download][Python]], [[https://root.cern.ch/downloading-root][ROOT]], and [[http://jupyter.readthedocs.io/en/latest/install.html][Jupyter]] on your laptop. In fact, Jupyter is meant to be a laptop tool; the container installations described above are to save you time. If you want to try your own installation:

   * These are _not_ applications that you can just double-click to install. The process requires some knowledge of the command shell. 
   * You'll need to read the documentation for the package installations and use some thought and initiative. The links in the previous paragraph point to the installation documentation.
   * The [[https://github.com/wgseligman/docker-jupyter-pyroot/blob/master/Dockerfile][Dockerfile]] used to create wgseligman/jupyter-pyroot may provide a clue to what you can do to create your own installation.
Topic revision: r8 - 2021-05-26 - WilliamSeligman
Main
Webs
ATLAS
DOE
Main
TWiki
Veritas