Fermi Analysis with FermiPy


Installation and Setup

The following instructions will set up an installation of the Fermi Science Tools (fermitools) and Fermipy using conda. Conda is a package manager that will automatically manage all of the dependencies needed to use the software and encapsulate them into an environment, which must be activated before use.

First, set up your environment for using the Fermipy pipeline by running the following commands:

cat /a/home/tehanu/brill/fermipipe/bash_setup.txt >> ~/.myprofile
source ~/.bashrc

In addition to initializing the conda environment for the fermi code for your account, this will provide you with several definitions and commands (you can see them by opening with a text editor the file .myprofile in your home directory).

FERMI_ANALYSIS_DIR=/a/data/tehanu/$USER/fermi_analysis: directory where your analysis will occur by default

fermisetup: set up your environment before running analyses

fermianalysis="nohup python $FERMIPIPE/run_analysis.py $FERMI_ANALYSIS_DIR/pipeline_config.yml &": run an analysis

NOTE: If your home directory is on Milne or another Nevis machine other than tehanu, see the special instructions "Customizing the Installation" below. If you're unsure where your home directory is, type echo $HOME.

Next, create a directory for running your analysis, and copy the configuration file templates there:

mkdir $FERMI_ANALYSIS_DIR
cp $FERMIPIPE/config.yml $FERMI_ANALYSIS_DIR
cp $FERMIPIPE/pipeline_config.yml $FERMI_ANALYSIS_DIR

Customizing the Installation

If you'd like to perform your analysis somewhere besides the tehanu data partition, change the FERMI_ANALYSIS_DIR to your preferred directory in your .myprofile file. For example, to perform the analysis in your home directory (recommended if your account is on milne), write:

export FERMI_ANALYSIS_DIR=$HOME/fermi_analysis

If you need to recreate the conda environment from scratch, use the setup_pipeline script to download and install fermitools and fermipy into an environment named "fermi". By default, the software will be installed to /a/data/tehanu/$USER/miniconda3, where $USER is your username. Run the following command anywhere:

$FERMIPIPE/setup_pipeline.sh

The script should take several minutes to run. Ignore any message saying to run source ~/.bashrc again, the script does that for you automatically.

If you'd like to change the installation path of miniconda, run the setup script using the -p flag to change the path, as follows:

$FERMIPIPE/setup_pipeline.sh -p <CONDA_PATH>

For example, to install to your home directory (recommended for milne), use -p $HOME/miniconda3. You can also use this option to create the fermi environment in an existing conda installation, if you have one, provided it doesn't already have an environment named fermi.

Configuring Your Analysis

When running an analysis with Fermipy, all parameters are set using a configuration file. Different sections contains the parameters for the data selection, likelihood fit, source model, and so on. There are many configuration parameters you can adjust when running an analysis. In your analysis directory, you should have a file called config.yml, which you can use as a template for your Fermipy configuration file. The settings in it are meant for a standard setup, which you can adjust as needed. The Fermipy documentation lists all possible configuration settings.

To perform an analysis, you will have to set a minimum of three parameters: the start time, the stop time, and the name of the target to analyze (if your target is not a known Fermi source, you can define the target using coordinates instead). The analysis start and stop times must be provided in Fermi mission elapsed time (MET). NASA provides a handy time converter that you can use to convert between a number of calendar/time formats and MET.

The pipeline around Fermipy is designed in a similar way. In your analysis directory, you should have a file called pipeline_config.yml, which you can use as a template for your pipeline configuration file.

fermipy_config: the name of your Fermipy config file

prefix: the name to be used for the output directory (within the fermi_analysis directory) and as a prefix for your output files; shouldn't contain spaces

Running Your Analysis

First, set your environment and activate the conda environment:

fermisetup

Now you can run an analysis using your config files:

fermianalysis

The fermianalysis command runs the analysis in the background, using nohup. This means that it will continue to run even if you close your terminal, and that instead of printing the output to the screen, it saves it to a file called nohup.out.

Useful links

Fermipy documentation

Fermipy Tutorials from Fermi Summer School

NASA mission time converter

Managing conda environments

-- Ari Brill - 2020-02-17

Comments

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2020-02-27 - AriBrill
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback