export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

> ssh <youruser>@tehanu.nevis.columbia.edu

> passwd

> /a/home/tehanu/santander/hs2017/fermi.sh

> cd

> gedit .bashrc

> vim .bashrc

:wq

> vim .screenrc

# GNU Screen - main configuration file 
# # All other .screenrc files will source this file to inherit settings.
# # Author: Christian Wills - cwills.sys@gmail.com
#
# # Allow bold colors - necessary for some reason
attrcolor b ".I"
#
# # Tell screen how to set colors. AB = background, AF=foreground
termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
#
# # Enables use of shift-PgUp and shift-PgDn
termcapinfo xterm|xterms|xs|rxvt ti@:te@
#
# # Erase background with current bg color
defbce "on"
#
# # Enable 256 color term
term xterm-256color
#
# # Cache 30000 lines for scroll back
defscrollback 30000
#
# # New mail notification
# backtick 101 30 15 $HOME/bin/mailstatus.sh
#
hardstatus alwayslastline 
# # Very nice tabbed colored hardstatus line
hardstatus string '%{= Kd} %{= Kd}%-w%{= Kr}[%{= KW}%n %t%{= Kr}]%{= Kd}%+w %-= %{KG} {KW}|%{KY}%101`%{KW}|%D %M %d {= Kc} %C%A%{-}'
#
# # change command character from ctrl-a to ctrl-b (emacs users may want this)
# #escape ^Bb
#
# # Hide hardstatus: ctrl-a f 
bind f eval "hardstatus ignore"
# # Show hardstatus: ctrl-a F
bind F eval "hardstatus alwayslastline"
# Fermi Science Tools
export HEADAS=$SOFTWARE/heasoft-6.16/BUILD_DIR
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r33p0-fssc-20140520-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r32p5-fssc-20130916-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v10r0p5-fssc-20150518-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export ENRICO_DIR=$SOFTWARE/fermi/enrico
alias fermisetup="source $FERMI_DIR/fermi-init.sh; source $ENRICO_DIR/enrico-init.sh"
export FERMI_DATA="/scratch/errando/allsky-fermi/data"
export FERMI_EVFILE="$FERMI_DATA/evlist"
export FERMI_SCFILE="$FERMI_DATA/spacecraft.fits"
alias ana="cd $VERITAS_EVNDISP_ANA_DIR"
export PATH=$PATH:$HOME/bin
export PYTHONPATH=$PYTHONPATH:$SOFTWARE/fermi/python
alias cta="cd /a/home/tehanu/santander/CTA/CameraServer/code"

> fermisetup

> mkdir <your username>

> mkdir PG1553113data (or something similarly descriptive)

> fermisetup

> enrico_config myanalysis.conf

> enrico_config myanalysis.conf
[Message]: Please provide the following required options [default] :
Output directory [/a/home/tehanu/santander/noimagination] :
Target Name : PG1553
Right Ascension: 238.929
Declination: 11.1901
redshift, no effect if null [0] :
Options are : [[PowerLaw][PowerLaw]], [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]], Generic
Generic is design to allow the user to fit with non-supported models
EBL absorption can be added for [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]] 
Spectral Model [PowerLaw] :
ROI Size [15] : 20
FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :
FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :
tag [LAT_Analysis] :
Start time [239557418] : 239557417
End time [334165418] : 256970880
Emin [100] :
Emax [300000] :
IRFs [CALDB] :
evclass [128] :
evtype [3] :
Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])
Is this ok? [y] :
Corresponding zmax = 90

> ls myanalysis.conf
myanalysis.conf

> enrico_xml myanalysis.conf

use the default location of the catalog
use the default catalog
Use the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit
(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')
[Message]: Summary of the XML model generation
Add 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees
4 sources have free parameters inside 3.0 degrees
0 source(s) is (are) extended
Iso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txt
Galactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits
[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

> screen

> fermisetup

> screen -r

> nohup <command> &

> nohup bash -c "<command> > <filename>.out" &

&& mail -s "<message subject>" <your email> < /dev/null

> nohup bash -c "<command> > <filename>.out && mail -s "<message subject>" <your email> < /dev/null" &

> enrico_sed myanalysis.conf

> enrico_plot_sed myanalysis.conf

> display Spectrum/SED_PG1553_PowerLaw.png

> enrico_lc myanalysis.conf

> enrico_plot_lc myanalysis.conf

Introduction to the Fermi analysis

Setting up your account

The Fermi data is openly available through the Fermi Science Center here: https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

To analyze the data, we'll use Enrico, a python wrapper for the Fermi Science Tools.

To get started you need to have a computer with ssh installed (to connect to a remote server). This is available by default in Mac OS X and Linux but you may need to download and install some software if you're using a Windows machine.

Open a terminal and connect with tehanu:

> ssh <youruser>@tehanu.nevis.columbia.edu

If you don't have a Nevis account, ask your mentor to contact Bill Seligman to create one for you.

When you do get your new account, you need to change the password to something that is unique, longer than 8 characters, and not your username. To do this, run this command:

> passwd

Follow the prompts, and remember your password. Password managers like Lastpass can make this easier.

The first time you log in (and only the first time) run this command:

> /a/home/tehanu/santander/hs2017/fermi.sh

This will add information to your account profile about where to find Enrico and the Fermi Science Tools. It should print out "Done!".

You will also want to add some certain things to your .bashrc file. The .bashrc is a script that runs each and every time you log in. We're going to add some paths to allow you to more easily certain programs.

First get to your home directory:

> cd

Then open up .bashrc in the text editor of your choice. If you would feel more comfortable with a GUI, try:

> gedit .bashrc

If you can't open a graphical application over ssh, try:

> vim .bashrc

You must excersize care here, as this is a rather important file.

To the end of the document, insert the following line by pressing "I" on the keyboard and pasting the below text with either "CTRL+"SHIFT"+"V" (Windows/Linux) or "CMD"+"V".

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

This allows you to launch ds9, a progam that lets you view fits image files, with a simple command.

Save and quit vim by hitting "ESC" to leave edit mode, and follow it with

:wq

The colon means that you will give vim a command. The w is a way to "write" the data, ie. save. The q quits.'

Now repeat the process with .screenrc.

> vim .screenrc

Paste into the file the following code:

# GNU Screen - main configuration file # # All other .screenrc files will source this file to inherit settings.# # Author: Christian Wills - cwills.sys@gmail.com## # Allow bold colors - necessary for some reasonattrcolor b ".I"## # Tell screen how to set colors. AB = background, AF=foregroundtermcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'## # Enables use of shift-PgUp and shift-PgDntermcapinfo xterm|xterms|xs|rxvt ti@:te@## # Erase background with current bg colordefbce "on"## # Enable 256 color termterm xterm-256color## # Cache 30000 lines for scroll backdefscrollback 30000## # New mail notification# backtick 101 30 15 $HOME/bin/mailstatus.sh#hardstatus alwayslastline # # Very nice tabbed colored hardstatus linehardstatus string '%{= Kd} %{= Kd}%-w%{= Kr}[%{= KW}%n %t%{= Kr}]%{= Kd}%+w %-= %{KG} {KW}|%{KY}%101`%{KW}|%D %M %d {= Kc} %C%A%{-}'## # change command character from ctrl-a to ctrl-b (emacs users may want this)# #escape ^Bb## # Hide hardstatus: ctrl-a f bind f eval "hardstatus ignore"# # Show hardstatus: ctrl-a Fbind F eval "hardstatus alwayslastline"# Fermi Science Toolsexport HEADAS=$SOFTWARE/heasoft-6.16/BUILD_DIR#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r33p0-fssc-20140520-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r32p5-fssc-20130916-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v10r0p5-fssc-20150518-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/export ENRICO_DIR=$SOFTWARE/fermi/enricoalias fermisetup="source $FERMI_DIR/fermi-init.sh; source $ENRICO_DIR/enrico-init.sh"export FERMI_DATA="/scratch/errando/allsky-fermi/data"export FERMI_EVFILE="$FERMI_DATA/evlist"export FERMI_SCFILE="$FERMI_DATA/spacecraft.fits"alias ana="cd $VERITAS_EVNDISP_ANA_DIR"export PATH=$PATH:$HOME/binexport PYTHONPATH=$PYTHONPATH:$SOFTWARE/fermi/pythonalias cta="cd /a/home/tehanu/santander/CTA/CameraServer/code"

Once this step is completed, log out of tehanu/milne by typing "exit" and hitting return.

Log back into tehanu/milne using the ssh command above and run this command, you'll have to run this command every time you want to start working with the Fermi analysis :

> fermisetup

This command should print out this:

Adding Enrico to PATH and PYTHONPATH
Setting environment variables for data file locations
Run enrico_setupcheck to check if you are set up correctly.
Check your PATH and PYTHONPATH if enrico_setup is not found or gives an error.

A primer on how to plot data with python in a jupyter notebook

You can use the Nevis python notebook server here: https://notebook.nevis.columbia.edu/ which you can access using your Nevis account and password. More info on jupyter (previously known as ipython) is available here: https://twiki.nevis.columbia.edu/twiki/bin/view/Main/IPython

An intro to how to create plots with jupyter notebooks is available here: https://www.nevis.columbia.edu/~santander/srmp/Python%20intro.html

An introduction to simple shell commands

The most important commands to know are those getting and looking around.

The directory "tree" of linux

Running your first analysis

For the first analysis we'll follow the tutorial written by Jonathan Katz available here: https://slack-files.com/T0BJXS8V8-F2G8JQR5W-d42135ee57

A useful tutorial to get you aquainted with the command line can be found here: https://www.codecademy.com/en/courses/learn-the-command-line/lessons/environment/exercises/bash-profile.

We'll analyze data for the blazar PG 1553+113 following the tutorial put together by Fermi here: https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/python_tutorial.html

First, navigate to /data/

If you already have your own data directory, navigate there and move on. Otherwise, you'll need to make one.

> mkdir <your username>

Now navigate there, and make another directory inside that will be for your analysis. This will help keep things organized later on. Avoid special characters or spaces. This is a good rule to follow always, it'll prevent headaches if you can avoid letting them get into any file names.

> mkdir PG1553113data (or something similarly descriptive)

Run fermisetup in the directory you want to use for your analysis

> fermisetup

Create a configuration file with enrico called myanalysis.conf by running the enrico_config command:

> enrico_config myanalysis.conf

It will ask for different paramaters associated with the source we want to analyze. The most important ones are the position of the source (Right Ascension and Declination), the energy range of the gamma rays we want to consider in our analysis (Emin, Emax), the times we want to consider (Tmin, Tmax), and the radius of the region in the sky to consider around the source (which we call the Region of Interest, or ROI). For our analysis, we'll get the values from the section of the Fermi tutorial called "Get the data". Namely:

• Search Center (RA,Dec) = (238.929,11.1901)

• Radius = 20 degrees

• Start Time (MET) = 239557417 seconds (2008-08-04T15:43:37)

• Stop Time (MET) = 256970880 seconds (2009-02-22T04:48:00)

• Minimum Energy = 100 MeV

• Maximum Energy = 300000 MeV

> enrico_config myanalysis.conf[Message]: Please provide the following required options [default] :Output directory [/a/home/tehanu/santander/noimagination] :Target Name : PG1553Right Ascension: 238.929Declination: 11.1901redshift, no effect if null [0] :Options are : PowerLaw, PowerLaw2, LogParabola, PLExpCutoff, GenericGeneric is design to allow the user to fit with non-supported modelsEBL absorption can be added for PowerLaw2, LogParabola, PLExpCutoff Spectral Model [PowerLaw] :ROI Size [15] : 20FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :tag [LAT_Analysis] :Start time [239557418] : 239557417End time [334165418] : 256970880Emin [100] :Emax [300000] :IRFs [CALDB] :evclass [128] :evtype [3] :Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])Is this ok? [y] :Corresponding zmax = 90

Once you enter all the values, you'll have the myanalysis.conf file in your folder:

> ls myanalysis.confmyanalysis.conf

The next step is to produce and XML file with all the sources in the vicinity of PG 1553:

> enrico_xml myanalysis.conf

This should print out:

use the default location of the cataloguse the default catalogUse the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')[Message]: Summary of the XML model generationAdd 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees4 sources have free parameters inside 3.0 degrees0 source(s) is (are) extendedIso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txtGalactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

Running long processes

The processing of the Fermi data can (and will) take several hours, depending on how much data you're analyzing. You may not want to leave your computer connected to Nevis during this time, but if you close the terminal running the analysis the processing stops. There are two main ways to run the analysis without your computer remaining logged in, screen and nohup. With screen, the server runs not only the commands but also the command line interface, so your computer is not needed to keep the process alive. screen's additional level of abstraction can add minor technical complication, but allows a persistent terminal session for running a series of commands. nohup is different. nohup (no-hangup) "daemonizes" the process so that it runs in the background. This way, you run the command and immediately return to a shell prompt. This is probably more straightforward for most analysis tasks.

Here's how to use screen:

> screen

Set up your Fermi variables again

> fermisetup

And now you can continue running the analysis inside the screen.

If you want to close your computer and/or disconnect from Nevis while the analysis is still running, you need to first "detach the screen" in which the analysis is running. To do that you need to first press "Ctrl + D" and then "A" in your keyboard. That should bring you back to the original session where you logged into tehanu/milne.

If you want to connect again with the session running the analysis, once in tehanu/milne you need to run:

> screen -r

Hit "Ctrl + D" and then "A" to exit.

Here's how to use nohup:

> nohup <command> &

This will make an output file called nohup.out. If you want to save your file to a specific name, try this:

> nohup bash -c "<command> > <filename>.out" &

Here's how this works:

nohup <command> runs the command with nohup. Because nohup doesn't need to stay connected, rather than outputting its contents into the terminal window, it outputs into a file. You can tell it what file to output with the > <filename>.out part of the command. The > says "take the output of this and put it into this file." bash -c executes the command in quotes ensuring nohup is being applied to the whole command, not just the enrico_lc portion.

You can also set it up to send you an email when the command is completed.

After the "<command> > <filename>.out, but before the second quote, add the following:

&& mail -s "<message subject>" <your email> < /dev/null

So the whole command looks like:

> nohup bash -c "<command> > <filename>.out && mail -s "<message subject>" <your email> < /dev/null" &

Running the SED creator

Once the xml file is created, run the SED step of the analysis to calculate the energy spectrum (which is the number of photons detected by Fermi from this source per unit area, unit time and unit energy) of PG 1553 during the time period considered:

> enrico_sed myanalysis.conf

This step should take several hours, so make sure you run it when you don't need to disconnect your computer from the network during that time.

Once this is finished, you should be able to run this command to plot the results from the analysis, which will be located in a folder called "Spectrum":

> enrico_plot_sed myanalysis.conf

To see the images produced by the analysis run this command:

> display Spectrum/SED_PG1553_PowerLaw.png

Running the Light Curve creator

We may also want to calculate the "light curve" of a source, namely how the flux (~brightness) of that source in gamma rays changed with time. To do that, we need to run:

> enrico_lc myanalysis.conf

and wait until it finishes. Once it completes, run:

> enrico_plot_lc myanalysis.conf

to see the results. Use the "display" command to look at the .png files created by the light curve step.

Issues?

1. Check that your commands are correctly typed.

2. Check to see if any of your filenames contain spaces or special characters.

3. Check to see if any errors were produced during the processing. Pay attention to what's being written to screen during the processing to see if any errors can be quickly fixed. For instance, a "file not found" error could mean that the name of a file was incorrectly entered.

4. Good ol' Google and/or post the question to our Slack channel https://veritas-gamma.slack.com/

Time conversion and source coordinates

*Time conversion tools*

The Fermi analysis uses MET (mission elapsed time) in seconds as the time frame of reference. To convert between standard dates, modified julian days, and MET please use this NASA webpage: https://heasarc.gsfc.nasa.gov/cgi-bin/Tools/xTime/xTime.pl

Source location in RA and Dec

The position of the source to analyze needs to be degrees for both Right Ascension (RA) and Declination (Dec), which are respectively the celestial equivalents of Longitude and Latitude for the Earth. For more information on this coordinate system check the wikipedia entry for the Equatorial Coordinate System: https://en.wikipedia.org/wiki/Equatorial_coordinate_system

There are several coordinate engines on the web but most of them provide the RA in hours, minutes, and seconds instead of degrees so the easiest way to find it is to use the Fermi database interface here: https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi Simply enter the name of your source in the "Object name or coordinates" field of the form and press enter. The position in RA, Dec in degrees should be given in a table in the next page under "Equatorial Coordinates (degrees)"

-- Marcos Santander - 2017-06-23

Edited by -- Jonathan Katz - 2017-07-10

Comments

<--/commentPlugin-->

> ssh <youruser>@tehanu.nevis.columbia.edu

> passwd

> /a/home/tehanu/santander/hs2017/fermi.sh

> cd

> gedit .bashrc

> vim .bashrc

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

:wq

The colon means that you will give vim a command. The w is a way to "write" the data, ie. save. The q quits.'

Now repeat the process with .screenrc.

> vim .screenrc

Paste into the file the following code:

# GNU Screen - main configuration file 
# # All other .screenrc files will source this file to inherit settings.
# # Author: Christian Wills - cwills.sys@gmail.com
#
# # Allow bold colors - necessary for some reason
attrcolor b ".I"
#
# # Tell screen how to set colors. AB = background, AF=foreground
termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
#
# # Enables use of shift-PgUp and shift-PgDn
termcapinfo xterm|xterms|xs|rxvt ti@:te@
#
# # Erase background with current bg color
defbce "on"
#
# # Enable 256 color term
term xterm-256color
#
# # Cache 30000 lines for scroll back
defscrollback 30000
#
# # New mail notification
# backtick 101 30 15 $HOME/bin/mailstatus.sh
#
hardstatus alwayslastline 
# # Very nice tabbed colored hardstatus line
hardstatus string '%{= Kd} %{= Kd}%-w%{= Kr}[%{= KW}%n %t%{= Kr}]%{= Kd}%+w %-= %{KG} %H%{KW}|%{KY}%101`%{KW}|%D %M %d %Y%{= Kc} %C%A%{-}'
#
# # change command character from ctrl-a to ctrl-b (emacs users may want this)
# #escape ^Bb
#
# # Hide hardstatus: ctrl-a f 
bind f eval "hardstatus ignore"
# # Show hardstatus: ctrl-a F
bind F eval "hardstatus alwayslastline"
# Fermi Science Tools
export HEADAS=$SOFTWARE/heasoft-6.16/BUILD_DIR
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r33p0-fssc-20140520-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r32p5-fssc-20130916-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v10r0p5-fssc-20150518-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export ENRICO_DIR=$SOFTWARE/fermi/enrico
alias fermisetup="source $FERMI_DIR/fermi-init.sh; source $ENRICO_DIR/enrico-init.sh"
export FERMI_DATA="/scratch/errando/allsky-fermi/data"
export FERMI_EVFILE="$FERMI_DATA/evlist"
export FERMI_SCFILE="$FERMI_DATA/spacecraft.fits"
alias ana="cd $VERITAS_EVNDISP_ANA_DIR"
export PATH=$PATH:$HOME/bin
export PYTHONPATH=$PYTHONPATH:$SOFTWARE/fermi/python
alias cta="cd /a/home/tehanu/santander/CTA/CameraServer/code"

> fermisetup

> mkdir <your username>

> mkdir PG1553113data (or something similarly descriptive)

> fermisetup

> enrico_config myanalysis.conf

> enrico_config myanalysis.conf
[Message]: Please provide the following required options [default] :
Output directory [/a/home/tehanu/santander/noimagination] :
Target Name : PG1553
Right Ascension: 238.929
Declination: 11.1901
redshift, no effect if null [0] :
Options are : [[PowerLaw][PowerLaw]], [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]], Generic
Generic is design to allow the user to fit with non-supported models
EBL absorption can be added for [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]] 
Spectral Model [PowerLaw] :
ROI Size [15] : 20
FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :
FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :
tag [LAT_Analysis] :
Start time [239557418] : 239557417
End time [334165418] : 256970880
Emin [100] :
Emax [300000] :
IRFs [CALDB] :
evclass [128] :
evtype [3] :
Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])
Is this ok? [y] :
Corresponding zmax = 90

> ls myanalysis.conf
myanalysis.conf

> enrico_xml myanalysis.conf

use the default location of the catalog
use the default catalog
Use the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit
(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')
[Message]: Summary of the XML model generation
Add 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees
4 sources have free parameters inside 3.0 degrees
0 source(s) is (are) extended
Iso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txt
Galactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits
[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

Running long processes

> screen

> fermisetup

> screen -r

> nohup <command> &

> nohup bash -c "<command> > <filename>.out" &

After the "<command> > <filename>.out, but before the second quote, add the following:

&& mail -s "<message subject>" <your email> < /dev/null

So the whole command looks like:

> nohup bash -c "<command> > <filename>.out && mail -s "<message subject>" <your email> < /dev/null" &

> enrico_sed myanalysis.conf

> enrico_plot_sed myanalysis.conf

> display Spectrum/SED_PG1553_PowerLaw.png

> enrico_lc myanalysis.conf

> enrico_plot_lc myanalysis.conf

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

An introduction to simple shell commands

The most important commands to know are those getting and looking around.

The directory "tree" of linux

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_

First get to your home directory:

> cd

Then open up .bashrc in the text editor of your choice. If you would feel more comfortable with a GUI, try:

> gedit .bashrc

If you can't open a graphical application over ssh, try:

> vim .bashrc

You must excersize care here, as this is a rather important file.

To the end of the document, insert the following line:

> export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/

This allows you to launch ds9, a progam that lets you view fits image files, with a simple command.

If you already have your own data directory, navigate there and move on. Otherwise, you'll need to make one.

> mkdir <your username>

Now navigate there, and make another directory inside that will be for your analysis. This will help keep things organized later on. Avoid special characters or spaces. This is a good rule to follow always, it'll prevent headaches if you can avoid letting them get into any file names.

> mkdir PG1553113data (or something similarly descriptive)

Run fermisetup in the directory you want to use for your analysis

> fermisetup

Create a configuration file with enrico called myanalysis.conf by running the enrico_config command:

Here's how this works:

1. Check to see if any of your filenames contain spaces or special characters.

> passwd

Follow the prompts, and remember your password. Password managers like Lastpass can make this easier.

> nohup <command>

This will make an output file called nohup.out. If you want to save your file to a specific name, try this:

> nohup <command> > <filename>

Time conversion and source coordinates

Time conversion tools

The Fermi analysis uses MET (mission elapsed time) in seconds as the time frame of reference. To convert between standard dates, modified julian days, and MET please use this NASA webpage: https://heasarc.gsfc.nasa.gov/cgi-bin/Tools/xTime/xTime.pl

Source location in RA and Dec

The position of the source to analyze needs to be degrees for both Right Ascension (RA) and Declination (Dec), which are respectively the celestial equivalents of Longitude and Latitude for the Earth. For more information on this coordinate system check the wikipedia entry for the Equatorial Coordinate System: https://en.wikipedia.org/wiki/Equatorial_coordinate_system

There are several coordinate engines on the web but most of them provide the RA in hours, minutes, and seconds instead of degrees so the easiest way to find it is to use the Fermi database interface here: https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi Simply enter the name of your source in the "Object name or coordinates" field of the form and press enter. The position in RA, Dec in degrees should be given in a table in the next page under "Equatorial Coordinates (degrees)"

Introduction to the Fermi analysis

Setting up your account

The Fermi data is openly available through the Fermi Science Center here: https://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

To analyze the data, we'll use Enrico, a python wrapper for the Fermi Science Tools.

To get started you need to have a computer with ssh installed (to connect to a remote server). This is available by default in Mac OS X and Linux but you may need to download and install some software if you're using a Windows machine.

Open a terminal and connect with tehanu

> ssh <youruser>@tehanu.nevis.columbia.edu

If you don't have a Nevis account, ask your mentor to contact Bill Seligman to create one for you.

The first time you log in (and only the first time) run this command:

> /a/home/tehanu/santander/hs2017/fermi.sh

This will add information to your account profile about where to find Enrico and the Fermi Science Tools. It should print out "Done!".

Once this step is completed, log out of tehanu/milne by typing "exit" and hitting return.

Log back into tehanu/milne using the ssh command above and run this command, you'll have to run this command every time you want to start working with the Fermi analysis :

> fermisetup

This command should print out this:

Adding Enrico to PATH and PYTHONPATH
Setting environment variables for data file locations
Run enrico_setupcheck to check if you are set up correctly.
Check your PATH and PYTHONPATH if enrico_setup is not found or gives an error.

A primer on how to plot data with python in a jupyter notebook

You can use the Nevis python notebook server here: https://notebook.nevis.columbia.edu/ which you can access using your Nevis account and password. More info on jupyter (previously known as ipython) is available here: https://twiki.nevis.columbia.edu/twiki/bin/view/Main/IPython

An intro to how to create plots with jupyter notebooks is available here: https://www.nevis.columbia.edu/~santander/srmp/Python%20intro.html

Running your first analysis

For the first analysis we'll follow the tutorial written by Jonathan Katz available here: https://slack-files.com/T0BJXS8V8-F2G8JQR5W-d42135ee57

We'll analyze data for the blazar PG 1553+113 following the tutorial put together by Fermi here: https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/python_tutorial.html

First, create a configuration file with enrico called myanalysis.conf by running the enrico_config command:

> enrico_config myanalysis.conf

It will ask for different paramaters associated with the source we want to analyze. The most important ones are the position of the source (Right Ascension and Declination), the energy range of the gamma rays we want to consider in our analysis (Emin, Emax), the times we want to consider (Tmin, Tmax), and the radius of the region in the sky to consider around the source (which we call the Region of Interest, or ROI). For our analysis, we'll get the values from the section of the Fermi tutorial called "Get the data". Namely:

• Search Center (RA,Dec) = (238.929,11.1901)
• Radius = 20 degrees
• Start Time (MET) = 239557417 seconds (2008-08-04T15:43:37)
• Stop Time (MET) = 256970880 seconds (2009-02-22T04:48:00)
• Minimum Energy = 100 MeV
• Maximum Energy = 300000 MeV

> enrico_config myanalysis.conf

[Message]: Please provide the following required options [default] :
Output directory [/a/home/tehanu/santander/noimagination] :
Target Name : PG1553
Right Ascension: 238.929
Declination: 11.1901
redshift, no effect if null [0] :
Options are : PowerLaw, PowerLaw2, LogParabola, PLExpCutoff, Generic
Generic is design to allow the user to fit with non-supported models
EBL absorption can be added for PowerLaw2, LogParabola, PLExpCutoff
Spectral Model [PowerLaw] :
ROI Size [15] : 20
FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :
FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :
tag [LAT_Analysis] :
Start time [239557418] : 239557417
End time [334165418] : 256970880
Emin [100] :
Emax [300000] :
IRFs [CALDB] :
evclass [128] :
evtype [3] :
Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])
Is this ok? [y] :
Corresponding zmax = 90

Once you enter all the values, you'll have the myanalysis.conf file in your folder:

> ls myanalysis.conf

myanalysis.conf

The next step is to produce and XML file with all the sources in the vicinity of PG 1553:

> enrico_xml myanalysis.conf

This should print out:

use the default location of the catalog
use the default catalog
Use the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit
(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')
[Message]: Summary of the XML model generation
Add 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees
4 sources have free parameters inside 3.0 degrees
0 source(s) is (are) extended
Iso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txt
Galactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits
[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

Running in a "screen" terminal

The processing of the Fermi data can (and will) take several hours, depending on how much data you're analyzing. You may not want to leave your computer connected to Nevis during this time, but if you close the terminal running the analysis the processing stops.

Luckily, we can use a "screen" terminal session to run our analysis. This is a terminal that you can connect and disconnect from without crashing what's running in it. Once you log into tehanu, start a screen:

> screen

Set up your Fermi variables again

> fermisetup

And now you can continue running the analysis inside the screen.

If you want to close your computer and/or disconnect from Nevis while the analysis is still running, you need to first "detach the screen" in which the analysis is running. To do that you need to first press "Ctrl + D" and then "A" in your keyboard. That should bring you back to the original session where you logged into tehanu/milne.

If you want to connect again with the session running the analysis, once in tehanu/milne you need to run:

> screen -r

Hit Ctrl + D" and then "A" everytime you want to exit.

Running the SED creator

Once the xml file is created, run the SED step of the analysis to calculate the energy spectrum (which is the number of photons detected by Fermi from this source per unit area, unit time and unit energy) of PG 1553 during the time period considered:

> enrico_sed myanalysis.conf

This step should take several hours, so make sure you run it when you don't need to disconnect your computer from the network during that time.

Once this is finished, you should be able to run this command to plot the results from the analysis, which will be located in a folder called "Spectrum":

> enrico_plot_sed myanalysis.conf

To see the images produced by the analysis run this command:

> display Spectrum/SED_PG1553_PowerLaw.png

Running the Light Curve creator

We may also want to calculate the "light curve" of a source, namely how the flux (~brightness) of that source in gamma rays changed with time. To do that, we need to run:

> enrico_lc myanalysis.conf

and wait until it finishes. Once it completes, run:

> enrico_plot_lc myanalysis.conf

to see the results. Use the "display" command to look at the png files created by the light curve step.

Issues?

1. Check that your commands are correctly typed.
2. Check to see if any errors were produced during the processing. Pay attention to what's being written to screen during the processing to see if any errors can be quickly fixed. For instance, a "file not found" error could mean that the name of a file was incorrectly entered.
3. Good ol' Google and/or post the question to our Slack channel https://veritas-gamma.slack.com/

-- Marcos Santander - 2017-06-23

Comments

<--/commentPlugin-->