Difference: AnalysisOfFermiDataWithEnrico (8 vs. 9)

Revision 92017-07-14 - JonathanKatz

Line: 1 to 1
 
META TOPICPARENT name="FermiAnalysis"

Introduction to the Fermi analysis

Line: 13 to 13
  Open a terminal and connect with tehanu
Changed:
<
<		> ssh <youruser>@tehanu.nevis.columbia.edu
>
>
> ssh <youruser>@tehanu.nevis.columbia.edu
  If you don't have a Nevis account, ask your mentor to contact Bill Seligman to create one for you.

When you do get your new account, you need to change the password to something that is unique, longer than 8 charecters, and not your username. To do this, run this command:

Changed:
<
<		> passwd
>
>
> passwd
  Follow the prompts, and remember your password. Password managers like Lastpass can make this easier.

The first time you log in (and only the first time) run this command:

Changed:
<
<		> /a/home/tehanu/santander/hs2017/fermi.sh
>
>
> /a/home/tehanu/santander/hs2017/fermi.sh
  This will add information to your account profile about where to find Enrico and the Fermi Science Tools. It should print out "Done!".
Line: 33 to 33
  First get to your home directory:
Changed:
<
<		> cd
>
>
> cd
  Then open up .bashrc in the text editor of your choice. If you would feel more comfortable with a GUI, try:
Changed:
<
<		> gedit .bashrc
>
>
> gedit .bashrc
  If you can't open a graphical application over ssh, try:
Changed:
<
<		> vim .bashrc
>
>
> vim .bashrc
  You must excersize care here, as this is a rather important file.
Changed:
<
<		To the end of the document, insert the following line: 

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_
>
>		To the end of the document, insert the following line by pressing "I" on the keyboard and pasting the below text with either "CTRL+"SHIFT"+"V" (Windows/Linux) or "CMD"+"V". 

export PATH=/a/home/tehanu/jkatz5671/HEASARC/heasoft-6.19/x86_64-unknown-linux-gnu-libc2.12/bin:/a/data/tehanu/santander/software/condor-utils:/a/data/tehanu/santander/software/utils/anaconda/bin:/a/data/tehanu/santander/veritas/ana/condor:/a/data/tehanu/santander/software/ctools-ctools-00-07-01/build/bin:/a/data/tehanu/santander/software/gammalib-00-08-01/build//bin:/a/data/tehanu/santander/software/cmake/build/bin:/a/home/jkatz5671/ds9:/a/data/tehanu/santander/software/bbftp-client-3.2.0/build/bin:/usr/nevis/root-05.34.26/bin:/usr/nevis/gcc-4.9.2/bin:/a/home/tehanu/jkatz5671/etc/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/a/home/tehanu/jkatz5671/bin:/a/data/tehanu/santander/software/VBF/build/bin:/a/data/tehanu/santander/software/EventDisplay/v470/scripts/VTS/:/a/data/tehanu/santander/software/EventDisplay/v470/bin:/a/data/tehanu/santander/software/protobuf-2.5.0/build/bin:/a/home/tehanu/jkatz5671/etc/fv5.3:/a/home/tehanu/jkatz5671/bin:/a/home/tehanu/jkatz5671/ds9dir/_
  This allows you to launch ds9, a progam that lets you view fits image files, with a simple command.
Added:
>
>		Save and quit vim by hitting "ESC" to leave edit mode, and follow it with 

:wq

The colon means that you will give vim a command. The w is a way to "write" the data, ie. save. The q quits.'

Now repeat the process with .screenrc. 

> vim .screenrc

Paste into the file the following code: 

# GNU Screen - main configuration file 
# # All other .screenrc files will source this file to inherit settings.
# # Author: Christian Wills - cwills.sys@gmail.com
#
# # Allow bold colors - necessary for some reason
attrcolor b ".I"
#
# # Tell screen how to set colors. AB = background, AF=foreground
termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
#
# # Enables use of shift-PgUp and shift-PgDn
termcapinfo xterm|xterms|xs|rxvt ti@:te@
#
# # Erase background with current bg color
defbce "on"
#
# # Enable 256 color term
term xterm-256color
#
# # Cache 30000 lines for scroll back
defscrollback 30000
#
# # New mail notification
# backtick 101 30 15 $HOME/bin/mailstatus.sh
#
hardstatus alwayslastline 
# # Very nice tabbed colored hardstatus line
hardstatus string '%{= Kd} %{= Kd}%-w%{= Kr}[%{= KW}%n %t%{= Kr}]%{= Kd}%+w %-= %{KG} %H%{KW}|%{KY}%101`%{KW}|%D %M %d %Y%{= Kc} %C%A%{-}'
#
# # change command character from ctrl-a to ctrl-b (emacs users may want this)
# #escape ^Bb
#
# # Hide hardstatus: ctrl-a f 
bind f eval "hardstatus ignore"
# # Show hardstatus: ctrl-a F
bind F eval "hardstatus alwayslastline"
# Fermi Science Tools
export HEADAS=$SOFTWARE/heasoft-6.16/BUILD_DIR
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r33p0-fssc-20140520-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12
#export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v9r32p5-fssc-20130916-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export FERMI_DIR=$SOFTWARE/fermi/ScienceTools-v10r0p5-fssc-20150518-x86_64-unknown-linux-gnu-libc2.12/x86_64-unknown-linux-gnu-libc2.12/
export ENRICO_DIR=$SOFTWARE/fermi/enrico
alias fermisetup="source $FERMI_DIR/fermi-init.sh; source $ENRICO_DIR/enrico-init.sh"
export FERMI_DATA="/scratch/errando/allsky-fermi/data"
export FERMI_EVFILE="$FERMI_DATA/evlist"
export FERMI_SCFILE="$FERMI_DATA/spacecraft.fits"
alias ana="cd $VERITAS_EVNDISP_ANA_DIR"
export PATH=$PATH:$HOME/bin
export PYTHONPATH=$PYTHONPATH:$SOFTWARE/fermi/python
alias cta="cd /a/home/tehanu/santander/CTA/CameraServer/code"
 Once this step is completed, log out of tehanu/milne by typing "exit" and hitting return.

Log back into tehanu/milne using the ssh command above and run this command, you'll have to run this command every time you want to start working with the Fermi analysis :

Changed:
<
<		> fermisetup
>
>
> fermisetup
  This command should print out this:
Line: 81 to 146
  If you already have your own data directory, navigate there and move on. Otherwise, you'll need to make one.
Changed:
<
<		> mkdir <your username>
>
>
> mkdir <your username>
  Now navigate there, and make another directory inside that will be for your analysis. This will help keep things organized later on. Avoid special characters or spaces. This is a good rule to follow always, it'll prevent headaches if you can avoid letting them get into any file names.
Changed:
<
<		> mkdir PG1553113data (or something similarly descriptive)
>
>
> mkdir PG1553113data (or something similarly descriptive)
  Run fermisetup in the directory you want to use for your analysis
Changed:
<
<		> fermisetup
>
>
> fermisetup
  Create a configuration file with enrico called myanalysis.conf by running the enrico_config command:
Changed:
<
<		> enrico_config myanalysis.conf
>
>
> enrico_config myanalysis.conf
  It will ask for different paramaters associated with the source we want to analyze. The most important ones are the position of the source (Right Ascension and Declination), the energy range of the gamma rays we want to consider in our analysis (Emin, Emax), the times we want to consider (Tmin, Tmax), and the radius of the region in the sky to consider around the source (which we call the Region of Interest, or ROI). For our analysis, we'll get the values from the section of the Fermi tutorial called "Get the data". Namely:
  • Search Center (RA,Dec) = (238.929,11.1901)
Line: 104 to 169
 
  • Maximum Energy = 300000 MeV
All other values (output directories, tag, spectral model, redshift, etc) can be left set to the default values (which enrico shows in square brackets) by just pressing enter when asked. This is how it would look when running enrico_config (notice that I didn't enter a value for those fields where the default was OK):
Changed:
<
<		> enrico_config myanalysis.conf

[Message]: Please provide the following required options [default] :
Output directory [/a/home/tehanu/santander/noimagination] :
Target Name : PG1553
Right Ascension: 238.929
Declination: 11.1901
redshift, no effect if null [0] :
Options are : PowerLaw, PowerLaw2, LogParabola, PLExpCutoff, Generic
Generic is design to allow the user to fit with non-supported models
EBL absorption can be added for PowerLaw2, LogParabola, PLExpCutoff
Spectral Model [PowerLaw] :
ROI Size [15] : 20
FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :
FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :
tag [LAT_Analysis] :
Start time [239557418] : 239557417
End time [334165418] : 256970880
Emin [100] :
Emax [300000] :
IRFs [CALDB] :
evclass [128] :
evtype [3] :
Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])
Is this ok? [y] :
Corresponding zmax = 90

>
>
> enrico_config myanalysis.conf
[Message]: Please provide the following required options [default] :
Output directory [/a/home/tehanu/santander/noimagination] :
Target Name : PG1553
Right Ascension: 238.929
Declination: 11.1901
redshift, no effect if null [0] :
Options are : [[PowerLaw][PowerLaw]], [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]], Generic
Generic is design to allow the user to fit with non-supported models
EBL absorption can be added for [[PowerLaw2][PowerLaw2]], [[LogParabola][LogParabola]], [[PLExpCutoff][PLExpCutoff]] 
Spectral Model [PowerLaw] :
ROI Size [15] : 20
FT2 file [/a/data/tehanu/santander/software/fermi/enrico/Data/download/lat_spacecraft_merged.fits] :
FT1 list of files [/a/data/tehanu/santander/software/fermi/enrico/Data/download/events.lis] :
tag [LAT_Analysis] :
Start time [239557418] : 239557417
End time [334165418] : 256970880
Emin [100] :
Emax [300000] :
IRFs [CALDB] :
evclass [128] :
evtype [3] :
Corresponding IRFs = ('P8R2_SOURCE_V6', ['BACK', 'FRONT'])
Is this ok? [y] :
Corresponding zmax = 90
  Once you enter all the values, you'll have the myanalysis.conf file in your folder:
Changed:
<
<		> ls myanalysis.conf

myanalysis.conf

>
>
> ls myanalysis.conf
myanalysis.conf
  The next step is to produce and XML file with all the sources in the vicinity of PG 1553:
Changed:
<
<		> enrico_xml myanalysis.conf
>
>
> enrico_xml myanalysis.conf
  This should print out:
Changed:
<
<		use the default location of the catalog
use the default catalog
Use the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit
(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')
[Message]: Summary of the XML model generation
Add 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees
4 sources have free parameters inside 3.0 degrees
0 source(s) is (are) extended
Iso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txt
Galactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits
[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

Running long processes

>
>
use the default location of the catalog
use the default catalog
Use the catalog : /a/data/tehanu/santander/software/fermi/enrico/Data/catalog/gll_psc_v16.fit
(84.002691737453972, 84.002691737453972, '3FGL J0921.8+6215')
[Message]: Summary of the XML model generation
Add 74 sources in the ROI of 22.0 ( 20.0 + 2 ) degrees
4 sources have free parameters inside 3.0 degrees
0 source(s) is (are) extended
Iso model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/iso_P8R2_SOURCE_V6_v06.txt
Galactic model file /a/data/tehanu/santander/software/fermi/enrico/Data/diffuse/gll_iem_v06.fits
[Message]: write the Xml file in /a/home/tehanu/santander/noimagination/PG1553_PowerLaw_model.xml

Running long processes

  The processing of the Fermi data can (and will) take several hours, depending on how much data you're analyzing. You may not want to leave your computer connected to Nevis during this time, but if you close the terminal running the analysis the processing stops. There are two main ways to run the analysis without your computer remainng logged in, screen and nohup. With screen, the server runs not only the commands but also the command line interface, so your computer is not needed to keep the process alive. screen's additional level of abstraction can add minor techincal complication, but allows a persisitent terminal session for running a series of commands. nohup is different. nohup (no-hangup) "daemonizes" the process so that it runs in the background. This way, you run the command and immediately return to a shell prompt. This is probably more straightforward for most analysis tasks.

Here's how to use screen:

Changed:
<
<		> screen
>
>
> screen
  Set up your Fermi variables again
Changed:
<
<		> fermisetup
>
>
> fermisetup
  And now you can continue running the analysis inside the screen.
Line: 140 to 235
  If you want to connect again with the session running the analysis, once in tehanu/milne you need to run:
Changed:
<
<		> screen -r
>
>
> screen -r
  Hit "Ctrl + D" and then "A" to exit.

Here's how to use nohup:

Changed:
<
<		> nohup <command> &
>
>
> nohup <command> &
  This will make an output file called nohup.out. If you want to save your file to a specific name, try this:
Changed:
<
<		> nohup bash -c "<command> > <filename>.out" &
>
>
> nohup bash -c "<command> > <filename>.out" &
  Here's how this works:

nohup <command> runs the command with nohup. Because nohup doesn't need to stay connected, rather than outputting its contents into the terminal window, it outputs into a file. You can tell it what file to output with the > <filename>.out part of the command. The > says "take the output of this and put it into this file." bash -c executes the command in quotes ensuring nohup is being applied to the whole command, not just the enrico_lc portion.

Added:
>
>		 You can also set it up to send you an email when the command is completed.

After the "<command> > <filename>.out, but before the second quote, add the following: 

&& mail -s "<message subject>" <your email> < /dev/null

So the whole command looks like: 

> nohup bash -c "<command> > <filename>.out && mail -s "<message subject>" <your email> < /dev/null" &
 

Running the SED creator

Once the xml file is created, run the SED step of the analysis to calculate the energy spectrum (which is the number of photons detected by Fermi from this source per unit area, unit time and unit energy) of PG 1553 during the time period considered:

Changed:
<
<		> enrico_sed myanalysis.conf
>
>
> enrico_sed myanalysis.conf
  This step should take several hours, so make sure you run it when you don't need to disconnect your computer from the network during that time.

Once this is finished, you should be able to run this command to plot the results from the analysis, which will be located in a folder called "Spectrum":

Changed:
<
<		> enrico_plot_sed myanalysis.conf
>
>
> enrico_plot_sed myanalysis.conf
  To see the images produced by the analysis run this command:
Changed:
<
<		> display Spectrum/SED_PG1553_PowerLaw.png
>
>
> display Spectrum/SED_PG1553_PowerLaw.png
 

Running the Light Curve creator

We may also want to calculate the "light curve" of a source, namely how the flux (~brightness) of that source in gamma rays changed with time. To do that, we need to run:

Changed:
<
<		> enrico_lc myanalysis.conf
>
>
> enrico_lc myanalysis.conf
  and wait until it finishes. Once it completes, run:
Changed:
<
<		> enrico_plot_lc myanalysis.conf
>
>
> enrico_plot_lc myanalysis.conf
  to see the results. Use the "display" command to look at the .png files created by the light curve step.

Issues?

View topic | History: r11 < r10 < r9 < r8 | More topic actions...
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback