Difference: 2022PabloDrakeResearchJournal (6 vs. 7)

Revision 72022-08-12 - PabloDrake

Line: 1 to 1
 
META TOPICPARENT name="2022Journals"

Pablo Drake Research Journal (Summer '22)

06/06-06/10

Line: 26 to 26
 06/10
  • Started writing a log/manual fermipy document, noting down the different errors that appear.
  • Found out the error was caused when simultaneously calling through my .myprofile doc VEGAS and fermipy code, possibly because of a conflict between subpackages. Another error appeared:
Changed:
<
<
Traceback (most recent call last):
>
>
Traceback (most recent call last):
 
Changed:
<
<
File "./run_analysis.py", line 348, in <module> free_sources=pipeline_config.get('free_sources'))
>
>
File "./run_analysis.py", line 348, in <module> free_sources=pipeline_config.get('free_sources'))
 
Changed:
<
<
File "./run_analysis.py", line 50, in run_analysis gta.setup()
>
>
File "./run_analysis.py", line 50, in run_analysis gta.setup()
 
Changed:
<
<
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 1082, in setup c.setup(overwrite=overwrite)
>
>
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 1082, in setup c.setup(overwrite=overwrite)
 
Changed:
<
<
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 5121, in setup self._ltc = LTCube.create(self.files['ltcube'])
>
>
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 5121, in setup self._ltc = LTCube.create(self.files['ltcube'])
 
Changed:
<
<
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/ltcube.py", line 192, in create ltc = cls.create_from_fits(files[0])
>
>
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/ltcube.py", line 192, in create ltc = cls.create_from_fits(files[0])
 
Changed:
<
<
IndexError: list index out of range
>
>
IndexError: list index out of range
 
  • Read EGRET (GeV) Blazars by Prof. Mukherjee. Wrote a set of reference notes.

06/13-06/17

Line: 52 to 52
  06/15
  • Started working on creating a full dataset lightcurve. The following error appeared:
Changed:
<
<
Traceback (most recent call last):
>
>
Traceback (most recent call last):
 
Changed:
<
<
File "/nevis/milne/files/pd2629/fermi_analysis/run_analysis_std.npy", line 349, in <module>
>
>
File "/nevis/milne/files/pd2629/fermi_analysis/run_analysis_std.npy", line 349, in <module>
 
Changed:
<
<
run_lightcurve(fermipy_config, prefix, num_sections, section)
>
>
run_lightcurve(fermipy_config, prefix, num_sections, section)
 
Changed:
<
<
File "/nevis/milne/files/pd2629/fermi_analysis/run_analysis_std.npy", line 253, in run_lightcurve
>
>
File "/nevis/milne/files/pd2629/fermi_analysis/run_analysis_std.npy", line 253, in run_lightcurve
 
Changed:
<
<
gta.load_roi('{}.npy'.format(prefix))
>
>
gta.load_roi('{}.npy'.format(prefix))
 
Changed:
<
<
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 3600, in load_roi roi_file,
>
>
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 3600, in load_roi roi_file,
 
Changed:
<
<
roi_data = utils.load_data(infile, workdir=self.workdir)
>
>
roi_data = utils.load_data(infile, workdir=self.workdir)
 
Changed:
<
<
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/utils.py", line 79, in load_data raise
>
>
File "/a/share/ged/src/miniconda3/envs/fermipy-1.0.1/lib/python3.7/site-packages/fermipy/utils.py", line 79, in load_data raise
 
Changed:
<
<
Exception('Input file does not exist.')
>
>
Exception('Input file does not exist.')
 
Changed:
<
<
Exception: Input file does not exist.
>
>
Exception: Input file does not exist.
 
  • Found out that the error described by this code had to do with a naming mismatch in Ari Brill's pipeline code. In particular, the input file for the lightcurve analysis had to be refered to as _initial.npy, as this was the output of the baseline analysis.
06/16
  • Regarding the full dataset lightcurve, the following error appeared:
Changed:
<
<
Traceback (most recent call last):
>
>
Traceback (most recent call last):
  File "/nevis/milne/files/pd2629/fer mi_analysis/run_analysis_std. py", line 350, in <module>
Changed:
<
<
run_lightcurve(fermipy_config, prefix, num_sections, section)
>
>
run_lightcurve(fermipy_config, prefix, num_sections, section)
  File "/nevis/milne/files/pd2629/fer mi_analysis/run_analysis_std. py", line 261, in run_lightcurve
Changed:
<
<
gta.lightcurve(fermipy_config[ 'selection']['target'], make_plots=True)
>
>
gta.lightcurve(fermipy_config[ 'selection']['target'], make_plots=True)
  File "/a/share/ged/src/miniconda3/e nvs/fermipy-1.0.1/lib/python3. 7/site-packages/fermipy/ lightcurve.py", line 289, in lightcurve
Changed:
<
<
o = self._make_lc(name, **config)
>
>
o = self._make_lc(name, **config)
  File "/a/share/ged/src/miniconda3/e nvs/fermipy-1.0.1/lib/python3. 7/site-packages/fermipy/ lightcurve.py", line 452, in _make_lc
Changed:
<
<
if not next_fit['fit_success']:
>
>
if not next_fit['fit_success']:
 
Changed:
<
<
KeyError: 'fit_success'
>
>
KeyError: 'fit_success'
 
  • This error happened during the computation of the first time bin in the lightcurve analysis, and so I tried changing the conditions of that bin regarding both TS requirements and length, to no avail. The error persisted.
06/17
Changed:
<
<
  • As I was not being able to carry out the whole lightcurve analysis,
>
>
 

06/21-06/24

Added:
>
>
06/22 06/23
  • However, another issue started popping up. The lightcurve analysis actually started, with the gtcube process finishing. However, when trying to fit the different time bins, the following sequence would happen (as can be checked in the log file):
Analysis failed in time range 675480000 675566400

<class 'TypeError'>

Analysis failed in time range 675566400 675652800

<class 'TypeError'>

Analysis failed in time range 675652800 675739200

<class 'TypeError'>

Analysis failed in time range 675739200 675825600

<class 'TypeError'>

Analysis failed in time range 675825600 675912000

<class 'TypeError'>

  • As a matter of fact, a fits file would result from this analysis, but it wouldn't include any datapoints.
06/24
  • Contacted Dr. Qi Feng about
 

06/27-07/01

07/05-07/08

Line: 109 to 136
 
  • I had a Zoom chat with Janeth Valverde to consult on the lightcurve problem. Janeth pointed out that the problem might have to do with some of fermitools not running properly for a source this faint. Instead of using Ari's pipeline, she recommended doing a general check on that by running fermitools directly on the command line. She also recommended focusing solely in the last 6 months, and doing an unbinned analysis.
  • Janeth also discussed trying out an adaptive binning strategy (in parallel to the other analyses). She shared this example as a demonstration of the usefulness of adaptive binning.
07/06
Changed:
<
<
  • Contacted Ari Brill regarding the problem. He asked the following questions, as a sanity check:

    "1) Did the baseline analysis using the entire dataset converge successfully? Were you able to look at the validation plots and see that the results seemed reasonable?

    2) To follow up on what Qi was saying, if you look at the monthly lightcurve, you can see that there are years-long stretches in between the flares when the source isn't detected at all. So there are parts of the lightcurve that might not be detected even with a 12-week time bin. That shouldn't cause the fit to fail though. It should converge and report an upper limit.

    3) Are you able to see the error message that's printed out? Just to make sure, do you have enough space in your output directory to store all of the output files?"

>
>
  • Contacted Ari Brill regarding the problem. He asked the following questions, as a sanity check:

    "1) Did the baseline analysis using the entire dataset converge successfully? Were you able to look at the validation plots and see that the results seemed reasonable?

    2) To follow up on what Qi was saying, if you look at the <a data-saferedirecturl="https://www.google.com/url?q=https://fermi.gsfc.nasa.gov/ssc/data/access/lat/LightCurveRepository/source.html?source_name%3D4FGL_J1350.8%2B3033&source=gmail&ust=1660253301149000&usg=AOvVaw2tgbwULgc8vg8pR_R3f9eF" href="https://fermi.gsfc.nasa.gov/ssc/data/access/lat/LightCurveRepository/source.html?source_name=4FGL_J1350.8+3033" target="_blank">monthly lightcurve</a>, you can see that there are years-long stretches in between the flares when the source isn't detected at all. So there are parts of the lightcurve that might not be detected even with a 12-week time bin. That shouldn't cause the fit to fail though. It should converge and report an upper limit.

    3) Are you able to see the error message that's printed out? Just to make sure, do you have enough space in your output directory to store all of the output files?"

 

-- Massimo Capasso - 2022-06-16

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback