Introduction
This page is about specific instructions to run distributed analysis from BNL, more detailed info on how to set up analysis code etc. for Nevis is at
RunningAnalysis. You can tar up the area you developed at Nevis and copy it over to BNL. Note that in principle it should be possible to run pathena from Nevis. I haven't tried that yet.
Basic analysis setup at BNL
Most of the info regarding computing at BNL can be accessed from
https://www.racf.bnl.gov/experiments/usatlas
ATLAS software support at BNL is at
http://www.usatlas.bnl.gov/twiki/bin/view/AtlasSoftware/SoftwareSupport
To log in at BNL, you have to go through the gateway, then
rterm -i acas+
It's best to do everything in bash, so
bash
Create a $HOME/cmthome directory, and in it create a requirements file to look like this:
#---------------------------------------------------------------------
set CMTSITE STANDALONE
macro PROJ_RELEASE "latest" \
11.3.0 "11.3.0" \
12.0.6 "12.0.6" \
12.0.7 "12.0.7" \
13.0.10 "13.0.10" \
13.0.20 "13.0.20" \
13.0.25 "13.0.25" \
13.0.30 "13.0.30" \
13.1.0 "13.1.0" \
13.2.0 "13.2.0" \
rel_0 "rel_0" \
rel_1 "rel_1" \
rel_2 "rel_2" \
rel_3 "rel_3" \
rel_4 "rel_4" \
rel_5 "rel_5" \
rel_6 "rel_6"
macro PROJ_BASE_RELEASE "$(PROJ_RELEASE)" \
13.0.25.2 "13.0.25" \
13.0.25.3 "13.0.25" \
13.0.25.4 "13.0.25" \
13.0.25.5 "13.0.25" \
13.0.25.6 "13.0.25" \
13.0.25.7 "13.0.25" \
13.0.25.8 "13.0.25" \
13.0.25.9 "13.0.25"
macro PROJ_SUBDIR "$(PROJ_BASE_RELEASE)" \
bugfix "bugfix/$(PROJ_BASE_RELEASE)" \
dev "dev/$(PROJ_BASE_RELEASE)"
set SITEROOT /opt/usatlas/kit_rel/${PROJ_SUBDIR}
macro ATLAS_DIST_AREA ${SITEROOT}
macro ATLAS_TEST_AREA ${HOME}/workarea/${PROJ_BASE_RELEASE}
apply_tag oneTest
apply_tag setupCMT
apply_tag setup
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
macro setup_slc3compat "" \
gcc323 "/opt/usatlas/kit_rel/SLC3/setup_slc3compat"
setup_script $(setup_slc3compat)
set PATHENA_GRID_SETUP_SH /afs/usatlas.bnl.gov/lcg/current/etc/profile.d/grid_env.sh
#---------------------------------------------------------------------
That last line is needed to run pathena later on.
Then, the first time (and presumably every time you edit that requirements file), do
cd $HOME/cmthome
source /afs/usatlas/cernsw/contrib/CMT/v1r19/mgr/setup.sh
cmt config
The following times, something like
source ~/cmthome/setup.sh -tag=12.0.6,gcc323
is enough.
Setting up for grid usage at BNL requires
source /afs/usatlas.bnl.gov/lcg/current/etc/profile.d/grid_env.sh
grid-proxy-init
To use panda you need to check out and compile the pandatools package where you also have your analysis code.
cmt co PhysicsAnalysis/DistributedAnalysis/PandaTools
cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
cmt config
source setup.sh
make
To use dq2:
source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.BNL
(Note that in csh or tcsh I have often observed "Word too long." errors when issuing these commands and subsequent problems.)
If you want to retrieve files while on the acas nodes, you need to
dq2_get --srmstreams=1 ...
(Sigh)
Now go back to your normal running area and test your code on a local file. If that works (with a reasonably large number of events), comment out the input file and number of events in you joboptions, and instead of
athena myoptions > log
run something like
pathena AnalysisSkeleton_topOptions.py --inDS trig1_misal1_csc11_V1.005014.J5_pythia_jetjet.recon.ESD.v12000604 --outDS user.GustaafBrooijmans.trig1_misal1_csc11_V1.005014.J5_pythia_jetjet.recon.AAN.v12000604
You may need to add something like
--site LYON
if most files are not at BNL.
Then you can go monitor your job at
http://gridui02.usatlas.bnl.gov:25880/server/pandamon/query
--
GustaafBrooijmans - 13 Sep 2007