Introduction

This page is about specific instructions to run distributed analysis from BNL, more detailed info on how to set up analysis code etc. for Nevis is at RunningAnalysis. You can tar up the area you developed at Nevis and copy it over to BNL. Note that in principle it should be possible to run pathena from Nevis. I haven't tried that yet.

Basic analysis setup at BNL

Most of the info regarding computing at BNL can be accessed from https://www.racf.bnl.gov/experiments/usatlas

ATLAS software support at BNL is at http://www.usatlas.bnl.gov/twiki/bin/view/AtlasSoftware/SoftwareSupport

To log in at BNL, you have to go through the gateway, then rterm -i acas+

It's best to do everything in bash, so bash

Create a $HOME/cmthome directory, and in it create a requirements file to look like this:

#---------------------------------------------------------------------
set CMTSITE STANDALONE
macro PROJ_RELEASE   "latest" \
         11.3.0         "11.3.0" \
         12.0.6         "12.0.6" \
         12.0.7         "12.0.7" \
         13.0.10        "13.0.10" \  
         13.0.20        "13.0.20" \  
         13.0.25        "13.0.25" \  
         13.0.30        "13.0.30" \  
         13.1.0         "13.1.0" \
         13.2.0         "13.2.0" \
         rel_0          "rel_0" \
         rel_1          "rel_1" \
         rel_2          "rel_2" \
         rel_3          "rel_3" \
         rel_4          "rel_4" \
         rel_5          "rel_5" \
         rel_6          "rel_6"

macro PROJ_BASE_RELEASE   "$(PROJ_RELEASE)" \
         13.0.25.2      "13.0.25" \
         13.0.25.3      "13.0.25" \
         13.0.25.4      "13.0.25" \
         13.0.25.5      "13.0.25" \
         13.0.25.6      "13.0.25" \
         13.0.25.7      "13.0.25" \
         13.0.25.8      "13.0.25" \
         13.0.25.9      "13.0.25"

macro PROJ_SUBDIR    "$(PROJ_BASE_RELEASE)" \
         bugfix      "bugfix/$(PROJ_BASE_RELEASE)" \
         dev         "dev/$(PROJ_BASE_RELEASE)"

set SITEROOT /opt/usatlas/kit_rel/${PROJ_SUBDIR}
macro ATLAS_DIST_AREA ${SITEROOT}
macro ATLAS_TEST_AREA ${HOME}/workarea/${PROJ_BASE_RELEASE}
apply_tag oneTest 
apply_tag setupCMT
apply_tag setup
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
macro setup_slc3compat "" \
      gcc323  "/opt/usatlas/kit_rel/SLC3/setup_slc3compat"
setup_script $(setup_slc3compat)
set PATHENA_GRID_SETUP_SH /afs/usatlas.bnl.gov/lcg/current/etc/profile.d/grid_env.sh
#---------------------------------------------------------------------

That last line is needed to run pathena later on.

Then, the first time (and presumably every time you edit that requirements file), do

cd $HOME/cmthome
source /afs/usatlas/cernsw/contrib/CMT/v1r19/mgr/setup.sh
cmt config

The following times, something like

source ~/cmthome/setup.sh -tag=12.0.6,gcc323

Setting up for grid usage at BNL requires

source /afs/usatlas.bnl.gov/lcg/current/etc/profile.d/grid_env.sh
grid-proxy-init

To use panda you need to check out and compile the pandatools package where you also have your analysis code.

cmt co PhysicsAnalysis/DistributedAnalysis/PandaTools
cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
cmt config
source setup.sh
make

To use dq2:

source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.BNL

(Note that in csh or tcsh I have often observed "Word too long." errors when issuing these commands and subsequent problems.)

If you want to retrieve files while on the acas nodes, you need to

dq2_get --srmstreams=1 ...

Now go back to your normal running area and test your code on a local file. If that works (with a reasonably large number of events), comment out the input file and number of events in you joboptions, and instead of

athena myoptions > log

pathena AnalysisSkeleton_topOptions.py --inDS trig1_misal1_csc11_V1.005014.J5_pythia_jetjet.recon.ESD.v12000604 --outDS user.GustaafBrooijmans.trig1_misal1_csc11_V1.005014.J5_pythia_jetjet.recon.AAN.v12000604

Then you can go monitor your job at http://gridui02.usatlas.bnl.gov:25880/server/pandamon/query

-- GustaafBrooijmans - 13 Sep 2007