Difference: ArCondNevis (6 vs. 7)

Revision 72012-05-18 - GustaafBrooijmans

Line: 1 to 1
 
META TOPICPARENT name="UsingNevisT3"

Using ArCond at Nevis

Line: 6 to 6
  There are of course some drawbacks. The first of which is that you can't monitor how the job is progressing. You can only wait for the job to finish and check the log files after the fact. Secondly, you can only run over the entire dataset (modulo telling your submission scripts to only run a certain number of events, per job). Also, if the datasets you're running on are not very large (or the jobs are not very cpu intensive), then the time it takes to set up the submission scripts (which of course you only have to do once and then you can re-use them as often as you need), to submit the jobs, for condor to copy the packages/output to and from the worker nodes, and for you to combine the output (by hand) will probably be longer than the time it takes to just run the job locally. Of course, you can set up scripts to do all of the above to make things easier/quicker.
Changed:
<
<
In my experience, a job that normally takes > 1 hour or so to run locally is worth submitting to ArCond (will take ~10 minutes to finish on ArCond). Also, if you are submitting many jobs over many different datasets, writing scripts to submit these all to ArCond rather than running them all sequentially will probably be much faster since you have 48 cores on the worker nodes vs. 16 interactively (that are more consistently in use by others).
>
>
In my experience, a job that normally takes > 1 hour or so to run locally is worth submitting to ArCond (will take ~10 minutes to finish on ArCond). Also, if you are submitting many jobs over many different datasets, writing scripts to submit these all to ArCond rather than running them all sequentially will probably be much faster since you have many many cores on the worker nodes vs. 16 interactively (that are more consistently in use by others).
 

The Tutorial

Line: 44 to 44
 

Tutorial Package

Changed:
<
<
Check out the ArCondNevis package. This directory has the code to run the jobs (Analysis package), plus the submission scripts (in arc_d3pd). You will probably want to do this on your xenia user directory. Condor copies the output of your jobs to wherever you submit from. If the output files are large and you do this from a karthur or kolya directory (i.e. where most of our home directories are mounted), bad things could happen when the output is copied over. Also, when you cd to your xenia user directory, make sure you go to the full NFS path /a/data/xenia/users/username/ rather than just /data/users/username/ so that the jobs know where the submission directory is on NFS. Otherwise, they won't know where to copy the output. Then, check out the package:
>
>
Check out the ArCondNevis package. This directory has the code to run the jobs (Analysis package), plus the submission scripts (in arc_d3pd). You will probably want to do this on your xenia user directory. Condor copies the output of your jobs to wherever you submit from. If the output files are large and you do this from a karthur or kolya directory (i.e. where most of our home directories are mounted), bad things could happen when the output is copied over. Also, when you cd to your xenia user directory, make sure you DO NOT go to the full NFS path /a/data/xenia/users/username/ but rather /data/users/username/. You need to use the condor built-in file transfer mechanism, documented here or there's a chance you will bring the system to its knees. Then, check out the package:
 
export SVNAP=svn+ssh://svn.cern.ch/reps/apenson
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback