Details of running condor at Nevis
If your condor script or program does any of the following, it's a warning sign that your job might crash (or worse, crash the cluster):
Bad idea: Referring explicitly to your home directory, even to read a file. To write a file directly to your home directly from a condor job is an even worse
idea. You probably want to read from a
partition, and write to a
If you change the default directory in the middle of a condor script or program, you'll wreak havoc on condor's standard file-transfer commands, and might have problems with disk sharing
. Stick to the directory that condor assigns you.
If you're clever, you can include lines like these in your script:
# Create a directory relative to the current directory
# ... do stuff ...
# ... perhaps do more stuff in the default directory, then:
rm -rf myTemporaryDirectory
If you can't figure out how the above lines work, then simply don't do it.
setup without initialization
The Nevis setup
command requires initialization. When you login, this initialization is done for you; look at your
if you use tcsh
). You have to explicitly include this line if you're submitting a batch job:
shopt -s expand_aliases # This line is only necessary if you're using bash
if you use tcsh
Submitting batch jobs
The systems on the condor batch cluster have enough RAM for 1GB/processing queue. This means if your job uses more than 1GB of memory, there can be a problem. For example, if your job required 2GB of memory, and a condor batch node had 16 queues, then your 16 jobs will require 32GB of RAM, twice as much as the machine has. The machine will start swapping memory pages continuously, and essentially halt.
To keep this from happening, condor will automatically cancel a job that requires more than 1GB of RAM. Unfortunately, condor has a problem estimating the amount of memory required by a running job: if a program uses threads, it will tend to overestimate; if a program uses shared libraries, it tends to underestimate.
Therefore, if you find that your large simulation program is being "spontaneously" canceled, look at its memory use.
Do you want 10,000 e-mails?
By default, condor will send you an e-mail message as each of your jobs completes. If you've submitted 10,000 jobs, that means 10,000 e-mails. This can clog the mail server, and make your life miserable. Therefore, the following has been made default at Nevis:
Notification = Error
This means that condor will only send you an e-mail if there's an error while running the job. Don't override it!
Use the vanilla environment
Unless you've specifically used the condor_compile
command to compile your programs, you must
submit your jobs in the "vanilla" universe. Any program that uses shared libraries cannot use condor_compile, and this includes most of the physics software at Nevis. Therefore, you are almost certainly required to have the following line in a command script:
universe = vanilla
condor log files
If you want to see the condor daemons' log files for a machine with the name
, look in
. For example:
# ls -blrth /a/data/karthur/condor/log
-rw-r--r-- 1 condor condor 153 2010-04-13 15:07 StarterLog
-rw-r--r-- 1 condor condor 473K 2010-04-13 16:29 SchedLog
-rw-r--r-- 1 root root 591K 2010-04-13 16:29 MasterLog
-rw-r--r-- 1 root root 788K 2010-04-13 17:15 StartLog
-rw-r--r-- 1 root root 562K 2010-04-13 17:25 NegotiatorLog
-rw-r--r-- 1 root root 296K 2010-04-13 17:25 CollectorLog
About the batch cluster
The system responsible for administering batch services on the general cluster is
. Users typically do not log in to this machine directly; you submit and monitor jobs from your local box on the Linux cluster. As far as job submission and execution are concerned, the existence of
may be completely transparent to you.
Condor status and usage
You can see how much of the batch cluster is in use, and by whom:
The cluster which executes a job is determined by the machine on which you issue the
command. For example, if you submit a job from a Neutrino system, it will run on the Neutrino cluster; if you submit a job from
, it runs on the general cluster; if you submit a job from
, it runs on the ATLAS cluster.
The condor system is most efficient when it's handling a large number of small jobs. Long jobs tend to clog up the queues, and prevent others from doing their work.
As of Feb-2010, there is no system that gives some groups or users higher priority than others. However, condor comes with a default scheme
for adjusting user priorities; here are the details
The practical upshots of condor's default priority scheme:
- If you use condor a lot, other users will tend to get higher priority when they submit jobs.
- If your job takes more than an hour to run, there's a chance it will be pre-empted; that chance increases the longer the job runs.
If you use the vanilla environment (see above), as most users at Nevis must, for a job to be "pre-empted" means that it is killed, and must start again from the beginning.
To get an idea of your user resource consumption and how it compares to other users, use these commands:
The larger the number, the lower
your priority in comparison to the other users listed.
What processing power is available
The following commands will show you the machines available to run your jobs, their status, and their resources:
Obviously, some machines are more powerful than others. Before you arbitrarily decide that only the most powerful machines are good enough for your jobs, consider:
- It's true, a machine that's 1/4 as fast will take 4 times as long to execute your jobs. However, the demand for the faster machine may be more than four times as much; it's possible that your job will sit waiting in the queue longer than it would have taken to run on the slower box.
- The CPU cycles on the slower machines are presently being wasted. You might be able to put them to some use.
- If you have a large number of jobs to submit, the slower machine can chug away at a couple of them while the rest are waiting to execute on the faster processors.
The best way to tell Condor that you'd prefer your job to execute on the faster machines is to use the Rank
attribute in your submit file:
Rank = Mips
With all that said, if you want to restrict your job to the faster machines, you can try a statement like the following in your submit file
Requirements = (Mips > 2000)
This would restrict your job to the fastest processors on the cluster.
All the machines on the batch farm are not the same
The batch farm is a heterogeneous collection of machines. If you're having problems with programs crashing on some systems but not on others, please read this page on compiler tools
that can help solve this problem.
"Why isn't my job running on all the machines in the batch farm?"
There may be several reasons:
The heterogenous cluster
Not all machines in the farm are the same; they have different amounts of memory, disk space, and occasionally even installed libraries. Make sure you use the standardized compiler
to compile your programs.
You'll also want to set up the standard Nevis environment explicitly in your jobs. If you look at the example described below, you'll see that the shell scripts all contain command such as:
# If you're using bash:
shopt -s expand_aliases
setup root geant4
Finally, don't forget to set
in your condor submit file.
The job requirements
There may be something explicit or implicit in the resources required to run your job. To pick an unrealistic example, if you job requires
and that shell isn't installed on machine, then it won't execute on the cluster. A more practical example: If you have the following in your job submit file:
Requirements = ( Memory > 1024 )
then your job won't execute if the amount of memory per job queue
is 1024 or less, including those machines with 1023 MB per queue to due rounding in the memory calculation.
If you think your job with ID 4402 should be able to execute on queue
, you can compare what condor thinks are the job's requirements against what the machine offers:
condor_q -long -global 4402
condor_status -long slot1@batch04
can come from using
. If you have a job held with an ID of 44.20:
condor_q -analyze 44.20
As noted elsewhere on this page, we generally use the vanilla universe at Nevis. This means if a job is suspended on a given machine, it can only continue on that particular machine. If that machine is running other jobs, then the suspended job must wait.