Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Disk sharing and the condor batch system | ||||||||
Line: 10 to 10 | ||||||||
You don't want to read this page. You just want to submit your jobs and not think about disk files or where they're located. Unfortunately, you have to think about it. Here's why. | ||||||||
Changed: | ||||||||
< < | Picture this: You submit a condor batch procedure that runs thousands of jobs. Each one of those jobs reads and/or writes directly into a directory on your server, accessed via an automount path like /a/data/tanya . Let's say there are 300 batch queues in the Nevis condor system. That means that 300 jobs are trying to access the disk on your server at once. | |||||||
> > | Picture this: You submit a condor batch procedure that runs thousands of jobs. Each one of those jobs reads and/or writes directly into a directory on your server, accessed via an automount path like /nevis/tanya/data/ . Let's say there are 300 batch queues in the Nevis condor system. That means that 300 jobs are trying to access the disk on your server at once. | |||||||
Sounds like a recipe for disaster, doesn't it? You can crash your server by writing 300 files at once over the network. It's happened several times at Nevis. | ||||||||
Line: 51 to 51 | ||||||||
The typical program/job development process looks like this:
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Line: 75 to 75 | ||||||||
There's one more thing to think about: If initialdir is located machine A (via automount), and you submit the job from machine B, then condor will use machine B to transfer the files to the directory on machine A. For example, if these lines are in your condor submit file: | ||||||||
Changed: | ||||||||
< < | initialdir = /a/data/kolya/jsmith | |||||||
> > | initialdir = /nevis/kolya/data/jsmith | |||||||
queue 10000 | ||||||||
Changed: | ||||||||
< < | and you submit the job on the machine karthur, then as each of the 10,000 jobs terminates karthur will automount /a/data/kolya/jsmith on kolya to write the file; see condor_shadow![]() | |||||||
> > | and you submit the job on the machine karthur, then as each of the 10,000 jobs terminates karthur will automount /nevis/kolya/data/jsmith on kolya to write the file; see condor_shadow![]() | |||||||
Plan where your log files will go | ||||||||
Line: 92 to 92 | ||||||||
These lines define where the job's output, error, and log files are written. If you submit one job, the above lines are fine. If you submit 10,000 jobs, you'll create 30,000 files. If you submit mySimulation1, mySimulation2, ... you'll create an indefinite number of files. If initialdir![]() /share , sooner or later you'll fill up that disk. Everyone in your group can be affected.
The general solution is to not write your output files into your /share directory. You can do this via one of the following: | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < | output = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).out error = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).err log = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).log | |||||||
> > | output = /nevis/amsterdam/data/jsmith/myjobs/mySimulation-$(Process).out error = /nevis/amsterdam/data/jsmith/myjobs/mySimulation-$(Process).err log = /nevis/amsterdam/data/jsmith/myjobs/mySimulation-$(Process).log | |||||||
There is another solution: delete these "scrap" files once you no longer need them. Only the most intelligent and clever physicists remember to do this... and we have high hopes for you! |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Disk sharing and the condor batch system | ||||||||
Line: 40 to 40 | ||||||||
Here's a diagram of how sharing works: ![]() | ||||||||
Changed: | ||||||||
< < | Important! If you're skimming this page, stop and read this paragraph! The /home and /share partitions are backed up![]() /scratch , /data , etc.) are not backed up at all. | |||||||
> > | Important! If you're skimming this page, stop and read this paragraph! The /home and /share partitions are backed up nightly. The other partitions (/scratch , /data , etc.) are not backed up at all. | |||||||
"OK, I'll put all my stuff where it's backed up." This is a bad strategy: you'll fill up the disk; your jobs will crash; your fellow users will complain. Put files that you personally create (e.g., programs, thesis, plots) on /home or /share . Files that are automatically created (e.g., Monte Carlo output, n-tuples, log files, output files) should be placed in /data or /scratch . |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Disk sharing and the condor batch system | ||||||||
Line: 61 to 61 | ||||||||
Let condor transfer the files | ||||||||
Changed: | ||||||||
< < | Don't rely on your job reading and writing files directly to a particular directory. Use commands like the following in your condor command file; look them up in the User's Manual![]() | |||||||
> > | Don't rely on your job reading and writing files directly to a particular directory. Use commands like the following in your condor command file; look them up in the User's Manual![]() | |||||||
should_transfer_files = YES when_to_transfer_output = ON_EXIT | ||||||||
Line: 69 to 69 | ||||||||
initialdir = ...where your inputs and outputs are located... | ||||||||
Changed: | ||||||||
< < | This will transfer your input files to the condor master server once, instead of 300 times; as your job is executing, it will write the output on a local area of the machine that's running the job. Once the job has finished executing, it will transfer the output file to the initialdir![]() | |||||||
> > | This will transfer your input files to the condor master server once, instead of 300 times; as your job is executing, it will write the output on a local area of the machine that's running the job. Once the job has finished executing, it will transfer the output file to the initialdir![]() | |||||||
Changed: | ||||||||
< < | Unless you specify a file's full pathname in the condor command file, the files will be copied to and from initialdir![]() | |||||||
> > | Unless you specify a file's full pathname in the condor command file, the files will be copied to and from initialdir![]() | |||||||
There's one more thing to think about: If initialdir is located machine A (via automount), and you submit the job from machine B, then condor will use machine B to transfer the files to the directory on machine A. For example, if these lines are in your condor submit file:
initialdir = /a/data/kolya/jsmith queue 10000 | ||||||||
Changed: | ||||||||
< < | and you submit the job on the machine karthur, then as each of the 10,000 jobs terminates karthur will automount /a/data/kolya/jsmith on kolya to write the file; see condor_shadow![]() | |||||||
> > | and you submit the job on the machine karthur, then as each of the 10,000 jobs terminates karthur will automount /a/data/kolya/jsmith on kolya to write the file; see condor_shadow![]() | |||||||
Plan where your log files will go | ||||||||
Line: 89 to 89 | ||||||||
log = mySimulation-$(Process).log | ||||||||
Changed: | ||||||||
< < | These lines define where the job's output, error, and log files are written. If you submit one job, the above lines are fine. If you submit 10,000 jobs, you'll create 30,000 files. If you submit mySimulation1, mySimulation2, ... you'll create an indefinite number of files. If initialdir![]() /share , sooner or later you'll fill up that disk. Everyone in your group can be affected. | |||||||
> > | These lines define where the job's output, error, and log files are written. If you submit one job, the above lines are fine. If you submit 10,000 jobs, you'll create 30,000 files. If you submit mySimulation1, mySimulation2, ... you'll create an indefinite number of files. If initialdir![]() /share , sooner or later you'll fill up that disk. Everyone in your group can be affected. | |||||||
The general solution is to not write your output files into your /share directory. You can do this via one of the following:
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Specifying the complete automount path is not a good idea for log files, because then every job will try to write to that directory and you run the risk of crashing the server. Do not do the following: |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Disk sharing and the condor batch system | ||||||||
Line: 52 to 52 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
If you think about this some more, you'll realize that this doesn't keep you from crashing the file server; you can still create a job that writes to the /data partition. But you'll only crash a file server, not a login server; the entire cluster won't crash. You'll only be in trouble with your group, not with everyone at Nevis. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Disk sharing and the condor batch system | ||||||||
Line: 16 to 16 | ||||||||
There's a bigger problem when that server is one on which users' /home directories are located. The mail server checks your home directory every time you receive a mail message. If your home directory is not available, it slows down the mail server. If the mail server receives a lot of mail for users with home directories on that server, the mail server can slow to the point of uselessness, or crash. If the mail server crashes, the entire Linux cluster can crash. | ||||||||
Changed: | ||||||||
< < | The choice is between potentially crashing the cluster and making you think about disk files. It seems pretty obvious that thinking is your only option. | |||||||
> > | The choice is between potentially crashing the cluster, and making you think about disk files. It's obvious: thinking is your only option. | |||||||
Types of systems | ||||||||
Line: 26 to 26 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > | ||||||||
Directories and how they're sharedNote: This describes the "ideal" case, which as of Jul-2011 only applies to the Neutrino group. As other groups continue to add and maintain systems on the cluster, I'm going to encourage moving to the separate login/file server configuration. Until then, be careful; crashing the cluster looks like more fun than it is. | ||||||||
Changed: | ||||||||
< < | There are many exceptions, but here are the general rules: | |||||||
> > | There are many exceptions (e.g., ATLAS T3![]() | |||||||
| ||||||||
Line: 100 to 100 | ||||||||
log = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).log | ||||||||
Added: | ||||||||
> > | There is another solution: delete these "scrap" files once you no longer need them. Only the most intelligent and clever physicists remember to do this... and we have high hopes for you! | |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Changed: | ||||||||
< < | -- WilliamSeligman - 06 Jul 2011
| |||||||
> > | Disk sharing and the condor batch system | |||||||
Changed: | ||||||||
< < |
| |||||||
> > | BackgroundYou don't want to read this page. You just want to submit your jobs and not think about disk files or where they're located. Unfortunately, you have to think about it. Here's why. Picture this: You submit a condor batch procedure that runs thousands of jobs. Each one of those jobs reads and/or writes directly into a directory on your server, accessed via an automount path like/a/data/tanya . Let's say there are 300 batch queues in the Nevis condor system. That means that 300 jobs are trying to access the disk on your server at once.
Sounds like a recipe for disaster, doesn't it? You can crash your server by writing 300 files at once over the network. It's happened several times at Nevis.
There's a bigger problem when that server is one on which users' /home directories are located. The mail server checks your home directory every time you receive a mail message. If your home directory is not available, it slows down the mail server. If the mail server receives a lot of mail for users with home directories on that server, the mail server can slow to the point of uselessness, or crash. If the mail server crashes, the entire Linux cluster can crash.
The choice is between potentially crashing the cluster and making you think about disk files. It seems pretty obvious that thinking is your only option.
Types of systemsIn general, these are the types of systems at Nevis:
Directories and how they're sharedNote: This describes the "ideal" case, which as of Jul-2011 only applies to the Neutrino group. As other groups continue to add and maintain systems on the cluster, I'm going to encourage moving to the separate login/file server configuration. Until then, be careful; crashing the cluster looks like more fun than it is. There are many exceptions, but here are the general rules:
![]() /home and /share partitions are backed up![]() /scratch , /data , etc.) are not backed up at all.
"OK, I'll put all my stuff where it's backed up." This is a bad strategy: you'll fill up the disk; your jobs will crash; your fellow users will complain. Put files that you personally create (e.g., programs, thesis, plots) on /home or /share . Files that are automatically created (e.g., Monte Carlo output, n-tuples, log files, output files) should be placed in /data or /scratch .
How this affects condor jobsThe only partitions that can be seen by the entire cluster are on your file servers. Therefore, any files that will be read or written by your job should be on/share (for reading) or /data (for reading or writing).
The typical program/job development process looks like this:
/data partition. But you'll only crash a file server, not a login server; the entire cluster won't crash. You'll only be in trouble with your group, not with everyone at Nevis.
If you don't want to crash any servers at all, here are some more things to consider:
Let condor transfer the filesDon't rely on your job reading and writing files directly to a particular directory. Use commands like the following in your condor command file; look them up in the User's Manual![]() should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = ...list of input files... initialdir = ...where your inputs and outputs are located...This will transfer your input files to the condor master server once, instead of 300 times; as your job is executing, it will write the output on a local area of the machine that's running the job. Once the job has finished executing, it will transfer the output file to the initialdir ![]() ![]() initialdir is located machine A (via automount), and you submit the job from machine B, then condor will use machine B to transfer the files to the directory on machine A. For example, if these lines are in your condor submit file:
initialdir = /a/data/kolya/jsmith queue 10000and you submit the job on the machine karthur, then as each of the 10,000 jobs terminates karthur will automount /a/data/kolya/jsmith on kolya to write the file; see condor_shadow![]() Plan where your log files will goAt the end of most condor batch files, you'll see lines that look like this:output = mySimulation-$(Process).out error = mySimulation-$(Process).err log = mySimulation-$(Process).logThese lines define where the job's output, error, and log files are written. If you submit one job, the above lines are fine. If you submit 10,000 jobs, you'll create 30,000 files. If you submit mySimulation1, mySimulation2, ... you'll create an indefinite number of files. If initialdir ![]() /share , sooner or later you'll fill up that disk. Everyone in your group can be affected.
The general solution is to not write your output files into your /share directory. You can do this via one of the following:
output = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).out error = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).err log = /a/data/amsterdam/jsmith/myjobs/mySimulation-$(Process).log
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Added: | ||||||||
> > |
|