Upgrading the Nevis Linux cluster to a 64-bit operating system
Upgrade schedule
Date |
Systems |
Mail affected |
Tue 17-Aug-2010 |
morningside, riverside |
Neutrino group, willis |
Thu 19-Aug-2010 |
shang, han |
DOE Group, shaevitz, annmarie, jsantini, bishop, capone |
Tue 24-Aug-2010 |
karthur |
D0/ATLAS, ban, zhang |
Wed 25-Aug-2010 |
kolya |
ATLAS group |
Thu 26-Aug-2010 |
tehanu |
VERITAS group |
Each upgrade will start at 4PM, and will hopefully take no more than two hours.
Important: While any of the above servers are being upgraded, there is a good chance that the mail server will slow down. It might be necessary to reboot the mail server before and/or after the upgrade.
The batch nodes (
hermesXX
,
kennelXX
) and 64-bit-compatible workstations will be upgraded during those two weeks. This will interfere with condor jobs running on those machines during that time; you may want to hold off running condor jobs during the day.
Which systems won't be upgraded
The following systems won't be upgraded to 64-bit; at least, not as part of this co-ordinated effort. Instead, as the systems are replaced, their replacements will receive a 64-bit version of Linux.
- westside.nevis.columbia.edu, a Neutrino server that will be reserved for compiling the SciBooNE framework. (If you need software compiled on a 32-bit system, you can request access to this server.)
- Any system that is not part of the physics-analysis portion of the Nevis Linux cluster; e.g., the mail server.
- The systems in the Nevis Annex, since they are not part of the condor batch system nor are they used to run physics software.
- The following systems, as of Jul-2010, have processors that are 32-bit-only:
machine |
room |
purpose |
anna |
106 |
Mikhail Lelthouk's desktop |
claremont |
116 |
Matt Toup's desktop |
cordwainer |
119 |
scanner workstation |
guinevere |
117 |
general student workstation |
harry |
117 |
general student workstation |
hermes00 |
119 |
batch farm |
hermes01 |
119 |
batch farm |
hermes02 |
119 |
batch farm |
hermes03 |
119 |
batch farm |
hermione |
117 |
general student workstation |
isolde |
117 |
general student workstation |
luigi |
120 |
Neutrino workstation |
mario |
120 |
Neutrino workstation |
qin |
117 |
general student workstation |
ron |
117 |
general student workstation |
toad |
120 |
Neutrino workstation |
yoshi |
120 |
Neutrino workstation |
Why the upgrade?
Here are the reasons:
- A number of groups are moving to 64-bit computing:
- The ATLAS analysis software will run on a 64-bit environment.
- The Double Chooz group is or will be moving to a 64-bit operating system.
- The VERITAS group's software functions best in a 64-bit environment.
- The DOE group already runs their software in a 64-bit environment at BNL, so the change means little to them.
- In general, a 64-bit operating system can be up to 40% faster than the equivalent 32-bit system running on the same hardware.
- The last two weeks of August have usually been the time for upgrades to the Nevis computer cluster. In 2009, it was the shift to Scientific Linux 5.
It boils down to: We would have to make the change someday, and now is as good a time as any.
Which version of Linux?
Scientific Linux 5.5
What about existing software?
As far as the
Scientific Linux
packages are concerned, there are 64-bit versions of all existing software. The upgrade will be automatic.
If you type the command
setup
on any system on the Nevis
Linux cluster, you'll see that I've prepared 64-bit versions of ROOT, Geant4, CLHEP, and the GCC 4.4 compiler. Scientific Linux 5 comes with the GCC 4.1 compiler.
The 32-bit versions of all software, both Linux- and HEP-related, will remain on the systems for as long as anyone still needs them. In particular, the GCC 3.2 compiler will remain available in a 32-bit version.
What do I have to change in my scripts?
If you are moving to 64-bit versions of your programs, and you use
/usr/lib
explicitly in your scripts, you'll probably have to change it to
/usr/lib64
.
If you want to keep using the 32-bit versions of your programs and the software, you'll probably have to add
-m32
as a compiler option; e.g.,
g++ -m32 myprog.cxx -o myprog
In general,
condor will only allow jobs submitted from 32-bit systems to run on 32-bit systems, and the same for 64-bit systems. You probably won't have to change any of your existing condor command files.