Upgrading the Nevis Linux cluster to a 64-bit operating system

Upgrade schedule

Each upgrade will start at 4PM, and will hopefully take no more than two hours.

Important: While any of the above servers are being upgraded, there is a good chance that the mail server will slow down. It might be necessary to reboot the mail server before and/or after the upgrade.

The batch nodes (hermesXX, kennelXX) and 64-bit-compatible workstations will be upgraded during those two weeks. This will interfere with condor jobs running on those machines during that time; you may want to hold off running condor jobs during the day.

Which systems won't be upgraded

The following systems won't be upgraded to 64-bit; at least, not as part of this co-ordinated effort. Instead, as the systems are replaced, their replacements will receive a 64-bit version of Linux.

• westside.nevis.columbia.edu, a Neutrino server that will be reserved for compiling the SciBooNE framework. (If you need software compiled on a 32-bit system, you can request access to this server.)
• Any system that is not part of the physics-analysis portion of the Nevis Linux cluster; e.g., the mail server.
• The systems in the Nevis Annex, since they are not part of the condor batch system nor are they used to run physics software.
• The following systems, as of Jul-2010, have processors that are 32-bit-only:

  machine	room	purpose
  anna	106	Mikhail Lelthouk's desktop
  claremont	116	Matt Toup's desktop
  cordwainer	119	scanner workstation
  guinevere	117	general student workstation
  harry	117	general student workstation
  hermes00	119	batch farm
  hermes01	119	batch farm
  hermes02	119	batch farm
  hermes03	119	batch farm
  hermione	117	general student workstation
  isolde	117	general student workstation
  luigi	120	Neutrino workstation
  mario	120	Neutrino workstation
  qin	117	general student workstation
  ron	117	general student workstation
  toad	120	Neutrino workstation
  yoshi	120	Neutrino workstation

Why the upgrade?

Here are the reasons:

• A number of groups are moving to 64-bit computing:
  • The ATLAS analysis software will run on a 64-bit environment.
  • The Double Chooz group is or will be moving to a 64-bit operating system.
  • The VERITAS group's software functions best in a 64-bit environment.

• The DOE group already runs their software in a 64-bit environment at BNL, so the change means little to them.

• In general, a 64-bit operating system can be up to 40% faster than the equivalent 32-bit system running on the same hardware.

• The last two weeks of August have usually been the time for upgrades to the Nevis computer cluster. In 2009, it was the shift to Scientific Linux 5.

It boils down to: We would have to make the change someday, and now is as good a time as any.

Which version of Linux?

Scientific Linux 5.5

What about existing software?

As far as the Scientific Linux packages are concerned, there are 64-bit versions of all existing software. The upgrade will be automatic.

If you type the command setup on any system on the Nevis Linux cluster, you'll see that I've prepared 64-bit versions of ROOT, Geant4, CLHEP, and the GCC 4.4 compiler. Scientific Linux 5 comes with the GCC 4.1 compiler.

The 32-bit versions of all software, both Linux- and HEP-related, will remain on the systems for as long as anyone still needs them. In particular, the GCC 3.2 compiler will remain available in a 32-bit version.

What do I have to change in my scripts?

If you are moving to 64-bit versions of your programs, and you use /usr/lib explicitly in your scripts, you'll probably have to change it to /usr/lib64.

If you want to keep using the 32-bit versions of your programs and the software, you'll probably have to add -m32 as a compiler option; e.g., g++ -m32 myprog.cxx -o myprog

In general, condor will only allow jobs submitted from 32-bit systems to run on 32-bit systems, and the same for 64-bit systems. You probably won't have to change any of your existing condor command files.