Upgrading the Nevis Linux cluster to a 64-bit operating system

Upgrade schedule

Date Systems Mail affected
Tue 17-Aug-2010 morningside, riverside Neutrino group, willis
Thu 19-Aug-2010 shang, han DOE Group, shaevitz, annmarie, jsantini, bishop, capone
Tue 24-Aug-2010 karthur D0/ATLAS, ban, zhang
Wed 25-Aug-2010 kolya ATLAS group
Thu 26-Aug-2010 tehanu VERITAS group

Each upgrade will start at 4PM, and will hopefully take no more than two hours.

Important: While any of the above servers are being upgraded, there is a good chance that the mail server will slow down. It might be necessary to reboot the mail server before and/or after the upgrade.

The batch nodes (hermesXX, kennelXX) and 64-bit-compatible workstations will be upgraded during those two weeks. This will interfere with condor jobs running on those machines during that time; you may want to hold off running condor jobs during the day.

Which systems won't be upgraded

The following systems won't be upgraded to 64-bit; at least, not as part of this co-ordinated effort. Instead, as the systems are replaced, their replacements will receive a 64-bit version of Linux.

  • westside.nevis.columbia.edu, a Neutrino server that will be reserved for compiling the SciBooNE framework. (If you need software compiled on a 32-bit system, you can request access to this server.)
  • Any system that is not part of the physics-analysis portion of the Nevis Linux cluster; e.g., the mail server.
  • The systems in the Nevis Annex, since they are not part of the condor batch system nor are they used to run physics software.
  • The following systems, as of Jul-2010, have processors that are 32-bit-only:

    machine room purpose
    anna 106 Mikhail Lelthouk's desktop
    claremont 116 Matt Toup's desktop
    cordwainer 119 scanner workstation
    guinevere 117 general student workstation
    harry 117 general student workstation
    hermes00 119 batch farm
    hermes01 119 batch farm
    hermes02 119 batch farm
    hermes03 119 batch farm
    hermione 117 general student workstation
    isolde 117 general student workstation
    luigi 120 Neutrino workstation
    mario 120 Neutrino workstation
    qin 117 general student workstation
    ron 117 general student workstation
    toad 120 Neutrino workstation
    yoshi 120 Neutrino workstation

Why the upgrade?

Here are the reasons:

  • A number of groups are moving to 64-bit computing:
    • The ATLAS analysis software will run on a 64-bit environment.
    • The Double Chooz group is or will be moving to a 64-bit operating system.
    • The VERITAS group's software functions best in a 64-bit environment.

  • The DOE group already runs their software in a 64-bit environment at BNL, so the change means little to them.

  • In general, a 64-bit operating system can be up to 40% faster than the equivalent 32-bit system running on the same hardware.

  • The last two weeks of August have usually been the time for upgrades to the Nevis computer cluster. In 2009, it was the shift to Scientific Linux 5.

It boils down to: We would have to make the change someday, and now is as good a time as any.

Which version of Linux?

Scientific Linux 5.5

What about existing software?

As far as the Scientific Linux packages are concerned, there are 64-bit versions of all existing software. The upgrade will be automatic.

If you type the command setup on any system on the Nevis Linux cluster, you'll see that I've prepared 64-bit versions of ROOT, Geant4, CLHEP, and the GCC 4.4 compiler. Scientific Linux 5 comes with the GCC 4.1 compiler.

The 32-bit versions of all software, both Linux- and HEP-related, will remain on the systems for as long as anyone still needs them. In particular, the GCC 3.2 compiler will remain available in a 32-bit version.

What do I have to change in my scripts?

If you are moving to 64-bit versions of your programs, and you use /usr/lib explicitly in your scripts, you'll probably have to change it to /usr/lib64.

If you want to keep using the 32-bit versions of your programs and the software, you'll probably have to add -m32 as a compiler option; e.g., g++ -m32 myprog.cxx -o myprog

In general, condor will only allow jobs submitted from 32-bit systems to run on 32-bit systems, and the same for 64-bit systems. You probably won't have to change any of your existing condor command files.

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2010-09-22 - WilliamSeligman
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback