What needs to be upgraded

• katya01

• hopper

What needs to be upgraded.

These are the lists of systems on the Nevis Linux cluster that need to be upgraded, organized by research group. If a system's name is bold, then it is exposed to the outside world and is vulnerable to being hacked.

ATLAS

These systems currently run SL6:

• xenia
• xenia2
• xeniaNN, where NN goes from 01 to 22

The following systems are already running CentOS 7:

• kolya
• xenia00

DOE

These systems currently run SL6:

• shang

Electronics design

This system is still running Scientific Linux 5:

• elecsim

These systems currently run SL6:

• elecsim3
• elecsim4
• license

The following systems are already running CentOS 7:

• elecdesign

Neutrino

These systems currently run SL6:

• houston
• bleeker
• riverside
• westside
• kennelNN, where NN goes from 00 to 21 (with some gaps)

The following systems are already running CentOS 7:

• amsterdam

VERITAS/CTA

These systems currently run SL6:

• tehanu
• vetch
• ged
• serret

Why do we need to upgrade?

Consequences

The obvious one is that compiled for Scientific Linux 6 will probably not run on CentOS 7. At minimum, your analysis software will have to be recompiled. It's likely there will be version issues with libraries and such.

It's possible that your analysis software is tied to specific versions of external packages are not available on CentOS 7. It may be necessary to rewrite some programs and scripts.

How to upgrade?

As of Sep-2020, there's no simple way for me to visit Nevis. There are two approaches we're pursuing for the upgrades:

KVM

If a system is attached to a BIOS-level KVM-over-IP (KVM = keyboard-video-monitor; BIOS-level = I can interact with the system while it boots), then I can upgrade the system remotely. There have been persistent hardware issues with this approach, but we're still trying.

Network boot

I'm working on scripts that would automatically upgrade a system to CentOS 7 if the system was booted over the network. I'm still validating this approach. It would also require someone to manually intervene at a system's console as it's being rebooted.

Questions

Why CentOS 7? Why not Scientific Linux 7?

The difference between CentOS and Scientfic Linux are a few configuration files that are only relevant if a given system is physically located at either Fermilab or CERN.

As a measure of insignificance of this difference, there will be no "Scientific Linux 8". The Scientific Linux project ends with SL7, and suggests that its users move to CentOS.

Why not CentOS 8?

There's some logic to this question. CentOS 7 will cease to be supported on 30-Jun-2024, which means in less than five years we have to go through the exercise again. If we go to CentOS 8 now, we wouldn't have to upgrade until 2029.

However, to my knowledge none of the national labs are supporting CentOS 8 yet. It's been out for only a year, and will take a while to permeate through the scientific community. Also, the "jump" from CentOS 7 to CentOS 8 is larger than the one from 6 to 7; an upgrade from 6 to 8 may be even more disruptive to analysis tasks.

What about Ubuntu?

None of the national labs have adopted the Ubuntu distribution for large-scale physics-analysis tasks.

CentOS 7 upgrades

Scientific Linux 6 (SL6) will reach the end of its maintenance life on 30-Nov-2020. We have to consider what to do with those systems that are still running SL6 before then.

--++ Why do we need to upgrade?

There are two reasons:

Security

Once SL6 is no longer being maintained, if there are any security holes in it, hackers will be able to exploit it without fear that the exploits will be patched. That means that any system that can be accessed from the outside world (see the list below) will be vulnerable.

It's possible that hackers have be holding on to "zero-day" exploits and will start actively using them on 1-Dec-2020. I already know from the system logs that any Nevis system that allows outside access is already being attacked several times a minutes.

Software

The national labs are closing down the SL6 versions of their software suites (e.g., LArSoft for MicroBooNE; Athena for ATLAS). Any system that runs SL6 might not be able to keep up with the latest versions of your software.

What can we do?

For any system still running SL6, there are two choices:

Upgrade to CentOS 7

This is the preferred solution. Unfortunately, it's complicated by the pandemic. I'm working on potential solutions (see below).

Cut off outside access

If I cut off outside access to a system via the firewall, then there's no particular security risk. You can still access the system via VPN.

This may be the best solution for systems that you need to keep running SL6 (e.g., upgrading would interfere with analysis effort, there's no SL6 support for a given software package, etc.).