FELIX Host Machines

Machines at Nevis are:

flx-srv-atlas -- the original machine with the FELIX-712

flx-alice and flx-bob -- the new machines with the FELIX-182

This page is a reference for setting up the 182.

Useful Links

FELIX User Manual (fairly recently updated): https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/felix-doc/felix-user-manual/5.1.x/felix-user-manual.html

Releases page (links to outdated git repos): https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/releases.html

Drivers repo (in case of manual build): https://gitlab.cern.ch/atlas-tdaq-software/ROSRCDdrivers

Host Machine Setup

BIOS

Set PCIe bifurcation in the BIOS (del at boot brings up the BIOS):

Advanced > CPU Configuration > CPU1 Information

Set all groups to 8x8 from Auto (if you know the slot you could set only that one). flx-info lspci should show two devices.

Turn IOMMU off

Advanced > NB Configuration > IOMMU

Switch to disabled from auto

OS

Install Alma9, setup atlas user with sudo rights. Make other users. Probably want a non-sudo user for production testing.

The following is done as the atlas user

Update packages

sudo yum update

Install some extra packages:

dnf config-manager --set-enabled crb
sudo dnf install epel-release

sudo yum install cmake librdmacm-utils libuuid-devel lm_sensors

sudo dnf install htop

sudo dnf install swig clang pybind11-devel
sudo dnf install zbar-libs libdmtx

Setup firefox to delete user logins:
about:preferences#privacy

Check > Delete cookies and site data when Firefox is closed

Uncheck in settings > clear history when firefox closes

Download Vivado lab from the AMD website:
https://www.xilinx.com/support/download.html

Install in /opt/Xilinx

Add to the .bashrc

source /opt/Xilinx/Vivado_Lab/2024.2/settings64.sh

CVMFS

Install CVMFS client https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html

Edit /etc/cvmfs/default.local to include:

CVMFS_QUOTA_LIMIT=10000

Maybe need to change CVMFS timeout / DNS settings. Getting a lot of input/output errors, which randomly appeared and stopped. /var/log/messages say it can't resolve cvmfs host names and keeps switching

TDAQ setup

Add to .bashrc

alias cm_setup='source /cvmfs/atlas.cern.ch/repo/sw/tdaq/tools/cmake_tdaq/bin/cm_setup.sh'

Now this should work, but it will be slow, but we don't actually set this up for the felix sw below, it does it's own setup. If this ends up too slow a container might be an option: https://atlas-tdaq-sw-releases.web.cern.ch/HOWTOS/Containers/

cm_setup tdaq-11-02-00

FELIX driver

grab rpm from releases website:

wget https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/dist/software/drivers/releases/el9/felix-drivers-04-20-00/tdaq_sw_for_Flx-4.20.0-2dkms.noarch.rpm
sudo dnf install tdaq_sw_for_Flx-4.20.0-2dkms.noarch.rpm

The Usual setup script is now here:

/etc/init.d/drivers_flx start

This should have made a derivers_flx_sd.service in /etc/systemd/system to automatically start the drivers at boot. If not make one as below:

Let's make this a service that starts on boot. Add the following to /etc/systemd/system/flx_derivers.service

[Unit]
Description=Start drivers for the FELIX card

[Service]
Type=oneshot
ExecStart=/etc/init.d/drivers_flx start
ExecStart=/etc/init.d/drivers_flx sttop
RemainAfterExit=yes
[Install]WantedBy=multi-user.target

Run:

sudo systemctl daemon-reload. 
sudo systemctl enable my_script.service
sudo systemctl start my_script.service
sudo systemctl status my_script.service

FELIX Software

CVMFS may be an option instead of building: /cvmfs/atlas-online-nightlies.cern.ch/felix/releases

Follow instructions in repo to build software distribution: https://gitlab.cern.ch/atlas-tdaq-felix/felix-distribution

git clone https://gitlab.cern.ch/atlas-tdaq-felix/felix-distribution.git

git submodule init
git submodule update
.....
make -j

source setup.sh from main directory

Follow instructions in repo to build lpgbt interface: https://gitlab.cern.ch/BNL-ATLAS/larphase2/feb2/libflx-lpgbt

Note this is not yet updated for the 182 and ALMA9....

Install extra packages

dnf --enablerepo=crb install swig

From FEB2 directory. Note cmake command is modified from repo to point to correct places, turn off docs, and turn off ROOT. In libflx-lpgbt below, make a setup.sh script with the following and run source ./setup.sh (the ./ is important otherwise it seems to find something else in the path)

if [ ! -z "$PS1" ]; then #only active in interactive mode
        export REGMAP_VERSION=0x0400
        export FELIXSWROOT=/home/atlas/FEB2/felix-distribution
        #source $FELIXSWROOT/setup.sh
fi

libflx-lpgbt is compiling with pybind11 after an edit to the cmake file:

--- a/pybind11/CMakeLists.txt
+++ b/pybind11/CMakeLists.txt
@@ -1,8 +1,9 @@
 ###################### lpGBTlib python binding ############################
 IF(BUILD_WITH_PYBIND11)
- message(STATUS "Building with pybind11 Python bindings")
- find_package(Python COMPONENTS Interpreter Development)
- find_package(pybind11 CONFIG)
+ message(STATUS "Building with pybind11 Python bindings")
+ find_package(Python 3.11 COMPONENTS Interpreter Development)
+ message($python) 
+ find_package(pybind11 2.10 CONFIG)

Then proceed to build

source ./setup.sh
mkdir libflx-lpgbt
cd  libflx-lpgbt
git clone https://gitlab.cern.ch/BNL-ATLAS/larphase2/feb2/libflx-lpgbt.git
mkdir build install
cd build
cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_DOC=OFF -DBUILD_WITH_ROOT=OFF  -DBUILD_TESTEXEC=ON -DBUILD_WITH_PYBIND11=ON -DBUILD_WITH_SWIG=OFF ../libflx-lpgbt

Add the install directory/pyFlxlpGBT to the python path, add this to the setup.sh file above in libflx-lpgbt

export PYTHONPATH=$PYTHONPATH:/home/atlas/FEB2/libflx-lpgbt/install/pyFlxlpGBT

Fiber Mapping

FEB2 MTP are numbered 3 at the top, 2 middle, and 1 bottom.

The left socket of the breakout box pair is a, and the right is b.

GUI Setup

Install UV (note local to user):

curl -LsSf https://astral.sh/uv/install.sh | sh
mkdir FEB2
cd FEB2

Clone gui repo

git clone https://gitlab.cern.ch/feb2/gui-new.git

venv setup
maybe only needed for python 3.8.5:

sudo yum install libnsl

uv venv venv-gui
source venv-gui/bin/activate
uv pip install -r gui-new/requirements.txt

You should also create ssh keys and copy them to feb2-prod.nevis.columbia.edu. The full hostname is used in the rsync and is what is checked for the host ID.

New login setup

• From the felix-distribution directory, source setup.sh
• source the libflx-lpgbt setup.sh
  -- Jonathan Long - 2025-04-16

Comments