River, the testbed

River is the Lab’s instance of the IRIS-HEP Scalable Systems Laboratory, the place where analysis software for the high-luminosity LHC era is run at realistic scale before it reaches a production facility. About seventy nodes, Kubernetes throughout, with Ceph storage integrated into the cluster.

What runs there is a catalogue of what the field is trying to build. ServiceX, in production and several parallel test instances, for columnar data delivery. Coffea-casa and BinderHub for interactive analysis. REANA for reproducible workflows. Dask and Parsl for scale-out. Caching proxies under test for wide-area data access. A machine learning portal and GPU-backed inference services. The perfSONAR analytics platform that archives worldwide network measurements.

River also carries infrastructure that serves the rest of the Lab: the collectors that feed the operations analytics platform, and a development instance of the Analysis Facility where changes are tried before they reach the physicists using the real one.

The point of River is that things are allowed to break there.

Pile, the PSD cluster

Pile serves people, not grid workflows. It provides interactive login nodes, HTCondor batch, notebook sessions through BinderHub and JupyterHub, Ceph storage with a separate pool for scratch, CVMFS for software delivery, and a Globus endpoint for moving datasets in and out.

Two communities use it. The University of Chicago group on SBND, the Short-Baseline Near Detector at Fermilab, has its analysis computing here, including transfers of detector data from Fermilab. And since 2024, PSD Connect has connected any researcher in the University’s Physical Sciences Division to the same cluster, with a login node and a batch queue available within about a day of signing up.

Work that fits the high-throughput pattern can overflow from Pile to the Open Science Pool. Pile returns the favor, contributing its idle cycles back to the national pool.

Tempest, the services layer

Tempest is the smallest of the three and the one everything else depends on. It runs on hardware procured through the Lab’s participation in the PATh project, and it carries the services that are not tied to any single facility:

• Identity, the Keycloak single sign-on that users meet when they reach any of the Lab’s web services
• A container registry, where the images the other clusters run are stored
• OSG compute entrypoints and the glideinWMS factory that places work onto the Open Science Pool
• Rucio for XENON, the global data management service the Lab operates for the dark matter experiment at Gran Sasso
• Data transfer services for the South Pole Telescope

It is a small cluster doing load-bearing work for several collaborations at once, which is the argument for keeping it separate from any of them.