Managing scientific data at the exascale with Rucio

Group photo of the Rucio community workshop

The first Rucio community workshop was held on 1 and 2 March at CERN and brought together computing experts from several experiments and organisations. The focus of the workshop was the collaborative use of Rucio for large-scale scientific data management.

Rucio, an open-source project for managing community data, was developed by the ATLAS collaboration and adopted by the AMS and XENON1T collaborations. Recently, several other upcoming and ongoing experiments have started evaluating it. During the workshop, EISCAT_3D, CMS, LIGO, NA62, IceCube, CTA, COMPASS, SKA, and DUNE presented and described their experience of using Rucio, as well as their requirements for handling petascale or even exascale data sets.

Two keynote speeches were given at the workshop. Dr Tiago Quintino from the European Centre for Medium-Range Weather Forecasts (ECMWF) spoke about “IO and Storage Challenges on the Path to Exascale Numerical Weather Prediction”. Dr Karan Bhatia from Google gave a talk on “The Google Cloud Platform and ATLAS – The Data Ocean Project”. The Rucio core team presented the architecture of the system and the plans for the future development of the project. The NorduGrid, Fermilab/FIFE and Open Science Grid teams described their plans to provide Rucio as a service to their user communities. A hands-on technical session with a live demo of the Rucio system and its functionality took place at the end of the workshop.