Using Rucio to find Protodune files at CERN

Thanks to Steven Timm

The Rucio Documentation: http://rucio.cern.ch/documentation/

Introduction--during the ProtoDUNE II run we attempted to hold all current raw data on disk at CERN in the Rucio Storage Element DUNE_CERN_EOS. However, the bulk of data that was taken forced us to spread it out over a large number of sites. This is similar to what was done in ProtoDUNE I except that the directory structure has changed to have some hex hashes in the directory paths. Rucio is the way going forward and we want to get people used to using it.

All current members of the DUNE VOMS now have read access to rucio.

For running inside SL7 apptainer container:

Follow the https://wiki.dunescience.org/wiki/SL7_to_Alma9_conversion instructions

These instructions should still work inside the stock DUNE RHEL7 container.

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

Then setup python 3_9, required for rucio v33_3

setup python v3_9_15

(note if you have dunesw set up already this is the python that comes with that. Necessary because system python on rhel7 machines is too old to work with rucio)

Then setup the rucio package

setup rucio
setup kx509

Next get your x509 grid proxy.

One can get their grid proxy by doing kx509 with a valid kerberos ticket.

kdestroy
kinit <username>@FNAL.GOV
kx509

If you want to read files anywhere other than CERN you will also have to do a voms-proxy-init

voms-proxy-init -voms dune:/dune/Role=Analysis -noregen

alternate solution on lxplus with apptainer:

If running the fermilab apptainer on lxplus (alma9)

Before you launch the apptainer

kdestroy
kinit <username>@FNAL.GOV
kx509

then launch the apptainer and things should work.

Define your Rucio account name – it should be your FNAL account

export RUCIO_ACCOUNT=<username>

Check to see if you can authenticate against the Rucio server

rucio whoami

created_at : 2021-10-19T20:01:12
account    : benjamin
status     : ACTIVE
email      : None
deleted_at : None
updated_at : 2021-10-19T20:01:12
account_type : USER
suspended_at : None

Rucio Alma9 instructions

There is a full working rucio, metacat, and data dispatcher client in Spack.

. /cvmfs/larsoft.opensciencegrid.org/spack-packages/setup-env.sh
spack load r-m-dd-config experiment=dune

spack load kx509

The r-m-dd config package sets up rucio and metacat and sets all the environment variables necessary.

How to find the locations of a file in EOS if you know the name of the file already:

In Rucio/MetaCat files are specified by scope:filename For ProtoDUNE II the 4 scopes that matter are vd-coldbox, hd-coldbox, vd-protodune, and hd-protodune

So for instance to find the location of file np02vdcoldbox_raw_run023784_0000_dataflow0_datawriter_0_20240118T140023.hdf5

you would say rucio list-file-replicas vd-coldbox:np02vdcoldbox_raw_run023784_0000_dataflow0_datawriter_0_20240118T140023.hdf5

+------------+------------------------------------------------------------------------------+------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | vd-coldbox | np02vdcoldbox_raw_run023784_0000_dataflow0_datawriter_0_20240118T140023.hdf5 | 13.390 MB | 308b18ba | DUNE_CERN_EOS: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/vd-coldbox/13/71/np02vdcoldbox_raw_run023784_0000_dataflow0_datawriter_0_20240118T140023.hdf5 | +------------+------------------------------------------------------------------------------+------------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You can access the file either by the root:// URL or by going directly on to EOSPUBLIC. Once the file has been replicated to Fermilab a URL at Fermilab will show up too.

Eventually these will also be visible in SAM but that is going to take some work.

Example: Check which RSE’s the dataset found using the scope and dataset discovered from a Metacat query

for example - dc4-vd-coldbox-bottom:dc4-vd-coldbox-bottom_307151901

rucio list-dataset-replicas --deep dc4-vd-coldbox-bottom:dc4-vd-coldbox-bottom_307151901

DATASET: dc4-vd-coldbox-bottom:dc4-vd-coldbox-bottom_307151901
+-------------------------+---------+---------+
| RSE                     |   FOUND |   TOTAL |
|-------------------------+---------+---------|
| PRAGUE                  |      60 |      60 |
| DUNE_ES_PIC             |      60 |      60 |
| DUNE_CERN_EOS           |      60 |      60 |
| DUNE_US_FNAL_DISK_STAGE |      60 |      60 |
| MANCHESTER              |      60 |      60 |
| DUNE_US_BNL_SDCC        |      60 |      60 |
+-------------------------+---------+---------+

Example: List the datasets at a given RSE for example DUNE_ES_PIC

rucio list-datasets-rse DUNE_ES_PIC