Rucio

Overview

Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. This tutorial walks through the steps needed to configure a Rucio Storage Element (RSE) with a demo of Rucio.

These configuration instructions were adapted from the draft instructions being developed in the Rucio GitHub project: Create documentation page for S3-type storage

Setup a demo environment

Prerequisites:

Already have a Rucio environment?

Skip ahead to Rucio

Clone Rucio and jump into the docker container

Note that if you get an error like No such container: dev_rucio_1 run docker ps to identify the name of your rucio container (it may be something like "dev-rucio-1)

git clone https://github.com/rucio/rucio
cd rucio
docker-compose --file ./etc/docker/dev/docker-compose.yml up -d
docker exec -it dev_rucio_1 /bin/bash
git clone https://github.com/rucio/rucio
cd rucio
docker-compose --file ./etc/docker/dev/docker-compose.yml up -d
docker exec -it dev_rucio_1 /bin/bash

The rest of the commands are to be executed within the Docker container

Initialize the database

tools/run_tests_docker.sh -i
tools/run_tests_docker.sh -i

Update the ca-certificates package

The image has python3 linked in /usr/bin/python but yum requires python2

One workaround is to change the first line of /usr/bin/yum and /usr/libexec/urlgrabber-ext-down to use /usr/bin/python2 .

vi /usr/bin/yum < now change "python" to "python2" in the first line

sdf

vi /usr/libexec/urlgrabber-ext-down < now change "python" to "python2" in the first line

yum update ca-certificates
yum update ca-certificates

Configure the Storj RSE

The gfal protocol was chosen because s3boto was recently removed and the draft rucio doc (linked here) suggested using it to support s3. gfal is used to send requests like get, put, delete, rename, and copy to the RSE.

# replace YOUR_BUCKET_NAME with your storj bucket
rucio-admin rse add STORJ
rucio-admin rse add-protocol --hostname gateway.storjshare.io --scheme https --port 443 --prefix YOUR_BUCKET_NAME --impl rucio.rse.protocols.gfal.NoRename --domain-json '{"wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}, "lan": {"read": 1, "write": 1, "delete": 1}}' STORJ
rucio-admin rse set-attribute --rse STORJ --key sign_url --value s3
rucio-admin rse set-attribute --rse STORJ --key skip_upload_stat --value True
rucio-admin rse set-attribute --rse STORJ --key verify_checksum --value False
rucio-admin rse set-attribute --rse STORJ --key strict_copy --value True
# replace YOUR_BUCKET_NAME with your storj bucket
rucio-admin rse add STORJ
rucio-admin rse add-protocol --hostname gateway.storjshare.io --scheme https --port 443 --prefix YOUR_BUCKET_NAME --impl rucio.rse.protocols.gfal.NoRename --domain-json '{"wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}, "lan": {"read": 1, "write": 1, "delete": 1}}' STORJ
rucio-admin rse set-attribute --rse STORJ --key sign_url --value s3
rucio-admin rse set-attribute --rse STORJ --key skip_upload_stat --value True
rucio-admin rse set-attribute --rse STORJ --key verify_checksum --value False
rucio-admin rse set-attribute --rse STORJ --key strict_copy --value True

Create rse-accounts.cfg

# get your rse_id
rucio-admin rse info STORJ

# use your rse_id and s3 credentials below
cat >> etc/rse-accounts.cfg <<EOL
{
    "YOUR_RSE_ID": {
        "access_key": "YOUR_ACCESS_KEY",
        "secret_key": "YOUR_SECRET_KEY",
        "signature_version": "s3v4",
        "region": "us-east-1"
    }
}
EOL
# get your rse_id
rucio-admin rse info STORJ

# use your rse_id and s3 credentials below
cat >> etc/rse-accounts.cfg <<EOL
{
    "YOUR_RSE_ID": {
        "access_key": "YOUR_ACCESS_KEY",
        "secret_key": "YOUR_SECRET_KEY",
        "signature_version": "s3v4",
        "region": "us-east-1"
    }
}
EOL

Upload and download a file

rucio upload --rse STORJ --scope archive tools/test.file.1000
rucio download archive:test.file.1000
rucio upload --rse STORJ --scope archive tools/test.file.1000
rucio download archive:test.file.1000

What Next?

Explore the Rucio Documentation

https://rucio.cern.ch/documentation/

Set up a Rucio Client

Getting additional Rucio Support

If you are looking for additional support, Rucio has a mailing list rucio-users@googlegroups.com as well as a slack support channel.

Previous
Restic
Next
S3 Browser