Globus

Integration guide for connecting Storj to Globus

Globus is a premier solution that empowers research by providing advanced data management capabilities. It offers data transfer, sharing, and discovery, all integrated into a unified, open-source platform. Globus ensures data security, seamless collaboration, and efficient management for individual researchers and larger research institutions.

Advantages of Globus with Storj

The synergy between Globus and Storj brings about a comprehensive data management solution at a competitive cost. Globus provides reliable data transfer and sharing tools for streamlined research processes, effectively integrating with Storj's S3 compatible storage for affordable storing and retrieval of vast datasets.

With Storj's cutting-edge security features, users can rest assured that their data managed on Globus is secure. Additionally, Globus offers metadata management, simplifying the handling and access of extensive datasets, while its efficient data sharing mechanisms help optimize storage costs.

Integration

The integration of Storj and Globus is facilitated via the S3 protocol, allowing Globus to directly write and read data to and from the Storj network. Users can configure the Globus endpoint to utilize Storj as the storage destination for their data.

To integrate Globus with Storj, you will need:

  • An active Storj account

  • Bucket(s) to be shared in the Globus ecosystem

  • Storj S3 compatible credentials

  • Globus Connect Server instance (see here)

  • Globus for AWS S3 Connector Subscription

  • Access to the Globus web interface for configuration and management

For more details, see https://www.globus.org/globus-connect.

Create a Storj Account

To begin, you will need to create a Storj account. If you already an account, go to https://storj.io/login.

Navigate to https://storj.io/signup to sign up. Enter your full name, email address, and a password, as shown below:

Create a Bucket

Once you have your Storj account you can create a bucket for your data to be stored in.

1. Navigate to “Buckets” on the left side menu.

2. Click “New Bucket” on the top right.

3. Assign the bucket an easily identifiable name, such as "my-bucket".

4. Click Create bucket

Generate S3 credentials

Storj has an Amazon S3 compatible API and you'll need generate S3 credentials to use it. S3 credentials consist of an access key, secret key, and endpoint.

Create S3 credentials in the Storj web console:

1. Navigate to Access on the left side menu.

2. Click Create S3 Credentials under the S3 Credentials block.

3. When the Create Access screen comes up, set specifications according to the following guidelines:

  • Type: S3 Credentials

  • Name: The name of the credentials (e.g. my-access)

4. Click Continue to provide permissions

  • Permissions: All

  • Buckets: Feel free to specify the bucket you created above (e.g. my-bucket), or leave as “All”

  • End date: provide an expiration date for these credentials (optional)

5. Click Continue to provide Access encryption Information

  • Use the current passphrase: this is default option

  • Advanced: you may provide a different encryption phrase either your own or generate a new one.

    • Enter a new passphrase: use this option, if you would like to provide your own new encryption phrase

    • Generate 12-word passphrase: use this option, if you would like to generate a new encryption phrase

In order to see the data uploaded to your bucket in the web console, you must unlock the bucket with the same encryption passphrase as the credentials.

6. Click Create Access to finish creation of your S3 credentials

7. Click Confirm the Confirm details pop-up message

8. Your S3 credentials are created. Write them down and store them, or click the Download all button. You will need these credentials for the following steps.

Connecting to Storj via Globus S3 Storage Gateway

The Globus Storage Gateway allows for a seamless integration with Storj's S3 compatible storage, creating an efficient bridge between your local data storage and Storj's decentralized cloud service.

Enable the Globus for AWS S3 Connector

To enable the S3 connector follow the instructions here

Create S3 Storage Gateway

Next run globus-connect-server storage-gateway create s3, which initiates the process of creating a new S3 storage gateway.

Replace my-bucket with the one created previously in the command below. You can specify more than one bucket if needed. For more details see here.

globus-connect-server storage-gateway create s3 \
"Storj Storage Gateway" \
--domain example.org \
--s3-endpoint https://gateway.storjshare.io \
--s3-user-credential \
--bucket my-bucket --bucket research-data-bucket-2
globus-connect-server storage-gateway create s3 \
"Storj Storage Gateway" \
--domain example.org \
--s3-endpoint https://gateway.storjshare.io \
--s3-user-credential \
--bucket my-bucket --bucket research-data-bucket-2

The command will output a unique id to identify the gateway.

Storage Gateway Created: 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab
Storage Gateway Created: 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab

If you forget the id of a storage gateway, use the command globus-connect-server storage-gateway list to get a list of the storage gateways on the endpoint.

Create S3 User Credential

First register user credentials with the Globus storage gateway using globus-connect-server user-credentials s3-create. For more details see here.

globus-connect-server user-credentials s3-create --globus-identity user@example.org
globus-connect-server user-credentials s3-create --globus-identity user@example.org

Follow prompts to input S3 Access Key and Secret Key (created previously)

S3 access key id: access_key # REPLACE ME
S3 secret key: secret_key # REPLACE ME
Created credential 9bb3d1d1-f506-41f1-b161-41c372b7da19 for

<user@example.org>
S3 access key id: access_key # REPLACE ME
S3 secret key: secret_key # REPLACE ME
Created credential 9bb3d1d1-f506-41f1-b161-41c372b7da19 for

<user@example.org>

Create Collection

Create a collection (additional details here)

The STORAGE_GATEWAY_ID is from the previous step of “Create S3 Gateway”

globus-connect-server collection create STORAGE_GATEWAY_ID BASE_PATH DISPLAY_NAME
globus-connect-server collection create STORAGE_GATEWAY_ID BASE_PATH DISPLAY_NAME

Viewing the Collection with Globus File Manager

Enter the COLLECTION ID from the previous step

To be able to view collection follow the instructions here

Optional - Increase Concurrency for Better Throughput

The Globus Endpoint can be modified with the "custom" network-use flag to modify the number of files to transfer concurrently. To make this modification, use the following command (replace N with an integer, not to exceed the number of CPU cores on your Globus Connect Server):

globus-connect-server endpoint update --network-use "custom" --max-concurrency N --preferred-concurrency N
globus-connect-server endpoint update --network-use "custom" --max-concurrency N --preferred-concurrency N
Previous
FileZilla Pro Integration