website logo
Create accountLogin
Home
DCS
Node
Resources
Help center
Navigate through spaces
Home
DCS
Node
Resources
Help center
⌘K
Storj DCS
Get Started
AWS CLI and Hosted Gateway MT
AWS SDK and Hosted Gateway MT
Uplink CLI
Satellite Admin Console
Object Browser
Downloads
Download Uplink CLI
Download Self-hosted S3 Compatible Gateway
Download Storj Client Libraries
SDKs & Reference
Tutorial
How-to Guides
Concepts
Support
Support Overview
FAQ
Community Forum
Status Page
Help Desk
Billing, Payment & Accounts
Resources
Moved Documents
Docs powered by archbee 
38min

Rclone Hosted Gateway

Selecting an Integration Pattern

Use our S3-compatible Hosted Gateway integration pattern to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only 1GB of data being uploaded from the client to the hosted gateway.

Use this pattern for

  • Reduced upload time
  • Reduction in network load

By selecting this integration pattern you are opting in to Server-side Encryption.

Prerequisites

Generate Credentials to the Gateway MT

Navigate to the Access page within your project and then click on Create S3 Credentials. A modal window will pop up where you should enter a name for this access grant.

Document image
Document image

Assign the permissions you want this access grant to have, then click on Encrypt My Access:

Document image

Enter the Encryption Passphrase you used for your other access grants. If this is your first access grant, we strongly encourage you to use a mnemonic phrase as your encryption passphrase. (The GUI automatically generates one on the client-side for you)

Document image

This passphrase is important! Encryption keys derived from it are used to encrypt your data at rest, and your data will have to be re-uploaded if you want it to change!



Importantly, if you want two access grants to have access to the same data, they must use the same passphrase. You won’t be able to access your data if the passphrase in your access grant is different than the passphrase you uploaded the data with.



Please note that Storj does not know or store your encryption passphrase, so if you lose it, you will not be able to recover your files.

Click either on the Copy to clipboard link or Download .txt and then confirm that you copied your Encryption Phrase to a safe place.

Document image

Click the Create my Access link to finish generating of S3 credentials.

Document image

Copy your Access Key, Secret Key, and Endpoint to a safe location or download them.

Now you are ready to configure Rclone

Setup

First, Download and extract the rclone binary onto your system.

Execute the config command:

|

A text-based menu will prompt. Type n and hit Enter to create a new remote configuration, select n (New Remote).

|

Enter a name for the new remote configuration, e.g. waterbear.

Text
|

A long list of supported storage backends will prompt. Enter s3 and hit Enter.

Text
|

A further list of S3 storage providers will prompt. Enter Storj and hit Enter.

Text
|

A choice will be given on how you will enter credentials. Hit Enter for the default choice of 1 / Enter AWS credentials in the next step.

Text
|

You will be asked for your Access Key ID followed by the Secret Access Key that you previously generated, follow the pattern in the code block below.

Text
|

You will be asked what Endpoint to connect to. Enter 1 to select the Global Hosted Gateway and hit Enter.

If you use an older Rclone release, you will see a list of regional gateway endpoints to select from. Ignore them and enter the global gateway.storjshare.io endpoint. Using the regional endpoints may cause performance issues.

Dockerfile
|

You will be asked if you want to edit the advanced config. Type y and hit Enter. Skip most of the questions by hitting Enter. The important ones to configure are chunk_size (64M) and disable_checksum (true).

Text
|

A summary of the remote configuration will prompt. Type yand hit Enter to confirm.

Text
|

Now you should see one remote configuration available. Enter q and hit Enter to quit the configuration wizard.

Text
|

For additional security, you should consider using the s) Set configuration password option. It will encrypt the rclone.conf configuration file. This way secrets like the access token, the encryption passphrase, and the access grant can't be stolen if an attacker gains access to your configuration file.

Create a Bucket

Use the mkdir command to create new bucket, e.g. mybucket.

|

List All Buckets

Use the lsf command to list all buckets.

|

Note the colon (:) character at the end of the command line.

Delete a Bucket

Use the rmdir command to delete an empty bucket.

|

Use the purge command to delete a non-empty bucket with all its content.

|

Upload Objects

Use the copy command to upload an object.

|

The --progress flag is for displaying progress information. Remove it if you don't need this information.

Use a folder in the local path to upload all its objects.

|

Only modified files will be copied.

List Objects

Use the ls command to list recursively all objects in a bucket.

|

Add the folder to the remote path to list recursively all objects in this folder.

|

Use the lsf command to list non-recursively all objects in a bucket or a folder.

|

Download Objects

Use the copy command to download an object.

|

The --progress flag is for displaying progress information. Remove it if you don't need this information.

Use a folder in the remote path to download all its objects.

|

Delete Objects

Use the deletefile command to delete a single object.

|

Use the delete command to delete all object in a folder.

|

Print the Total Size of Objects

Use the size command to print the total size of objects in a bucket or a folder.

|

Sync Two Locations

Use the sync command to sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.

|

The --progress flag is for displaying progress information. Remove it if you don't need this information.

Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.

The sync can be done also from Storj DCS to the local file system.

|

Or between two Storj DCS buckets.

|

Or even between another cloud storage and Storj DCS.

|
Updated 23 Jan 2023
Did this page help you?
Yes
No
UP NEXT
How to Fine-tune File Transfer Performance on Storj DCS
Docs powered by archbee 
TABLE OF CONTENTS
Selecting an Integration Pattern
Use this pattern for
Prerequisites
Generate Credentials to the Gateway MT
Setup
Create a Bucket
List All Buckets
Delete a Bucket
Upload Objects
List Objects
Download Objects
Delete Objects
Print the Total Size of Objects
Sync Two Locations