Rclone Hosted Gateway
Selecting an Integration Pattern
Use our S3-compatible Hosted Gateway integration pattern to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only 1GB of data being uploaded from the client to the hosted gateway.
Use this pattern for
- Reduced upload time
- Reduction in network load
By selecting this integration pattern you are opting in to Server-side Encryption.
Prerequisites
Generate Credentials to the Gateway MT
Navigate to the Access page within your project and then click on Create S3 Credentials. A modal window will pop up where you should enter a name for this access grant.
Assign the permissions you want this access grant to have, then click on Encrypt My Access:
Enter the Encryption Passphrase you used for your other access grants. If this is your first access grant, we strongly encourage you to use a mnemonic phrase as your encryption passphrase. (The GUI automatically generates one on the client-side for you)
This passphrase is important! Encryption keys derived from it are used to encrypt your data at rest, and your data will have to be re-uploaded if you want it to change!
Importantly, if you want two access grants to have access to the same data, they must use the same passphrase. You won’t be able to access your data if the passphrase in your access grant is different than the passphrase you uploaded the data with.
Please note that Storj does not know or store your encryption passphrase, so if you lose it, you will not be able to recover your files.
Click either on the Copy to clipboard link or Download .txt and then confirm that you copied your Encryption Phrase to a safe place.
Click the Create my Access link to finish generating of S3 credentials.
Copy your Access Key, Secret Key, and Endpoint to a safe location or download them.
Now you are ready to configure Rclone
Setup
First, Download and extract the rclone binary onto your system.
Execute the config command:
A text-based menu will prompt. Type n and hit Enter to create a new remote configuration, select n (New Remote).
Enter a name for the new remote configuration, e.g. waterbear.
A long list of supported storage backends will prompt. Enter s3 and hit Enter.
A further list of S3 storage providers will prompt. Enter Storj and hit Enter.
A choice will be given on how you will enter credentials. Hit Enter for the default choice of 1 / Enter AWS credentials in the next step.
You will be asked for your Access Key ID followed by the Secret Access Key that you previously generated, follow the pattern in the code block below.
You will be asked what Endpoint to connect to. Enter 1 to select the Global Hosted Gateway and hit Enter.
If you use an older Rclone release, you will see a list of regional gateway endpoints to select from. Ignore them and enter the global gateway.storjshare.io endpoint. Using the regional endpoints may cause performance issues.
You will be asked if you want to edit the advanced config. Type y and hit Enter. Skip most of the questions by hitting Enter. The important ones to configure are chunk_size (64M) and disable_checksum (true).
A summary of the remote configuration will prompt. Type yand hit Enter to confirm.
Now you should see one remote configuration available. Enter q and hit Enter to quit the configuration wizard.
For additional security, you should consider using the s) Set configuration password option. It will encrypt the rclone.conf configuration file. This way secrets like the Create Access Grant in CLI, the encryption passphrase, and the access grant can't be stolen if an attacker gains access to your configuration file.
Create a Bucket
Use the mkdir command to create new bucket, e.g. mybucket.
List All Buckets
Use the lsf command to list all buckets.
Note the colon (:) character at the end of the command line.
Delete a Bucket
Use the rmdir command to delete an empty bucket.
Use the purge command to delete a non-empty bucket with all its content.
Upload Objects
Use the copy command to upload an object.
The --progress flag is for displaying progress information. Remove it if you don't need this information.
Use a folder in the local path to upload all its objects.
Only modified files will be copied.
List Objects
Use the ls command to list recursively all objects in a bucket.
Add the folder to the remote path to list recursively all objects in this folder.
Use the lsf command to list non-recursively all objects in a bucket or a folder.
Download Objects
Use the copy command to download an object.
The --progress flag is for displaying progress information. Remove it if you don't need this information.
Use a folder in the remote path to download all its objects.
Delete Objects
Use the deletefile command to delete a single object.
Use the delete command to delete all object in a folder.
Print the Total Size of Objects
Use the size command to print the total size of objects in a bucket or a folder.
Sync Two Locations
Use the sync command to sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.
The --progress flag is for displaying progress information. Remove it if you don't need this information.
Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.
The sync can be done also from Storj DCS to the local file system.
Or between two Storj DCS buckets.
Or even between another cloud storage and Storj DCS.
