The Storj Developer Hub

Welcome to the Storj developer hub. You'll find comprehensive guides and documentation to help you start working with Storj as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    Support

FileZilla

How to up- and download files to Storj with FileZilla

Introduction

Storj is a decentralized peer-to-peer file storage system, and new protocol being added to FileZilla.

  • Download the FileZilla client
  • Add a new site in the Site Manager and select Storj as the protocol
  • Enter api.storj.io as the host and 443 as the port
  • Enter the bridge user name and password
  • If you have not created an Encryption Key before, generate a new one, otherwise add your existing key
  • Connect to the server and begin uploading and downloading files

FileZilla is a fast and reliable cross-platform FTP, FTPS, SFTP and Storj client with many useful features and an intuitive graphical user interface. A server client is also available.

Below are some of the FileZilla client features:

  • Easy to use
  • Supports Storj, FTP, FTP over SSL/TLS (FTPS) and SSH File Transfer Protocol (SFTP)
  • Cross-platform. Runs on Windows, Linux, *BSD, Mac OS X and more
  • IPv6 support
  • Available in many languages
  • Supports resume and transfer of large files >4GB
  • Tabbed user interface
  • Powerful Site Manager and transfer queue
  • Bookmarks
  • Drag & drop support
  • Configurable transfer speed limits
  • Filename filters
  • Directory comparison
  • Network configuration wizard
  • Remote file editing
  • Keep-alive
  • HTTP/1.1, SOCKS5 and FTP-Proxy support
  • Logging to file
  • Synchronized directory browsing
  • Remote file search

FileZilla incorporates the Storj asynchronous multi-platform C library for encrypted file transfer on the Storj network into its back end to ensure the highest level of performance and stability. This library is also the back end to the native libstorj CLI.

In this guide, we will run through the process of setting up FileZilla to be able to up- and download painlessly to the Storj Cloud.

Note: please do not upload more than to 1000 files to a single bucket/folder. If you want to upload thousands of files either zip them or create a new bucket. Uploading more than 1000 files to a single bucket will result in a very slow upload/download process or even the inability to access the bucket at all.

Notice: Storj is still in the early stages of development and undergoing testing to ensure high reliability of files. There will be upcoming improvements to the Storj network and Bridge for improved stability, scalability and the reliability of the service. Any early adopters that will be working with us to improve these areas are greatly appreciated.

Configure billing to upload/download beyond the free tier limit

Currently the free tier for Storj renters is structured as follows:

  • Uploading limit: 3GB/hour, 10GB/day and 25GB/month.
  • Downloading limit: 9GB/hour, 30GB/day and 180GB/month.

(Data obtained from: storj-bridge)

If you want to upload or download beyond the free tier, a payment option can be added to the Billing tab of your account on app.storj.io.

Payments options currently include:

  1. STORJ token (this currently needs to be requested by email to hello@storj.io, automatic deposit option will be added later to the GUI)
  2. Bitcoin
  3. Credit Card
*Figure 3.4. Adding a payment option to 'Billing' allows you to upload and download beyond the free tier limit.*

Figure 3.4. Adding a payment option to 'Billing' allows you to upload and download beyond the free tier limit.

Once your billing information is added, the free tier limit will be lifted and you will now be able to upload and download as much as you like/can afford!

These are the current storage and bandwidth prices (**they apply only after you exhausted your monthly free tier limits**):

  • Storage -> $0.015 per GB per month
  • Download Bandwidth -> $0.05 per GB downloaded
  • Upload bandwidth -> free

This information was extracted from pricing.

Important things to note regarding the billing structure:

  • You only pay for the time span that your data was stored on the network, if you delete some of your files today you will not pay for them anymore tomorrow, you will stop getting charged for storage right after you delete the files.
  • You storage usage is added daily to your "Current Usage" in the billing section. However, if your billing method is by credit card, you will only be billed once per month (normally on the 22nd of every month). For pre-paid billing methods like STORJ token or Bitcoin, your pre-paid balance will be reduced daily according to your usage but again actual amount deducted from your balance will be reconciled at the end of the billing cycle taking into account your free tier allotment.
  • The free tier allotment is subtracted from your paid tier every month, thus if you upload 100GB in a given month, you will only pay for 75GB (100GB – 25GB free tier) at the end of the billing cycle.

General Setup

Setup a Storj Account

Login

Before we can start uploading and downloading to the Storj network, it is necessary to open an account on Storj. You can either login or sign up

Sign Up

To sign up, enter an email address and desired password, then click the check box to agree to the terms of service. Afterwards, click on the "Sign Up" button.

Sign Up page

Sign Up page

Now check your email inbox and click on the account activation link sent to you by Storj. A web page should open with the account activation confirmation. Note that sometimes the email automatically gets moved to the spam/trash folder by some email providers, so if the email is not in the inbox, make sure to check the spam folder

Client Installation

Download the appropriate FileZilla client for your computer

Windows

After the download of the latest release here, a FileZilla installation guide for Windows OS can be found on the official FileZilla Wiki page.

GNU & Linux

  1. Download the latest release here.
  2. Unpack archive
    for 64x:
    tar xjf FileZilla_3.27.1_x86_64-linux-gnu.tar.bz2
    or for x86:
    tar xjf FileZilla_3.27.1_i586-linux-gnu.tar.bz2

  3. Copy to the /usr folder with super user's rights:

    cd FileZilla3
    sudo cp * -R /usr

  4. Download this 64x or this x86 library depending on your system architecture.

  5. Install the library
    for 64x:
    sudo dpkg -i libpng12-0_1.2.50-1ubuntu2.14.04.2_amd64.deb
    or for x86:
    sudo dpkg -i libpng12-0_1.2.50-1ubuntu2.14.04.2_i386.deb
    After the installation is complete, exit the installation window and launch the client.

Mac OSX

  1. Download the FileZilla client for all platforms.
  2. Unzip it & launch the client

Connect the client to the server for all computer platforms

  1. Add a new site in the Site Manager on File --> Site Manager and select Storj as the protocol
*Figure 4.1. Go to `File` -> `Site Manager...`*

Figure 4.1. Go to File -> Site Manager...

  1. In the resulting Site Manager Popup Window, first click on the New Site button at the bottom left-hand side. You can Rename the site name to a name of your liking, for example Storj. Then in the General tab on the right side, for Host: enter api.storj.io, for Port: enter 443 and for Protocol: select Storj - Decentralized Cloud Storage.
*Figure 4.2. Site Manager menu.*

Figure 4.2. Site Manager menu.

  1. In the User field enter your api.storj.io email address. In the Password field enter the matching password to the previous email address.
  2. If you are new to Storj click on Generate... to the right of the Encryption key field. This will generate an encryption key which will be used to encrypt all the data you upload to Storj, including all the folders. This key will also be used to decrypt the data you download from your Storj account. Be sure to write down the encryption key in a very safe place as you will lose access to your uploaded data if you don´t have the encryption key.
    If you were previously running libstorj see the next chapter to export the encryption key from libstorj. Once the encryption key is exported, you can paste it into the Encryption key field in FileZilla. This way you will be able to access any files you previously uploaded to Storj with libstorj. The same applies if you were using core-CLI or any other toolkit to upload to Storj.

  3. On the bottom right, there is a space to add Comments to the account which is particularly handy if you want to add more accounts to FileZilla.
  4. Finally, click on Connect. FileZilla Should now connect to your account. If all goes well and the login details are correct, no errors will be returned and the Site Manager will close (Figure 4.3).

Note: If already added any buckets (folders in FileZilla) to you account with another application, you will get the following error:

  • Error: Wrong encryption key for at least one bucket

You do not have to worry about this error, it just means you cannot access one of the folders because you do not have a valid decryption key for it. In certain use cases this is actually desirable because sometimes you want to keep the file encryption key private for a certain machine/environment.

*Figure 4.3. Dashboard after successfully logging in to our Storj account.*

Figure 4.3. Dashboard after successfully logging in to our Storj account.

Secure the FileZilla keys with a master password

To protect the encryption keys and bridge password, it is advisable to encrypt these passwords with a top-layer master password (and/or use full disk encryption and home directory encryption). This can be done as follows:

  1. In the FileZilla top menu, click on Edit -> Settings....
  2. Once in the Settings menu, click on Interface.
  3. Under Passwords select Save passwords protected by a master password.

Note: It is absolutely vital that you backup the encryption key to a safe location and preferentially offline. Printing the key on paper is a very secure common practice.

  1. Now enter a password of your choice (preferentially a long one), repeat the password for confirmation and click OK.

Done!. Now in case your computer gets compromised, it will be impossible for the hacker to retrieve your bridge keys and encryption key without using a key logger while you're unlocking the password.

Adjust simultaneous transfers and timeout details

We can adjust the number of simultaneous transfers to a value from 1 to 10. Setting this value to 10 allows maximum bandwidth utilization during transfers of a large number of files, both for uploading and downloading.

This can be done as follows:

  1. In the FileZilla top menu, click on Edit -> Settings....
  2. Once in the Settings menu, click on Transfers.
  3. In the Transfers section set the Maximum simultaneous transfers to a number of your choice between 0 and 10.

Note: You can also set transfer speed limits in KiB/s for both downloads and uploads in this settings menu. However, this isn't currently available with Storj, please watch https://github.com/Storj/libstorj/issues/9 for when it's added.

Timeouts can occur at the default timeout value. We can adjust this value upwards to where timeout errors no longer occur. In the example below the timeout was set to 60 seconds. This will affect all commands except for uploads and downloads.

Creating a folder, uploading and downloading files

This step is very easy. Once connected to our account, there are two main sections of the GUI, namely:

  • Local site (your computer) - Left pane
  • Remote site (the Storj cloud) - Right pane

If you already have some folders (buckets) in your Storj account, they will be displayed in the Remote site pane. You can easily create a folder by right clicking somewhere in the remote site pane and selecting Create directory (Figure 5.1).

Note: You can add as many folders as you want and also add sub-folders!.

*Figure 5.1. Adding folders to Storj for uploading files.*

Figure 5.1. Adding folders to Storj for uploading files.

You can now browse to the files/folders you want to upload in the Local site pane and upload them by dragging-and-dropping them to the Storj folder you created in the Remote site pane.

Note: please do not upload more than to 1000 files to a single bucket/folder. If you want to upload thousands of files either zip them or create a new bucket. Uploading more than 1000 files to a single bucket will result in a very slow upload/download process or even the inability to access the bucket at all.

*Figure 5.2. Uploading files to a folder/bucket.*

Figure 5.2. Uploading files to a folder/bucket.

Tip: If you can't add a directory, you can reconnect to Storj by doing the following:

  • Go to Server -> Reconnect (Figure 5.3).
*Figure 5.3. Reconnecting to Storj.*

Figure 5.3. Reconnecting to Storj.

Downloading files is just as easy, only this time drag-and-drop the files from your folder in your Remote Site pane to a folder of your choice in the Local Site pane. That's it!

Arch Linux (not required)

Huge thanks to reddit/u/ortango and rocketchat @joffamac for his guidance. Assuming that people know how to or read the AUR user guidelines for Arch in the wiki.

  1. Make a PKGBUILD file for libstorj containing the following:
pkgname=libstorj
pkgver=1.0.1
pkgrel=1
pkgdesc="Asynchronous multi-platform C library and CLI for encrypted file transfer on the Storj network"
arch=('x86_64')
url="https://github.com/Storj/libstorj"
license=('GPL')
depends=('curl' 'libuv' 'nettle' 'json-c' 'libmicrohttpd')
makedepends=('libtool' 'automake' 'autoconf')
source=(https://github.com/Storj/libstorj/archive/v$pkgver.tar.gz)
sha256sums=('63e245e68fe6fd6fa25380d70bf25134a41f9748bebdcd010e1db9ac7acc93e9')

build() {
cd "$pkgname-$pkgver"
sh ./autogen.sh
./configure --prefix=/usr
make
}
package() {
cd "$pkgname-$pkgver"
make DESTDIR="$pkgdir" install
}
  1. Make a folder named 'libstorj' in Downloads and saved that PkGBUILD file in there. CD into that folder and run makepkg -si. That downloads & builds the libstorj package which then can be installed.

  2. From AUR library, downloaded filezilla-svn build file (into Downloads), extracted it, CD into that folder, edited that PKGBUILD config parameter to include --enable-storj, then run makepkg -si.

  3. After +-10 minutes, it eventually allows to continue and install the Filezilla package. When started, the option for Storj is there. Follow the rest of the Storj - Filezilla docs and after that all should be working.

Registering an account via Bridge-GUI (app.storj.io)

Once you've successfully logged in, you will be taken to the bridge-GUI:

*Figure 3.3. Bridge-GUI (app.storj.io) dashboard.*

Figure 3.3. Bridge-GUI (app.storj.io) dashboard.

The bridge acts as an interface to the Storj network. It also manages all the complicated stuff in the background, making it completely painless for a user to start using the Storj network. Please note that the bridge maintained by Storj Labs is not the only bridge the user can connect to, it is also possible to set up a Storj network in a contained environment using the Storj development Kit. Also, other bridges exist on the network (e.g. for testing) to which one may choose to connect.

Note: If you want to register to a bridge that does not have a graphical user interface, you will have to register via libstorj CLI first and then import the keys into FileZilla.

Your bridge credentials is the same credentials you used to sign up in Storj.

Frequently Asked Questions

What is an Encryption Key?

The Encryption Key is a series of 12 to 24 words that represents 128 to 256 bits of entropy that is used to encrypt files before they are uploaded into the network.
Here is an example Encryption Key:

legal winner thank year wave sausage worth useful legal winner thank year wave sausage worth useful legal will

If you're familiar with BTC, this will be recognizable as it's based on the BIP39 standard.

How do I export settings from libstorj CLI?

First run this command to get the bridge user, password and the encryption key:

storj export-keys

And then enter those settings into FileZilla in the Getting Started section.

What does it mean if I see "Wrong encryption key for at least one bucket"?

This means that a bucket was added using a different Encryption Key and you won't be able to download the files within that bucket as it won't be able to decrypt or verify their integrity.

Why does the progress bar go above 100% for uploads?

Files are encoded with Reed Solomon error correction before they are uploaded to the network, so there is more data being uploaded compared to the original size of the file. This provides the ability to repair parts of the file that are missing or inaccessible. The number of parity shards is currently calculated as 2/3 of the original file size.

How are files encoded before being uploaded to the Storj network?

Files are first encrypted using the Encryption Key and then encoded using Reed Solomon into various size shards. For specific details please read SIP5. These shards are then uploaded to different nodes in the Storj network directly after communicating with the Storj bridge to negotiate which nodes to communicate with. After the file is uploaded, it is then mirrored to other nodes in the network, so that in the event that a shard goes missing after this initial upload, a shard can be repaired. The total number of shards that can be repaired is around 1/2 of the original file.

Note: For larger files the time it takes to do this preparation step can be more noticeable.

Can I rename or move files?

Not currently, this feature can be supported in the future, when file ids are unique, see deprecation messages at https://github.com/Storj/service-storage-models/blob/master/lib/models/bucketentry.js#L115-L119

What do I do if I see "Transfer rate limit" error?

This means that a rate limit on the number of bytes uploaded or downloaded has been reached. These are calculated time spans by hour, day and month. Accounts that have billing information entered currently do not have these rate limits, and instead have a rate limit on the total number of requests, which can be seen as a "Too many requests" error.

How many files can I upload?

There currently isn't a hard limit an the total number of bucket entries that can be added to a bucket. However there is limited support for more than 5000 buckets and more than 2000 bucket entries.

Why can't I upload a file at the root?

FileZilla currently maps buckets as directories at the root level and each file within those buckets is a bucket entry. Sub-directories are created by having a forward slash in the bucket entry name, which are parsed and shown as a directory within FileZilla. For this reason, it's not possible to add a file to the root because there isn't a bucket to add a bucket entry.

Debugging

It's possible to view full detailed logs for file transfers. To enable it (you'll need to do this each time you start up):

  • Open up the Preferences/Settings
  • Go to Logging and optionally enable timestamps and select a file to log
  • Go to Debug and select "4 - Debug"
  • View logs in the command console

Conclusion

In this simple guide, we first created a Storj account and added a billing option. We then imported our account details into FileZilla and adjusted a few parameters in the client settings. Finally, we learned how to add a folder to out Storj account and how to up- and download files to and from this folder. FileZilla has many more advanced features and settings, so if you want to learn more about FileZilla, please have a look at the official FileZilla wiki page.

We are constantly improving Storj and all of its tools to make sure that both renters and farmers can interact harmoniously in the Storj ecosystem. We would love to hear from you, no matter if you have suggestions, comments, questions or criticisms you would like to share# #.

FileZilla

How to up- and download files to Storj with FileZilla