website logo
Create accountLogin
Home
DCS
Node
Resources
Help center
Navigate through spaces
Home
DCS
Node
Resources
Help center
⌘K
Storj DCS
Get Started
AWS CLI and Hosted Gateway MT
AWS SDK and Hosted Gateway MT
Uplink CLI
Satellite Admin Console
Object Browser
Downloads
Download Uplink CLI
Download Self-hosted S3 Compatible Gateway
Download Storj Client Libraries
SDKs & Reference
Tutorial
How-to Guides
Concepts
Support
Support Overview
FAQ
Community Forum
Status Page
Help Desk
Billing, Payment & Accounts
Resources
Moved Documents
Docs powered by archbee 
32min

Splunk

Introduction 

Splunk is a data analytics platform that provides data-driven insights across all aspects of a company.

Visit https://www.splunk.com/ for more information.

Advantages of Splunk with Storj

  1. Adds powerful features to your data storage. Monitor, analyze, and visualize data.

  2. Access your data from anywhere thanks to Splunk's unified hybrid experience.

Integration Requirements

Integrating Splunk with Storj requires S3 credentials from Storj that will be added to the indexes.conf in Splunk.

Splunk Enterprise integrates with any S3-compatible cloud storage platform.

To complete the integration, you will need:

  • A Storj account
  • An on-premises instance of Splunk

Splunk is compatible with Windows, Mac, and Linux OS.

To complete the integration, follow the steps below.

Create a Storj Account

To begin, you will need to create a Storj account. If you already an account, go to https://storj.io/login.

Navigate to https://storj.io/signup to sign up. Enter your full name, email address, and a password, as shown below:

Document image

Create a Bucket 

Once you have your Storj account, create a Storj bucket for Splunk following the steps below.

1. Navigate to “Buckets” on the left side menu.

2. Click “New Bucket” on the top right.

Document image

3. Choose a name for your bucket, such as "splunk".

Document image

4. Select “Continue”.

5. Generate a passphrase or enter your own.

Document image

6. Select “Continue”.

7. Record the passphrase somewhere safe.

Remember your passphrase as you will need it for future access of your data. Storj is unable to recover your passphrase for you.

Generate S3 credentials

You will need to generate S3 credentials for LucidLink to access your bucket in Storj. S3 credentials consist of an access key, secret key, and endpoint. You will need to store them somewhere safe, as they cannot be recovered.

Create S3 credentials in the Storj web console:

  1. Navigate to “Access” on the left side menu
  2. Click “Create S3 Credentials” under the S3 Credentials block.
Document image

3. When the Create Access screen comes up, set specifications according to the following guidelines:

  • Type: S3 Credentials
  • Name: The name of the credentials (e.g. iconik)
  • Permissions: All
  • Buckets: Feel free to specify the bucket you created above (e.g. iconik), or leave as “All”
Document image

4. Check the "Encryption Information" pop-up message that says, "By generating S3 credentials, you are opting in to server-side encryption". The click "Continue".

Document image

5. Select your passphrase encryption from either the "Generate Passphrase" or "Create My Own Passphrase" options. This is the passphrase used to access the files in your bucket.

6. You can then click one of the "Copy to Clipboard" or "Download .txt" options.



Document image

7. Check the message to acknowledge that you have read it: "I understand that Storj does not know or store my encryption passphrase. If I lose it, I won't be able to recover files."

8. Click the "Create My Access" button.

Document image

9. Your S3 credentials are created. Write them down and store them, or click the "Download .txt" button. You will need these credentials for the following steps.

Document image

Integrating Splunk with Storj

To complete the integration, you will need the S3 credentials created in the previous steps and an instance of Splunk Enterprise on your local machine.

Splunk Access

To get started with Splunk Enterprise, visit https://www.splunk.com/en_us/products/splunk-enterprise.html. Either request a free trial or contact the Splunk sales team.

Connect Storj remote storage

1. To connect Storj remote storage to Splunk, add Storj volume information to indexes.conf. This is usually added at the top of the file.

See Splunk's indexes.conf documentation for more details.

Name this volume Storj and specify credentials underneath.

  • The access key, secret key, and endpoint are those generated in Storj in the previous steps of this tutorial.
  • For the path , use s3://splunk/ .
  • Set maxGlobalDataSizeMB to 5 for optimal performance.
Shell
|

2. Restart Splunk

Verify Connectivity

1. Create a test file using the following command:

Shell
|

2. Use Splunk to attempt to push the test file into Storj using the Storj volume just created in Splunk:

Shell
|

3. You should see the file listed in the shell and in your Storj web UI.

Shell
|

Add the remote storage to a provisioned index

1. In Splunk, create an index and name it something memorable such as "Storj". This is the index you will add the Storj volume to.

2. Mount the Storj volume under the Storj index stanza in indexes.conf:

Shell
|

3. Restart Splunk

Shell
|

4. Force a data roll from hot to warm for testing purposes by performing an internal rest call. You will need to authenticate with your Splunk username and password.

Shell
|

Alternate call without credentials. You will still be prompted for credentials:

Shell
|

Success!

Once the bucket is rolled to warm, it will populate in its own folder within the Storj bucket. Smart Store has been fully enabled for the index. Smartstore allows many other items to be configured, please reference the following documentation for additional configuration options:

  • Indexes.conf https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Indexesconf 
  • Smart Store https://docs.splunk.com/Documentation/Splunk/9.0.1/Indexer/ConfigureSmartStore 
Updated 26 Jan 2023
Did this page help you?
Yes
No
UP NEXT
TrueNAS - iX Systems
Docs powered by archbee 
TABLE OF CONTENTS
Introduction 
Advantages of Splunk with Storj
Integration Requirements
Create a Storj Account
Create a Bucket 
Generate S3 credentials
Integrating Splunk with Storj
Splunk Access
Connect Storj remote storage
Verify Connectivity
Add the remote storage to a provisioned index
Success!