Splunk
Introduction
Splunk is a data analytics platform that provides data-driven insights across all aspects of a company.
Visit https://www.splunk.com/ for more information.
Adds powerful features to your data storage. Monitor, analyze, and visualize data.
- Access your data from anywhere thanks to Splunk's unified hybrid experience.
Integrating Splunk with Storj requires S3 credentials from Storj that will be added to the indexes.conf in Splunk.
Splunk Enterprise integrates with any S3-compatible cloud storage platform.
To complete the integration, you will need:
- A Storj account
- An on-premises instance of Splunk
Splunk is compatible with Windows, Mac, and Linux OS.
To complete the integration, follow the steps below.
To begin, you will need to create a Storj account. If you already an account, go to https://storj.io/login.
Navigate to https://storj.io/signup to sign up. Enter your full name, email address, and a password, as shown below:
Create a Bucket
Once you have your Storj account, create a Storj bucket for Splunk following the steps below.
1. Navigate to “Buckets” on the left side menu.
2. Click “New Bucket” on the top right.
3. Choose a name for your bucket, such as "splunk".
4. Select “Continue”.
5. Generate a passphrase or enter your own.
6. Select “Continue”.
7. Record the passphrase somewhere safe.
Remember your passphrase as you will need it for future access of your data. Storj is unable to recover your passphrase for you.
You will need to generate S3 credentials for LucidLink to access your bucket in Storj. S3 credentials consist of an access key, secret key, and endpoint. You will need to store them somewhere safe, as they cannot be recovered.
Create S3 credentials in the Storj web console:
- Navigate to “Access” on the left side menu
- Click “Create S3 Credentials” under the S3 Credentials block.
3. When the Create Access screen comes up, set specifications according to the following guidelines:
- Type: S3 Credentials
- Name: The name of the credentials (e.g. iconik)
- Permissions: All
- Buckets: Feel free to specify the bucket you created above (e.g. iconik), or leave as “All”
4. Check the "Encryption Information" pop-up message that says, "By generating S3 credentials, you are opting in to server-side encryption". The click "Continue".
5. Select your passphrase encryption from either the "Generate Passphrase" or "Create My Own Passphrase" options. This is the passphrase used to access the files in your bucket.
6. You can then click one of the "Copy to Clipboard" or "Download .txt" options.
7. Check the message to acknowledge that you have read it: "I understand that Storj does not know or store my encryption passphrase. If I lose it, I won't be able to recover files."
8. Click the "Create My Access" button.
9. Your S3 credentials are created. Write them down and store them, or click the "Download .txt" button. You will need these credentials for the following steps.
To complete the integration, you will need the S3 credentials created in the previous steps and an instance of Splunk Enterprise on your local machine.
To get started with Splunk Enterprise, visit https://www.splunk.com/en_us/products/splunk-enterprise.html. Either request a free trial or contact the Splunk sales team.
1. To connect Storj remote storage to Splunk, add Storj volume information to indexes.conf. This is usually added at the top of the file.
See Splunk's indexes.conf documentation for more details.
Name this volume Storj and specify credentials underneath.
- The access key, secret key, and endpoint are those generated in Storj in the previous steps of this tutorial.
- For the path , use s3://splunk/ .
- Set maxGlobalDataSizeMB to 5 for optimal performance.
2. Restart Splunk
1. Create a test file using the following command:
2. Use Splunk to attempt to push the test file into Storj using the Storj volume just created in Splunk:
3. You should see the file listed in the shell and in your Storj web UI.
1. In Splunk, create an index and name it something memorable such as "Storj". This is the index you will add the Storj volume to.
2. Mount the Storj volume under the Storj index stanza in indexes.conf:
3. Restart Splunk
4. Force a data roll from hot to warm for testing purposes by performing an internal rest call. You will need to authenticate with your Splunk username and password.
Alternate call without credentials. You will still be prompted for credentials:
Once the bucket is rolled to warm, it will populate in its own folder within the Storj bucket. Smart Store has been fully enabled for the index. Smartstore allows many other items to be configured, please reference the following documentation for additional configuration options:
