Connectors → Splunk

About Splunk

Splunk is a software product that captures, indexes, and correlates real-time, machine-generated data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. Currently, the Splunk connector extracts data represented as Splunk reports.

Splunk Connector

The Incorta Splunk connector uses the Splunk Software Development Kit (SDK) for Java, which is built as a layer on top of the Splunk REST API. Version 1.0 of the connector supports Splunk reports. The Splunk connector creates a search job to retrieve the list of reports created in the system. When a report is selected during schema design, the Splunk Connector creates another search job to retrieve the fields of that report. Splunk retrieves the fields by discovering them from the last loaded job of the report, a mechanism that works for both scheduled and unscheduled reports.

The Splunk connector supports the following Incorta specific functionality:

Feature Supported
Incremental loading
Encryption at ingest
Performance Optimization
Webhook Callbacks

Deployment Steps

The Splunk connector is an external connector. You deploy an external connector as a JAR file to each Incorta Node in an Incorta cluster as well as to Cluster Management Console (CMC) host. A System Administrator with root access to the operating systems for each host in the Incorta cluster, including the CMC, will need to deploy the external JAR file for the Splunk Connector. A CMC Administrator will need to restart the Analytics and Loader Services in the cluster. A Systems Administrator will need to restart the CMC.

Deployment to an Incorta Node

Here are the steps to deploy the incorta.connector.splunk.jar file to the extensions directory of an Incorta Node that is running the Analytics and/or Loader Services in an Incorta cluster.

  • Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.
  • As the root user for the hosts running Incorta Nodes, use Secure Copy for shell or similar to copy the incorta.connector.splunk.jar to the /tmp directory of the hosts.

    PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
    INCORTA_NODE_HOST_IPv4_LIST='1.1.1.1  2.2.2.2  3.3.3.3  4.4.4.4'
    PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
    HOST_ROOT_USER='ec2-user'
    for i in ${INCORTA_NODE_HOST_IPv4_LIST}
    do
    echo $i
    scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${i}:/tmp/${PATH_JAR_FILE}
    wait
    done
  • Secure shell in to each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

    sudo su incorta
    sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar
  • For each Incorta Node, as the incorta user, create the splunk directory in the /extensions/connectors/ folder.

    INCORTA_NODE_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/IncortaNode'
    mkdir ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk
  • For each Incorta Node, as the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

    mv /tmp/incorta.connector.splunk.jar ${INCORTA_NODE_INSTALLATION_PATH}/extensions/connectors/splunk
Restart the Analytics and Loader Services

Here are the steps to restart the Analytics and Loader Services in an Incorta Cluster from the Cluster Management Console (CMC).

  • As the CMC Administrator, sign in to the CMC.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • Select the Details tab, if not already selected.
  • In the footer, select ** Restart**.

Deployment to the Cluster Management Console

  • Download the Splunk JAR file (incorta.connector.splunk.jar) from the latest version of your Incorta customer release distribution.
  • Using Secure Copy for Shell, copy the incorta.connector.splunk.jar to the /tmp directory of the host running the CMC.

    PATH_JAR_FILE='~/Downloads/incorta.connector.splunk.jar'
    CMC_HOST_IPv4='5.5.5.5'
    PATH_PEM_KEY_FILE='~/.ssh/incorta_2020.pem'
    HOST_ROOT_USER='ec2-user'
    scp -o StrictHostKeyChecking=no -i ${PATH_PEM_KEY_FILE} ${PATH_JAR_FILE} ${HOST_ROOT_USER}@${CMC_HOST_IPv4}:/tmp/${PATH_JAR_FILE}
  • Secure shell each Incorta Node, and if needed, change the ownership of the file to that of the incorta user.

    sudo su incorta
    sudo chown incorta:incorta /tmp/incorta.connector.splunk.jar
  • As the incorta user, create the splunk directory in the /extensions/connectors/ folder.

    CMC_INSTALLATION_PATH='/home/incorta/IncortaAnalytics/cmc'
    mkdir ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk
  • As the incorta user, move the incorta.connector.splunk.jar from the \tmp file to the splunk directory.

    mv /tmp/incorta.connector.splunk.jar ${CMC_INSTALLATION_PATH}/extensions/connectors/splunk
  • As the incorta user, stop the CMC

    cd ${CMC_INSTALLATION_PATH}
    ./stop-cmc.sh
  • As the incorta user, start the CMC

    cd ${CMC_INSTALLATION_PATH}
    ./start-cmc.sh

Connect Splunk and Incorta

To connect Splunk and Incorta, here are the high level steps, tools, and procedures:

Create an external data source

Here are the steps to create a external data source with the Splunk connector:

  • Sign in to the Incorta Direct Data Platform.
  • In the Navigation bar, select Data.
  • In the Action bar, select + NewAdd Data Source.
  • In the Choose a Data Source dialog, in Application, select Splunk.
  • In the New Data Source dialog, specify the applicable connector properties.
  • To test, select Test Connection.
  • Select Ok to save your changes.

Splunk connector properties

Here are the properties for the Splunk connector:

Property Control Description
Data Source Name text box Enter the name of the data source
Authentication Method drop down list Options are:
  • Using Splunk Username and Password
  • Using AppleConnect
  • Username text box Splunk Username and Password authentication only
    Password text box Splunk Username and Password authentication only
    IdMS Account Name text box Splunk AppleConnect authentication only
    IdMS Account Password text box Splunk AppleConnect authentication only
    IdMS AppID Key text box Splunk AppleConnect authentication only
    TOTP Secret Code text box Splunk AppleConnect authentication only
    Hostname text box Splunk hostname
    Port text box Splunk port

    Create a schema with the Schema Wizard

    Here are the steps to create a Splunk schema with the Schema Wizard:

    • Sign in to the Incorta Direct Data Platform.
    • In the Navigation bar, select Schema.
    • In the Action bar, select + New → Schema Wizard
    • In (1) Choose a Source, specify the following:

      • For Enter a name, enter the schema name.
      • For Select a Datasource, select the Splunk external data source.
      • Optionally create a description.
    • In the Schema Wizard footer, select Next.
    • In (2) Manage Tables, in the Data Panel, first select the name of the Data Source, and then check the Select All checkbox.
    • In the Schema Wizard footer, select Next.
    • In (3) Finalize, in the Schema Wizard footer, select Create Schema.

    Create a schema with the Schema Designer

    Here are the steps to create a Splunk schema using the Schema Designer:

    • Sign in to the Incorta Direct Data Platform.
    • In the Navigation bar, select Schema.
    • In the Action bar, select + New → Create Schema.
    • In Name, specify the schema name, and select Save.
    • In Start adding tables to your schema, select Splunk.
    • In the Data Source dialog, specify the Splunk table data source properties.
    • Select Add.
    • In the Table Editor, in the Table Summary section, enter the table name.
    • To save your changes, select Done in the Action Bar.

    Splunk table data source properties

    For a schema table in Incorta, you can define the following Splunk specific data source properties as follows:

    Property Control Description
    Type drop down list Default is Splunk
    Data Source drop down list Select the Splunk external data source
    Report Entry Method drop down list Select an option for specifying the report to create the schema table from:
  • Fully qualified name
  • Select from list
  • Report’s Fully Qualified Name text box This property appears when the value of Report Entry Method is Fully qualified name. Enter the full name of the report.
    Report drop down list This property appears when the value of Report Entry Method is Select from list. Select an available report from the list.
    Start Date drop down list Select the time window of the report
    Full Load Start Date text box This property appears when the value of Start Date is Custom Date. Enter the custom date in yyyy-mm-dd format.
    Page Size (in rows) text box Enter the number of records in a page for the REST API request
    Callback toggle Enables the Callback URL field
    Callback URL text box This property appears when the Callback toggle is enabled. Specify the URL.

    Start date options

    The start date options apply to unscheduled reports only:

    • Report’s Default Start Time: This option will use the default time window of the report.
    • All Time: This option will run the report to retrieve all available data without restricting the time window.
    • Custom Date: This option allows the user to enter a custom date to get the data from that date.

    For scheduled reports, data is extracted from the last load job. In other words, incremental and full loading is supported for unscheduled reports, and full loading only is supported for scheduled reports.

    View the schema diagram with the Schema Diagram Viewer

    Here are the steps to view the schema diagram using the Schema Diagram Viewer:

    • Sign in to the Incorta Direct Data Platform.
    • In the Navigation bar, select Schema.
    • In the list of schemas, select the Splunk schema.
    • In the Schema Designer, in the Action bar, select Diagram.

    Load the schema

    Here are the steps to perform a Full Load of the Splunk schema using the Schema Designer:

    • Sign in to the Incorta Direct Data Platform.
    • In the Navigation bar, select Schema.
    • In the list of schemas, select the Splunk schema.
    • In the Schema Designer, in the Action bar, select Load → Load Now → Full.
    • To review the load status, in Last Load Status, select the date.

    Explore the schema

    With the full load of the Splunk schema complete, you can use the Analyzer to explore the schema, create your first insight, and save the insight to a new dashboard.

    To open the Analyzer from the schema, follow these steps:

    • In the Navigation bar, select Schema.
    • In the Schema Manager, in the List view, select the Splunk schema.
    • In the Schema Designer, in the Action bar, select Explore Data.

    © Incorta, Inc. All Rights Reserved.