Upgrade to Incorta 4.8

This document details how to upgrade a standalone Incorta Cluster from 4.2, 4.3, 4.4, or 4.5 to 4.8.

For details about how to upgrade from 4.6 or 4.7 to 4.8, please review Upgrade from Incorta 4.6 to 4.8.

Upgrade from Incorta 4.2, 4.3, 4.4, or 4.5

In order to upgrade for Incorta 4.2, 4.3, 4.4, or 4.5, you will need to:

  • Prepare for the upgrade
  • Stop the Incorta cluster
  • Upgrade the Incorta cluster
  • Verify the successfully upgrade

Prepare for the upgrade

In order to prepare for the upgrade, you must first download several files and tools to help resolve any potential issues with the schemas, business schemas, session variables, and dashboards for all the tenants in the Incorta cluster.

Create an export of all tenants

Create an export of all tenants using the Tenant Management Tool (TMT). The default installation path for the TMT is ~/IncortaAnalytics/cmc/tmt.

./tmt.sh -clnm <cluster-name> --export <tenant-name>  <export-file.zip>
./tmt.sh -clnm localCluster --export ebs_cloud ebscloud.zip

Resolve issues with Alias tables

To resolve issues with Alias tables, you must download and run the Alias Sync Tool iteratively for each tenant in your cluster using the following steps:

  • Download the Alias Sync Tool
  • Copy the alias_sync.py file to the IncortaNode/bin directory
  • Run the script python alias_sync.py <Incorta_URL> <Tenant> <username> <password>

python alias_sync.py https://rel4_3.dev1.incorta.cloud/incorta/ ebsgold admin admin

Identify formula issues with the Formula Validation Tool

The Formula Validation Tool checks the formula syntax in dashboards, business schemas, and schemas. For example, the tool identifies issues with formula columns and runtime security filters in a schema table. One such issue is with aggregation formulas and missing commas between input values of the type integers, longs, and doubles. In previous builds of Incorta, the Formula Builder accepted spaces between input values. The Formula Validation Tool identifies this and other issues with formula syntax.


The Formula Validation Tool does not identify issues with load filters or session variables. In order to identify issues with formulas that depend on presentation variables, you must supply a username and password for a user that belongs to the SuperUser role. This SuperUser must have View access permissions for all dashboards as this access allows the Formula Validation Tool to read and parse all presentation variables.

In its initial execution, the Formula Validation Tool creates a timestamp directory with the following output files:

  • extractedFormulas.tsv
  • failedFormulas.tsv
  • succeededFormulas.html
  • succeededFormulas.tsv

Each file captures specific details about the formulas in a given tenant: extracted formulas, formulas that successfully passed validation, and also formulas that failed validation. For example, the failedFormulas.tsv file contains the following columns:

  • Formula Type
  • Context Formula Field
  • Formula Label
  • Report Name
  • Report GUID
  • Link
  • Error Message
  • Formula Text
  • Valid Syntax


If the user does not have View access permission to all dashboards, the Link column for some formula rows may contain “UNAUTHORIZED ACCESS”.

Using the Error Message field, resolve all issues in the failedFormulas.tsv file. In many cases, resolving one instance of an issue will resolve issues in dependent objects.

To determine if you successfully resolved all the issues with formulas that failed validation, you must again export the given tenant and run the Formula Validation Tool. For the second execution, you can optionally identify the previous directory as a reference folder which in turn instructs the tool to generate an additional file that compares previous and current failedFormulas.tsv files. The comparison file is failedFormulasDifference.tsv. Depending on the number of schemas, business schemas, and dashboards in a given tenant, it may take several iterations to successfully resolve all formula validation issues.

Formula Validation Tool Releases

There are two releases of this Formula Validation Tool.

Release Supports Incorta Version Notes
FormulaValidationTool_48.jar <= 4.8.1
  • Download from Google Drive
  • Standalone Jar
  • incorta.formula.extractor.jar >= 4.8.2
  • Included in the installer
  • Run shell script
  • FormulaValidationTool_48.jar is the standalone JAR. The JAR is for Incorta Releases 4.8.1 and earlier. The file available for download

    For Incorta Releases 4.8.2 and higher, you can find the Formula Validation Tool in in ~/IncortaAnalytics/IncortaNode/formulaParsingTool/. Within the packaged formulaParsingTool directory are three files: incorta.formula.extractor.bat, incorta.formula.extractor.jar, and incorta.formula.extractor.sh.

    Both the standalone JAR and the packaged directory JAR support working with either a tenant compressed .ZIP export file or an uncompressed directory of the tenant export that includes the tenant.xml file.

    Upon execution, both JARs requires the following details:

    Input Parameter Example Description
    Tenant Name myTenant The tenant name for the tenant export
    Username admin The username for a user that belongs to the SuperUser role
    Password Incorta#1 Password for username
    Host URL http://incorta.mycompany.com:8080/incorta The Incorta Analytics Service host for
    the tenant
    Incorta Version Number 4.5.3 The Incorta release version number.
    This becomes part of the export
    folder name for the validation results
    Tenant file (.zip) or
    folder (containing tenant.xml)
    absolute path
    /tmp/2020-07-29-16-08-50/ebs_gold.zip A path to the .ZIP
    or the folder that contains the tenant.xml file
    Output Folder absolute
    /tmp/ A path to an existing directory
    for the formula validation tool results
    Do you have a
    reference to compare to?
    Y Y is Yes. N is No.
    Y requires specifying an existing
    validation result for comparison
    Reference folder path /tmp/Rel4.5.3_myTenant_1595989474708 A path to a previously generated
    formula validation output;
    required for comparison usage
    Formula validation with the standalone JAR for Incorta 4.8.1 or below

    Follow these steps to download and run the standalone JAR file:

    • Download the Formula Validation Tool JAR.
    • Copy FormulaValidationTool_48.jar to the same parent directory of the tenant .ZIP file or the directory of the uncompressed tenant export.
    • From the terminal, enter the following command:

      java -jar FormulaValidationTool_48.jar
    • Enter the required values for…
    • Tenant Name
    • Username
    • Password
    • Host URL
    • Incorta Version Number
    • Tenant file (.zip) or folder (containing tenant.xml) absolute path
    • Output Folder absolute path
    • Do you have a reference to compare to?

      • If Yes, specify the Reference folder path
    • View the results in the Output Folder.
    • Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
    • Repeat this process to confirm validation success.
    Formula validation with the packaged directory for Incorta 4.8.2 and above

    To run formula validation from the packaged directory, follow these steps:

    • From the terminal, navigate to ~/IncortaAnalytics/IncortaNode/formulaParsingTool/.
    • Enter one of the following commands:

      java -jar incorta.formula.extractor.jar
    • Enter the required values for…

      • Tenant Name
      • Username
      • Password
      • Host URL
      • Incorta Version Number
      • Tenant file (.zip) or folder (containing tenant.xml) absolute path
      • Output Folder absolute path
      • Do you have a reference to compare to?
      • If Yes, specify the Reference folder path
    • View the results in the Output Folder.
    • Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
    • Repeat this process to confirm validation success.

    Identify issues with the Inspector Tool

    The Incorta Inspector Tool is a lineage and a consistency check tool. Follow these steps to download and run the Inspector Tool for each tenant in your cluster.

    • Download the Inspector Tool
    • Unzip the file
    • In the Inspector_1_0_63_Build_971 directory, edit the config.properties file as required:

    • Next, run the Inspector_1_0_63_Build_971.jar to create the related Inspector CSV files for each tenant:

      java -jar Inspector_1_0_63_Build_971.jar -v -l -p  <tenantname>/data
    • From the Inspector_1_0_63_Build_971/importdirectory, import the following into the tenant

      • Import Schema.zip into Schema
      • Import business_schema.zip into Business Schemas
      • Import Dashboards.zip into Content
    • For each Inspector Schema in each tenant, perform a** Full Load.**
    • In the Inspector dashboard folder, view the 1-Validation UseCases dashboard.
    • In Error & Warning Codes, identify all the Severity-1 issues.
    • Resolve all the Severity-1 issues

    Create a backup of the Incorta Metadata Database

    To create a backup of the incorta metadata database, use mysqldump command line utility:

    mysqldump -u [user] -p [database_name] > [filename].sql

    Here is example with the MySql user as root with the password incorta_root:

    mysqldump -uroot -pincorta_root incorta_metadata > /tmp/incorta_metadata.sql

    Create a backup of the Incorta installation directory

    To create a backup of the Incorta installation directory, use the following command

    zip -r <Zip-File>.zip <Directory to be Zipped>
    zip -r Incorta-Installation_Backup.zip /incorta/IncortaAnalytics

    Create a backup of the Apache Spark configuration files

    Create a backup of the following spark configuration files present in the $SPARK_HOME/conf directory:

    • spark-defaults.conf
    • spark-env.sh
    cd $SPARK_HOME/conf
    zip -r Spark-Conf_Backup.zip spark-defaults.conf spark-env.sh

    Stop the Incorta Cluster

    Next, stop the Incorta Cluster including the Cluster Management Console, Apache Spark, and Incorta Nodes.

    Stop the Incorta Node

    The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with the stopNode.sh shell script:


    Stop Apache Spark

    The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with thestopSpark.sh shell script:


    Stop the Cluster Management Console (CMC)

    The default directory for the CMC is /home/incorta/IncortaAnalytics/cmc. Stop the CMC with thestop-cmc.sh shell script:


    Upgrade the Incorta cluster

    Next, you can run the run the Java incorta-installer JAR file:

    java -jar incorta-installer.jar -i console

    In the Incorta Installer console, enter these values for a standalone (Typical) upgrade:

    Welcome                     : Enter
    License Agreement/Copyright : Enter
    License Agreement/Copyright : Y
    Installation Type           : 2- Upgrade
    Installation Set            : 1- Typical
    Choose Installation Folder  : Enter- Default
    Installation Status         : Enter

    Upgrade the Cluster Metadata Database

    After upgrading, sign in to the Cluster Management Console (CMC) and upgrade the Cluster Metadata database. To sign in to the CMC, visit your CMC host at one of the following:

    The default port for the CMC is 6060. Sign in to the CMC using your administrator username and password. To upgrade the Cluster Metadata database, follow the steps:

    • In the Navigation bar, select Clusters.
    • For each cluster name in the Cluster list, in the Actions column, select Upgrade Cluster Metadata.


    A dialog indicates to restart the Incorta Services. In the dialog, select OK.

    Upgrade Apache Spark

    If the Incorta Cluster is using an external Apache Spark environment, you must also upgrade the Apache Spark environment by following these steps:

    • Zip the bundled spark directory under IncortaNode:

      zip -r Incorta-Bundled-Spark.zip /incorta/IncortaAnalytics/IncortaNode/spark
    • Zip the bundled hadoop directory under IncortaNode:

      zip -r Incorta-Bundled-Hadoop.zip /incorta/IncortaAnalytics/IncortaNode/hadoop
    • Remove the spark directory completely in the eternal Apache Spark environment.
    • Unzip Incorta-Bundled-Spark.zipto recreate the Spark environment.
    • Unzip Incorta-Bundled-Hadoop.zipto recreate the Hadoop environment.
    • Copy to the $SPARK_HOME/conf directory the spark configuration files, spark-defaults.conf and spark-env.sh

    Optional: Run the Migration Snapshot Tool Script

    You can skip the Migration Snapshot Tool script if you do not have a high volume of data in each tenant in your cluster. Instead of running the scripts, you can simply perform a full load of all the schemas.


    Either using the CMC or shell scripts, stop the Analytics and Loader Services. The Incorta Metadata database must be available and running.

    Create a backup of Parquet and Snapshot directories

    Before running the Migration Snapshot script, backup the tenant parquet and snapshot folders.

    Here are examples:

    zip -r tenant-parquet-backup.zip /incorta/IncortaAnalytics/Tenants/ebs_cloud/parquet
    zip -r tenant-snapshot-backup.zip /incorta/IncortaAnalytics/Tenants/ebs_cloud/snapshots
    Run migrateSnapshotsTool.sh

    To run the shell script, execute the following:

    cd $INCORTA_HOME/IncortaNode

    Provide the metadata database URL, username & password, and tenants to migrate. Follow the instructions and use defaults for the other parameters. Optionally, you can use migration.properties file and pass it as a parameter to the tool and pass it as a parameter.

    cd $INCORTA_HOME/IncortaNode
    ./migrateSnapshotsTool.sh migration.properties

    If any table fails to migrate, the tool will recommend you to take one of the following actions

    • Load the table from staging
    • Perform a full load on the table
    Review the Migration Snapshot Tool logs

    The logs are available as follows:

    • Redirect of stdout screen output

    • Log across all tenants

    • Log for specific tenants


    Review the migration snapshot tool log files for issues:

    • If you see the error message that a particular table was not able to migrate, it will ask you to load the table from staging.
    • Ignore the message saying Duplicate Joins.
    Start the Incorta Services

    Either using the CMC or the shell scripts, start the Analytics and Loader Services for the cluster.

    Review the Upgrade logs

    Review the following logs to make sure that there are no critical errors with the Incorta Upgrade.

    • Installer logs (resides just beside the installer package)
    • Incorta Node upgrade logs

      ls -l <IncortaNode>/logs/
    • CMC logs

      ls -l <CMC>/logs/

    Verify the successfully upgrade to Incorta 4.8

    Sign into each tenant and confirm that scheduled jobs are running.

    Review and Monitor scheduled jobs

    Sign in to each tenant and review the scheduled jobs.

    Run the Inspector Tool

    Run Inspector Tool again to make sure that there are no Severity-1 issues introduced as part of the upgrade.

    Export Tenants

    In order to create a tenant backup of the successful upgrade, either using the CMC or the TMT, export each tenant.

    © Incorta, Inc. All Rights Reserved.