Upgrade to Incorta 4.8

This document details how to upgrade a standalone Incorta Cluster from 4.6 or 4.7 to 4.8.

For details about how to upgrade from 4.2, 4.3, 4.4, or 4.5 to 4.8, please review Upgrade from Incorta 4.2 to 4.9.

Upgrade from Incorta 4.6 or 4.7

In order to upgrade for Incorta 4.6 or 4.7, you will need to:

• Prepare for the upgrade
• Stop the Incorta cluster
• Upgrade the Incorta cluster
• Verify the successfully upgrade

Prepare for the upgrade

In order to prepare for the upgrade, you must first download several files and tools to help resolve any potential issues with the schemas, business schemas, session variables, and dashboards for all the tenants in the Incorta cluster.

Pause all scheduled jobs

Enable this setting to pause active scheduled schema loads, dashboards, and data alerts. This is helpful when importing or exporting an existing tenant. Here are the steps to enable this option as default tenant configuration:

• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Cluster Configurations.
• In the panel tabs, select Default Tenant Configurations.
• In the left pane, select Data Loading.
• Enable the Pause Scheduled Jobs setting.
• Select Save.

Create an export of all tenants

Create an export of all tenants using the Tenant Management Tool (TMT). The default installation path for the TMT is ~/IncortaAnalytics/cmc/tmt.

./tmt.sh -clnm <cluster-name> --export <tenant-name>  <export-file.zip>

Example

./tmt.sh -clnm localCluster --export ebs_cloud ebscloud.zip

Resolve issues with Alias tables

To resolve issues with Alias tables, you must download and run the Alias Sync Tool for each tenant in your cluster using the following steps:

• Download Alias Sync Tool
• Copy the alias_sync.py file to the IncortaNode/bin directory
• Run the script python alias_sync.py <Incorta_URL> <Tenant> <username> <password>

Example

python alias_sync.py https://cloud.incorta.com/incorta/ ebsgold admin admin

Identify formula issues with the Formula Validation Tool

The Formula Validation Tool checks the formula syntax in dashboards, business schemas, and schemas. For example, the tool identifies issues with formula columns and runtime security filters in a schema table. One such issue is with aggregation formulas and missing commas between input values of the type integers, longs, and doubles. In previous builds of Incorta, the Formula Builder accepted spaces between input values. The Formula Validation Tool identifies this and other issues with formula syntax.

IMPORTANT:

The Formula Validation Tool does not identify issues with load filters or session variables. In order to identify issues with formulas that depend on presentation variables, you must supply a username and password for a user that belongs to the SuperUser role. This SuperUser must have View access permissions for all dashboards as this access allows the Formula Validation Tool to read and parse all presentation variables.

In its initial execution, the Formula Validation Tool creates a timestamp directory with the following output files:

• extractedFormulas.tsv
• failedFormulas.tsv
• succeededFormulas.html
• succeededFormulas.tsv

Each file captures specific details about the formulas in a given tenant: extracted formulas, formulas that successfully passed validation, and also formulas that failed validation. For example, the failedFormulas.tsv file contains the following columns:

• Formula Type
• Context Formula Field
• Formula Label
• Report Name
• Report GUID
• Link
• Error Message
• Formula Text
• Valid Syntax

NOTE:

If the user does not have View access permission to all dashboards, the Link column for some formula rows may contain “UNAUTHORIZED ACCESS”.

Using the Error Message field, resolve all issues in the failedFormulas.tsv file. In many cases, resolving one instance of an issue will resolve issues in dependent objects.

To determine if you successfully resolved all the issues with formulas that failed validation, you must again export the given tenant and run the Formula Validation Tool. For the second execution, you can optionally identify the previous directory as a reference folder which in turn instructs the tool to generate an additional file that compares previous and current failedFormulas.tsv files. The comparison file is failedFormulasDifference.tsv. Depending on the number of schemas, business schemas, and dashboards in a given tenant, it may take several iterations to successfully resolve all formula validation issues.

Formula Validation Tool Releases

There are two releases of this Formula Validation Tool.

FormulaValidationTool_48.jar is the standalone JAR. The JAR is for Incorta Releases 4.8.1 and earlier. The file available for download

For Incorta Releases 4.8.2 and higher, you can find the Formula Validation Tool in in ~/IncortaAnalytics/IncortaNode/formulaParsingTool/. Within the packaged formulaParsingTool directory are three files: incorta.formula.extractor.bat, incorta.formula.extractor.jar, and incorta.formula.extractor.sh.

Both the standalone JAR and the packaged directory JAR support working with either a tenant compressed .ZIP export file or an uncompressed directory of the tenant export that includes the tenant.xml file.

Upon execution, both JARs requires the following details:

Formula validation with the standalone JAR for Incorta 4.8.1 or below

Follow these steps to download and run the standalone JAR file:

• Download the Formula Validation Tool JAR.
• Copy FormulaValidationTool_48.jar to the same parent directory of the tenant .ZIP file or the directory of the uncompressed tenant export.

• From the terminal, enter the following command:

  java -jar FormulaValidationTool_48.jar

• Enter the required values for…
• Tenant Name
• Username
• Password
• Host URL
• Incorta Version Number
• Tenant file (.zip) or folder (containing tenant.xml) absolute path
• Output Folder absolute path

• Do you have a reference to compare to?

  • If Yes, specify the Reference folder path
• View the results in the Output Folder.
• Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
• Repeat this process to confirm validation success.

Formula validation with the packaged directory for Incorta 4.8.2 and above

To run formula validation from the packaged directory, follow these steps:

• From the terminal, navigate to ~/IncortaAnalytics/IncortaNode/formulaParsingTool/.

• Enter one of the following commands:

  ./incorta.formula.extractor.bat

  java -jar incorta.formula.extractor.jar

  ./incorta.formula.extractor.sh

• Enter the required values for…

  • Tenant Name
  • Username
  • Password
  • Host URL
  • Incorta Version Number
  • Tenant file (.zip) or folder (containing tenant.xml) absolute path
  • Output Folder absolute path
  • Do you have a reference to compare to?
  • If Yes, specify the Reference folder path
• View the results in the Output Folder.
• Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
• Repeat this process to confirm validation success.

Identify issues with the Inspector Tool

The Incorta Inspector Tool is available in the CMC. The Incorta Inspector Tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables.

Download the Inspector Tool Dashboards, Schema, and Business Schema

For a given tenant with the Inspector Tool Scheduler enabled, you need to first download the Inspector Tool Dashboards, Schema, and Business Schema. Here are the steps to download the Inspector Tool Dashboards, Schema, and Business Schema:

• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Cluster Configurations.
• In the panel tabs, select Default Tenant Configurations.
• In the left pane, select Incorta Labs.
• In the right pane, in the description of the Inspector Tool Scheduler, select the download link.

• In the Box folder, select the following files to download:

  • dashboards.zip
  • business_schema.zip
  • schema.zip

After successfully downloading the zip files, you must import the schema, business schema, and dashboards into a given tenant.

Enabling the Inspector Tool

Having successfully imported the Inspector Tool Dashboards, Schema, and Business Schema, you can now enable the Inspector Tool Scheduler in the Cluster Management Console as both a default tenant configuration or a tenant configuration. Here are the steps to enable this option for a specific tenant configuration in the CMC:

• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select the Tenants tab.
• In the Tenant list, for the given tenant, select Configure.
• In the left pane, select Incorta Labs.
• In the right pane, toggle Inspector Tool Scheduler to enabled.
• Specify the schedule.
• Select Save.

Load data and Review errors

Next, you will need to load the data for the schema and review a dashboard for errors and warnings.

• For each Inspector Schema in each tenant, perform a Full Load.
• In the Inspector dashboard folder, view the 1-Validation UseCases dashboard.
• In Error & Warning Codes, identify all the Severity-1 issues.

• Resolve all the Severity-1 issues

Create a backup of the Incorta Metadata Database

To create a backup of the incorta metadata database, use mysqldump command line utility:

mysqldump -u [user] -p [database_name] > [filename].sql

Example

Here is example with the MySql user as root with the password incorta_root:

mysqldump -uroot -pincorta_root incorta_metadata > /tmp/incorta_metadata.sql

Create a backup of the Incorta installation directory

To create a backup of the Incorta installation directory, use the following command:

zip -r <Zip-File>.zip <Directory to be Zipped>

Example

zip -r Incorta-Installatio_Backup.zip /incorta/IncortaAnalytics

Create a backup of the Apache Spark configuration files

Create a backup of the following spark configuration files present in the $SPARK_HOME/conf directory:

• spark-defaults.conf
• spark-env.sh

Example

cd $SPARK_HOME/conf
zip -r Spark_Conf_Backup.zip spark-defaults.conf spark-env.sh

Stop the Incorta Cluster

Next, stop the Incorta Cluster including the Cluster Management Console, Apache Spark, and Incorta Nodes.

Stop the Incorta Node

The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with the stopNode.sh shell script:

./stopNode.sh

Stop Apache Spark

The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with the stopSpark.sh shell script:

./stopSpark.sh

Stop the Cluster Management Console (CMC)

The default directory for the CMC is /home/incorta/IncortaAnalytics/cmc. Stop the CMC with the stop-cmc.sh shell script:

./stop-cmc.sh

Upgrade the Incorta cluster

Next, you can run the run the Java incorta-installer JAR file:

java -jar incorta-installer.jar -i console

In the Incorta Installer console, enter these values for a standalone (Typical) upgrade:

Welcome                     : Enter
License Agreement/Copyright : Enter
License Agreement/Copyright : Y
Installation Type           : 2- Upgrade
Installation Set            : 1- Typical
Choose Installation Folder  : Enter- Default
Installation Status         : Enter

Upgrade the Cluster Metadata Database

After upgrading, sign in to the Cluster Management Console (CMC) and upgrade the Cluster Metadata database. To sign in to the CMC, visit your CMC host at one of the following:

• http://<Public_IP>:6060/cmc
• http://<Public_DNS>:6060/cmc
• http://<Private_IP>:6060/cmc
• http://<Private_DNS>:6060/cmc

The default port for the CMC is 6060. Sign in to the CMC using your administrator username and password.

To upgrade the Cluster Metadata database, follow the steps:

• In the Navigation bar, select Clusters.
• For each cluster name in the Cluster list, in the Actions column, select Upgrade Cluster Metadata.

NOTE:

A dialog indicates to restart the Incorta Services. In the dialog, select OK.

Upgrade Apache Spark

If the Incorta Cluster is using an external Apache Spark environment, you must also upgrade the Apache Spark environment by following these steps:

• Zip the bundled spark directory under IncortaNode:

  zip -r Incorta-Bundled-Spark.zip /incorta/IncortaAnalytics/IncortaNode/spark

• Zip the bundled hadoop directory under IncortaNode: \

  zip -r Incorta-Bundled-Hadoop.zip /incorta/IncortaAnalytics/IncortaNode/hadoop

• Remove the spark directory completely in the eternal Apache Spark environment.
• Unzip Incorta-Bundled-Spark.zip to recreate the Spark environment.
• Unzip Incorta-Bundled-Hadoop.zip to recreate the Hadoop environment.

Review the Upgrade logs

Review the following logs to make sure that there are no critical errors with the Incorta Upgrade.

• Installer logs (resides just beside the installer package)

• Incorta Node upgrade logs

  <IncortaNode>/logs/

• CMC logs

  ls -l <CMC>/logs/

Verify the successfully upgrade to Incorta 4.8

Unpause all scheduled jobs

Here are the steps to disable this option as a default tenant configuration:

• In the Navigation bar, select Clusters.
• In the cluster list, select a Cluster name.
• In the canvas tabs, select Cluster Configurations.
• In the panel tabs, select Default Tenant Configurations.
• In the left pane, select Data Loading.
• Disable the Pause Scheduled Jobs setting.
• Select Save.

Review and Monitor scheduled jobs

Sign in to each tenant and review the scheduled jobs.

Run the Inspector Tool

Run Inspector Tool again to make sure that there are no Severity-1 issues introduced as part of the upgrade.

Export Tenants

In order to create a tenant backup of the successful upgrade, either using the CMC or the TMT, export each tenant.