Upgrade to Incorta 4.8

This document details how to upgrade a standalone Incorta Cluster from 4.2, 4.3, 4.4, or 4.5 to 4.8.

For details about how to upgrade from 4.6 or 4.7 to 4.8, please review Upgrade from Incorta 4.6 to 4.8.

Upgrade from Incorta 4.2, 4.3, 4.4, or 4.5

In order to upgrade for Incorta 4.2, 4.3, 4.4, or 4.5, you will need to:

• Prepare for the upgrade
• Stop the Incorta cluster
• Upgrade the Incorta cluster
• Verify the successfully upgrade

Prepare for the upgrade

In order to prepare for the upgrade, you must first download several files and tools to help resolve any potential issues with the schemas, business schemas, session variables, and dashboards for all the tenants in the Incorta cluster.

Create an export of all tenants

Create an export of all tenants using the Tenant Management Tool (TMT). The default installation path for the TMT is ~/IncortaAnalytics/cmc/tmt.

./tmt.sh -clnm <cluster-name> --export <tenant-name>  <export-file.zip>

Example

./tmt.sh -clnm localCluster --export ebs_cloud ebscloud.zip

Resolve issues with Alias tables

To resolve issues with Alias tables, you must download and run the Alias Sync Tool iteratively for each tenant in your cluster using the following steps:

• Download the Alias Sync Tool
• Copy the alias_sync.py file to the IncortaNode/bin directory
• Run the script python alias_sync.py <Incorta_URL> <Tenant> <username> <password>

Example

python alias_sync.py https://rel4_3.dev1.incorta.cloud/incorta/ ebsgold admin admin

Identify formula issues with the Formula Validation Tool

The Formula Validation Tool checks the formula syntax in dashboards, business schemas, and schemas. For example, the tool identifies issues with formula columns and runtime security filters in a schema table. One such issue is with aggregation formulas and missing commas between input values of the type integers, longs, and doubles. In previous builds of Incorta, the Formula Builder accepted spaces between input values. The Formula Validation Tool identifies this and other issues with formula syntax.

IMPORTANT:

The Formula Validation Tool does not identify issues with load filters or session variables. In order to identify issues with formulas that depend on presentation variables, you must supply a username and password for a user that belongs to the SuperUser role. This SuperUser must have View access permissions for all dashboards as this access allows the Formula Validation Tool to read and parse all presentation variables.

In its initial execution, the Formula Validation Tool creates a timestamp directory with the following output files:

• extractedFormulas.tsv
• failedFormulas.tsv
• succeededFormulas.html
• succeededFormulas.tsv

Each file captures specific details about the formulas in a given tenant: extracted formulas, formulas that successfully passed validation, and also formulas that failed validation. For example, the failedFormulas.tsv file contains the following columns:

• Formula Type
• Context Formula Field
• Formula Label
• Report Name
• Report GUID
• Link
• Error Message
• Formula Text
• Valid Syntax

NOTE:

If the user does not have View access permission to all dashboards, the Link column for some formula rows may contain “UNAUTHORIZED ACCESS”.

Using the Error Message field, resolve all issues in the failedFormulas.tsv file. In many cases, resolving one instance of an issue will resolve issues in dependent objects.

To determine if you successfully resolved all the issues with formulas that failed validation, you must again export the given tenant and run the Formula Validation Tool. For the second execution, you can optionally identify the previous directory as a reference folder which in turn instructs the tool to generate an additional file that compares previous and current failedFormulas.tsv files. The comparison file is failedFormulasDifference.tsv. Depending on the number of schemas, business schemas, and dashboards in a given tenant, it may take several iterations to successfully resolve all formula validation issues.

Formula Validation Tool Releases

There are two releases of this Formula Validation Tool.

FormulaValidationTool_48.jar is the standalone JAR. The JAR is for Incorta Releases 4.8.1 and earlier. The file available for download

For Incorta Releases 4.8.2 and higher, you can find the Formula Validation Tool in in ~/IncortaAnalytics/IncortaNode/formulaParsingTool/. Within the packaged formulaParsingTool directory are three files: incorta.formula.extractor.bat, incorta.formula.extractor.jar, and incorta.formula.extractor.sh.

Both the standalone JAR and the packaged directory JAR support working with either a tenant compressed .ZIP export file or an uncompressed directory of the tenant export that includes the tenant.xml file.

Upon execution, both JARs requires the following details:

Formula validation with the standalone JAR for Incorta 4.8.1 or below

Follow these steps to download and run the standalone JAR file:

• Download the Formula Validation Tool JAR.
• Copy FormulaValidationTool_48.jar to the same parent directory of the tenant .ZIP file or the directory of the uncompressed tenant export.

• From the terminal, enter the following command:

  java -jar FormulaValidationTool_48.jar

• Enter the required values for…
• Tenant Name
• Username
• Password
• Host URL
• Incorta Version Number
• Tenant file (.zip) or folder (containing tenant.xml) absolute path
• Output Folder absolute path

• Do you have a reference to compare to?

  • If Yes, specify the Reference folder path
• View the results in the Output Folder.
• Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
• Repeat this process to confirm validation success.

Formula validation with the packaged directory for Incorta 4.8.2 and above

To run formula validation from the packaged directory, follow these steps:

• From the terminal, navigate to ~/IncortaAnalytics/IncortaNode/formulaParsingTool/.

• Enter one of the following commands:

  ./incorta.formula.extractor.bat

  java -jar incorta.formula.extractor.jar

  ./incorta.formula.extractor.sh

• Enter the required values for…

  • Tenant Name
  • Username
  • Password
  • Host URL
  • Incorta Version Number
  • Tenant file (.zip) or folder (containing tenant.xml) absolute path
  • Output Folder absolute path
  • Do you have a reference to compare to?
  • If Yes, specify the Reference folder path
• View the results in the Output Folder.
• Sign in to the Incorta Analytics Service and resolve all issues in failedFormulas.tsv.
• Repeat this process to confirm validation success.

Identify issues with the Inspector Tool

The Incorta Inspector Tool is a lineage and a consistency check tool. Follow these steps to download and run the Inspector Tool for each tenant in your cluster.

• Download the Inspector Tool
• Unzip the file

• In the Inspector_1_0_63_Build_971 directory, edit the config.properties file as required:

  incortaDir=./example
  enginePort=5436
  port=8080
  offlineDataDir=./example.zip
  caseSensitiveWords=
  username=admin
  sparkPort=5442
  timeout=600000
  tenant=demo
  password=admin
  host=http://localhost

• Next, run the Inspector_1_0_63_Build_971.jar to create the related Inspector CSV files for each tenant:

  java -jar Inspector_1_0_63_Build_971.jar -v -l -p  <tenantname>/data

• From the Inspector_1_0_63_Build_971/importdirectory, import the following into the tenant

  • Import Schema.zip into Schema
  • Import business_schema.zip into Business Schemas
  • Import Dashboards.zip into Content
• For each Inspector Schema in each tenant, perform a** Full Load.**
• In the Inspector dashboard folder, view the 1-Validation UseCases dashboard.
• In Error & Warning Codes, identify all the Severity-1 issues.

• Resolve all the Severity-1 issues

Create a backup of the Incorta Metadata Database

To create a backup of the incorta metadata database, use mysqldump command line utility:

mysqldump -u [user] -p [database_name] > [filename].sql

Example

Here is example with the MySql user as root with the password incorta_root:

mysqldump -uroot -pincorta_root incorta_metadata > /tmp/incorta_metadata.sql

Create a backup of the Incorta installation directory

To create a backup of the Incorta installation directory, use the following command

zip -r <Zip-File>.zip <Directory to be Zipped>

Example

zip -r Incorta-Installation_Backup.zip /incorta/IncortaAnalytics

Create a backup of the Apache Spark configuration files

Create a backup of the following spark configuration files present in the $SPARK_HOME/conf directory:

• spark-defaults.conf
• spark-env.sh

Example

cd $SPARK_HOME/conf
zip -r Spark-Conf_Backup.zip spark-defaults.conf spark-env.sh

Stop the Incorta Cluster

Next, stop the Incorta Cluster including the Cluster Management Console, Apache Spark, and Incorta Nodes.

Stop the Incorta Node

The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with the stopNode.sh shell script:

./stopNode.sh

Stop Apache Spark

The default directory for the CMC is /home/incorta/IncortaAnalytics/IncortaNode. Stop the Incorta Node with thestopSpark.sh shell script:

./stopSpark.sh

Stop the Cluster Management Console (CMC)

The default directory for the CMC is /home/incorta/IncortaAnalytics/cmc. Stop the CMC with thestop-cmc.sh shell script:

./stop-cmc.sh

Upgrade the Incorta cluster

Next, you can run the run the Java incorta-installer JAR file:

java -jar incorta-installer.jar -i console

In the Incorta Installer console, enter these values for a standalone (Typical) upgrade:

Welcome                     : Enter
License Agreement/Copyright : Enter
License Agreement/Copyright : Y
Installation Type           : 2- Upgrade
Installation Set            : 1- Typical
Choose Installation Folder  : Enter- Default
Installation Status         : Enter

Upgrade the Cluster Metadata Database

After upgrading, sign in to the Cluster Management Console (CMC) and upgrade the Cluster Metadata database. To sign in to the CMC, visit your CMC host at one of the following:

• http://<Public_IP>:6060/cmc
• http://<Public_DNS>:6060/cmc
• http://<Private_IP>:6060/cmc
• http://<Private_DNS>:6060/cmc

The default port for the CMC is 6060. Sign in to the CMC using your administrator username and password. To upgrade the Cluster Metadata database, follow the steps:

• In the Navigation bar, select Clusters.
• For each cluster name in the Cluster list, in the Actions column, select Upgrade Cluster Metadata.

NOTE:

A dialog indicates to restart the Incorta Services. In the dialog, select OK.

Upgrade Apache Spark

If the Incorta Cluster is using an external Apache Spark environment, you must also upgrade the Apache Spark environment by following these steps:

• Zip the bundled spark directory under IncortaNode:

  zip -r Incorta-Bundled-Spark.zip /incorta/IncortaAnalytics/IncortaNode/spark

• Zip the bundled hadoop directory under IncortaNode:

  zip -r Incorta-Bundled-Hadoop.zip /incorta/IncortaAnalytics/IncortaNode/hadoop

• Remove the spark directory completely in the eternal Apache Spark environment.
• Unzip Incorta-Bundled-Spark.zipto recreate the Spark environment.
• Unzip Incorta-Bundled-Hadoop.zipto recreate the Hadoop environment.
• Copy to the $SPARK_HOME/conf directory the spark configuration files, spark-defaults.conf and spark-env.sh

Optional: Run the Migration Snapshot Tool Script

You can skip the Migration Snapshot Tool script if you do not have a high volume of data in each tenant in your cluster. Instead of running the scripts, you can simply perform a full load of all the schemas.

IMPORTANT:

Either using the CMC or shell scripts, stop the Analytics and Loader Services. The Incorta Metadata database must be available and running.

Create a backup of Parquet and Snapshot directories

Before running the Migration Snapshot script, backup the tenant parquet and snapshot folders.

Here are examples:

Examples

zip -r tenant-parquet-backup.zip /incorta/IncortaAnalytics/Tenants/ebs_cloud/parquet
zip -r tenant-snapshot-backup.zip /incorta/IncortaAnalytics/Tenants/ebs_cloud/snapshots

Run migrateSnapshotsTool.sh

To run the shell script, execute the following:

cd $INCORTA_HOME/IncortaNode
./migrateSnapshotsTool.sh

Provide the metadata database URL, username & password, and tenants to migrate. Follow the instructions and use defaults for the other parameters. Optionally, you can use migration.properties file and pass it as a parameter to the tool and pass it as a parameter.

cd $INCORTA_HOME/IncortaNode
./migrateSnapshotsTool.sh migration.properties

If any table fails to migrate, the tool will recommend you to take one of the following actions

• Load the table from staging
• Perform a full load on the table

Review the Migration Snapshot Tool logs

The logs are available as follows:

• Redirect of stdout screen output

  $INCORTA_HOME/IncortaNode/migrationTool.20190726-151032.log

• Log across all tenants

  $INCORTA_HOME/IncortaNode/migration/incorta-migration.2019-07-26.log

• Log for specific tenants

  $INCORTA_HOME/IncortaNode/migration/<Tenant_Name>/incorta-migration.2019-07-26.log

Review the migration snapshot tool log files for issues:

• If you see the error message that a particular table was not able to migrate, it will ask you to load the table from staging.
• Ignore the message saying Duplicate Joins.

Start the Incorta Services

Either using the CMC or the shell scripts, start the Analytics and Loader Services for the cluster.

Review the Upgrade logs

Review the following logs to make sure that there are no critical errors with the Incorta Upgrade.

• Installer logs (resides just beside the installer package)

• Incorta Node upgrade logs

  ls -l <IncortaNode>/logs/

• CMC logs

  ls -l <CMC>/logs/

Verify the successfully upgrade to Incorta 4.8

Sign into each tenant and confirm that scheduled jobs are running.

Review and Monitor scheduled jobs

Sign in to each tenant and review the scheduled jobs.

Run the Inspector Tool

Run Inspector Tool again to make sure that there are no Severity-1 issues introduced as part of the upgrade.

Export Tenants

In order to create a tenant backup of the successful upgrade, either using the CMC or the TMT, export each tenant.