Migration Upgrade from 3.x to 4.6

To upgrade from 3.x to 4.6, you must upgrade from 3.x to 4.3, then upgrade from 4.3 to 4.6. Incorta 4.6 processes information in a way that requires information in release 4.3 and later.

Follow the steps outlined here for the complete process from beginning to end.

The installation for the Incorta Unified Data Analytics Platform 3.x took place on one server, utilizing all the resources within that machine to perform all the data loading and query processing operations.

Incorta Unified Data Analytics Platform 4.6 enhanced the installation process, separating data loading from query processing services. This allows for query processing while data loading is in progress, minimizing dashboards downtime. Also, this provides users with an option to install as many loader and analytic services as they need on separate nodes providing a more scalable architecture.

The following is the high level process. Details for each step follow the high-level process:

A: Prepare for Installation

  1. Create a shared folder that all Incorta analytics nodes, loader nodes, and Spark clusters can read. The following steps call this folder 4.6 <TENANT_SHARED_FOLDER>.
  2. Back up your Incorta Release 3.x metadata database.
  3. Back up the tenant folders. For each tenant, create a folder with the following folders and files:

    • Direct Data Mapping (Snapshot) Files <TENANT_SHARED_FOLDER>/<TENANT_NAME>/snapshots
    • Parquet Files <TENANT_SHARED_FOLDER><TENANT_NAME>/parquet
    • Compacted Parquet Files <TENANT_SHARED_FOLDER><TENANT_NAME>/compacted
    • Data Files <TENANT_SHARED_FOLDER><TENANT_NAME>/data
    • Schema Folders <TENANT_SHARED_FOLDER><TENANT_NAME>/<SchemaName>
  4. Ensure that you can access your metadata database from both the 3.x node and the 4.3 node.
  5. Determine the upgrade type, which is one of the following:

    • Single node installation. One node containing both the analytics and loader services. This is not recommended for UAT or production environments.
    • Two node installation. One node for the analytics service, one node for the loader service.
    • Multi-node installation. More than two nodes for either or both of the analytics and loader services.
    • High availability installation. At least 2 nodes with duplicates of the analytics service and at least 2 nodes with duplicates of the loader service, for redundancy.
  6. Open the ports:

    • HTTP ports: 8005, 8009, 8080
    • HTTP redirect port: 8443
    • Required ports:
    • 4500 for the node agent
    • 5442, 5436 for SQL interface only
    • 7077, 7078, 9091, 9092 if you plan to use materialized views
    • 7777 to send a dashboard in HTML format by email
  7. Install and configure Java. For information on how to install and configure Java, see: Install Java.

B: Install Incorta

For information on how to install Incorta, see:

C: Upgrade from 3.x to 4.3

  1. Export the 3.x tenant by using the TMT script:
Linux Windows
<INSTALLATION_PATH>/cmc/tmt/tmt.sh -ex <TENANT_NAME> <EXPORT_FILE_NAME.zip> <INSTALLATION_PATH>/cmc/tmt/tmt.bat -ex <TENANT_NAME> <EXPORT_FILE_NAME.zip>
  1. Import the 3.x tenant into the 4.3 tenant,cd <INSTALLATION_PATH>/cmc/tmt. Both the loader service and the analytics service will shut down after the tenant completes importing.
  2. Copy the exported zip file into <INSTALLATION_PATH>/cmc/tmt folder:
Linux Windows
<INSTALLATION_PATH>/cmc/tmt/tmt.sh -i <EXPORT_FILE_NAME.zip> -clnm <CLUSTER_NAME> -op <TENANT_SHARED_FOLDER> <INSTALLATION_PATH>/cmc/tmt/tmt.bat -i <EXPORT_FILE_NAME.zip> -clnm <CLUSTER_NAME> -op <TENANT_SHARED_FOLDER>
  1. Verify that the cluster is created by signing in to the CMC to see the existing clusters: http://<cmchostname>:6060/cmc/clusters.
  2. Move the Release 3.x tenant folder to the Release 4.3 tenant shared folder. You must move Direct Data Map (snapshot) files, Parquet files, compacted Parquet files, data files, and schema folders.
  3. Run the Direct Data Mapping (snapshot) File migration tool.
Linux Windows
<incorta_installation>/IncortaNode/migrateSnapshotsTool.sh < <incorta_installation>/IncortaNode/migrateSnapshotsTool.bat
  1. Use the max off-heap size parameter to specify the amount of memory allocated for the migration tool during the migration process. If you do not set the max off-heap size parameter, Incorta sets the max off-heap-size parameter to the default value, 50 GB.
Linux Windows
./migrateSnapshotsTool.bat <3.x_SNAPSHOTS_FILE_PATH> <MAX_OFFHEAP_SIZE_GB(OPTIONAL)> ./migrateSnapshotsTool.bat <3.x_SNAPSHOTS_FILE_PATH> <MAX_OFFHEAP_SIZE_GB(OPTIONAL)>

Before performing the following steps, wait until the migration tool migrates all text and timestamp columns.

D: Start One Loader Service

IMPORTANT:

Start only one loader service. If you start more than one loader service or any other services, the upgrade will not succeed.

To start one loader service:

  1. Sign in to the CMC: http://<cmchostname>:6060/cmc/.
  2. Navigate to nodes > loaderNode > loaderService.
  3. Select Start.
  4. Verify that the loader service started by monitoring the Incorta tenant sign from the loader service by locating Signs at the following file path: <Installation_Path>/IncortaNode/services/loaderServiceindex/Signs/incorta/<tenant_name>/incorta.current_timestamp.Sign

After Incorta completes loading data from the Direct Data Mapping (snapshot) files to memory, a message displays like this:

INFO: [12:07:28] [bi | System | Startup] Tenant: bi started in: 0.11 Mins [com.incorta.server.Server.startupTenant]

Starting the loader service for the first time after you complete the steps to upgrade from 3.x to 4.3 can take longer than usual.

IMPORTANT: Do not stop, interrupt, kill, or start another loader or analytics service before the migration completes.

E: Start Other Services

After the loader service completes loading data, you can start other services (for example, the analytics services or other loader services).

F: Verify that the Upgrade Migration to 4.3 Completed Successfully

Verify that the CMC works correctly and contains your data as expected by performing the following steps:

  1. Sign in to the CMC: http://<cmchostname>:6060/cmc/.
  2. Open the Nodes tab.
  3. Open the Clusters tab.
  4. Open the Tenants tab.
  5. Verify that your migrated tenants are in the Tenant List.

Verify that your Tenants migrated as expected by performing the following steps:

  1. Open the Analytics service.
  2. Sign in to each migrated tenant.
  3. Open the Schema tab and verify that your existing schemas display.
  4. Run a dashboard to verify that it looks as expected.
  5. Load schema and verify that it loaded into the dashboard as expected.

G: Upgrade from 4.3 to 4.6

Perform the following steps to run the upgrade migration tool for Incorta Release 4.6.

  1. (For Windows installations only):

    • Install Microsoft Visual C++ 2015 redistributable.
    • Set the HADOOPHOME environmental variable to `<Incortainstallation_path>/IncortaNode/hadoop`.
    • Add to the PATH environmental variable: %HADOOP_HOME%/bin.
  2. Sign into the 4.3 CMC: http://<cmchostname>:6060/cmc/.
  3. Stop all 4.3 services.
  4. Verify that the 4.3 loader services and analytics services are not running.
  5. Backup the folder that contains your Incorta installation.
  6. Upgrade the CMC and every Incorta node to Release 4.6. by running the ./install script.
  7. Start CMC.
  8. Sign in to the CMC: http://<cmchostname>:6060/cmc/.
  9. Select the cluster.
  10. Select the Upgrade Cluster Metadata button.
  11. Run the 4.6 upgrade migration tool <IncortaNode>/migrateSnapshotsTool.sh in the 4.6/IncortaNode directory on one of the loader services.
  12. Replace Spark with the new version bundled with Incorta and reapply any custom configurations.

NOTE: You can use migration.properties to default some of the configuration parameters used for the 4.6 upgrade migration tool migrateSnapshotsTool.sh migration.properties.

  1. Follow the prompts to determine whether to run for all or specific objects.

What to Expect After Running the Upgrade Migration Tool

When the upgrade migration tool completes running, it prints a result summary for each tenant you specified. If any failures occurred they will be listed with the results.

You must view the log file (named migrationTool.<timestamp>.log in the same directory as the Incorta Node) to see the results for individual tables.

The upgrade migration tool creates backup files of Direct Data Mapping files (snapshots) for each schema:

Original Direct Data Mapping (Snapshot) File Location Backup Direct Data Mapping (Snapshot) File Location
snapshots/<schema_name>.*.zxt snapshots/backup.<timestamp>/<schema_name>.all.zxt.zip
snapshots/<schema_name>.*.zxi snapshots/backup.<timestamp>/<schema_name>.all.zxi.zip
snapshots/<schema_name>.*.zxc snapshots/backup.<timestamp>/<schema_name>.all.zxc.zip

Verify that the Upgrade Migration to 4.6 Completed Successfully

Verify that the CMC works correctly and contains your data as expected by performing the following steps:

  1. Sign in to the CMC: http://<cmchostname>:6060/cmc/
  2. Open the Nodes tab.
  3. Open the Clusters tab.
  4. Open the Tenants tab.
  5. Verify that your migrated tenants are in the Tenant List.

Verify that your Tenants upgraded as expected by performing the following steps:

  1. Open the Analytics service.
  2. Sign in to each migrated tenant.
  3. Open the Schema tab and verify that your existing schemas display.
  4. Run a dashboard to verify that it looks as expected.
  5. Load schema and verify that it loaded into the dashboard as expected.

For information on troubleshooting and known issues during upgrade, see Installation and Upgrade Troubleshooting.


© Incorta, Inc. All Rights Reserved.