Upgrade from 3.x to 4.3

The installation for Incorta Analytics 3.x took place on one server, utilizing all the resources within that machine to perform all the data loading and query processing operations. Refer to the following diagram for a visual representation of an Incorta Analytics 3.x setup:

Incorta Analytics 4.3 enhanced the installation process, separating data loading from query processing services. This allows for query processing while data loading is in progress, minimizing dashboards downtime. Also, this provides users with an option to install as many loader and analytic services as they need on separate nodes providing a more scalable architecture. Additionally, the 4.3 line of releases includes the Cluster Management Console (CMC). CMC is a graphical UI that helps users to easily manage all the nodes within a cluster, and determine which nodes to use for loading operations and which to use for analytics operations. Refer to the following diagram for a visual representation of an Incorta Analytics 4.3 setup:

With release 4.3, users have many topology options to choose from. In order to set up a cluster, choose Custom installation when running the installation wizard, then choose to install the CMC (once per cluster) or a node (server). This would be ideal for a dev and UAT environments. As for a demo/POC environment, you can choose a typical installation, which would automatically install one loader service, one analytic service, and a CMC. Zookeeper must be running, because zookeeper communicates between the loader and analytics services. The node agent for each node has to be up and running to connect it to the CMC.

To upgrade to Release 4.x from 3.x, you must install Incorta on a separate machine and then migrate your tenant.

Release 4.3 Highlights

  • Incorta 4.3 is a multi-node architecture.  
  • Determine the number of nodes you need and create a shared disk or NFS-mounted disk to share with different nodes.
  • Incorta Analytics services, the CMC, and the Loader Service require an HTTP port. Access the Analytics service and CMC using the Internet.
  • The node agent doesn’t need a web app port because you cannot access the Node agent or Loader service from the Internet.

  • Incorta recommends that you:

    • Use a Clustered Zookeeper setup with Incorta Release 4.3
    • Run Spark on a different machine than the machine running Incorta
  • Even though you can install all services on one server, this is not recommended.

Checklist

  • Servers Time Zone. The time zone in your 3.x release must be the same as the time zone on your 4.3 release nodes.
  • Create a shared folder for Release 4.3. All Release 4.3 nodes must have access to the shared folder.
  • (Optional) Back up the metadata database for your Release 3.x. You do not need to back up the metadata database because you will use a new database for Release 4.3.

  • Backup the Tenant folders. For each tenant, create a folder with the following folders and files:

    • Snapshots Files <TenantName>/snapshots
    • Parquet Files <TenantName>/parquet
    • Compacted Parquet Files <TenantName>/compacted
    • Data Files <TenantName>/data
    • Schema Folders <TenantName>/<SchemaName>

  • Prepare 4.3 Metadata Database. Store system data in a shared metadata database. Incorta Release 4.3 uses the following databases:

    • MySql v5.6
    • Oracle 12rc
    • Derby (for standard installations only)

  • Select the 4.3 topology (depending on how you installed Incorta), which is one of the following:

    • Standard installation of Incorta
    • Single node installation (1 x Analytics, 1 x Loader)
    • Two nodes installation (1 x Analytics, 1 x Loader)
    • Multi-node installation (n x Analytics, m x Loader)
    • Multi-node + DR installation (2 x n x Analytics, 2 x m x Loader)

  • Identify and open the ports you need. Open the following ports:

    • HTTP ports
    • HTTP redirect ports

  • Install and configure Java

    • Oracle JDK 1.8. Required on the CMC and all Incorta nodes.
    • Set the <JAVA_HOME> and <Path> environment variables.
    • Oracle JDK 1.8u144 or newer (except Oracle JDK1.8u151 through JDK1.8u171) Incorta does not support Open JDK 8.0 and up. If you use Spark clusters with Incorta, you must be able to access the Incorta tenant folder from each machine that the Spark cluster runs.

Migration Steps

  1. Export the 3.x Tenant using the Tenant Management Tool (TMT). Cd <INSTALLATION_PATH>/tmt ./tmt.sh -ex <TENANT_NAME> <EXPORT_FILE_NAME.zip>
  2. Import the 3.x Tenant into the 4.3 Tenant using the TMT or CMC. Both the loader service and the analytics service shut down when you import the tenant into the 4.3 instance. Cd <INSTALLATION_PATH>/cmc/tmt
  3. Copy the exported zip file into a new folder, using the following syntax: ./tmt.sh -i <EXPORT_FILE_NAME.zip> -clnm <CLUSTER_NAME> -op <TENANT_SHARED_PATH>
  4. Find the Cluster name. You can use the following syntax to find the Cluster name. cd <INSTALLATION_PATH>/cmc/cmcData/clusters cat index.properties
  5. Move the Release 3.x Tenant folder to the Release 4.3 Tenant Shared folder. You must move snapshot files, parquet files, compacted parquet files, data files, and schema folders. In the Release 3.x installation environment, run the command: cd <TENANT_PATH> <TENANT_SHARED_PATH>/<TENANT_NAME>
  6. Run the snapshots migration tool. Run the “migrateSnapshotsTool.sh” script. The migrateSnapshotsTool.sh script is in the IncortaNode folder.

Perform this step if the columns contain TEXT or Date/TIMESTAMP as the data type. The snapshot migration tool converts text columns into strings and date/timestamps into two columns.

./migrateSnapshotsTool.sh <3.x_SNAPSHOTS_FILE_PATH> <MAX_OFFHEAP_SIZE_GB(OPTIONAL)>

Use the max off-heap size parameter to specify the amount of memory allotted for the migration tool during the migration process. If you do not set the max off-heap size parameter, Incorta sets the max off-heap-size parameter to the default value, 50 GB.

Wait till the migration tool migrates all text and timestamp columns.

5. Start one Loader Service

To start a loader service:

  1. In Incorta, navigate to the CMC UI.
  2. On the Nodes tab, navigate to nodes > loaderNode > loaderService**`.
  3. Select Start.
  4. Verify that the loader service started bymonitoring the Incorta tenant log from the loader service. Locate logs at the following file path:

<Installation_Path>/IncortaNode/services/loaderServiceindex/logs/incorta/<tenant_name>/incorta.current_timestamp.log

When Incorta completes loading data from snapshot files to memory, a message displays like this:

INFO: [12:07:28] [bi | System | Startup] Tenant: bi started in: 0.11 Mins [com.incorta.server.Server.startupTenant

Notes:

  • When you start the loader service from the CMC UI, the loader service status moves through the following statuses (which come from tomcat, not Incorta):

    • Stopper
    • Processing
    • Started on refresh
  • The first time you start the loader service after migration can take longer than usual.

IMPORTANT: Do not stop, interrupt, or kill the loader service before the migration complete. Do not start another service loader or analytics service until the migration is completed.

6. Start Other Services

After migration completes, you can start other services, like analytics services and other loader services.

7. Verify that the Migration Completed Successfully

Verify that the CMC works correctly and contains your data as expected by performing the following steps:

  1. Log in to the CMC Console.
  2. Open the Nodes tab.
  3. Open the Clusters tab.
  4. Open the Tenants tab.
  5. Verify that your migrated tenants are in the Tenant List.

Verify that your Tenants migrated as expected by performing the following steps:

  1. Open the Analytics service.
  2. Log in to each migrated tenant.
  3. Open the Schema tab and verify that your existing schemas display.
  4. Run a dashboard to verify that it looks as expected.
  5. Load schema and verify that it loaded into the dashboard as expected.
Related Links
© Incorta, Inc. All Rights Reserved.