Guides → Migrate

Schema Developers, Dashboard Developers, Tenant Super Users, System Administrators, and a Cluster Management Console (CMC) Administrator migrate specified objects of a tenant, entire tenants, host files, and CMC configurations to ensure that tenant functionality is maintained between environments.

Types of Environments

Each environment represents a stage in the software development life cycle, from development to production. Migration centers around how to move content from a development stage to production. Dashboards, physical schemas, business schemas, session variables, external data sources, users and groups, tenant configurations, server configurations, and system configuration file are examples of migratable content.

For enterprise deployments of Incorta, there are typically three environments with each environment representing its own stage:

Development and the DEV environment

Schema and dashboard developers create new content in the DEV environment. The DEV environment represents a desired ideal for the business community in production. The DEV environment often has a subset copy of production data. Developers safely explore, learn, demo, develop, and unit test new content in this environment.

User Acceptance Testing and the UAT environment

Functional experts, business users, and quality assurance engineers perform tests in the UAT environment. Functional experts such as product owners, project managers and program managers, oversee the technical side of development and help ensure that specifications are met. Quality assurance engineers administer performance tests to measure optimizations. Business users know exactly what the outcome should look like. Together, these users test content in UAT to determine its acceptance for production. UAT may have a subset or copy of production data.

Production and the PROD environment

Having passed UAT, new content migrates to production in the PROD environment. Administrators and Super Users migrate content from UAT to PROD. In production, ad-hoc changes are often highly regulated or prohibited. Real-world users are dashboard consumers. Dashboard consumers personalize, filter, bookmark, and share dashboards in production.

There are ideal configurations for each of the deployment environments. When you use and maintain these configurations, you will mitigate failures during migration.

Incorta Version

You should maintain the same Incorta version across all environments. This guide will assume that the Incorta version for all of your environments is the same.

Migration Path

The ideal environment migration path is DEV → UAT → PROD. It is important to maintain consistent functionality between environments. There may be use cases where it is necessary to migrate changes from PROD back to UAT and DEV, but these cases should be minimal. An example use case is a dashboard change made directly in PROD that is migrated back to UAT and DEV. Permissible operations by environment and user role may differ by organization, and require that you establish clear change management guidelines.

A change management process should also require the following for every change:

  • A test case, proven test results, and user acceptance
  • An impact analysis, or comparison between the UAT and PROD environments

Tenant Backups

You should export a tenant backup in the DEV, UAT and PROD environments on a regular basis. You can schedule these backups to occur with a utility such as crontab. Tenant backups allow you to revert to an older, working version of the environment in the event that an issue is introduced during development or migration.

Group and User Permissions

User and group permissions are a useful way to mitigate unplanned changes in the PROD tenant and maintain data security.

In the DEV and UAT environments, group level permissions should be used to share content. Since permissions are not migrated along with content, it is simpler to manage group level permissions than individual level permissions.

In the PROD environment, primarily Administrators or Super Users should have Can Edit permissions. User accessibility use cases should be tested in the UAT environment before migrating to the PROD environment. Restrict edit controls in the PROD environment to encourage the practice of changing the Incorta tenant in DEV, and migrating the changes to UAT and PROD.

Note

If you are using Lightweight Directory Access Protocol (LDAP) or Single Sign-On (SSO), group and user integration will need to be performed on the UAT and PROD environments. Refer to Secure Login Access for LDAP and SSO configurations.

Prerequisites for Migration

Run the Inspector Tool

The Inspector tool checks the lineage references of Incorta metadata objects including tables, schemas, business schemas, business schema views, dashboards, and session variables. For the purposes of migration, the CMC administrator should run the Inspector tool before and after a migration. The Inspector Tool will identify broken references, unused objects, and other issues that may result from a migration. Refer to the Inspector Tool documentation for additional information on how to run the tool and view results.

Run the Formula Validation Tool

The Formula Validation tool identifies issues with the formula expression syntax in dashboards, business schemas, and schemas. In order to validate the formula expression syntax, the tool requires a tenant export file. The CMC Administrator can create the tenant export using the Cluster Management Console (CMC) or the System Administrator can create the tenant export using the Tenant Management Tool (TMT). For the purposes of migration, the System Administrator should run the Formula Validation tool before and after a migration. Refer to the Formula Validation Tool documentation for additional information on how to run the tool and view results.

Types of Migration

Migrate as a Developer

Developers typically migrate specific tenant objects, which is the simplest form of migration. Ideally use specified object migration in late phase development or in a continuous integration continuous deployment (CICD), or similar process. Also use it in early phase development if the DEV environment has a large number of extraneous objects not relevant to the UAT or PROD environments. Following are the tenant objects that can be independently moved from one environment to another, and the migration steps.

Migrate External Data Sources

Migrate external data sources with the Command Line Interface (CLI) tool. Use the export_datasources command to export the source environment data source to a ZIP file, and the import_datasources command to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Schemas

Migrate schemas with the Incorta Direct Data Platform™ user interface. Here are the schema migration steps in the Incorta Direct Data Platform™:

  • In the Navigation bar of the source environment, select Schema.
  • Select the checkbox to the left of the schema you would like to export.
  • In the Action menu that appears (Kebab icon), select Export.
  • In the Navigation bar of the target environment, select Schema.
  • In the Action bar, select select + New → Import Schema.
  • Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing schema checkbox if you are importing a schema with the same name as an existing schema.
  • The Import Results dialog will contain a status of Created if the import was successful. Select Close.

You can also migrate schemas with the Command Line Interface (CLI) tool. Use the export_schemas command to export the source environment schema to a ZIP file, and the import_schemas command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Business Schemas

Migrate business schemas with the Incorta Direct Data Platform™ user interface. Here are the business schema migration steps in the Incorta Direct Data Platform™:

  • In the Navigation bar of the source environment, select Business Schema.
  • Select the checkbox to the left of the business schema you would like to export.
  • In the Action menu that appears (Kebab icon), select Export.
  • In the Navigation bar of the target environment, select Business Schema.
  • In the Action bar, select select + New → Import Business Schema.
  • Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing business schema checkbox if you are importing a business schema with the same name as an existing business schema.
  • The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Schema Load Jobs

Migrate schema load jobs manually from one environment to another. You can recreate the schema load in the target environment based on the load job configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Schema Alerts

Migrate schema alerts manually from one environment to another. You can recreate the schema notification in the target environment based on the schema notification configuration in the source environment. Refer to the Scheduler documentation for additional information.

Migrate Dashboards

Migrate dashboards with the Incorta Direct Data Platform™ user interface. Here are the dashboard migration steps in the Incorta Direct Data Platform™:

  • In the Navigation bar of the source environment, select Content.
  • Select the dashboard you would like to migrate to open it.
  • In the More Options menu (Kebab icon), select Export.
  • In the Export Dashboard dialog, select Export.
  • In the Navigation bar of the target environment, select Content.
  • In the Action bar, select select + New → Import Folder/Dashboard.
  • Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing dashboard checkbox if you are importing a dashboard with the same name as an existing dashboard.
  • The Import Results dialog will contain a status of Created if the import was successful. Select Close.

You can also migrate dashboards with the Command Line Interface (CLI) tool. Use the export_dashboards command to export the source environment dashboard to a ZIP file, and the import_dashboards command to import the ZIP file to the target environment. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Dashboard Alerts

Migrate dashboard alerts with the Incorta Direct Data Platform™ user interface. Here are the Dashboard Alert migration steps in the Incorta Direct Data Platform™:

  • In the Navigation bar of the source environment, select Scheduler, and then the Data Notifications tab.
  • Select the checkbox to the left of the dashboard alert you would like to export.
  • In the Action menu that appears, select Export (Download icon).
  • In the Navigation bar of the target environment, select Scheduler, and then the Data Notifications tab.
  • In the Action bar, select select + New → Import Notifications.
  • Select the File Upload panel, the source ZIP file from your computer and Open, or drag and drop the source ZIP file to the File Upload panel. Select the Overwrite existing alerts checkbox if you are importing a data alert with the same name as an existing data alert.
  • The Import Results dialog will contain a status of Created if the import was successful. Select Close.

Migrate Dashboard Delivery Jobs

Migrate dashboard delivery jobs with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Scheduled Jobs checkbox in the Export dialog.

Migrate Bookmarks

Migrate bookmarks with the Incorta Direct Data Platform™ user interface when you migrate the associated dashboard. Select the Include Bookmarks checkbox in the Export dialog. Both public and private bookmarks will be preserved.

Migrate Global Variables

Migrate global variables manually from one environment to another. You can recreate the global variable in the target environment based on the global variable configuration in the source environment. Refer to the Schema Manager documentation for additional information.

Migrate Session Variables

Migrate internal and external session variables with the Command Line Interface (CLI) tool. The export_session_variables command can be used to export the session variables to a ZIP file from the source environment, and the import_session_variables command can be used to import the ZIP file to the target system. Refer to the CLI tool documentation for more information on accessing the tool and command usage.

Migrate Groups

Migrate groups manually from one environment to another. You can recreate the groups in the target environment based on the group configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

Migrate Users

Migrate users manually from one environment to another. You can recreate the users in the target environment based on the user configuration in the source environment. Refer to the User Profile Manager documentation for additional information.

A more complex scenario involves the migration of more than one of the tenant objects listed above. It is critical to ensure that dependent objects are not broken when a change is made to a tenant object. For example, when you change a business schema column, it is important to migrate the affected dashboards. As mentioned above, the Inspector Tool will help to identify any migration issues.

Migrate as a Tenant Super User

Super users typically migrate full tenants. Ideally use full tenant migration in early phase development for the initial creation of the UAT or PROD environments. The following are considerations when you migrate a full tenant:

  • All objects are migrated with the original ownership. It is recommended that the objects in PROD are owned by Administrators or Super Users, not by developers.
  • Data sources point to databases in DEV or UAT that are different than in PROD.
  • There is not an option to exclude certain objects from migration.

There are two options for migrating a full tenant:

  • Use the CMC
  • Use the TMT

The CMC option is user interface driven, while the TMT is a command line utility. The other difference is the TMT has an option to copy local data files and folders as part of the migration, while the CMC does not. Before you migrate the full tenant with either option, it is important to first create a tenant backup.

Backup the Existing Tenant in the Target Environment

Following are the steps to create a tenant backup using the CMC:

  • Sign in to the CMC of the source environment.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Tenants.
  • In the More Options (Kebab icon) menu at the end of the tenant row, select Backup.

Use the CMC to Migrate the Full Tenant

Following are the steps to migrate the full tenant using the CMC:

  • Sign in to the CMC of the source environment.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Tenants.
  • In the More Options (Kebab icon) menu on the right side of the tenant row, select Execute inspector now to review and verify the tenant object lineage.

  • In the More Options (Kebab icon) menu on the right side of the tenant row, select Export. The tenant will be packaged into a tenant_<TENANT_NAME>.zip folder containing:

    • <TENANT_NAME>.properties file

    • <TENANT_NAME>.zip file

      • dashboards folder containing XML
      • datasources folder containing XML
      • schemas folder containing XML
      • tenant.xml file
  • The exported ZIP file will be downloaded to your browser’s default download folder.
  • Sign in to the CMC of the target environment.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Tenants.
  • Select + → Import Tenant.
  • Drag and drop the tenant_<TENANT_NAME>.zip to the Click or drag a file here to upload panel in the Import a tenant to the cluster dialog.
  • Select Next.
  • If there is a <TENANT_NAME>.properties file located within tenant_<TENANT_NAME>.zip, select Yes, Use it to autofill the tenant property values. The tenant property values can be changed later. Otherwise, select No, ignore it.
  • Verify or enter the tenant properties:
Property Control Description
Name text box Enter the tenant name. Select Check to determine if a Tenant already exists with the name entered.
Username text box Enter the username for the Super User
Password text box Enter the password for the Super User
Email text box Enter the email address for the Super User
Path text box Enter the shared storage path for tenant related data
Pause scheduled jobs toggle Enable this property if the imported tenant will have all scheduled jobs paused on import
  • Enter and verify the tenant email properties:
Property Control Description
Sender’s Username Auth toggle Enable this property if the email requires username authentication
System Email Username text box Enable Sender’s Username Auth to configure this property. Enter the username for the system email.
System Email Address text box Enter the email address for the system email
System Email Password text box Enter the password for the system email address
SMTP Host text box Enter the Simple Mail Transfer Protocol (SMTP) host for the system email
SMTP Port text box Enter the SMTP port number
Share Notifications toggle Enable this property to share notifications
  • Select Create to finalize the tenant migration.
  • If the tenant contained data files that were not part of the migration, they should be added to the indicated directory for tenant data:
<CMC_INSTALLATION_PATH>/IncortaAnalytics/Tenants/<Tenant_Name>/data
  • Run the Inspector Tool following the migration to identify any lineage errors that may have occurred from moving the tenant between environments.

Use the TMT to Migrate the Full Tenant

Another option to migrate a full tenant is to use the TMT command line tool. A System Administrator with root privileges to the host running the CMC is able to run the TMT.

TMT file path

Following is the default path to the TMT tool in your CMC:

<CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt
TMT Command for Tenant Export

Here are the TMT commands to export the full tenant:

cd <CMC_INSTALLATION_PATH>/IncortaAnalytics/cmc/tmt/
./tmt.sh --cluster-name <CLUSTER_NAME> --export <TENANT_EXPORT_NAME>.zip --cf

TMT Command for Tenant Import

Here is the TMT command to import the full tenant:

./tmt.sh -clnm <CLUSTER_NAME> -i <PATH_TO_TENANT_EXPORT_NAME>.zip

Refer to the TMT documentation for more information on command usage.

Migrate as a System Administrator

System Administrators typically migrate host files that contain configurations, properties, customizations, and more. This requires root access to the hosts. As necessary, copy host files manually from one environment to another, or edit the file manually in the target environment as indicated in the table below:

Host File Migration Action
Linux Host Configurations - Ulimit Manually edit
/etc/security/limits.conf
CMC Node Configuration files Manually edit all files under
<CMC_INSTALLATION_PATH>/cmc/cmcData

Manually edit
<CMC_INSTALLATION_PATH>/cmc/conf/catalina.properties
Analytics and Loader Service Properties Manually edit each of the following files:
<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
node.properties

<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
services/<SERVICE_GUID>/conf/catalina.properties

<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
services/<SERVICE_GUID>/incorta/service.properties

<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
services/<SERVICE_GUID>/incorta/engine.properties

<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
services/<SERVICE_GUID>/incorta/options.properties

<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
nodeAgent/nodeAgent.cfg
Spark Configuration Manually edit
<SPARK_NODE_INSTALLATION_PATH>/spark/conf/
spark-defaults.conf
Zookeeper Configuration Manually edit
<ZOOKEEPER_NODE_INSTALLATION_PATH>/zookeeper/conf/
zoo.cfg
Independent JARs Copy the file to the appropriate connectors subdirectory
<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
extensions/connectors/
Tomcat server XML Manually edit
<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
services/<SERVICE_GUID>/conf/server.xml
CSS Customizations Manually edit
<INCORTA_NODE_INSTALLATION_PATH>/IncortaNode/
runtime/webapps/ROOT/css/custom-new.css
Certificates, SSL PCKS12 or Java Keystore Files for an External Data Source Copy the file to the certs directory
/home/incorta/IncortaAnalytics/certs/

Migrate as a CMC Administrator

CMC Administrators typically migrate CMC configurations. CMC configurations are migrated manually from one environment to another. You can replicate the CMC configurations in the target environment based on the CMC configurations in the source environment. Adjust any configuration values as necessary for the target environment. Following are the CMC configurations to consider for migration.

Important

An update of CMC configurations may require a restart of the analytics service, loader service, or all services.

Migrate General Configurations

Migrate Node Services Configurations

Migrate the On Heap Memory (GB), Off Heap Memory (GB), and CPU Utilization (%) for both the Analytic and Loader services as follows:

  • Sign in to the CMC.
  • In the Navigation bar, select Nodes.
  • In the Services tab, select either an Analytics or Loader Service.
  • Select Edit (Pen icon).
  • Update the node services configurations as necessary.
  • Select Update.
Migrate Notebook Add-on

Enable Notebook Add-on for for the Analytics service as follows:

  • Sign in to the CMC.
  • In the Navigation bar, select Nodes.
  • Select the Add-ons tab.
  • Select + → Add Notebook.

Migrate Server Cluster Configurations

Migrate the server cluster configurations as follows:

  • Sign in to the CMC.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Cluster Configurations.
  • In the panel tabs, select Server Configurations.

  • In the left pane, select the one of the following options to update the associated server configurations:

    • Kafka Consumer Service Name
    • SQL Interface
    • Spark Integration
    • Tuning
    • UI Customizations
    • Diagnostics

Migrate Default Tenant Cluster Configurations

Migrate the default tenant configurations as follows:

  • Sign in to the CMC.
  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Cluster Configurations.
  • In the panel tabs, select Default Tenant Configurations.

  • In the left pane, select one of the following options to update the associated default tenant configurations:

    • Security - Authentication Type
    • Regional Settings
    • Email - SMTP Host, Enable SMTP SSL, Enable Notifications, Enable Internal Error Notifications
    • Data Loading - Stop Loading on First Error, Pause Scheduled Jobs
    • Integration - Google Maps API Key, Apple Maps Developer Team ID, Apple MapKit JS Key ID, Apple Maps API Key, Google Drive Client ID, Google Drive Client Secret, Dropbox App Key, Dropbox App Secret, Box Client ID, Box Client Secret, Mapbox API KeyTenant EFS
    • Advanced - Insight Max Groups UI Default, Insight Max Groups Limit, Insight Max Values in Filter Subquery, Force Reload Columns, Sync In Background, Warmup Mode
    • Tuning - Schema Pool Size, Table Maximum Parallel Chunks, Evaluate Session Variables At Login
    • Incorta Labs - Enable Insight ‘View As’ Menu, Notebook Integration
    • External Visualization - Incorta Host, Incorta Port
Related Links
© Incorta, Inc. All Rights Reserved.