Release Notes 4.7

Release Notes 4.7

IMPORTANT: PLEASE READ AND FOLLOW

BEFORE UPGRADE

Before upgrading to Incorta 4.7, please run the Inspector Tool.

To learn more about the Inspector Tool, please review the Inspector Tool documentation.

In the Inspector Dashboard, first identify all Severity 1 issues such as null pointer exceptions for invalid joins and formulas. Next, resolve the Severity 1 issues by recreating schema joins when applicable and resolving issues with specific formulas. After resolving the Severity 1 issues, then upgrade to Incorta 4.7.

AFTER UPGRADE

After upgrading to Incorta 4.7, sign in to the Cluster Management Console (CMC) and upgrade the Cluster Metadata database.

To sign in to the CMC, visit your CMC host at one of the following:

  • http://<Public_IP>:6060/cmc
  • http://<Public_DNS>:6060/cmc
  • http://<Private_IP>:6060/cmc
  • http://<Private_DNS>:6060/cmc

The default port for the CMC is 6060. Sign in to the CMC using your administrator username and password. To upgrade the Cluster Metadata database, follow the steps:

  • In the Navigation bar, select Clusters.
  • For each cluster name in the Cluster list, in the Actions column, select Upgrade Cluster Metadata.

The goal of the Incorta 4.7 release is to enhance analytical capabilities, empower business users, and improve data management and administration. To that end, the 4.7 release introduces several key improvements to the Cluster Management Console (CMC), Incorta Loader Service, and the Incorta Analytics Service. In addition, the release includes an Incorta Labs offering for enabling an Organizational Chart insight visualization. The release also introduces a command line tool that merges numerous small Apache Parquet files into a few large files.

Release Highlights

There are several important features in this release:

Additional Improvements and Enhancements


Cluster Management Console (CMC)

The following new configurations and enhancements are available in the Cluster Management Console (CMC) for this release:

Auto Start Services

For the selected Incorta Cluster, you can now enable Auto Start Services. Enable this setting to have the Analytics and Loader Services start automatically when the service process terminates unexpectedly.

After three unsuccessful consecutive attempts to start a service, the Auto Start Services becomes disabled. A tooltip provides this detail: “Auto Start has been disabled due to failure of three consecutive restart trials”.

In addition, with SMTP configured to send email for the default tenant configuration, the CMC sends email alerts to the Cluster administrator email. The email subject is: “Cluster:[ClusterName];Service[serviceName] crash report”.

Attached to the email is a .zip file that contains a folder with several log files including the catalina.YYYY-MM-DD.log and various incorta.YYYY-MM-DD.logs (tenant, system, and sql~i).

When Auto Start Services becomes disabled, the crash report email contains this additional note.

As of this release, Auto Start Services does not support the Notebook Add-on Service.

To enable this option for an existing Cluster, follow these steps:

  • In the Navigation bar, select Clusters.
  • In the cluster list, select a cluster name.
  • In the canvas tabs, select Details.
  • In the view panel, to edit the cluster details, select the Pen.
  • In the edit panel, enable Auto Start Services.
  • Select Update.

Here are the steps to enable this option when creating a new Cluster using the wizard:

  • In the Monitoring & Alerts step, enable Auto Start Services.
  • Select Next.
  • In the Review step, review the new Cluster details.
  • Select Create.

Configure email alerts

In order to receive email Auto Start Services alerts associated with the Loader or Analytics Service, you must configure SMTP Email for the Cluster’s Default Tenant configuration.

To enable SMTP Email for Auto Start Services alert in the CMC, follow these steps:

  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Cluster Configurations.
  • In the panel tabs, select Default Tenant Configurations.
  • In the left pane, select Email.
  • In the right pane, specify the SMTP details such as:

    • SMTP Host, e.g., smtp.gmail.com
    • SMTP Server Port
    • Email Server Protocol
    • If required, enable Email Host Requires Authentication and specify the System Email Address, Person Name, System Email Password.
    • If required, enable Sender’s Username Auth, and specify the System Email Username.
  • Select Save.

Enhanced downloads for Log files

The 4.7 release contains enhancements in the Cluster Management Console (CMC) for selecting and downloading log files as follows:

  • In the Navigation bar, select Logs.
  • Specify a Date range, Cluster name, Service name, and Tenant Name.
  • Scroll through log file results.
  • Select one or more file for download
  • To download the selected log files in .zip format, select Download.

Incorta Labs

Incorta Labs are experimental features and functionality that Incorta supports for non-production use. As such, some experimental features will potentially become part of an Incorta release and others potentially will be deprecated. Incorta Support will investigate issues with Incorta Labs features.

In this release, there is one new feature in Incorta Labs to enable and Organizational Chart visualization.

Organizational Chart visualization

To display hierarchical relationships using the new Organizational Chart insight visualization, you must first enable the visualization in the Cluster Management Console (CMC).

The visualization requires a physical schema table that makes a self-referential join representative of a hierarchy where the child foreign key left outer joins to the parent primary key, for example, ManagerID left outer joins to EmployeeID.

Here are the steps to enable this option as default tenant configuration in the CMC:

  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select Cluster Configurations.
  • In the panel tabs, select Default Tenant Configurations.
  • In the left pane, select Incorta Labs.
  • In the right pane, toggle Org-charts to enabled.
  • Select Save.

Here are the steps to enable this option for a specific tenant configuration in the CMC:

  • In the Navigation bar, select Clusters.
  • In the cluster list, select a Cluster name.
  • In the canvas tabs, select the Tenants tab.
  • In the Tenant list, for the given tenant, select Configure.
  • In the left pane, select Incorta Labs.
  • In the right pane, toggle Org-charts to enabled.
  • Select Save.

To create an insight using the Organization Chart visualization, follow these steps:

  • If not already open, open the dashboard.
  • In the Action bar, select +.
  • In the Choose a Visualization dialog, in Charts, select Organizational Chart.
  • In the Grouping dimension tray, open properties for the pill.
  • Optionally, specify value for the Image URL property, typically a URL reference and a reference to a column literal, e.g., http://my.website.com/{SchemaName.TableName.ColumnName}.
  • In Hierarchy Depth, enter a value.
  • In the Action bar, select Done.

Incorta Analytics and Loader Service

The 4.7 release introduces several key improvements to the Incorta Analytics and Loader Services such as:

Schema description tooltips

In the Schema Manager, you can now view the entire schema description in a tooltip as follows:

  • In the Navigation bar, select Schema.
  • In the Schema list, for a given schema row, hover over the Description text in the Description column.

To create or edit a Schema Description, follow these steps:

  • In the Navigation bar, select Schema.
  • In the Schema list, for a given schema row, select More Options (Kebab icon).
  • In the More Options menu, select Edit Description.
  • In the Edit Description dialog, enter the schema description.
  • Select Save.

Specify a Custom Chunk Period for a SQL table

In this release, for an SQL table (a table with a SQL Data Source), you can now configure a Custom Chunk Period for a specific number of days. This configuration instructs the Loader Service to perform a parallelized extraction of the source table which is especially helpful when the source table contains a large number of rows.

To enable and configure a Custom Chuck Period for an incremental table, follow these steps:

  • For a given schema, in the Schema Designer, select the SQL table, and open the Schema Table Editor.
  • In the Table Editor, in the Summary section, select the Table Data Source.
  • In the Data Source dialog, in Chunking Method, select By Timestamp.
  • In Order Column [Date/Timestamp], select an applicable timestamp column.
  • In Chunk Period, select Custom.
  • A new property, Number of Days, appears.
  • Specify a value.
  • In the dialog, select Save.

Join Editor validates joins

In this release, when you try to create an unsupported join, the Join Editor shows an error. Here are two examples of unsupported joins:

  • A child table key column joins to the same table as the parent table on the same on the same key column
  • Multiple joins between a single key column in a parent table with two or more columns in the child table

Enhanced Job Error dialog for schema load errors

In this release, you can now view and copy the Job Errors for a schema load with errors. In addition, you can view and copy the error details which contain the stack trace for the exception.

To view the Job Errors dialog, follow these steps:

  • In the Navigation bar, select Schema.
  • In the Schema Manager, select a schema with a reported error.
  • In the Schema Designer, in the Summary section, in Last Load Status, select the date.
  • In Jobs, select View Errors.
  • In the Job Errors dialog, review the error message.
  • Optionally, select Copy to Clipboard.
  • To view the error details, select View Details.
  • In Error Trace, review the stack trace for the exception.
  • Optionally, select Copy to Clipboard.
  • To close the dialog, select Ok.

With the Analyzer, create an Incorta View

In the 4.7 release, there are now two ways to create a Runtime Business View:

  • Use the Business Schema Designer to create a Business Schema View
  • Use the Analyzer to create an Incorta View

The term, Business Schema View, indicates a Business Schema Designer created view. The term, Incorta View, indicates an Analyzer created view.

The term, Runtime Business View, encompasses both types of views — Business Schema View and Incorta View — in a Business Schema.

Both view types — Business Schema View and Incorta View — can exist within the same Business Schema.

The Business Schema Designer distinguishes between both view types: a table icon indicates a Business Schema Designer created view and table with search icon indicates an Analyzer created view.

In the 4.7 release, you can now use the Analyzer to create a runtime business view, known as an Incorta View. The approach is similar to creating an Incorta Table, a table in a physical schema created with the Analyzer.

The new feature encourages business analysts to easily create ad-hoc, runtime business views. For example, a business analyst can create an Incorta View and then reference the Incorta View in an InQuery insight filter expression.

An Incorta View supports two Table visualization types: Table and Aggregated. Table is the default visualization.

You can edit an Incorta View only with the Analyzer.

Here are the steps to create an Incorta View in a new Business Schema:

  • In the Navigation bar, select Business Schema.
  • In the Action bar, select + New.
  • In the Add New menu, select Create Business Schema.
  • In the Create Business Schema dialog, for Name, enter a value.
  • In the Canvas, select Add New View, then select Create in Analyzer.
  • Optionally, in the Action bar, select the Visualization icon, and in the Choose a Visualization dialog, select Aggregated.
  • In the Data panel, select +.
  • In the Edit panel, select a physical schema, physical schema table, business schema, and/or business schema view, and then select Done. Note: You cannot select another Incorta View.
  • From the Data panel, drag & drop or add the required columns to the measure tray.
  • Optionally, from the Data panel, drag & drop New Formula to the measure tray, and edit the formula with the Formula Builder.
  • Optionally, from the Data panel, drag & drop or add additional columns to the grouping dimension tray.
  • Optionally, from the Data panel, drag & drop New Formula to the grouping dimension tray, and edit the formula with the Formula Builder.
  • Optionally define Insight filters.
  • Optionally define Measure filters.
  • To confirm your changes, in the Action bar, select Done.
  • In the view header, enter a view name.
  • To save the business schema, in the Action bar, select Done.

Here are the steps to edit an Incorta View in an existing Business Schema:

  • In the view header, select the More Options (Kebab icon).
  • In the More Options menu, select Edit in Analyzer.
  • To confirm your changes, in the Action bar, select Done.
  • To save the business schema, in the Action bar, select Done.

Additional considerations for an Incorta View

In the Analyzer for an Incorta View, you are not able to reference another Incorta View, a view created with the Analyzer.

Additional considerations for Runtime Business Views

If using the Incorta Labs feature for SQL Joins to read non-performance optimized tables with Apache Spark in conjunction with a Runtime Business View, Incorta does not cache the results.

For a Runtime Business View with performance optimized tables, Incorta does cache the query view results.

To increase the related cache for runtime business views and dashboard insights, for a given Tenant, adjust the following properties in the Cluster Management Console:

Maximum Cached Entries

Specify the number of distinct runtime business views and dashboard insights for the Analytics Service cache. The default value is 2,000.

Maximum Cached Memory (%)

Specify the maximum percentage of memory available for cached runtime business views and dashboard insights.

Analytics Services uses both the Maximum Cached Entries and the Maximum Cached Memory (%) to determine the lower limit of the cache size. For example, if the Analytics Service off-heap memory is set to 100GB, and the Maximum Cached Memory (%) to 5%, then the cache size is 5GB.

Compact mode for a dashboard layout

In this release, it is now possible to enable Compact Mode for a dashboard layout. Compact Mode removes the extra whitespace for insights on a dashboard. A dashboard with Compact Mode layout enabled shows COMPACT in the dashboard’s Action bar.

Here are the steps to enable Compact Mode:

  • If not already open, open the dashboard.
  • In the Action bar, select More Options (Kebab icon).
  • In the More Options menu, select Edit Layout.
  • In the Action Bar, select Compact, then select Save.

Here are the steps to disable Compact Mode:

  • If not already open, open the dashboard.
  • In the Action bar, select More Options (Kebab icon).
  • In the More Options menu, select Edit Layout.
  • In the Action Bar, select Relaxed, then select Save.

Hide and show the dashboard filter bar

In this release, for a given Dashboard, a user can now hide or show the Filter bar. The Filter bar contains dashboard runtime filters, Search, the Filter panel, and Bookmarks.

To hide the Filter bar, simply select the center Up arrow directly below the Filter bar.

When hidden, below the Action bar, a center Down arrow appears. A filter icon indicates one or more dashboard runtime filters.

To show the Filter bar, directly below the Action bar, select the center Filter icon or Up arrow.

Support for dashboard prompt operators

The 4.7 release supports all dashboard prompt operators including Starts With, Does Not Start With, Contains, Does Not Contains, or Ends With.

To create a dashboard prompt, follow these steps:

  • In the Action bar for the dashboard, select More Options (Kebab icon).
  • In the More Options menu, select Manage Filter & Prompts.
  • For Dashboard filters, from the Data panel, drag & drop a column to the Prompts tray.
  • Optionally, edit the Prompt properties.
  • In the Action bar, select Done.

To view the operator support for the Prompt, follow these steps:

  • In the Filter bar for the dashboard, select Filter (Filter icon).
  • In the Filter dialog, in the left pane, select the Prompt column.
  • In the right pane, select an operator such as Starts With, Does Not Start With, Contains, Does Not Contains, or Ends With.
  • Select Paste Values and enter a comma or line separated list of values, then select +.
  • Select Apply.

InQuery insight filter expression support for dynamic presentation variables

In this release, for a given insight, you can now specify an InQuery insight filter expression that utilizes a dynamic value from a dashboard presentation variable.

To add an InQuery insight filter expression, follow these steps:

  • For the given dashboard insight, select Edit (Pen icon).
  • In the Analyzer, in the Action bar, select Filter (Filter icon).
  • From the Data panel, drag & drop New Formula to the Filter Individual Values tray.
  • In the New Formula properties, select Edit Operator / Values.
  • Select the default operator, for example [True].
  • In Choose Operator, select InQuery.
  • To open the Formula Builder, select the text box.
  • In the Formula Builder, specify a query function such as query() or queryDistinct().
  • For the Filter Expression, specify the Presentation Variable using the $ syntax, e.g., $pv_Name.
  • In the Formula Builder, select Done.
  • In the Action bar, select Done.

Improved filtering for insight interactions

In the 4.7 release, for certain insight visualizations, when a user selects a value in an insight, the interaction applies two dashboard runtime filters: one for the grouping dimension column selected value and one for the coloring dimension column selected value, when applicable. These grouping and coloring column selected values represent a slice.

The following insight visualizations support the ability for users to drill into a slice, i.e., apply two dashboard runtime filters with one interaction:

  • Column Chart
  • Stacked Column Chart
  • Percentage Column Chart
  • Bar Chart
  • Stacked Bar Chart
  • Percentage Bar Chart
  • Area Chart
  • Stacked Area Chart
  • Percentage Area Chart
  • Line Chart
  • Stacked Line Chart
  • Percentage Line Chart
  • Time Series
  • Dual X-Axis
  • Line Time Series
  • Treemap Chart
  • Heatmap Chart
  • Combination Chart

Copy specific columns and rows for table insights

In this release, you can copy specific columns and rows for the following Table visualizations:

  • Pivot Table
  • Table
  • Aggregated
  • Aggregated - Total
  • Aggregated - Subtotal
  • Aggregated - Subtotal - Total

To copy columns and rows follow these steps:

  • If not already open, open the dashboard.
  • Scroll to the Table insight.
  • Select the whitespace of a table cell.
  • Select the lower right corner square so that the mouse icon turns from a pointer icon to a crosshair icon.
  • Drag the crosshair icon down the cell column, across the cell row, or across columns and rows.
  • To copy the selection, use the cmd + C or ctrl + C keystroke.
  • Use either the cmd + V or ctrl + V keystroke to paste the selection into a new Numbers or MS Excel worksheet.

For a Pivot Table insight, you can also select and copy the Pivot Table Row and Column headers.

Schedule dashboard delivery to multiple recipient email addresses

In this release, you can now schedule delivery of a dashboard to multiple email recipients as follows:

  • If not already open, open the dashboard.
  • In the Action bar, select Sharing (Share icon).
  • In the Sharing menu, select Schedule Delivery.
  • Specify the following:

    • Schedule Name
    • Delivery frequency, Starting at time, Starting From date, Does not end checkbox
    • Email Subject, Body, Hide NotificationText, Use HTML
    • Export Folders File Name, Append Timestamp
    • Data Format
  • For Recipients, in the To: field, enter one or more recipient email addresses.
  • For multiple email addresses, use either a comma (,) or semicolon (;) separator.
  • In the Action bar, select Schedule.

Schedule dashboard delivery as a single HTML page

In this release, you can now schedule delivery of a dashboard as a single HTML file attachment as follows:

  • If not already open, open the dashboard.
  • In the Action bar, select Sharing (Share icon).
  • In the Sharing menu, select Schedule Delivery.
  • Specify the following:

    • Schedule Name
    • Recipients
    • Delivery frequency, Starting at time, Starting From date, Does not end checkbox
    • Email Subject, Body, Hide NotificationText, Use HTML
    • Export Folders File Name, Append Timestamp
  • For Data Format, select HTML FILE.
  • In the Action bar, select Schedule

Command Line Tools

Apache Parquet Merge Tool

Over time, an Incorta table configured for incremental loads can generate hundreds of Apache Parquet files that are small in size. The Apache Parquet Merge tool is an interactive, command line tool that merges multiple Parquet table increment files into a single table increment file that contains the merged segments. As the merged files are typically about 1GB in size, the result is improved performance for reading Parquet files from Shared Storage.

There are three options for running the Parquet Merge Tool:

  • For a specific Tenant, merge all schema table increments
  • For a specific Tenant, merge all table increments for one or more Schemas
  • For a specific Tenant and Schema, merge one or more table increments

The tool merges backs up the original files into a backup directory and merges the table increments into the original directory.

To access the Parquet Merge Tool, follow these steps:

  • In the terminal for the Incorta Loader Node host, navigate to the default installation path as the incorta user:

    cd /home/incorta/IncortaAnalytics/IncortaNode/parquetMergeTool/

Merge file

For Linux operating systems, use the merge.shfile. For Windows operating systems, use the merge.bat file.

The interactive shell script has the following parameters:

  • Tenants path : -path or --p
  • Tenant name : -tenant or --t
  • Schema(s) name : -schema or --s
  • Table(s) name : -table or --tab
  • Min table increments count for merge : -increments or --i

For a specific Tenant, to merge all schema table increments, execute the following:

./merge.sh -path /home/incorta/IncortaAnalytics/Tenants -tenant TenantName

For a specific Tenant, merge all table increments for one Schema, execute the following:

./merge.sh -path /home/incorta/IncortaAnalytics/ -tenant TenantName -schema SchemaName_1 SchemaName_2

For a specific Tenant and Schema, merge one or more table increments, execute the following:

./merge.sh -path /home/incorta/IncortaAnalytics/ -tenant TenantName -schema SchemaName_1 -table TableName

Log files

The Parquet Merge Tool generates a log file in the following default installation directory:

/home/incorta/IncortaAnalytics/parquetMergeTool/work/

The file contains the Source Segment, Target Segment, and Source Offset. Use this log file to determine the results of the merge activities.


Additional Improvements and Enhancements

In the 4.7 release, there are additional performance enhancements:

  • Improved metadata discovery for Apache Hive tables
  • Embedded IFRAME support for a dashboard URL

Improved metadata discovery for Apache Hive tables

In previous versions of Incorta, in order to determine the table schema definition, Incorta sent a discovery query to an Apache Hive data source that returned an empty result set with the table schema definition.

Because Apache Hive often supports an Apache Hadoop cluster with many nodes, the discovery query could take a significant amount of time.

In the 4.7 release, Incorta now queries the Apache Hive metadata database object directly to retrieve table schema results.

Embedded IFRAME support for a dashboard URL

When referencing the URL for the Analytics Service in an HTML IFRAME, you can specify a specific Dashboard URL.

In the 4.7 release, when a dashboard is embedded in this manner, the following are absent:

  • the Navigation bar
  • the breadcrumb in the Action bar typically used to navigate back to the parent dashboard folder or back to Content

© Incorta, Inc. All Rights Reserved.