Connector SDK guide

The Incorta Connector Software Development Kit, or Connector SDK, allows Incorta customers and partners to develop their own custom connectors. 

With the Connector SDK, you can implement metadata discovery and data extraction that enables you to:

  • Discover source database tables using the schema wizard
  • Discover table columns while manually editing a schema table
  • Execute full and incremental load queries on a source system
  • Fetch data records to store in Incorta’s persistence layer

Follow the instructions in this article to learn how to implement, build, and deploy the connector.

Connector SDK Versions

Ensure you use the correct Connector SDK version with the correct release version of Incorta. The following table lists the Connector SDK versions supported by each Incorta release.

Connector SDK Version Incorta Release Version
1.6 4.9
1.5 4.7.2+
1.4 4.7.0, 4.7.1
1.4 4.6
1.3 4.5
1.1 4.4
1.0 4.3

Implement the Connector

  1. Download the Connector SDK JAR file. The JAR file, incorta.connector.api.jar, is offered independently from the installation script and is located in the build directory.  If you need assistance with locating this file, contact your account manager.
  2. Implement the following interfaces:

    • com.incorta.connector.ConnectorFactory

      • Main entry point for the connector. It is the factory class used for accessing the connector model and for instantiating the connector.
      • A single factory class can support multiple connector types.
    • com.incorta.connector.Connector

      • Main connector object that manages connection to the data source.
      • Use this parameter to discover source tables using the schema wizard and to create data set objects.
    • com.incorta.connector.DataSet

      • Extracts data for a single Incorta table.
      • It contains the necessary information for extracting the table data. For example, for an SQL-based connector, a data set definition contains the SQL query for that table.
      • A data set can support incremental extraction.
    • com.incorta.util.Record.RecordSet

      • A record set object represents the set of records returned from a single execution of a data set. Create a record set during the ETL job. The record set is consumed by Incorta’s loader. The record set provides an interface for the loader to iterate over the returned records.

Build the Connector

Package the connector into a JAR file. The connector JAR must have a manifest file that:

  • Points to the connector factory class
  • Specifies the connector API version

If the connector requires third-party libraries, you do not need to package them inside the connector JAR. Only deploy these libraries with the connector JAR.

You do not need to deploy the Connector API JAR with the connector and its dependencies, because it is included in your Incorta installation.

Below is a sample connector JAR manifest file. It specifies the connector factory class named SampleConnectorFactory, and indicates that it targets API version 1.0:

Manifest-Version: 1.0
Connector-Factory-Class: com.incorta.connector.sample.SampleConnectorFactory
Connector-API-Version: 1.0

Deploy the Connector

Deploy the connector on each Incorta node and the Cluster Management Console (CMC). The CMCs need a connector to access its model when you import or export tenants.

  1. Copy the connector JAR, containing the connector implementation, manifest file, and any third-party library JARs into the following directories:

    */IncortaNode/extensions/connectors/<connector_directory>/<connector.jar file>
    
    */cmc/extensions/connectors/<connector_directory>/<connector.jar file>

    Optionally, you can create a subdirectory in the connectors directory and copy the files there.

  2. Restart Incorta services.
  3. After restarting your services you can add the connector as a data source using the instructions in Connect to a Data Source.

The connector framework scans all directories recursively. A special class loader is used to load classes from the directories to mirror the directory structure. JARs located in separate directories are loaded using separate class loaders. JARs in a subdirectory are loaded using a child class loader.

This is done to ensure isolation between connectors. For example, if one connector uses one version of Apache Commons, while another connector uses a different version of Apache Commons, we want to make sure that each connector code accesses its own correct version.

Connector Model XML Structure

Configuring the connector is done with an XML file.

It is structured as follows: Root element is a <connectorModel> tag, containing one <connectorProperties> tag and one <datasetProperties> tag.

Each one of them contains zero or more <propertyModel> tags that describe the properties of the connector or data set.

<?xml version="1.0" ?>
<connectorModel version="1" typeName="sql" typeLabel="SQL Database" subTypeName="mysql" subTypeLabel="MySQL">
   <connectorProperties>
       <propertyModel type="TEXT" name="user" label="Username" required="true" />
       <propertyModel ... />
       ...
   </connectorProperties>
   <dataSetProperties>
       <propertyModel type="LONG_TEXT" name="query" label="Query" required="true" />
       <propertyModel ... />
       ...
   </dataSetProperties>
</connectorModel>

Tags and Attributes

Tag: connectorModel

This is the root tag of the connector model.

<connectorModel> tag attributes

Attribute Required Default Value Description
version No 1 The connector version number (not the API version)
typeName Yes Connector type name (e.g. “sql”)
typeLabel Yes Connector type label displayed in UI (e.g. “SQL”)
subTypeName Yes Connector sub-type name (e.g. “mysql”)
subTypeLabel Yes Connector sub-type label displayed in UI (e.g. “MySQL”)
displaySourceFormat No ${connector} Format text under datasets icon. See displaySourceFormat below.
predefined No false Flag indicating connector ownership. See predefined below.
active No false If set to false (default), the connector object will not be created until it is needed. When set to true, the connector object is initialized as soon as it is defined, or right after server startup, and stays available in memory.

displaySourceFormat

This is the format of the text displayed under the data set icon that appears when you edit a schema table.

Usually it is just the connector name, that is why the default value is ${connector}, which is just a placeholder for the connector name.

In some cases, however, it is desired to display more details, such as in the file system connector where the file name is also displayed.

Placeholders in the format string are substituted with the connector name (in case of ${connector} placeholder) or data set property names (e.g. ${filename}).

predefined

This flag indicates that the connector instance is created and maintained by Incorta, not by the user. There can be a single connector instance of this type per tenant, and it is named after the subTypeLabel value.

This field is used mainly by local file system connector, because local files are stored in one place that can’t be changed.

Tag: connectorProperties

A required container tag for connector properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: datasetProperties

A required container tag for data set properties (propertyModel tags). The order of the properties inside this tag is the same as the order of the property fields as displayed in the UI. This tag has no attributes.

Tag: propertyModel

This tag is used to define a single property field that must be displayed in the UI.

Table of supported types for the <propertyModel> tag attributes

Type Description
INTEGER A numeric integer field.
BOOLEAN A yes/no toggle.
HIDDEN Not displayed in the UI. Used when there is a generated property that needs to be maintained.
TEXT A small text field.
MEDIUM_TEXT A medium text field.
LONG_TEXT A long text field (e.g. for SQL queries).
PASSWORD A password field.
LIST A dropdown list where user can select one of predefined values.
LIST_WITH_OTHER Similar to LIST, except that user can also choose the option “Other” which shows a text field where he enters a text value.
DYNAMIC_LIST Similar to LIST, except that the values of this list are obtained dynamically from the backend, by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector.
FILE A file upload field.
CATALOG_BROWSER Used for browsing and selecting a catalog. To navigate catalogs, the UI invokes getCatalogs() in the connector.
SCHEMA_BROWSER Used for browsing and selecting a schema. To navigate schemas, the UI invokes getCatalogs() and getSchemas() in the connector.
OAUTH OAuth access token. The UI renders an “Authorize” button which opens a new browser tab for obtaining OAuth authorization.

In case a property is of type LIST or LIST_WITH_OTHER, the property must also have a child possibleValues tag to list the property values.

A property can also be dependent on other properties. It can be displayed by the UI only if specific values were selected in other properties. This is done by specifying those dependencies in the child dependencies tag.

Property Attributes

Table of <propertyModel> attributes

Attribute Required Default Value Description
type Yes The property type. See the table above.
name Yes The name of the property.
label Yes The property label to be displayed in the UI.
prefix No Static text to be displayed before the property field.
suffix No Static text to be displayed after the property field.
tooltip No Tooltip text to be displayed next to the property label.
defaultValue No Default property value. The UI is pre-populated with this value the first time the user edits the connector or data set.
minValue No Minimum value for INTEGER property.
maxValue No Maximum value for INTEGER property.
extensions No Comma-separated list of file extensions. Used only for FILE properties.
extensionsProperty No Name of another property, the value of which is a comma-separated list of file extensions to be used by the current property. Used only with FILE properties.
usernameProperty No Used with OAUTH properties to store the name of another HIDDEN property used to store the authenticated user name.
dependsOnProperties No A comma-separated list of property names. When any of these properties are modified, the current property is refreshed by invoking either refreshConnectorProperty() in the connector factory, or refreshDataSetProperty() in the connector. Used for DYNAMIC_LIST, CATALOG_BROWSER and SCHEMA_BROWSER property types.
required No false A flag that indicates whether or not this property is required.

Tag: possibleValues

A container tag for predefined property values (propertyValue tags). This tag is used only for properties of type LIST or LIST_WITH_OTHER.

Tag: propertyValue

Represents a single predefined value for a property of type LIST or LIST_WITH_OTHER.

Attribute Required Default Value Description
value Yes Property value.
label Yes Label for property value to be displayed in the UI.

Tag: dependencies

An optional container tag for property dependencies (propertyDependency tags). It is used when we need to make a field appear only when user selects a specific value for another property.

For example, the queryUpdate property that contains the incremental query should be displayed only if the incremental property is set to true.

There can be more than one dependency. The current property will be displayed only if all dependencies are satisfied.

Tag: propertyDependency

Represents a single property dependency.

Attribute Required Default Value Description
propertyName Yes Name of dependency property to be checked.
propertyValue Yes Value of dependency property to be checked.
inverse No false When set to true, the dependency condition is inverted, which means that the current property will be displayed only when this dependency condition is NOT satisfied. For example, this can be used when we need to check that a field is not empty.

© Incorta, Inc. All Rights Reserved.