Materialized Views are abstractions that behave like tables in Incorta. Materialized Views allow users to write python code that can be executed against the Spark server using the pySpark API. Use Materialized Views to apply additional transformations to the data before loading the data into Incorta.
Materialized views can be used for any complex transformational logic and usually implemented with languages like pl/SQL.Incorta runs the pySpark code on the data in the staging area, then loads it to the Incorta engine. The new table can be joined to other tables like any other table in the schema. You can also create formula columns on Materialized views and add security filters.
When using Materialized Views and the Spark Server for data transformation, install the Spark server as a separate server in a standalone cluster mode. For more information about installing the Spark server in a standalone cluster mode, see Spark documentation.
Configuring Incorta to work with a Spark server requires that you:
- Modify the Spark configuration files
- Start Spark
- Start the Spark slave
Materialized Views and Spark
A Materialized View in Incorta is an object type that contains the results from an advanced transformation program. For example, you can write Spark Python code to summarize the data using an aggregated function. The materialized view can be scheduled to load periodically like any other Incorta object. Once the materialized view is populated, it can be used like other object. You can define aliases, joins, and define formula columns. You can query the materialized view in an Insight, and you can define a business schema using a materialized view.
The Materialized View read operation requires a specific Hive configuration. To ensure the Spark SQL used to access Hive can be executed, the Hive metastore used by the Spark instance should use either an Oracle or MySQL database instance.
Create a Materialized View
To create a Materialized View:
- Navigate to the “Schemas” tab of the “Schema” page.
- Select the schema where you want to add a materialized view.
- Select the + New icon in the upper right-hand corner of the screen, and select “Materialized View”.
- Choose a language from the “Language” drop-down list.
- Paste the script in the “Script” field.
- Select Add Property to provide the “key” and “value” in the respective fields.
- Select Add.