Data-as-a-Service (DaaS)

Pipelines and Schemas

Ennov InSight uses different types of pipelines and schemas for DaaS architecture.

Pipeline

Database pipelines allow the flow of data from source to destination. There are two types of pipelines:
  • Full Load: In the Full Load pipeline, a current dataset is replaced by an updated one in the next data loading. All datasets have an initial full load.
  • Incremental Load: In the Incremental Load pipeline, the data coming from the source is compared with the data in the destination. Only changed data is replaced.

Schemas

The database schema structure used by Ennov InSight includes:
  • Audit Schema: The Audit schema includes an overview of the fields used, the data, and the source of the data. All the audit tables from the Ennov InSight Oracle database are copied to the s_audit table of the SQL database using Azure Data Factory.
  • Reporting Schema: The Reporting schema defines the tables, the fields in each table, and the relationships between fields and tables. All the MGR and ODS tables from the Ennov InSight Oracle database are copied to s_reporting of SQL database using Azure Data Factory.

Work with Data

Before copying, replacing, or further data processing, check if the data is current by selecting the last_refresh_date from dbcommon.df_run_parameters where run_key=<run_key>. Run_key is a unique string key used to get pipeline settings from df_run_parameters.Run_key defines the Table_owner, Include_list, and Exclude list.

Tables are refreshed every five minutes.