Pipelines and Schemas
Calyx RIM uses different types of pipelines and schemas for DaaS architecture.
Pipeline
A database pipeline enables flow of data from source to destination. There are two types of pipelines:
- Full Load: In the Full Load pipeline, an existing dataset is replaced by the newly updated dataset in the next data loading. All datasets will have an initial full load.
- Incremental Load: In the Incremental Load pipeline, the data coming from the source is compared with the existing data in the destination. Only data that has been changed from the source will be replaced.
Schemas
The database schema structure used by
Calyx RIM includes:
- Audit Schema: The Audit schema include an overview of the fields used, the data and the source of the data. All the audit tables from the Calyx RIM Oracle database are copied to the s_audit table of SQL database using Azure Data Factory.
- Reporting Schema: The Reporting schema defines the tables, the fields in each table, and the relationships between fields and tables. All the MGR, ODS tables from the Calyx RIM Oracle database are copied to s_reporting of SQL database using Azure Data Factory.
Work with Data
Before copying, replacing or further data processing, check if the data is current by selecting the last_refresh_date from dbcommon.df_run_parameters where run_key=<run_key>. Run_key is a unique string key used to get pipeline settings from df_run_parameters.Run_key defines the Table_owner,, Include_list and Exclude list.
A table is refreshed every five minutes.