Setting Up Data Sync
Set up full and incremental data sync pipelines in Matillion to load data from external applications into PTC Orbit through Snowflake.
Matillion performs full and incremental data syncs from external applications to PTC Orbit stage objects stored in Snowflake. This topic covers enabling data ingestion modes on objects, configuring orchestration and transformation pipelines for Salesforce data, syncing from AWS S3, creating custom connectors, and importing pre-built pipelines.
Ingestion Modes
PTC Orbit stage objects support two ingestion modes:
• Staging Review: Data is staged for user review before committing to the database. For example, Asset object.
• Pre-DB Sync: Data is ingested, temporarily staged, then written to the database via sync system jobs automatically. For example, Account object.
Perform the following steps to enable an ingestion mode and configure sync pipelines for a customer.
1. Log in to the customer PTC Orbit tenant.
2. In Max Designer, click Objects on the Developer Tools launchpad menu. In the list view, click the object record to configure.
3. Set the Data Ingestion Mode field to Staging Review or Pre-DB Sync, then click Save.
4. On the Development Actions launchpad menu, click Synchronize to apply the change.
The object is ready for data ingestion through Matillion pipelines.
System Jobs for Data Sync
PTC Orbit uses two system jobs to synchronize staging records from Snowflake to the PTC Orbit PostgreSQL database:
• Pull Pre-DB Sync Records from Snowflake: Syncs Pre-DB Sync records with a status of New or Failed to Sync from Snowflake to their corresponding business object tables.
• Sync Unified Records: Syncs Data Review Stage Unified records with a status of Auto Mapped, Reviewed, or Sync Failed from Snowflake to the PTC Orbit database.