MyLab: Database replication to BigQuery using change data capture

Databases like MySQL, Oracle, and SAP are the most often discussed CDC data sources. However, any system can be considered a data source if it captures and provides changes to data elements that are identified by primary keys. If a system doesn’t provide a built-in CDC process, such as a transaction log, you can deploy an incremental batch reader to get changes.

This document discusses CDC processes that meet the following criteria:

  1. Data replication captures changes for each table separately.
  2. Every table has a primary key or a composite primary key.
  3. Every emitted CDC event is assigned a monotonically increasing change ID, usually a numeric value like a transaction ID or a timestamp.
  4. Every CDC event contains the complete state of the row that changed.

For more detail, please see the references below.


You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *