CDC for Event-Driven Architecture

Moved a data platform from batch ETL to real-time streaming using Change Data Capture on MongoDB and Postgres, with reusable Python packages that hid the integration complexity from product teams.

Overview

Our data platform ran on nightly batch ETL, which meant the rest of the system was always a few hours behind the database. I led a small team to replace that with Change Data Capture, so a change in MongoDB or Postgres becomes an event the moment it happens and everything downstream can react in real time.

Why we did it

The old approach leaned on updating several APIs at once whenever data changed, and that turns out to be fragile. One slow service or a dropped request and you end up with state that disagrees across the platform. We wanted something that kept downstream systems in sync without product teams hand writing fan out logic, survived network blips, restarts and deploys without losing its place, and told consumers enough about each event that they could read it correctly without knowing how the source stored it.

What we built

Two Python packages did most of the work. CDC-Watcher tails the change streams on MongoDB and Postgres and publishes every change as an event. CDC-Consumers sits on the other side and hands those events to product code, keeping the awkward parts like offsets, retries and schema handling out of sight.

There is also a CLI built with Click for running and inspecting watchers. We later exposed the same operations as Django management commands so they fit the workflows people already had.

Keeping it reliable

Each stream carries a resume token, a marker for the last event it processed. If a watcher dies, a link drops or a cluster goes down, it picks back up from that token instead of replaying old data or skipping new data. The one case it cannot resume from is a long outage. MongoDB’s op log has a fixed size, so if a watcher stays offline long enough for the log to roll over, there is nothing left to resume against and the stream restarts from now.

Deploys needed the same care. The watcher listens for SIGINT and SIGTERM, and on either signal it stops taking new work and waits, usually 30 to 60 seconds, for the messages already in flight to finish before it exits. Nothing gets dropped in the middle of a release.

Working with async

The watcher runs on asyncio, which keeps it fast but means events do not arrive in a tidy line. Messages move through an async queue, and we drop a sentinel into that queue to mark a safe point to save progress. Resume tokens get persisted when a sentinel comes through or on a short timer, whichever lands first, so the saved position never drifts far behind what has actually been handled.

Events and schemas

Published events land on RabbitMQ streams, which behave like append only logs, close to a Kafka topic, and are distributed and highly available. Alongside every event we store its schema in a dedicated cdc_schema table. When someone adds a field to a watcher, a new schema version is written automatically, so the change shows up downstream without anyone hand editing a migration.

That part is what made the system pleasant to build on. Because each event travels with its field types and structure, a new product or UI can start reading a stream and interpret it correctly without knowing anything about how the source database is laid out.

Impact

Product teams could subscribe to data changes without learning any of the CDC machinery underneath, and that is what let event driven features spread across the platform. The data stayed in sync in real time, the watchers recovered on their own after failures, and schema changes flowed through without manual migrations.