Automatic Mid-Flight Schema Propagation in Live CDC Pipelines

Automatic mid-flight schema propagation keeps live CDC pipelines aligned with upstream schema changes by treating DDL as ordered events within the pipeline. The engine detects each change, applies it at the destination before the row events that depend on it, and keeps processing without stopping the whole pipeline. For automatic schema evolution to be safe, it also needs idempotent recovery, log capture that can be trusted, and clear rules for destructive changes.

Why schema changes break CDC pipelines in ways that are hard to detect

An architect hands over revised drawings midway through construction, and every trade has to stop while the plans are reconciled. A data pipeline faces the same mismatch when a developer adds a column, a DBA renames a field, or an ORM migration drops a legacy table. But the pipeline may not stop at all. Its consumers can keep running with assumptions that no longer match the source and produce wrong data with no clear sign that something has failed.

The consequences depend on the kind of change. An added column can make a fixed-schema consumer fail to parse events, or the consumer may silently discard the new field. A dropped column can break downstream SQL, while a stale destination schema can allow queries to keep succeeding against an obsolete structure. A rename is especially dangerous when interpreted as a drop followed by an add, because that interpretation can cause irreversible data loss.

Older and newer CDC events stored in a streaming platform may use different schema versions, so the current schema may not be enough to read the whole history. Pinterest Engineering describes how an unhandled change in a distributed pipeline can also break Flink jobs, block Spark upserts, and leave online and offline representations out of step. The failure is therefore not confined to the altered table.

What the transaction log actually contains, and why DDL events live there alongside DML

PostgreSQL records changes in its write-ahead log, or WAL, while MySQL uses its binlog and MongoDB uses its oplog. These logs record committed changes in the order they were committed. That ordering gives log-based CDC the basis for relating a change in structure to the data written under it, although the events exposed to a connector differ by database.

Log-based capture also has low latency, puts little load on the source, and reliably captures deletes. For schema propagation, however, commit order is the decisive property. Query-based and trigger-based CDC cannot see the same DDL events, so schema detection has to take place outside their capture path. This is a structural limitation rather than a problem that faster polling alone can solve.

PostgreSQL has an important gap between logging and decoding. Logical decoding converts WAL changes into row-level events through an output plugin, but it does not include DDL directly in that stream. An ALTER TABLE on the source therefore does not arrive as a decoded DDL event. A propagation layer must have its own DDL detection process and keep track of how the changed schema relates to the rows that follow.

MongoDB presents a different model through Change Streams. This high-level API over the oplog uses resume tokens rather than LSN or GTID positions for resumability, and its events reflect actual document structure, including nested fields. Its schema-free design changes what schema evolution means. The downstream system still has to interpret changing structures, even when the source does not impose a fixed table definition.

How do you handle upstream schema changes without stopping your CDC pipeline

Automatic propagation begins by detecting the structural change at its position in the change history. Where the capture stream exposes DDL, the engine intercepts that event directly. Where it does not, as with PostgreSQL logical decoding, a separate detection process must close the gap. Detection alone is insufficient because the engine also needs to establish which row events depend on the revised schema.

The next requirement is ordering at the destination. The change to the structure must take effect before any later DML that relies on it. An insert containing a new column cannot safely precede creation of that column in the destination table. Treating DDL as first-class means routing it through the ordered processing path, rather than skipping it or leaving it in a separate backlog while dependent rows pass.

Once the destination applies the change, it can reopen the gate for the rows that depend on it while the rest of the pipeline keeps processing. Automatic handling also does not mean approving every possible operation, because an additive change has a different risk from a rename or type narrowing. Artie, a managed CDC-as-a-service platform that streams row-level changes from operational databases to cloud data warehouses without teams having to build their own capture infrastructure, handles additive schema changes mid-flight while routing destructive ones through guard logic. It is usually safe to propagate column additions automatically, while destructive changes need guard logic.

Flink CDC makes that distinction configurable by event type. Its schema synchronization controls cover create table, add column, alter column type, drop column, rename column, drop table, and truncate table. Operators can choose which categories to include or exclude for a sink, so some changes can propagate automatically and others need a human decision. A rename must keep its meaning, because turning it into a drop-plus-add sequence can destroy data.

Pinterest's framework handles changes across more than one destination table. Its propagation based on service-level commitments preserves availability while schema and data correctness converge across layers, rather than halting the entire stream until every layer confirms the change. Local ordering still has to hold wherever rows depend on a revised structure. PR-based rollout, versioning, auditing, and rollback paths extend handling beyond passing ALTER TABLE statements through to one sink, making those changes controlled artifacts across that convergence process.

What happens when schema propagation is under-engineered, with AWS DMS as a reference case

AWS DMS supports DDL statements but does not provide comprehensive schema evolution. Its design reflects migration work rather than schema evolution as a governing requirement. What is supported depends on which source and target engines are used together. That makes its behavior an instructive architectural contrast while leaving room for successful schema operations.

AWS DMS documentation lists drop column and change column data type among the CDC statements it supports, but says some source engines do not capture all supported DDL. AWS Prescriptive Guidance also states that schema changes require ongoing maintenance. The useful distinction is therefore not “supports DDL” versus “does not support DDL.” It is whether a particular source-to-target path reliably propagates additions, drops, type changes, and nested structures without manual reconfiguration.

Where automatic handling falls short, recovery can require stopping the task, editing its JSON mapping, rebuilding the target schema, and relaunching. That workflow does not itself provide schema versioning, rollback, or an audit trail. Throughput can compound the interruption. AWS describes relational-target CDC processing as single-threaded per task, unlike full-load parallelism, leaving replication-instance sizing and task sharding by table as manual responses to high-change-rate workloads.

A pipeline either treats the relevant DDL as part of its controlled change stream or leaves operators responsible for stopping and rebuilding around it. A healthy task status alone does not establish that the destination structure is correct.

The PostgreSQL WAL slot hazard that schema propagation systems must account for

A PostgreSQL connector can fail in a way that threatens the source database itself. An inactive logical replication slot retains WAL that the connector has not consumed. With the default max_slot_wal_keep_size = -1, PostgreSQL places no limit on that retention. If a dead connector goes unnoticed, the pg_wal directory can grow until the disk fills and the database stops accepting writes.

Schema propagation depends on the same capture path, so reliable DDL handling cannot be evaluated independently of slot health. A system that understands schema changes but leaves an abandoned slot retaining WAL is not operationally safe.

PostgreSQL 17, released September 26, 2024, added a concrete improvement for source failover. Slots created with failover = true propagate to physical standbys through the slotsync worker, allowing a promoted standby to resume CDC without a full re-snapshot. Prior versions required external tooling or a complete re-snapshot after primary failover. This improves continuity but does not remove the need to account for schema detection.

The propagation system still needs to resume cleanly. In particular, its DDL detection process must account for changes made during the primary-to-standby transition, rather than assuming slot continuity alone proves schema continuity. Slot monitoring and failover recovery are therefore part of the evidence needed to trust automatic propagation.

Why delivery semantics determine whether schema propagation is safe to automate at all

CDC delivery is inherently at-least-once, so a failure or restart can cause an event to arrive again. The destination must know when a schema or data event has been replayed and avoid applying its effect twice.

That guarantee depends on idempotent sink logic. For PostgreSQL, an LSN can identify replay position, while MongoDB uses a resume token. The sink must use that identity to tell when an event has arrived again and avoid treating it as a fresh operation. An ALTER TABLE ADD COLUMN that succeeded before a crash must become a no-op on replay, just as a repeated insert must not create a duplicate.

A schema change may succeed at Snowflake and fail at BigQuery, leaving the destinations inconsistent until recovery resolves the partial failure. Each destination must independently handle re-delivered DDL and DML without double application. A shared upstream checkpoint is not, by itself, evidence of that behavior.

Teams assessing automatic propagation should therefore evaluate delivery semantics at both the sink and the transport. An at-least-once stream can have exactly-once effects if it uses idempotent application, but a claim of exactly-once must hold for both changes to the structure and row writes. Otherwise, the first crash can invalidate the ordering guarantees that appeared sound during uninterrupted processing.

How schema stability becomes a hard dependency for AI agents and feature pipelines

A dashboard query may expose a schema mismatch by failing. An AI feature pipeline can instead pass incomplete data to a model that continues producing outputs. If a feature store silently drops a column, an agent may receive a vector with a missing dimension and no error signal. The input no longer means what the model was trained to expect.

Freshness and structural correctness are separate dependencies here. A 60-second replication lag leaves an agent acting on a world model that is slightly out of date. An unhandled schema change can leave that model structurally wrong, even when events otherwise arrive promptly.

Databricks' AUTO CDC INTO construct in Lakeflow Declarative Pipelines allows dashboards and AI models to consume continuously updated, historically rich data. For that to work, upstream schema changes must reach it correctly. Continuous processing does not establish semantic continuity. The propagation path must keep the structure that gives the arriving data its meaning.

HIPAA audit logs must be retained and regularly reviewed, so a schema change that breaks audit-record writes creates a compliance gap as well as missing data. In these settings, a pipeline that continues running is not necessarily a pipeline that continues fulfilling its function.

What to evaluate in a CDC tool's schema propagation claims

A claim to support schema evolution should be tested against five properties. First, the documented event set should include at minimum add column, alter column type, drop column, and rename column, with clear guard behavior for destructive operations. Coverage must be specific to the source and destination. Flink CDC's event-type controls illustrate the distinction between a supported operation and one permitted to propagate automatically.

Second, the tool must preserve DDL ordering relative to DML. The test is whether the destination applies a structural change before processing rows that depend on it, including when schema detection sits outside the decoded log stream. Third, recovery must provide idempotent application for both DDL and DML. A delivery label alone is insufficient without an explanation of what happens when a successful operation is replayed after a crash.

Fourth, the tool must check replication slot health and recover from source failover while catching all changes to the structure made during the transition. PostgreSQL deployments need an answer covering both WAL continuity and the separate DDL detection path. Fifth, schema changes need versioning, rollback paths, alerts, and an audit trail.

The best fit depends on the operating requirement. A migration workflow may allow for manual maintenance that is documented, while a feature pipeline that runs continuously needs ordered propagation and safe recovery as part of how it normally works. A purpose-built managed replication service should have schema evolution, exactly-once effects, and slot monitoring in place by default when it goes into production, without needing to add them later. The decisive evidence is what the system does when a schema change and a failure occur together.