Exactly-Once Delivery Semantics in CDC Pipelines

A connector restarts after a crash, replays the last thousand events from its confirmed log position, and revenue metrics double, but no alert fires. Exactly-once delivery in a CDC pipeline is never a configuration flag. It is a coordinated guarantee that must hold simultaneously at the source, the transport, and the destination, and most production pipelines fail at least one layer silently.

The term "exactly-once" appears throughout marketing copy, architectural diagrams, and documentation, but the promise it makes varies by layer. At the source it means the change stream is ordered and replayable from a durable cursor. At the transport it means retries and rebalances cannot produce duplicate records. At the destination it means repeated writes cannot alter final table state. Each layer answers a different question, and conflating them is how pipelines ship with gaps.

Streaming pipelines operate under three delivery postures. At-most-once delivery may drop events but is simplest to implement, and it is unacceptable for production data replication. Exactly-once delivery guarantees each event arrives exactly one time, even across crashes and retries, and it requires active coordination at every layer. Conduktor's CDC fundamentals treats this distinction as a core implementation consideration: CDC may deliver duplicate events, so it recommends idempotent processing or deduplication as the practical defense.

The strongest objection deserves acknowledgment early: most systems marketed as exactly-once are in practice at-least-once pipelines with idempotent sinks. The observable outcome can be identical, but the failure modes are not, and that distinction carries real consequences. The remainder of this argument resolves it.

How log-based CDC creates the conditions for exactly-once delivery at the source

Log-based CDC reads the database's own transaction log rather than polling tables or firing triggers. It is the prerequisite for exactly-once delivery because it is the only CDC pattern that provides ordered, complete, replayable change events without modifying the source schema or adding write overhead. The same guide's warehousing discussion notes that log-based CDC is generally preferred for data warehousing because it captures all changes reliably without requiring schema modifications.

The log position is the atomic unit of the exactly-once contract at the source. Whether it is an LSN in Postgres, an offset in Kafka, or an SCN in Oracle, it tells the consumer exactly where its last confirmed read ended, and the database will not discard log data the consumer still needs. This creates a critical discipline: a consumer must confirm durability of a message before it advances its log position. The cursor only protects delivery if the pipeline treats confirmation as the gate, not the afterthought.

How Postgres, MongoDB, and DynamoDB each implement the source-layer guarantee differently

Postgres replication slots, MongoDB change streams, and DynamoDB Streams each provide the ordered, replayable log that exactly-once delivery requires, but each carries its own operational failure mode that the pipeline must account for. Postgres implements the source-layer guarantee through logical decoding, which reads the WAL through a named, crash-safe cursor called a replication slot. The slot records how far the consumer has confirmed, and Postgres will not remove WAL the slot still needs.

Two edge cases break naive implementations. Schema evolution requires a separate, coordinated process outside the replication protocol, a constraint that cascades through the remaining layers. Postgres 17 introduced logical replication failover so slot state can be synchronized to promotion candidates, but a standby only becomes eligible once its synced slot transitions from temporary to persistent. That transition requires the standby's retained WAL to cover the slot's restart_lsn and confirmed_flush_lsn, so freshly added standbys remain ineligible until the subscriber has advanced and the WAL positions are consistent.

MongoDB change streams consume the database's operations log directly, providing near-real-time data change events natively. Consuming a change log to feed downstream systems is a first-class MongoDB capability, not a workaround. OneUptime's CDC implementation guide identifies change streams as the recommended mechanism for MongoDB CDC.

DynamoDB Streams takes the managed, serverless approach. There is no slot to manage and no WAL retention risk, but the service carries a hard constraint: stream data is retained for only 24 hours, making it mandatory that downstream consumers keep pace. The NEW_AND_OLD_IMAGES stream type provides before-and-after row images, which CDC consumers need to reconstruct a full change event. Each database mechanism is replayable and ordered, but each fails in its own distinctive way, and pipeline architecture must be designed around that specific failure mode rather than a generic one.

What the transport layer must do to preserve the guarantee the source established

The transport layer, the message bus that carries CDC events from the source connector to the downstream processor, is where exactly-once guarantees are most commonly broken in practice. Network failures, producer retries, and consumer rebalances each create opportunities for duplication that require active coordination to suppress. A consumer group rebalance can cause a consumer to reprocess events it already handled if offsets were not committed before the rebalance.

Apache Kafka addresses these failure modes with two mechanisms. Transactional APIs provide atomic writes across multiple partitions, so a batch of CDC events either all lands or none do. Kafka 4.0 now runs in KRaft mode without ZooKeeper, using its built-in Raft consensus protocol for metadata management. This simplifies CDC deployments by eliminating ZooKeeper as a separate failure domain and improving failover speed.

Partitioning is where the transport layer either preserves ordering or silently destroys it. A consumer that processes partition A and partition B in parallel will corrupt state if an UPDATE to row 42 in partition A races with a DELETE to row 42 in partition B. Schema Registry integration extends the transport layer's responsibility beyond delivery: Avro serialization with a Schema Registry attached to the event bus provides type safety and schema evolution support, so a column addition at the source does not produce undeserved parse failures in consumers.

How the destination layer closes the exactly-once loop with idempotent writes

A transport layer that delivers events exactly once is insufficient if the destination writes them in a way that allows duplicates to accumulate. The warehouse must either participate in a distributed transaction with the transport or implement idempotent write patterns that produce the same table state regardless of how many times an event arrives. The two destination-layer approaches differ in what they preserve and what they expose. MERGE, by contrast, applies inserts, updates, and deletes to the target table in a single atomic statement, matching on primary key. Conduktor's CDC warehousing guide provides this MERGE pattern explicitly, handling UPDATE, DELETE, and INSERT cases in one statement, and it is the standard idempotent write pattern for CDC into a warehouse.

Apache Flink often sits between transport and destination as a stream-processing layer, consuming CDC events from Kafka and providing exactly-once processing semantics that guarantee each change event is processed once even during failures. That interval is typically acceptable for analytical workloads but becomes a tuning variable for near-real-time requirements. Slowly Changing Dimensions Type 2 offers another destination pattern for analytical use cases requiring historical context: each CDC change generates a new row with validity timestamps rather than overwriting the existing row, preserving history so queries can reconstruct any past state.

When the transport layer cannot guarantee exactly-once delivery, deduplication at the destination serves as the fallback. The pipeline tooling must detect DDL out-of-band and propagate it to the warehouse. Tools that require manual intervention for schema changes create the same data-loss risk as a delivery gap.

Why "exactly-once" in most production pipelines means idempotent sinks, not distributed transactions — and when that distinction matters

The strongest objection to the three-layer exactly-once model is also largely correct. The real goal is exactly-once semantics at the table level, even if the compute executes at-least-once. This is defensible because an idempotent MERGE into a warehouse table produces the same row state whether it runs once or ten times. From the analyst's or AI model's perspective, the table is correct.

The distinction matters in three specific domains. Financial transaction counts fail first: if an event bus replays 1,000 payment events and the sink is append-only rather than idempotent, revenue figures are wrong before anyone notices. AI agents and real-time dashboards present a subtler risk, because models or dashboards trained on a table that briefly contained duplicate state during a replay window may cache incorrect outputs that persist after the duplicates are cleaned up.

The practitioner's decision framework follows from those distinctions. If the workload is analytical and the destination supports MERGE, idempotent sinks are sufficient, and the investment belongs in sink correctness rather than full transactional exactly-once. If the checkpoint interval, sixty seconds for Flink in the common case, is acceptable latency, the checkpoint-based approach is both simpler and sufficient. In CDC pipelines, it is the only way to survive retries, reprocessing, and late-arriving data without manual cleanup, and building it into the sink from the start is lower risk than attempting to coordinate distributed transactions across heterogeneous systems.

Schema evolution as the layer of exactly-once delivery that most pipeline documentation ignores

A pipeline that delivers every row change exactly once but mishandles a source schema change will corrupt the destination table just as surely as a duplicate event. Schema evolution is a delivery correctness problem, not a configuration problem, and it is the failure mode that most pipeline documentation leaves unaddressed. The failure takes several forms. A column rename produces events where a key the destination expects is absent and a new key the destination ignores is present, so the row lands with a null where a value should be. A data type change, INTEGER to BIGINT for example, may pass through the pipeline without error and corrupt the destination column silently if the destination type is not widened to match.

The Postgres-specific complication makes this failure mode structural rather than incidental. Logical decoding does not stream DDL events, so an ALTER TABLE at the source does not appear in the replication stream at all. Conduktor's CDC fundamentals identifies Avro serialization with Schema Registry as the mechanism for schema evolution support: new versions of a schema are registered before the first event using the new shape is emitted, giving consumers a chance to adapt. Breaking changes, column renames, type narrowing, or column removal, require explicit handling, and pipelines that pause and require operator intervention are safer than ones that silently continue with a mismatched schema.

The operational implication is direct. A CDC pipeline that handles delivery guarantees perfectly but requires manual intervention for every schema change is not production-grade. Managed CDC services that handle schema evolution automatically remove this class of silent corruption from the operational burden entirely. This is where a purpose-built replication service earns its cost over a DIY pipeline.

What a production-grade exactly-once CDC pipeline looks like end to end

A production-grade exactly-once CDC pipeline requires deliberate design at every layer: source log position management, transport-layer idempotence, idempotent sink writes, and automatic schema evolution. The gap between a DIY assembly of these layers and a purpose-built managed service is typically measured in months of engineering time, not days. The full stack assembled from the argument has five layers. The source uses log-based CDC with a crash-safe log position cursor, whether a replication slot, change stream, or managed stream, and the cursor advances only after downstream durability is confirmed. The transport uses idempotent producers partitioned by primary key, with transactional writes for workloads that cannot tolerate transient duplicates, and a Schema Registry attached for schema evolution support. A stream processor such as Flink, optional but common, provides checkpoint-based exactly-once semantics with a tunable checkpoint interval that sets the pipeline's maximum lag. The destination uses idempotent MERGE on primary key for operational tables, SCD Type 2 for historical analytical tables, and automatic schema propagation for additive DDL changes. A deduplication layer keyed by LSN serves as a safety net for at-least-once transport paths, discarding events whose LSN has already been confirmed.

The DIY cost is not trivial. DMS was designed for both database migrations and continuous data replication, and under sustained CDC workloads its architecture shows its limits: schema evolution support is limited, and structural changes at the source may require full-table resyncs. It prices by instance-hour and remains the path of least resistance for fully AWS-native pipelines, but it is not purpose-built for continuous exactly-once replication.

The passthrough architecture matters as a security property. For teams operating under HIPAA or SOC 2 constraints, where data residency and access controls are audit findings, this distinction is not theoretical. A pipeline that requires manual recovery from any of these events is not production-grade regardless of what its documentation claims.