Why Custom Fan-In Pipelines Collapse Under Multi-Source CDC and What Replaces Them

Fan-in CDC replication merges changes from multiple operational databases into a single warehouse. Custom pipelines become easy to break because each new source has its own schema, ordering, and recovery needs that affect one another at the merge point. The durable replacement is a managed replication layer that treats multi-source merging, schema evolution, and exactly-once delivery as built-in infrastructure.

What fan-in means in a CDC context and why teams reach for it

Most companies do not run a single database. Their analytics teams nevertheless need a unified view of the data those databases hold. Fan-in CDC meets that need by capturing changes from several sources and bringing them into one warehouse, sometimes into a single destination table. A typical deployment might combine Postgres in one region, MySQL in another, and MongoDB behind a third service.

Streamkap’s multi-source CDC guide identifies three drivers of this pattern, namely database-per-service microservices, regional database sharding, and parallel-running legacy-to-modern migrations. A retailer might merge identical orders tables from us-east, eu-west, and ap-southeast Postgres clusters into a global_orders table in Snowflake. An e-commerce platform might bring users from Postgres, orders from MySQL, and payments from MongoDB together for a revenue dashboard. During a migration, CDC from both the legacy system and Postgres can keep analytics continuous.

Teams often start with streaming components they already know. Each source connector writes to its own Kafka topic, while Flink, ksqlDB, or a custom consumer reads the topics and normalizes their schemas. That processing layer writes to a single output topic for the destination connector to consume. In a controlled environment, the result is straightforward, with one destination serving a unified view of production data. At two or three sources, however, that clean arrangement can conceal the structural exposure that emerges as more branches converge.

How the fan-in topology multiplies failure exposure with every source added

A schema mismatch in one branch can disrupt shared normalization, while a poisoned source can corrupt the merged output. Adding a source therefore adds more than another connector to maintain. It adds another set of changes, retries, and recovery states that must coexist with those already feeding the destination.

Latency becomes coupled when the merge waits for branches. In a width-8 configuration with that dependency, the critical path follows the slowest branch. A connector stuck retrying can hold back the combined result even while the other connectors remain healthy.

A Kafka cluster, stream processor, or destination connector can impair several branches in the same window, so treating connector failures as independent understates the exposure. More sources also create more combinations of schema versions, retry positions, and ordering conditions for the merge to reconcile. These states affect one another in more combinations as sources are added, but there is no universal numerical failure formula. Fixing an individual connector bug does not remove the dependencies that allow the next branch-level fault to propagate.

The three specific failure modes that surface in production fan-in pipelines

Schema conflicts begin with differences that normalization must reconcile. Postgres might represent created_at as a timestamp while MySQL supplies creation_date as a datetime string. Adding an upstream column then changes one input, but the shared transformation that consumes it may stay the same. Postgres logical decoding makes this harder because it does not emit DDL itself, even though the WAL records the underlying catalog changes. An ALTER TABLE therefore requires a separate process to detect and propagate the schema change.

Without that process, the normalization layer can get data with a structure it no longer expects. A shared merge makes this more consequential than an isolated mapping error. That same transformation handles streams from other sources too, even though their schemas have not changed. When teams handle schemas by hand, each upstream change is another chance for something to break. Ordinary database evolution can cause this failure even when the application has no unusual defect.

Independent databases have their own clocks and commit sequences, with no global transaction order between them. DynamoDB Streams orders events only within each shard, and auto-scaling can close a parent shard and open child shards. MongoDB Change Streams adds a separate complication because update events contain modified fields by default. Its fullDocument: 'updateLookup' option reads the current document after the event, so another intervening change can make that document differ from the state at the event.

Streamkap’s guide identifies partitioning by entity key as a mitigation for out-of-order events. Routing an entity’s events to the same partition does not, however, erase differences in source guarantees or make a later document lookup an event-time snapshot. Those cases require explicit source handling. Silent data loss is another possible result of incomplete handling, as a DynamoDB consumer that fails to enumerate child shards misses their events entirely. If an outage lasts beyond the stream’s retention window, a full table re-scan is needed instead.

In Postgres, a replication slot persists on the production database and, by default, retains WAL without a limit until the consumer advances. A stalled consumer can therefore let files accumulate until the disk fills and Postgres crashes or refuses new writes. If the shared pipeline stops, every connected Postgres slot can accumulate WAL at once. Monitoring can reveal these conditions, but it cannot supply missing schema propagation, shard traversal, or recovery behavior.

Why exactly-once delivery and at-least-once delivery are not the same problem, and what that means for fan-in

Even correct schema handling and ordering leave delivery unresolved. At-least-once delivery allows an event to arrive again, whereas exactly-once processing must prevent that retry from producing a second effect in the destination. Network retries after connector crashes, Kafka producer retries on timeout, and consumer group rebalances can all lead to repeated processing. A pipeline that just forwards those events leaves downstream systems to handle the duplicates. In fan-in, each source contributes its own retry behavior.

The accepted approach combines a stream processor’s exactly-once mode with idempotent sinks, so replay does not create an additional destination effect. That design must account for each source’s sequence identifiers and failure modes. A checkpoint or log sequence number records where consumption stopped, but it does not by itself establish that recovery will avoid loss or duplication. With N sources, a restart must pick up correctly from each of the N independent cursors.

Schema evolution makes replay more difficult still. A duplicate that arrives after a schema change may be read differently than it was the first time, so destination records can differ even though they come from the same source event. Deduplication and schema handling therefore cannot be designed as unrelated features. Their interaction belongs in the recovery model from the start. Adding another source extends that model.

What AWS DMS does and does not handle in a multi-source replication context

AWS DMS has a clear strength in migration speed. It reduces upfront engineering for migrations to AWS, helping shorten cutovers and limit associated business disruption. Continuous multi-source replication needs to keep working after a migration is complete. It has to keep the merged destination correct as sources change and fail again and again.

DMS supports several DDL operations, but support has qualifications. AWS DMS’s supported DDL page lists operations including dropping columns and changing column data types. The AWS Knowledge Center describes scenarios in which such changes can still suspend tables or require manual remediation. More complex nested structures and certain mid-stream changes may require intervention or halt replication.

Configuration and recovery add to the operating burden. Complex CDC setups can involve manual JSON configuration, although some settings are available through the DMS console. Limited automation for schema changes and failure recovery leaves teams responsible for handling more of the ongoing replication work. Each new source has its own configuration, schema history, and needs for recovery. DMS can reduce the effort of getting data moving without resolving the shared merge and delivery problems that define long-running fan-in.

Migration-oriented tooling is useful when migration speed is the main objective. A continuously updated warehouse instead needs source-specific recovery and schema handling to remain dependable after the initial load. In that context, manual intervention recurs whenever an independent source changes in a way the replication task cannot absorb.

How do you consolidate CDC streams from five production databases into one Snowflake warehouse?

A managed replication layer replaces the shared custom merge with built-in multi-source handling. Streamkap’s guide describes a connector-level pattern in which each source connector performs field renames, type conversions, and metadata injection before writing directly to the destination. This removes the need for a separate stream processing layer just to normalize and merge topics. Source differences are handled where they enter the replication system.

Schema detection and propagation must accompany those transformations. Applying supported changes automatically keeps destination tables in step with their sources, so a team does not have to edit and restart tasks for every upstream change. Initial loading also needs to coexist with production workloads. Historical backfills that avoid table locks and can read from replicas without interrupting ongoing replication protect source health during that phase.

The managed layer still has to handle recovery for each source. Postgres requires slot monitoring and controlled backpressure, DynamoDB requires child shard enumeration, and MongoDB document handling requires consistent read strategies. A service must incorporate those behaviors rather than leave them to separate consumer implementations. Exactly-once delivery must also cover retries and recovery without silently skipping ingestion errors. Otherwise, managed connectors merely relocate the custom pipeline’s unresolved work.

For a team consolidating five databases into Snowflake, those are the criteria for the replacement architecture. The evaluation turns on connector-level normalization, automatic schema handling, non-disruptive backfills, source-aware recovery, and destination delivery semantics. The managed layer must take responsibility for the interactions between them, including the processing infrastructure needed for deduplication, filtering, or enrichment. That is the distinction between buying connectors and adopting fan-in as a pre-solved replication capability.

How real-time fan-in replication connects to the downstream systems that depend on fresh data

With reliable fan-in, warehouse consumers can use a combined view of production that is up to date. Traditional batch ETL can leave hours between a source change and its arrival in analytics. CDC removes that scheduled delay, provided the merge does not reintroduce it through stalled branches or unresolved schema changes. For the retailer’s global_orders table, freshness must hold across regions.

That expectation extends beyond conventional analytics workloads. Streamkap’s guide identifies agentic data streaming infrastructure as a category built to keep delivering real-time data to AI agents. Agents operating on merged operational data depend on both its freshness and its consistency. A current record from one service does not compensate for stale related data from another. Reliable replication gives those applications a sounder input, without claiming to guarantee the quality of every downstream output.

Real-time dashboards and A/B testing frameworks have the same dependency. Their warehouse must keep up with changes across all its inputs, since it can look live even when some inputs have stopped moving. When replication preserves that relationship, teams can use the combined data without building a separate recovery project around each new source. Sub-minute, exactly-once, multi-source replication should be the default expectation.