Warehouse-Native vs Pipeline-Based Real-Time Ingestion

Most data teams ask themselves how fast their data needs to move, and then they build their ingestion strategy around the answer. That question matters, but it doesn't settle the architecture. A team can hit sub-minute latency with either of two fundamentally different setups, and the choice between them has far more to do with where the ingestion logic lives than with how quickly rows show up in a table. Does the scheduling, the secrets management, the compute, and the schema handling all sit inside the warehouse, or does it sit in a separate layer the warehouse never touches and never controls? Where the ingestion logic lives decides how a team builds, staffs, and operates its data infrastructure. Warehouse-native ingestion pulls scheduling, credentials, and compute into one system, where code, execution, and destination all live in the same place. Pipeline-based streaming ingestion keeps that layer separate, moving data continuously through an event broker as changes happen at the source, with a sink connector delivering it downstream. Both can be fast. Only one of them keeps everything under the warehouse's own roof.

Warehouse-native ingestion

Warehouse-native ingestion is a consolidation strategy. It trades the flexibility of an external pipeline for the simplicity of running everything, extraction, loading, and often transformation, inside the warehouse's own compute. The underlying pattern is ELT: get the data in first, then use the warehouse's own scalable compute to transform it, rather than routing it through a separate processing engine before it ever lands. The warehouse owns the compute, so it also owns the scheduling and the credentials needed to reach source systems. This model reads as unified rather than assembled from parts.

Warehouse vendors have been building toward this model deliberately. Databricks offers Auto Loader as a lakehouse-native ingestion feature (distinct from a warehouse-native one, since it operates on the lakehouse rather than a traditional warehouse). Amazon Redshift has auto-copy for automatically loading new files as they land. BigQuery has its Storage Write API for high-throughput, API-driven ingestion directly into the platform. Each of these exists to pull more of the ingestion job inside the walls of the platform itself, so customers have less reason to reach for a separate tool.

But none of this makes warehouse-native ingestion a slow or outdated choice. For a large share of analytics work, a well-tuned batch or micro-batch job delivers exactly the freshness the business needs, and it does so with far less to operate. Scheduled batch loads run at fixed intervals, and they remain the cheapest and most predictable way to populate a daily sales report or a weekly marketing dashboard, where freshness measured in hours is perfectly adequate. Micro-batching, where small loads run every few minutes rather than once a night, is where warehouse-native ingestion performs best: it gets a team into near-real-time territory for dashboards without asking them to operate anything outside the warehouse itself.

What the research shows about pipeline-based real-time ingestion

Pipeline-based ingestion splits the job into distinct stages, capture, transport, and load, each of which can scale, fail, or get replaced on its own. The pattern that shows up across most production deployments runs from the source database, through a log reader, into an event stream, out through a sink connector, and into the warehouse. Each stage is a separate piece of infrastructure, which is the opposite of the warehouse-native model's single, consolidated system.

Log-based change data capture is what makes this approach work. In Postgres, that means reading the Write-Ahead Log. In MySQL, it means reading the binlog. In MongoDB, it means reading change streams. These logs already exist for crash recovery, so when a CDC tool reads them, it reads something the database was going to write anyway, not new query load on production tables. That's the core advantage of log-based capture over the alternative: it captures every insert, update, and delete, in the order they happened, while adding almost no burden to the source system.

The pipeline-based family also has an alternative, query-based incremental capture: it periodically asks the source for any rows changed since the last run, usually using an updated-at timestamp. It's simpler to run and doesn't touch the transaction log, but it carries two real weaknesses: it can miss hard deletes unless the application is set up to soft-delete rows instead of removing them outright, and its latency is tied directly to how often it runs. A large share of teams assume they need full streaming log-based CDC when what they actually need is a reliable incremental load running every few minutes, and this is the single biggest cost decision a team makes in this category. Streaming log-based CDC earns its keep in fraud detection, real-time personalization, IoT sensor monitoring, and operational dashboards where the data has to reflect the current state of the system, not yesterday's.

Where each approach breaks down under production conditions

Each one has a specific point where it stops working cleanly, and that point is what a team should actually be evaluating against its own source systems and its own tolerance for operational risk.

Warehouse-native ingestion runs into trouble at the boundary of the source system itself. It has no way to reach into a production database's transaction log directly. It depends entirely on the database exposing its data through an API, a file export, or a replication slot the warehouse can poll or receive from, and if none of those exist in a form the warehouse can consume, the model simply has no path in. Micro-batch and incremental loads also miss hard deletes unless the application has already been built with a soft-delete convention, and that's a schema decision you have to make upstream, in the application itself, long before the warehouse ever sees the data. And once you need latency below a few minutes, the fixed scheduling cadence of warehouse-native compute turns into a hard floor. You can't tune that floor away; the team has to leave the warehouse-native model behind.

Pipeline-based ingestion runs into trouble on the operations side. Managing connectors, handling schema evolution, and tracking consumer offsets all cost real engineering time, and teams that underestimate that cost end up running a second system in parallel with the warehouse they were trying to simplify. Postgres surfaces this concretely: a replication slot created for a CDC consumer holds onto WAL data until that consumer confirms it has read it, so if the consumer falls behind or disconnects, the WAL keeps accumulating and the publisher's disk fills up. TOAST columns raise a subtler, more dangerous problem. When a TOASTed column, typically a large jsonb or text field, doesn't change during a given update, Postgres leaves it out of the WAL. Some CDC tools read that absence as a NULL value rather than "unchanged," which quietly corrupts the warehouse without triggering any error. Schema changes carry their own structural risk: renaming a column is a breaking change that forces a consumer migration, and because the pipeline sits between the source and the destination, the failure appears downstream, far from where the actual change happened.

The decision criteria that determine which model fits a team

The right model follows from four concrete factors: what the source system exposes, how fast the data needs to move, whether deletes have to be captured, and how much operational capacity the team actually has to spend. None of these have anything to do with which approach sounds more current.

Source exposure comes first. If the source is a SaaS application, a flat file, or an API, warehouse-native ingestion through connectors is usually enough, and it's simpler to run. If the source is a production relational database like Postgres or MySQL, or a document store like MongoDB, and deletes need to show up in the warehouse, log-based CDC inside a pipeline-based model is the fit that actually works.

Latency requirements come next. Batch replication is fine when freshness needs are measured in hours. Sub-minute freshness calls for a purpose-built CDC tool, full stop on that tradeoff. Micro-batching remains the warehouse-native sweet spot for near-real-time dashboards, so if your freshness needs are in the low-minutes range, a full CDC deployment may not be necessary.

Delete semantics decide the next branch. Query-based incremental loads miss hard deletes unless the application soft-deletes everything, so if the application performs physical deletes and the warehouse has to reflect them accurately, you can't treat log-based CDC as optional.

Operational capacity is the factor teams most often underestimate. Building and running a Kafka-based pipeline takes more engineering effort than most teams expect going in, and the real decision isn't whether to adopt CDC at all, it's whether to build and operate that infrastructure directly or hand it to a managed service. If a managed pipeline-based CDC platform handles schema evolution, merge logic, and failure recovery on its own, it removes the operational ceiling that makes DIY log-based CDC expensive to sustain. For teams choosing the pipeline-based route, the cost of maintaining that infrastructure from scratch, custom log readers, custom consumer code, can become its own failure mode well before the architecture itself does anything wrong. Managed CDC-as-a-service platforms exist to remove that operational surface, and they keep the benefits of log-based capture intact.

Destination variety matters last. Teams sending the same source changes to multiple destinations, a warehouse, a feature store, a search index, benefit from a pipeline-based model because the event stream becomes a shared resource all of those destinations can draw from. Teams with a single warehouse destination often find that warehouse-native ingestion gets them the same practical outcome, and they have far less to manage.

The hard parts pipeline-based CDC tools handle that warehouse-native cannot reach

Every log-based CDC tool can capture changes. That part of the job is table stakes. What separates a tool that works in production from one that collapses under its own proof-of-concept is how much of the downstream pipeline it handles without someone stepping in by hand, schema evolution, merge logic, monitoring, and recovery from failure.

A handful of capabilities mark that line clearly. Schema evolution means the tool detects a new column automatically and propagates it to the destination without manual intervention, so adding a field doesn't stall replication. Merge and delete handling means applying real MERGE logic and physical deletes at the destination, not simply appending every event and leaving the table to grow unbounded and inconsistent. Backfill and resync mean the tool can handle the initial historical load, and it can recover from a failed consumer without a full pipeline rebuild. Failure recovery means tracking the exact position in the WAL stream where a consumer stopped, so a restart picks up from that point instead of reprocessing events or skipping them. Exactly-once semantics mean the merge itself is idempotent, so reprocessing an event is a no-op rather than a duplicate row, which matters enormously for revenue tables and any destination where a duplicate throws off an aggregate. Backpressure handling means the system keeps buffering when the destination slows down, so it never blocks the source or silently drops events.

Log-based CDC tools like Artie take on these parts of the pipeline, schema evolution, failure recovery, exactly-once delivery, so a team isn't building or operating its own log readers and consumer code from zero. Artie is a fully managed CDC streaming platform that handles the full path from reading a database's transaction logs through delivering and merging changes into Snowflake, BigQuery, Redshift, Databricks, and other destinations. It automates schema evolution, merge logic including MERGE and DELETE handling, and backfills, running at typical latency under a minute. Internally, it uses Apache Kafka as a buffer, so ingestion keeps running even when the destination itself slows down, which avoids the backpressure failures that trip up pipelines built without that buffering layer. It reads transaction logs from Postgres, MySQL, MongoDB, SQL Server, and other sources, and the customer never has to stand up a Kafka cluster or connector infrastructure directly. It fits teams running production CDC pipelines where reliability and low latency into a cloud warehouse are the priority, though it carries fewer connectors than broad ELT platforms and focuses specifically on database-to-warehouse replication rather than wide SaaS app ingestion.

AWS DMS sits in the same pipeline-based category for teams already operating inside AWS infrastructure, and you can use it as another route into log-based replication without a fully independent build.

The entire batch-versus-stream debate is really about who operates what. A team can choose to run pipeline infrastructure itself, with all the WAL management, schema migration, and backpressure handling that entails, or it can choose ingestion that deploys in minutes with no separate infrastructure layer to staff and maintain. That decision determines architecture, and it's the one worth making deliberately rather than inheriting by accident.