DynamoDB Streams for Real-Time Replication to Data Warehouses

DynamoDB Streams captures a time-ordered record of item-level changes to a DynamoDB table and holds them for up to 24 hours. Moving that data into a warehouse in real time means building a pipeline around a set of fixed mechanical constraints, not treating the stream like an infinite log you can query at will.

DynamoDB Streams: what it captures and how its internals are structured

A DynamoDB stream is a change data capture mechanism built directly into the database. When a table has streaming turned on, every write produces a stream record, and AWS stores that record for up to 24 hours. That window is the first fact to internalize, because it sets the outer limit on how long a consumer can lag before data disappears for good.

Three kinds of events move through the stream: INSERT, MODIFY, and REMOVE. A pipeline that assumes global ordering will eventually process events out of sequence.

Underneath, stream records live in shards, the same conceptual unit as a Kafka partition. Tracking that shard lineage correctly is one of the more tedious problems in writing a stream consumer from scratch, and it's exactly the kind of detail that's easy to get wrong under load.

Turning on streams for an existing table doesn't interrupt the table, but it doesn't backfill history either. And if a stream gets disabled and later re-enabled, AWS issues a new stream ARN. None of this makes DynamoDB Streams a general-purpose transaction log or a replayable archive. It's a durable, time-bounded, item-scoped record of change, and every architectural decision that follows has to respect that definition.

Choosing a stream view type

Before any of the above matters operationally, an engineer has to pick what the stream actually emits for each change. DynamoDB offers four stream view types, and the choice determines what information every downstream consumer will ever have access to for a given event.

KEYS_ONLY sends just the partition key and sort key of the item that changed. NEW_AND_OLD_IMAGES sends both the before and after state together, and it's the only option that lets a consumer compute a diff without going back to query the source table.

View type is fixed at the moment the stream is enabled. Treat the view type choice as close to permanent from the start.

The 24-hour retention ceiling and the two-consumer fan-out limit as hard pipeline constraints

Two limits on DynamoDB Streams are not configurable, and both have real consequences in production. Records expire after 24 hours, and each shard allows at most two consumers reading from it at the same time.

The retention limit means there's no replay mechanism and no way to extend the window on the native stream. A pipeline that goes down for a weekend and comes back up on Monday hasn't just lost a weekend of throughput, it has lost that data permanently, with no way to recover it from the source stream.

The two-consumer limit shapes architecture just as directly. A team can't wire a replication pipeline, an audit logger, and a search indexer straight into the same DynamoDB stream, because the stream only supports two simultaneous readers per shard. Any fan-out beyond that has to go through an intermediary, typically SNS or SQS, or by moving to Kinesis Data Streams for DynamoDB, which raises both the consumer count and the retention window to as long as one year, at additional cost. That tradeoff makes Kinesis the right fit for high-throughput tables or for organizations that need more than two things reading the same change feed.

Throughput ceilings reinforce the same push. Each native shard supports up to 1,000 change events per second and allows up to two read transactions per second with up to 1 MB of data. None of these limits make DynamoDB Streams unusable. They make it a system that has to be designed around from day one, because discovering the 24-hour ceiling or the two-consumer cap after the pipeline is already in production usually means discovering it through data loss.

Three pipeline architectures for getting DynamoDB changes into a warehouse

Given those constraints, three architectural patterns dominate how teams actually move DynamoDB changes into a warehouse.

The first runs DynamoDB Streams into a Lambda function, which transforms each record and writes it to S3, where Snowpipe (or an equivalent loader for another warehouse) picks it up and loads it into Snowflake or Redshift. This is the pattern AWS's own database team documents end to end, including the mapping from DynamoDB rows to SQL tables, the event source mapping configuration, and the translation from INSERT, MODIFY, and REMOVE into INSERT, UPDATE, and DELETE. It's serverless and gives fine control over every transformation, but the team owns every line of that transformation code, along with Lambda concurrency limits, cold starts, error handling, and the small-file problem that comes from writing many individual records to S3, which in turn hurts warehouse query performance.

The second architecture adds Kinesis Data Streams and Firehose between DynamoDB and S3. Firehose buffers and batches records before writing to S3, which largely solves the problem of too many small files, and Kinesis extends retention well past the native stream's 24-hour limit. The cost is more infrastructure: Kinesis and Firehose are each separately configured, separately billed, and each comes with its own failure modes, so this path carries more moving parts than the Lambda-only approach. It also inherits Kinesis's duplicate-record behavior, so deduplication logic has to sit somewhere in the pipeline before data lands in the warehouse.

The third option is AWS's Zero-ETL integration from DynamoDB to Redshift, which removes most of the custom pipeline work. Internally, it pulls data on a roughly 15 to 30 minute cycle using DynamoDB's native export-to-S3 capability and point-in-time recovery, which has to be enabled on the source table beforehand. Fifteen to thirty minute freshness is fine for a lot of analytical reporting, but it rules this path out for anything that needs near-real-time data: live dashboards, AI feature pipelines, or operational use cases where stale numbers cause real problems. The activation fee for Zero-ETL is $0, though ongoing costs accrue from PITR, the exports themselves, and Redshift compute.

A fourth pattern alongside these three is a managed CDC platform that connects to DynamoDB Streams directly as a source and writes to the warehouse as a destination, handling shard discovery, checkpointing, type conversion, and loading internally. Because this path skips the S3-and-Snowpipe staging step, some managed platforms deliver end to end to Snowflake in 10 to 30 seconds, noticeably faster than the Lambda-based pattern. It removes the Lambda functions, the S3 staging buckets, the Snowpipe configuration, and the custom transformation code a team would otherwise write and maintain, in exchange for depending on a third party to get the mechanics right.

One more wrinkle affects teams running multi-account setups. Resource-based policy support for DynamoDB Streams, which reached general availability in early 2026, now lets a Lambda function in one AWS account read a stream from a DynamoDB table in a different account directly, with no need for Kinesis or custom relay infrastructure in between. For organizations that keep application workloads in isolated accounts and centralize analytics elsewhere, that's a meaningful simplification of what used to require custom cross-account plumbing.

Translating DynamoDB's typed JSON format into a relational schema (the structural problem at the center of warehouse replication)

Every one of those architectures runs into the same unresolved problem downstream: DynamoDB's data format and a relational warehouse's data format don't match, and nothing about picking an architecture solves that on its own.

DynamoDB doesn't store plain JSON. Every value in a stream record comes wrapped in a type descriptor, so a string shows up as {"S": "value"} and a number shows up as {"N": "123"}. This typed wire format is, by the Snowflake replication guide's own account, the single biggest source of friction in getting DynamoDB data into a warehouse. Loading the raw stream record naively leaves the warehouse storing the wrapper structure itself rather than the value it contains, which is useless for querying.

Resolving that requires explicit type mapping decisions, not defaults. DynamoDB's Set type has no clean equivalent in a relational schema at all, so it needs its own handling strategy.

Compounding this, DynamoDB is schema-less: two items in the same table can carry entirely different sets of attributes. No architecture choice removes that responsibility.

REMOVE events need their own handling path. The correct response is to issue a DELETE against the warehouse row by primary key, not to try reconstructing a row from attributes that were never sent.

Schema evolution in production: how DynamoDB's flexible model creates downstream instability

Schema translation is an ongoing requirement, not a one-time task a pipeline finishes and moves past. Because DynamoDB enforces no schema, producers can add, remove, or restructure item attributes at any time, with no migration step and no warning to anything downstream. Each of those changes is a potential breaking event for a warehouse pipeline that wasn't built to absorb it.

Some changes are easy to live with. Structural changes cause more damage: a field that changes from a scalar value to a map, or a list whose elements change type, is much harder to detect automatically, and applying the old mapping logic to the new shape can silently corrupt rows in the warehouse.

AWS's own Zero-ETL integration from DynamoDB to Redshift shows the cost of getting this wrong: certain schema changes there trigger a full resynchronization of the table, and during testing, this has left high-write tables unavailable for querying in Redshift for extended periods. That's a concrete argument for strict DDL guardrails and automated schema checks built into CI/CD, even for a managed integration where the vendor handles most of the pipeline mechanics.

AWS DMS, used by some teams for ongoing replication, can handle some schema changes, but typically needs manual configuration and extra scripting to keep the downstream tables consistent. Teams running DMS for continuous replication commonly end up maintaining separate schema migration scripts, custom merge logic, deduplication logic, and tooling to replay or backfill data when something goes wrong, essentially rebuilding, by hand, the schema evolution handling that a more automated approach would provide natively. The second requires a person on call for every structural change upstream, indefinitely.

Exactly-once delivery and checkpointing: why the stream's at-least-once behavior requires deliberate design

DynamoDB Streams guarantees at-least-once delivery, not exactly-once. Lambda retries records on failure, and a consumer that restarts after a crash replays from its last checkpoint. That means a pipeline that doesn't build in idempotent writes or deduplication will, sooner or later, write duplicate rows into the warehouse.

Checkpointing, recording the sequence number of the last record successfully processed, is what lets a consumer resume correctly after a crash, without either reprocessing records it already delivered or silently skipping ones it never got to. A MODIFY event delivered twice because of a retry should produce exactly one updated row in the target table, never two rows and never a row stuck in a conflicting state.

The Kinesis path adds a wrinkle here: because Kinesis Data Streams can produce duplicate records at the stream level itself, deduplication has to happen before the warehouse load step, not only as part of it. Exactly-once delivery is a property the pipeline has to construct deliberately, through idempotent writes, sequence-number tracking, and correct handling of Lambda's retry behavior.

AWS DMS illustrates what happens when a tool was built for periodic batch replication rather than continuous, high-throughput streaming. In long-running CDC workloads, DMS has run into memory saturation, replication tasks stalling during write spikes, and crashes when handling schema changes upstream. It also lacks native, automated merge logic for cloud warehouses, which pushes engineers toward building custom S3 staging architectures just to get the upsert behavior a warehouse pipeline needs.

Requirements for a production-ready DynamoDB-to-warehouse pipeline

A pipeline that handles stream mechanics, schema translation, schema evolution, and delivery guarantees from the outset avoids an entire category of failure that teams building incrementally tend to discover only once data has already been lost or corrupted in production. The decisions belong at the design stage, not as fixes applied after something breaks.

On the stream configuration side, the view type should be NEW_AND_OLD_IMAGES whenever the destination is a relational warehouse, since narrower view types can't reconstruct updates without going back to query the source table. The choice between native DynamoDB Streams, with its two-consumer limit and 24-hour retention, and Kinesis Data Streams for DynamoDB, with higher fan-out and retention up to a year, should match the actual number of downstream consumers and how much recovery time the organization can tolerate. And the bootstrapping sequence matters: a full table scan has to load the historical data first, with the stream enabled afterward to pick up incremental changes, because starting from the stream alone silently leaves out every row that existed before it was turned on.

On the schema and type system side, a typed-JSON unmarshalling layer needs to exist before any warehouse loading logic runs, since raw stream records can't be loaded as-is. And the pipeline should actively watch incoming stream records for new attributes it hasn't seen before, either applying additive schema changes automatically or raising an alert before loading data that could corrupt the target.

On delivery guarantees, the warehouse layer needs MERGE, or upsert, logic so that a retried MODIFY event produces exactly one updated row. And failure and restart scenarios deserve explicit testing before launch, since silent duplicate accumulation is the most common defect found in at-least-once stream pipelines once they're already running in production.

Operationally, a pipeline needs alerting on consumer lag well before it approaches the 24-hour retention ceiling, because by the time records have actually expired, the data loss has already happened and there's nothing left to recover. And for organizations running multi-account setups, the cross-account Lambda pattern built on resource-based policies for DynamoDB Streams, now generally available, removes the need for Kinesis relay infrastructure when the account running analytics differs from the account running the application.

Taken together, these requirements describe real engineering work: schema layers, checkpointing logic, monitoring, and failure handling that someone has to design, build, and maintain. A managed CDC platform takes on a meaningful share of that work in exchange for a third-party dependency, while the DIY architectures keep full control in exchange for owning every one of these problems directly. Neither path is free. The choice is about where the operational cost sits, and a team that understands the mechanics covered here is positioned to make that choice deliberately.