Column Type Changes in Replicated Warehouse Tables
Type changes in replication pipelines risk silent data corruption across three layers.
For automatic schema evolution in live CDC pipelines, the pipeline must detect changes at the source, update its streaming schema state, and change warehouse columns while keeping the data’s meaning. In “Column Type Changes in Replicated Warehouse Tables,” the central conclusion is that type changes need stricter handling than routine structural changes because a running pipeline can still deliver corrupted data.
Why column type changes are uniquely dangerous among schema evolution events
Schema evolution can feel like a solved problem after a team has added nullable columns without incident. Type changes break that intuition in a specific way. Adding or dropping a column changes its structure, so a mismatch that goes unhandled often leaves a field missing, breaks a query, or stops the pipeline. Changing a type can leave the row format intact while changing how values must be interpreted. The result may be silent truncation or lost precision.
A source can accept the new type while the decoder and warehouse continue using different definitions. Each component may appear healthy because it accepts the values presented to it. Even a replication service that handles routine add-column and drop-column events safely still needs separate protection against these semantic changes.
The three-layer architecture where type-change failures originate
The pipeline map is source → streaming → sink. At the source, DDL changes the column before downstream systems necessarily learn about it. The streaming layer decodes, serializes, and routes events using its own schema state. At the sink, the warehouse column has an independent type, storage format, and constraints. It can reject, coerce, or correctly store an incoming value.
These layers do not fail in lockstep. Even if the registry accepts a schema, a warehouse column may be unable to store the new range, and a correct warehouse definition cannot fix an event decoded with the wrong precision. For managed platforms such as Artie, operating across all three layers means detecting source changes, tracking them through schema-aware streaming, and validating sink compatibility before replication begins. That coordination prevents one layer’s outdated definition from propagating downstream unnoticed.
How Postgres logical replication handles — and mishandles — type changes at the source
Postgres logical replication does not replicate DDL. Instead, when data changes, its Relation message describes the table’s structure, including its column types. Executing an ALTER TABLE does not itself tell the downstream system to update its schema, so the connector must interpret the later metadata correctly before processing affected values.
The critical blind spot is OID-only detection. A tracker that compares only the type OID misses changes to the type modifier, atttypmod. For example, with numeric(10,2) → numeric(14,4), the numeric type identity stays the same, but precision and scale change. The tracker can then decide that nothing changed and keep decoding with the old definition. The required comparison is (type_oid, type_modifier).
This is a connector shortcut. A CDC platform that compares both fields can catch the precision change before it corrupts values in the warehouse. Artie’s handling of type changes should be tested in this way too, because automatic evolution alone does not prove that it tracks modifiers.
TOAST exposes a related schema-trust problem. When a large jsonb or text value is unchanged in an update, its value may be omitted from the WAL. A connector that interprets that omission as NULL creates holes in the warehouse. This demonstrates why correct interpretation requires more than accepting the apparent shape of an event.
How MongoDB and other sources introduce their own type-change failure modes
MongoDB presents the same problem in a different form. Its Change Streams tap the oplog, which contains BSON-typed documents. A price field can be a number in one document and a string in another. The pipeline gets both without a DDL event to tell it that the type has changed.
One platform may pick a dominant type and cast the other values to it, while another may put them in separate columns. Either choice can distort downstream analytics if its consequences are not explicit. A Postgres-specific metadata fix is therefore insufficient for a pipeline serving multiple source types, because document sources require a policy for conflicting values as well as changed definitions.
The type-change risk taxonomy: widening, narrowing, and class-swapping
The first distinction is whether the new type can represent all existing values. INT → BIGINT and VARCHAR(50) → VARCHAR(200) are widening changes because the new type is a superset. Existing values remain valid after the change. However, INT → DECIMAL also changes semantics, so consumer parsing and registry compatibility can still prevent automatic adaptation.
Narrowing reverses that relationship and introduces possible loss. With BIGINT → INT, existing values may exceed the new range, and BACKWARD or FULL compatibility checks can reject the change. A type-class swap such as STRING → INTEGER or INTEGER → BOOLEAN is more disruptive because consumers must interpret values differently. Operationally, that calls for a new-column migration.
With precision changes such as numeric(10,2) → numeric(14,4), a check is needed to see if the decoder tracks atttypmod, even though the change looks like widening. A change is safe only when both the destination type and the pipeline’s understanding of it are correct.
What happens at the streaming layer: schema registries, compatibility enforcement, and Flink's policy surface
A schema registry checks new schema versions against previous ones. BACKWARD compatibility asks whether the new schema can read old data, making it a practical default for CDC topics. Widening and optional-field additions are the intended compatible cases, while narrowing and incompatible renames or removals require rejection. The actual protection depends on the schema representation. A precision change that the connector never registers cannot be caught by compatibility enforcement.
The proposed schema should pass the registry’s compatibility endpoint before the source DDL runs, with incompatible results resolved first. This test complements metadata detection. Otherwise, a registry can correctly approve the schema it receives while remaining unaware of the change that creates the risk.
Flink CDC makes propagation policy explicit through exception, evolve, lenient, and ignore. These respectively fail on schema changes, apply upstream changes downstream, adapt changes to avoid destructive sink operations, or continue propagating changes from unchanged columns. Flink also supports individual controls for table creation, column addition, type alteration, column deletion, column renaming, table deletion, and table truncation. Ordering remains a separate responsibility. Additive changes should reach the subscriber before the publisher, while destructive changes should stop at the publisher before the subscriber loses the ability to receive them.
How type changes land at the warehouse sink: rewrites, coercions, and silent data loss
PostgreSQL’s ALTER TABLE documentation states that changing an existing column’s type normally rewrites the table and its indexes. That rewrite is a local DDL operation. It does not become a corresponding flood of row-level changes in logical replication, so the subscriber retains its existing physical data and converts independently. The warehouse definition stays the same until something updates it.
That independence creates a dangerous mismatch window. If incoming values exceed the warehouse column’s precision or range, implicit coercion can silently lose information. The earlier numeric example therefore has a second failure point. Even when the decoder understands four decimal places, a sink retaining the old scale can still lose them.
Snowflake, BigQuery, Redshift, and ClickHouse have different rules for casts, in-place alterations, and column recreation. Teams must check the rules for the destination they use, since those rules differ from one warehouse to another. A staging-table MERGE keyed by primary key remains the appropriate upsert pattern, but it does not validate type meaning. An upsert that silently coerces every incoming value accumulates corruption with each event.
AWS DMS as a case study in what happens when type-change handling is an afterthought
AWS DMS does not automatically propagate every schema change. Complex changes, including type alterations and nested-structure additions, can need work by hand and may pause replication for the affected table or stop a task. Recovery can involve stopping the task, editing JSON mappings, rebuilding the target schema, and relaunching from scratch. Those steps alone do not establish that the outage missed or duplicated no rows.
An AWS re:Post incident from March 2026 illustrates a narrower type-handling failure. The report says that after a Redshift maintenance update to v1.0.232773, DMS created primary-key segment columns in batch-apply staging tables with their type hardcoded as varchar(20). Only those segment columns were affected. When primary-key values went over the limit, they caused overflow and the CDC task failed at once. The case shows that infrastructure updates can introduce type mismatches.
The safe-handling playbook for each category of type change
For widening changes, the sequence starts before source DDL. First test registry compatibility, then widen the subscriber or sink column so it can accept the new values. Next run the source alteration and allow the CDC engine to detect the updated Relation metadata. Finally, check the events and stored values after the change to see if they use the new range or precision. A pipeline that reports no errors has not passed this final check.
Numeric widening adds a specific verification requirement. For numeric(10,2) → numeric(14,4), confirm that the streaming layer tracks the modifier as well as the OID. With tools that lack this tracking, the column must be recreated at the sink, even if widening it looks safe. Recreation does not remove the need to validate how subsequent events are decoded.
For narrowing, the safe path is a new-column migration. Add the target type as a new nullable column, backfill and validate it, then cut over the application before deprecating the old column. An in-place narrowing on a live pipeline risks values that no longer fit and registry rejection. Destructive cleanup must start at the publisher so consumers can still receive fields that are being sent.
For a type-class swap, add a column of the target type and migrate data at the application layer. Keep both columns until every consumer has updated, then deprecate the old one. Confirm that the configured BACKWARD or FULL policy rejects the incompatible direct change before proceeding. Across all three categories, idempotent upserts keyed by primary key prevent replay during migration from creating duplicates.
Why silent type-change corruption is especially harmful for AI and real-time analytics consumers
AI agents and ML feature pipelines can take the values in the warehouse as ground truth. A feature that has lost precision through truncation still looks like a valid number, so a healthy pipeline status gives no warning that the training data’s meaning has changed. Freshness can amplify this problem as data that keeps being replicated gives fresh wrong answers if its type is read the wrong way.
An agent’s schema cache creates another point of divergence. With a schema cache that predates a type change, it can query the warehouse successfully while misinterpreting the returned column. Real-time dashboards and A/B testing frameworks face a similar risk when truncated metrics remain plausible enough to pass checks. The mitigation extends beyond pipeline monitoring. Consumers should verify the schema version they read rather than assume the warehouse definition matches the source.
How can a managed replication service handle upstream schema changes without stopping CDC?
A managed service should take care of the routine work of tracking types and evolving the sink. That includes detecting precision-only changes, updating internal schema state, checking compatibility, and propagating supported types without manual task restarts or JSON mapping edits. Exactly-once delivery ensures that an event is applied once, but a value can still be decoded incorrectly. Automatic schema evolution must protect meaning as well as continuity.
Artie handles schema evolution, including type changes, fan-in architectures, and failure recovery, as part of its managed service. Flink offers explicit propagation policies for teams managing streaming behavior, while DMS requires manual handling for some complex changes. These are different operating models. The relevant selection criteria are detection accuracy, compatibility enforcement, destination behavior, and recovery burden.
The practical test is the same for any tool, including Artie. Run numeric(10,2) → numeric(14,4) on a live table and verify that the next event is stored with four decimal places in the warehouse. If the old precision silently persists, the pipeline has failed the precision-sensitive schema-evolution test regardless of whether replication continued.
