How Do You Know a CDC Vendor’s Recovery Claims Will Hold Up? 

Test restart state, catch-up, validation, replay behavior, diagnostics, and recovery limits before choosing a CDC platform. 

Most CDC demonstrations show the path working as designed. Data moves. The target updates. The dashboard stays green. But a CDC recovery evaluation needs to test a different question:

What happens when the path breaks?

AWS Well-Architected guidance recommends testing resilience under adverse conditions in environments that are as close to production as practical. It warns against designing for resilience without verifying how the workload behaves when faults and unexpected load occur.*

A platform proof of concept, or POC, gives your team an opportunity to observe how the system behaves when replication falls behind, stops during active processing, loses immediate access to required source history, or must prove that the recovered target is correct.

The goal isn’t to construct an artificial failure that would never occur in production. It’s to evaluate whether the platform makes recovery visible, testable, and defensible under conditions that reflect your environment. 

Quick answer: How should you test CDC recovery during a POC?

A practical CDC recovery POC should test eight areas:

1. Lag and backlog visibility
2. Restart and recovery-state visibility
3. Historical catch-up
4. Catch-up progress and stalled recovery
5. Diagnostics and failure evidence
6. Post-recovery validation
7. Replay and downstream-consumer impact
8. Recovery limits and resynchronization boundaries

For every test, ask what condition was introduced, what operators could see, what state the platform retained, what happened during recovery, what evidence remained afterward, and what the platform could not establish. Every conclusion should be tied to the source, target, topology, version, configuration, and failure scenario tested.  

The standard is simple: Show us what happens when the path breaks. Then show us how we would know the recovery succeeded. 

Before the POC: Start with the gaps you need to test  

Reaching this stage means you’ve likely already determined that recovery deserves closer evaluation. Don’t repeat the entire readiness assessment inside the POC. Instead, carry the documented gaps in the current environment into the test plan.  

If those gaps haven’t been documented, begin with When Does a CDC Recovery Problem Become a Platform Problem?  And document the recovery gaps in the current environment before defining the POC.  

Then define the environment the POC is intended to represent.  At minimum, document:  

• Source and target technologies and versions
• Capture and apply methods
• Replication topology
• Normal and peak transaction rates
• Source-log, journal, or retained-history policies
• Network and target conditions
• Transformation and supported schema-change requirements
• Recovery objectives and unacceptable reload or resynchronization scenarios
• Downstream consumers, replay sensitivity, and business dependencies

A generic demonstration provides limited evidence.  

The important question isn’t: Does the platform support recovery? It’s: How does recovery work for our source, target, configuration, workload, and operating requirements?

From capabilities to evidence 

Capability names are not proof. The tests below move the conversation from “Do you support this?” to “Show us how it works when recovery becomes necessary.”

POC Test 1: Make replication fall behind

Create a controlled condition in which replication remains active but cannot keep pace with source activity. The purpose is to determine whether operators can distinguish normal operation, growing lag, progressing catch-up, and stalled catch-up.

Ask the vendor to demonstrate lag magnitude, backlog size and direction, where delay is occurring, whether processing is advancing, and what happens to source-history exposure as the gap grows.  

What a strong demonstration shows

Both the magnitude and direction of the problem are visible. Operators don’t have to infer recovery progress from a green status indicator. They can tell how far behind the target is, where the delay exists, and whether conditions are improving. 

POC Test 2: Stop replication during active processing

Introduce a safe, supported interruption while transactions are actively moving. A clean shutdown may demonstrate normal restart behavior, but it may not show what happens during an abrupt termination, network interruption, target failure, or another realistic event.

Ask the vendor to show what restart or process state was retained, what each marker means, where processing is expected to resume, what transaction boundary applies, and whether work may be retried or reprocessed.

What a strong demonstration shows

Operators can inspect the relevant state before restart and understand what the platform will do. The vendor can distinguish source capture, processing, transport, and target-apply state where those distinctions apply. 

POC Test 3: Test historical recovery support

For log-based CDC, accumulated changes may depend on active logs, archived logs, journals, or other retained source history. Test the dependency chain under the retention model your production environment will use.

Ask the vendor to explain which history is required, where it resides, who controls retention, how it’s retrieved, whether the platform can process it directly, and where the supported recovery boundary closes.

What a strong demonstration shows

The vendor demonstrates how the platform locates and processes required history and what operators can monitor. The answer is specific to the source and configuration, not a generic statement that archived-log recovery is supported. 

POC Test 4: Create a stalled catch-up

Begin catch-up, then constrain source, network, apply, or target conditions so the existing gap stops shrinking or grows. This tests whether the platform distinguishes activity from actual progress.  

Ask the vendor to show backlog direction, current processing and apply rates, position advancement, retries or throttling, source and target impact, and the basis for any completion estimate.  

What a strong demonstration shows

Progressing and stalled catch-up look meaningfully different. Operators can see whether the gap is closing, holding steady, or worsening, and completion estimates are based on observed conditions rather than theoretical throughput. 

POC Test 5: Preserve and investigate failure evidence

Trigger a representative source, network, replication, or target failure and evaluate what evidence remains available before and after restart.

Ask the vendor to demonstrate how operators investigate the failure, how processing context and errors are correlated, how long evidence is retained, what can be exported, and what additional source, network, or target evidence is still required.

What a strong demonstration shows

Operators can connect what the pipeline was processing, how lag or throughput changed, which component raised the error, what state remained afterward, and which recovery action followed. Useful evidence doesn’t silently disappear when the process restarts. 

POC Test 6: Validate the recovered target

A successful restart proves that replication resumed. It doesn’t prove that the target is complete, current, and correct. Seed known discrepancies where practical and safe, such as a missing insert or update, an unexpected record, a duplicate, a missing delete, a stale value, a transformation difference, or an ordering issue where sequence matters.

Ask the vendor to demonstrate how source-to-target differences are identified, what objects or incident windows can be scoped, what keys and values are compared, how discrepancies are presented, and what additional validation remains the operating team’s responsibility.

What a strong demonstration shows

The platform identifies specific differences and clearly explains what the comparison proves and doesn’t prove. A matching row count or completed process is not presented as universal evidence of integrity. 

POC Test 7: Test replay and downstream-consumer impact

Accumulate changes during an interruption and process them during catch-up. Include replay-sensitive event types where doing so is safe. Replay is not automatically a problem, but invisible or poorly understood replay can be.

Ask the vendor to show the replay window, affected records or event types, expected volume and rate, retry or repeated-processing behavior, ordering considerations, and what evidence remains after catch-up.

Why idempotency matters  

A replication platform cannot make a payment, notification, inventory, or workflow application idempotent. It can provide enough information for downstream teams to decide whether suppression, deduplication, sequencing, or coordinated activation is required.

What a strong demonstration shows

Replay is visible and attributable. Downstream teams can estimate what will arrive and when, understand whether repeated processing may occur, and assess whether consumer-side safeguards are required. 

POC Test 8: Demonstrate the recovery boundary

Every CDC platform has conditions in which restart or catch-up is not the appropriate path. A targeted resynchronization, broader resynchronization, or full reload may be required when recovery state is unavailable, necessary source history cannot be processed, the target is inconsistent, unsupported changes occurred, or validation reveals broad discrepancies.

Ask the vendor to explain which scenarios support restart, historical catch-up, targeted resynchronization, broader resynchronization, or reload, and what evidence tells operators that the normal recovery path no longer applies.

What a strong demonstration shows

The response is specific, qualified, and tied to the proposed environment. An honest recovery boundary is a sign of maturity, not weakness

Document what the POC actually proved

A rigorous POC should leave behind more than screenshots, meeting notes, and a statement that the test passed. For every scenario, capture the test conditions, evidence observed, recovery result, evaluation, limitations, and required follow-up.

The goal isn’t to assign a generic recovery score. It’s to determine whether the platform demonstrated the capabilities your environment requires under the conditions you tested.

Use the CDC Recovery POC Evaluation Worksheet. The worksheet is the working record for the evaluation. Use it to:

• Define the source, target, topology, workload, retention model, and recovery objectives
• Plan the eight recovery scenarios
• Capture observable behavior and retained evidence
• Rate capabilities as Demonstrated, Partially demonstrated, Not demonstrated, or Not applicable
• Document limitations, dependencies, and open questions
• Compare vendors only under equivalent conditions
• Support the final recommendation with evidence

A capability should not be marked Demonstrated based only on product documentation, presentation material, or verbal explanation. The rating should reflect observable behavior and retained evidence from the tested configuration.

Could our normal operating team use the evidence demonstrated here to understand the failure, manage recovery, and defend the all-clear without depending entirely on the vendor or one internal specialist?  

Common CDC recovery POC mistakes

Testing only the happy path: A successful initial load and steady-state replication test establish basic functionality. They don’t establish recovery behavior. Include controlled lag, interruption, catch-up, validation, and recovery-boundary scenarios.

Using only clean shutdowns: Clean shutdowns may not expose the same restart-state, transaction, diagnostic, or retry behavior as abrupt interruptions. Test supported failure conditions that reflect realistic production risk.

Accepting labels without definitions: Terms such as exactly once, zero data loss, self-healing, automatic recovery, seamless catch-up, and built-in validation may describe different behaviors across platforms. Require a definition tied to the tested configuration and failure scenario.  

Treating process status as evidence of recovery: Running doesn’t mean caught up. Caught up doesn’t mean validated. Validated within one scope doesn’t prove every dimension of correctness.

Not seeding known discrepancies: If every test begins and ends with correct data, the POC doesn’t establish what the platform can detect when something is wrong. Include missing, unexpected, deleted, stale, or repeated data where practical.

Ignoring downstream behavior: Recovery doesn’t stop at the target table. Delayed and replayed changes may trigger downstream applications, events, notifications, payments, workflows, or decisions.

Not documenting what was not demonstrated: An unanswered question should not silently become an assumed capability. Record it as: 

Put the recovery claims to the test

A CDC POC should demonstrate more than successful replication under normal conditions. It should show what operators see when the pipeline falls behind, what recovery state remains after failure, where processing can resume, what historical recovery depends on, how catch-up progress is measured, what diagnostic evidence survives, how the target is validated, what replay means for downstream consumers, and where the supported recovery path ends.

Apply the same tests to every platform, including the one already operating in your environment. Recovery capability should not be assumed from a feature name or a polished demonstration. It should be established through observable behavior and documented evidence.

Show us what happens when the path breaks. Then show us how we would know the recovery succeeded. 
 

*AWS Well-Architected: Test resiliency using chaos engineering  November, 2024