The Most Dangerous CDC Recovery Mistake Happens After the Failure 

When a CDC pipeline fails, the instinct is automatic: Get it running again. Fast.

Meanwhile, the source may still be processing transactions. The backlog may still be growing. Reports, applications, and downstream processes may already be working from stale data.

And sooner or later, someone asks the question: Can we just restart it?

That’s usually the moment recovery gets harder.

Because the real problem is often not the failure itself. It’s everything that happened while replication wasn’t running.

• What happened during the gap?  
• Can we safely resume from where we left off?  
• Do we need a resynchronization or full reload?  
• How do we know nothing was missed?  

A successful restart proves only one thing: Replication is moving again.

It does not prove the target is complete, current, or correct.

The most dangerous CDC failures aren’t always the ones where replication stops. They’re the ones where replication starts moving again and nobody can explain what happened during the outage or prove the recovered data can be trusted.

The goal is not simply to restart as quickly as possible. It’s to avoid turning one failure into a harder recovery. A stopped replication job doesn’t automatically mean transactions were lost. And a successful restart doesn’t automatically mean the target is correct.

How to check whether a CDC pipeline failure caused data loss, and whether you can restart safely

The first step is not to press Restart.

NIST’s current incident-response guidance separates detection, response, recovery, and continuous improvement. That broader discipline supports establishing what happened and preserving enough context to make a defensible recovery decision before changing the environment.*

It’s to establish where processing stopped, what continued happening at the source, what state exists for recovery, and what evidence you will need afterward.

Taken in order, the seven questions below will  help you work through that process. 

What exactly failed and when?

“Replication stopped” is not specific enough to choose a recovery path.

First determine where the problem occurred. Was replication:

• completely stopped?  
• still running but falling behind?  
• capturing changes but unable to move them forward?  
• unable to apply changes at the target?  
• affected by a source-system event?  
• affected by the target, network, or another part of the environment?  

Then establish the timeline. The time someone noticed the problem may not be the time the problem started.

Check

• When did the last confirmed successful replication activity occur?  
• Is source capture still progressing?  
• Is downstream processing progressing?  
• Is target apply progressing?  
• Did the source continue processing transactions?  
• Is the target available?  
• Did anything change around the same time — schema, configuration, security, infrastructure, maintenance, or workload?  

Why it matters

A capture failure, an apply failure, a growing backlog, and a target outage are different problems. They may require different teams and different recovery actions. You’re ready to move on when you can say: This is where processing stopped, this is when it happened, and this is what is affected. Not simply: “Replication is broken somewhere.” 

Where can replication safely resume?

This is one of the most important questions in the recovery. Replication platforms may store process or restart state that helps determine where processing can resume after an interruption.

Depending on the source and platform, you may encounter terms such as:

• Restart state  
• Checkpoint  
• Bookmark  
• Log position  
• Resume position  
• Another source-specific recovery coordinate  

Don’t assume those terms mean exactly the same thing across every platform. The important question is whether you can identify where processing last left off and whether that information is reliable enough to support recovery.

Mainframe CDC recovery markers: restart state, log positions, and Db2 for z/OS RBA/LRSN

Depending on the source and replication platform, teams may encounter different recovery markers and processing-position terminology. For Db2 for z/OS environments, those concepts may include RBA or LRSN.

The important question is whether valid saved recovery information exists and whether the team understands what that information represents. For supported RDRS Change Data Capture processes, saved status information allows processing to continue at the correct source-specific position after restart. The position and terminology depend on the source technology.

Check

• Is saved restart/process state available?  
• What exactly does that state represent?  
• When was it last updated?  
• Does it correspond to the processing state you expect?  
• Is the relevant source history still available?  
• Has anything happened at the target that changes the recovery assumptions?  

Why it matters

A known resume position is very different from reconstructing where you think the pipeline left off during an incident. You’re ready to move on when the team has an identifiable restart/process state and understands what it means for this source and target. 

What happened while replication was down, and is the required source history still available?

If the source kept running, transactions may have continued while the target stopped receiving changes. That means the team needs to understand both the replication gap and the downstream impact.

Check the Gap

• How long has the target been behind?  
• How large is the backlog?  
• Is the backlog growing, stable, or shrinking?  
• What source history remains available for the affected period?  
• Did inserts, updates, deletes, or supported schema changes occur?  
• Does the platform provide a supported catch-up path for this source and target combination?  

For log-based CDC, check source log retention and archived history. For some log-based CDC environments, retained or archived source logs may be relevant to processing changes that accumulated during an interruption. The exact prerequisites vary by:

• source  
• platform  
• configuration  
• retained history  
• topology  
• duration and nature of the interruption  

Don’t assume that the existence of archived logs automatically means every recovery path remains available. Confirm what the platform supports for the specific environment. Check who is already using the target:

• Which applications, reports, analytics, APIs, or operational processes depend on it?  
• Are they still consuming the data?  
• Do those teams know it may be stale?  
• Is a settlement, reporting, batch, customer, or other business cutoff approaching?  

Why it matters

A pipeline can still be technically recoverable while stale downstream data is already becoming a business problem. You’re ready to move on when you understand what happened during the gap, how far behind you are, and who may already be affected. 

Checkpoint restart or full resynchronization: which recovery path applies?

This is where teams often feel pressure to make a quick decision. Depending on the platform, the saved recovery state may be called a checkpoint, bookmark, restart state, processing position, or something else.

What matters is whether a supported recovery path exists for the actual failure scenario. Sometimes replication can resume from saved state and catch up. Sometimes a more targeted comparison or resynchronization is required. And sometimes a broader reload is appropriate.

Depending on the source, target, configuration, retained history, and failure scenario, the available paths may include:

• Restart from saved process state  
• Process retained historical changes  
• Compare source and target to identify differences  
• Repair or resynchronize affected data using an appropriate method  
• Partially resynchronize affected datasets  
• Perform a broader resynchronization or full reload  

How to restart a CDC pipeline after a failure without a full initial load

In some environments, a full initial load may not be necessary. Whether recovery can proceed without one depends on conditions such as:

• Usable restart/process state  
• Required source history remaining available  
• The state of the target  
• Supported source and target behavior  
• The type of failure  
• The replication platform’s documented recovery mechanisms  

The goal isn’t to avoid a reload at all costs. It’s to determine whether a supported restart or catch-up path is appropriate for this incident.

Check

• Is the saved restart/process state available and understood?  
• Is the source history needed for the proposed recovery still available?  
• Is the target in a state that supports recovery forward?  
• Did anything occur at the source or target that changes the recovery assumptions?  
• Can a comparison help establish whether only part of the data is affected?  
• How long is catch-up expected to take?  
• Can downstream systems tolerate that period of staleness?  
• What recovery paths does the platform support for this exact source/target configuration?  

Why it matters

Practitioners describe the painful alternative very plainly: “full reload” or even “resync from scratch.”  The question to answer is: When can we recover without starting over, and when can we not? You’re ready to move on when the team can explain why the selected recovery path fits this particular incident. 

What could be missing, duplicated, deleted, stale, or out of order?

A successful restart does not by itself prove that the target is correct. Depending on the data and architecture, teams should look for:

• Missing changes  
• Duplicate or reprocessed changes  
• Missing deletes  
• Stale values  
• Partial or related-record inconsistencies  
• Ordering issues where sequence matters  
• Supported schema changes that occurred around the incident  

One especially easy problem to miss is the delete. A pipeline can appear healthy while a record that was deleted at the source is still sitting in the target.

Check

• What transaction/change behavior does the platform document for this source and target?  
• What happens during retry or reprocessing?  
• How are source deletes handled?  
• Which schema-change scenarios are supported?  
• Which ordering behaviors matter to your workload?  
• What could a normal status check fail to detect?  

Why it matters

“Everything is moving again” doesn’t tell you whether something was missed along the way. You’re ready to move on when you know which failure modes matter in your environment and what evidence could expose them. 

How will we prove nothing was lost?

Don’t wait until after recovery to decide how you’ll validate it. Before you restart, define what evidence will be needed before downstream teams are told the target is ready to use again. Depending on the environment, validation may involve:

• Supported source-to-target comparison  
• Records or business keys  
• Counts  
• Hashes or checksums where appropriate  
• Timestamps or processing positions  
• Replication-platform diagnostics  
• Business-critical transaction samples  
• Downstream validation  

No single method proves everything. The important question for every validation check is:

What does this prove — and what does it not prove?

Ask

• What recovery period is in scope?  
• Which data needs to be checked?  
• Which checks address the risks identified in Question 5?  
• What discrepancies would require further investigation?  
• Who needs to review the result?  
• What evidence should be retained?  

Why it matters

The recovery mistake teams often make is assuming that “Replication is running again” and “The target can be trusted again” are the same conclusion.

They aren’t.

One is about system health. The other is about data integrity. Recovery is not complete until you have evidence for both. You’re ready to move on when the team has a validation approach appropriate to the source, target, incident, and business risk. 

How do we know recovery is actually complete?

A pipeline can be running while the backlog is still there. It can be catching up while downstream data remains stale. And it can show a healthy process state while questions about the recovered data remain unresolved. Before closing the incident, check both the process and the outcome.

Check

• Is lag returning to the expected range?  
• Is the backlog actually shrinking?  
• Is replication continuing to advance?  
• Are errors or retries still occurring?  
• Has the relevant target data been validated?  
• Have affected downstream owners been told what is safe to use?  
• Are any discrepancies or exceptions still open?  
• Has the incident state been documented?  
• Do we understand enough about what happened to respond better if it happens again?  

Why it matters

Closing an incident too early can leave unresolved data, validation, or downstream-impact questions hidden behind a healthy process status. You’re ready to close the incident when the team can say replication is stable, the relevant data has been validated, downstream teams understand the current state, and we can explain what happened.

Recovery planning vs. recovery validation

The seven questions in this article help identify what could have gone wrong and what recovery depends on. They do not prove the recovered target is correct. After recovery, the next step is determining whether any of those risks actually occurred and whether the target can be trusted.

Related resource: The Pipeline Is Running Again. Can You Trust the Data? 

The decision you’re really making

These seven questions aren’t about whether you can press Restart. They’re about whether you can answer: Where can we resume? What happened during the gap? Can we recover without starting over? And how will we prove nothing was lost?

That’s the difference between getting the pipeline running again and recovering to a state you can trust.

CDC recovery decision checklist

When a recovery problem becomes a platform problem

Every CDC platform will eventually experience failures, interruptions, or recovery events.

The question is not whether incidents happen. The question is how much effort, uncertainty, and risk follows when they do.

One bad incident does not automatically mean you need a different CDC platform. But repeated recovery pain deserves a different conversation.

Ask whether you are repeatedly seeing:

• A growing backlog on the source  
• Full reloads or broad resynchronizations because no other recovery path is trusted  
• Recovery that depends on one or two people knowing the right steps  
• Hours spent figuring out whether the issue is at the source, replication layer, or target  
• Manual or inconsistent source-to-target validation  
• Uncertainty about whether changes or deletes were missed  
• Poor visibility into pipeline or recovery state  
• Source-system impact during normal processing or catch-up  
• Recurring downstream questions about whether the data can be trusted  

For mainframe environments where Db2 for z/OS, VSAM, IMS, or ADABAS are systems of record, recovery behavior should be evaluated for the specific source — not assumed from a generic “mainframe support” claim.

At that point, the question changes from “How do we fix this incident?” to “Is our current replication approach giving us the recovery capability and visibility we actually need?” 

Where Rocket Data Replicate & Sync fits

Recovery is one of the most important capabilities to evaluate in any CDC platform.

Rocket Data Replicate & Sync (RDRS) manages enterprise replication between mainframe environments and modern distributed and cloud platforms.

If you’re evaluating RDRS for an environment like the one described above, apply the same standard you would apply to any replication platform.

The seven questions in this article:  what failed, where can we resume, what happened during the gap, which recovery path applies, what could be wrong, how will we prove it, and how do we know we’re done are the same questions to answer about any platform, including RDRS, before trusting a recovered target.

Ask for a demonstration of how recovery works in your specific source and target configuration:

• Where processing can resume after an interruption  
• What happens to changes that accumulate during the gap  
• What operators can see while recovery and catch-up are underway  
• What options exist when a normal restart is not sufficient  
• What source-to-target comparison or validation capabilities are available  
• What happens to source-system impact during normal operation and catch-up  
• What evidence the team can use before telling downstream users the data is ready again  

Whether a particular failure can be recovered without a broader resynchronization or full reload depends on the source, target, topology, retained history, configuration, and nature of the failure.

The most useful question isn’t: “Does the platform support recovery?”

It’s: “Show us how recovery works in the failure scenarios our environment actually faces.”

*NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations

Next: Prove the recovery

Getting replication moving again is only part of the job. The next question is:

How Do We Prove the Recovered Data Is Complete?

Learn how to define the recovery window, determine what evidence is available, investigate discrepancies, and establish when the target is ready for downstream use.