The Pipeline Is Running Again. Can You Trust the Data?

After a replication failure or interruption, a successful restart proves only one thing: Replication is running again. It doesn’t prove that every change arrived. It doesn’t prove deletes were processed. It doesn’t prove values are current. And it doesn’t prove downstream teams should trust the data.

That's why post-recovery validation matters.

NIST’s data-integrity guidance treats recovery as more than restoring availability. It emphasizes timely detection and response to integrity loss and the role of preparation, logging, and appropriate tools in minimizing downtime and loss when data may have been corrupted or changed.

Before declaring recovery complete, teams need enough evidence to answer a simple question: What are we relying on when we tell people the data is safe to use again?

Quick Answer: How do you validate data after CDC recovery?

Validation needs to answer two different but equally important questions

• Did replication recover successfully? 
  This is the operational question. The process restarted. Lag returned to expected levels. The backlog cleared.
• Can the recovered data be trusted? 
  This is the data-integrity question. The target reflects the expected changes. Important records are present. Deletes were processed. No unexplained discrepancies remain.

A healthy process helps establish that recovery occurred. Validation helps establish that recovery produced trustworthy data. A practical post-recovery validation process has five parts:

• Define the recovery window.  
• Identify the data and downstream systems that were exposed.  
• Use the evidence available from the source, replication platform, and target.  
• Check specifically for the failure modes that matter.  
• Document what was validated, what was found, and what remains uncertain.  

Don’t rely on one signal alone.

A running process tells you replication is operating. A row count tells you something about the resulting dataset. A source-to-target comparison identifies differences within its supported scope. Platform logs and diagnostics tell you what happened operationally. Business-level checks tell you whether the downstream result makes sense.

Together, those signals can support a defensible conclusion. Let’s break these down.

Validation Is a five-part process

The sections that follow expand on these five areas in more detail. Some steps require multiple checks. For example, comparing source and target data may involve validating counts, identifying missing changes, checking for duplicates, confirming deletes, and investigating stale values.

The goal isn’t to complete thirteen disconnected tasks. It’s to build enough evidence to answer a single question: Can we trust the recovered data? 

Define the recovery window 

Before comparing anything, establish what period you are actually trying to prove. Start with the last point before the incident where replication was known to be operating as expected. Then identify when recovery and catch-up reached the point you are evaluating.

That gives you a recovery window: From the last known-good state → through the interruption → through catch-up → to the state you now want to trust. Depending on the source and platform, useful references can include:

• timestamps  
• restart or process state  
• checkpoints  
• bookmarks  
• source log positions  
• platform-specific recovery coordinates  

For Db2 for z/OS environments, teams may also work with source-specific concepts such as RBA or LRSN where appropriate. The exact terminology varies. The important thing is that everyone evaluating the recovery is talking about the same period and the same logical boundary.

How source log positions, including Db2 for z/OS RBA/LRSN, can help anchor post-recovery validation

Source log positions can help teams identify the processing period they are trying to evaluate. In Db2 for z/OS environments, RBA or LRSN may be relevant to understanding source-log position, depending on configuration and environment.

These coordinates can help anchor the investigation. They should not be treated as universal proof that the target is complete. The team still needs to connect the relevant source state, replication-platform evidence, and resulting target state.

Ask:

When did the problem actually begin?  

What was the last state we trusted?  

When did catch-up begin?  

When did the target return to the state we are now evaluating?  

What transactions, datasets, tables, files, or downstream processes fall inside that window?  

Why it matters

If the validation window is vague, the conclusion will be vague too so, you can’t confidently answer: “Did we miss anything?” until you agree on: “Between which two points?” 

Identify what was actually exposed

Don’t automatically validate everything with the same level of effort. First understand what changed during the recovery window and what downstream systems relied on that data.

Ask:

• Which sources and targets were affected?  
• Which tables, files, datasets, or data domains changed during the window?  
• Which downstream applications, reports, analytics, APIs, or operational processes depend on them?  
• Was any stale target data used while replication was behind?  
• Which data is business-critical enough to require deeper validation?  
• Are there customer, reporting, financial, operational, regulatory, or other high-impact uses in scope?  

This helps determine the depth of validation required. A short interruption affecting low-risk data may justify a different validation approach than a long outage involving settlement, claims, regulatory reporting, or another critical process.

The practical question: “What actually matters in this gap?”

That’s a more useful starting point than applying the same checklist blindly to every table. 

Inventory the evidence you actually have

Post-recovery validation should draw from three places:

A. The source

What does the authoritative source tell you about the affected period? Relevant evidence can include:

• Source records  
• Transaction/change history  
• Timestamps  
• Keys  
• Source-specific log or sequence information  
• Business totals  
• Other source-system evidence  

B. The replication platform 

What can the platform show about what happened? Depending on the product and configuration, that may include:

• Saved restart/process state  
• Status and monitoring information  
• Errors and diagnostics  
• Lag or catch-up information  
• Supported comparison or validation capabilities  
• Other platform-specific operational evidence  

C. The target 

What does the target actually contain now? Depending on the data and use case, you may need to inspect:

• Records  
• Business keys  
• Counts  
• Values  
• Timestamps  
• Target exceptions  
• Downstream results  
• Application-level behavior  

The strongest validation usually comes from connecting evidence across these layers, not taking one system’s “success” status as the final answer. 

Start broad — but don’t stop at row counts

Row counts are useful. They’re also one of the first things experienced practitioners warn against treating as sufficient. Two datasets can have the same number of rows and still be wrong. 
You could have:

• one missing row and one unexpected row  
• the correct key with an incorrect value  
• a record that should have been deleted but still exists  
• a later update without an earlier expected state  
• differences that happen to cancel each other out in the total count  

So use broad checks to identify where to look more deeply. Useful broad checks can include:

• expected row or record counts  
• expected change volumes  
• key ranges  
• high-level business totals  
• timestamps  
• unexpected exceptions  
• platform-reported status  

A mismatch tells you immediately that something requires investigation. A match tells you this particular check passed. It doesn’t mean everything is proven correct.

Row counts are a signal, not a clean bill of health. 

Compare source and target where it matters

Once you have defined the recovery window and affected scope, compare the source and target at a logically appropriate point. Depending on your environment, validation could use:

• Business keys  
• Record-level comparison  
• Selected columns  
• Hashes or checksums where appropriate  
• Totali di controllo
• Timestamps  
• Source and target values  
• Platform-supported comparison capabilities  

The purpose isn’t necessarily to compare every byte of every dataset. It’s to identify whether the data that should be represented at the target is represented correctly enough to support the required conclusion.

Be Careful About Timing

Source and target need to be compared at meaningful, logically compatible states. If the source continues to change while the target is still catching up, a difference does not automatically mean recovery failed. It may simply mean the two sides are being measured at different points in time.

That’s why the recovery window and comparison boundary matter.

How to validate mainframe-to-cloud CDC replication completeness after recovery: checksums, log positions, and reconciliation

There is no universal validation recipe. But teams commonly combine several forms of evidence when validating data after a mainframe CDC recovery.

Counts and Control Totals: Useful for identifying obvious differences in expected volumes or business aggregates.

• Good for: broad anomaly detection.
• Does not prove: that the correct individual records or values are present.

Keys and Record-Level Comparisons: Useful for identifying missing, unexpected, or mismatched records within a defined scope.

• Good for: finding specific differences.
• Does not automatically prove: transaction sequence or every form of historical correctness.

Hashes or Checksums: Where appropriate, hashes or checksums can help compare sets of values efficiently.

• Good for: detecting that source and target values differ.
• Does not automatically tell you: why they differ or whether sequence-specific behavior was correct.

Timestamps, Restart State, and Source Log Positions: These may help establish the period or processing state being evaluated. For mainframe sources, the specific position terminology depends on the underlying platform.

• Good for: anchoring the recovery investigation.
• Does not automatically prove: that the final target data is correct.

Platform Diagnostics and Comparison Capabilities Your replication platform may provide operational evidence or supported source-to-target comparison functionality.

• Good for: adding platform-specific evidence to the validation process.
• Important: understand exactly what the feature covers for your source and target. Don’t assume a platform comparison capability proves every dimension of correctness.

Business-Level Validation For critical workflows, check whether important business records, totals, events, or outcomes make sense after recovery.

• Good for: determining whether the data is usable in the context that matters.
• Does not replace: technical validation where technical proof is required.

The most defensible approach is often layered validation: several checks answering different questions rather than one check claiming to answer everything.

How to Detect Missing Transactions or Changes After CDC Recovery

The question users ask is simple: “How do we know nothing was missed?”

Start with the recovery window. Then compare what should have reached the target against what actually did. Depending on the environment, that may mean examining:

• Affected source records  
• High-value business keys  
• Change activity during the gap  
• Target records  
• Platform diagnostics  
• Supported source-to-target comparison results  
• Downstream outcomes  

Look especially at data that changed while replication was interrupted or catching up.

Red Flags

• Expected records are absent  
• Expected updates are not reflected  
• Downstream totals no longer reconcile  
• Platform or target exceptions remain unexplained  

Specific high-value transactions cannot be accounted for  

Don’t broaden the conclusion beyond the evidence. If you validated a defined subset of tables or a specific recovery window, say that. A defensible statement is: “We validated the affected datasets and recovery window using these checks, and found no unexplained discrepancies.”

That’s stronger than making an absolute statement the evidence cannot support.

How to Identify Duplicate or Reprocessed Changes After Pipeline Restart

A restart or recovery can create concern about whether work was processed more than once. The exact retry and duplicate behavior depends on the replication platform, source, target, and configuration.

Don’t assume a universal delivery guarantee. Instead, identify where duplicate or repeated application would be visible in your environment. Potential indicators include:

• Unexpected additional records  
• Duplicate business keys  
• Target constraint errors  
• Inconsistent counts  
• Repeated business events  
• Platform errors or diagnostics  
• Application-level anomalies  

For datasets without natural uniqueness constraints, duplicates can be harder to detect.

That is why teams should understand which keys or business rules make duplication visible before the incident occurs.

How to detect missing deletes after CDC recovery

Deletes deserve their own check because they are easy to miss. A target can have all the records you expect to see and still be wrong because it contains records that should no longer exist. Practitioner language for this problem is very direct: “Records that were deleted from the source are still in the target.”

When deletes occurred during the recovery window, verify that the relevant target state reflects them appropriately. Depending on the data model, that can involve:

• Checking affected business keys  
• Comparing source and target presence  
• Reviewing supported platform comparison results  
• Validating downstream behavior where deleted records matter  

Don’t assume a successful count comparison will always surface missing deletes.

How to identify stale or incorrect values

Sometimes the record is present on both sides. The problem is that the target has the wrong version of it. Look for:

• Values that don’t match the source at the intended comparison point  
• Updates from the recovery window that are not reflected  
• Downstream results using an earlier value  
• Differences that cannot be explained by normal timing or transformation  

This matters because a simple existence check can pass while the target is still stale.

The question isn’t just: “Is the record there?” It’s: “Is the record in the state we expect it to be in?” 

How to identify out-of-order changes after CDC recovery

Ordering matters in some workloads more than others. If the business outcome depends on sequence, final-state comparison may not answer every question. For those scenarios, teams may need additional evidence appropriate to:

• The source  
• The target  
• The replication architecture  
• The affected transaction pattern  
• The specific ordering requirement  

Don’t assume that a checksum or matching final row state proves historical order. Likewise, don’t assume that every workload requires transaction-by-transaction sequence validation.  

The right standard is: Validate the ordering behavior that actually matters to the affected business process and architecture.

Investigate Discrepancies — Don’t Average Them Away

If a validation check finds a difference, treat it as evidence to investigate.

Ask:

• Is this an expected timing difference?  
• Is transformation involved?  
• Did the comparison use equivalent source and target states?  
• Is the target still catching up?  
• Is there an unexplained missing record?  
• Is something present that should have been deleted?  
• Was a change processed more than once?  
• Is a value stale?  
• Is the discrepancy isolated or part of a larger pattern?  

Then determine the appropriate remediation. Depending on the environment, that could involve:

• Correcting an identified record or dataset  
• Repeating a validation step  
• Resynchronizing affected data  
• Performing a broader resynchronization  
• Keeping downstream use restricted until the issue is understood  

Don’t let a large number of successful checks hide one unexplained discrepancy in business-critical data.

Create a Post-Recovery Evidence Record

The team that handled the incident may know exactly what happened today. Three months later, somebody else may ask: “How did we know the data was safe to use again?”

Don’t make them reconstruct the answer from memory. Create a concise Post-Recovery Evidence Record that includes:

Incident Window: When was replication affected?

Recovery Point: Where did processing resume, using the terminology appropriate to the source and platform?

Affected Scope: Which sources, targets, datasets, tables, files, or downstream processes were involved?

Recovery Method: What recovery path was used? For example:

• Restart from saved state  
• Catch-up  
• Targeted resynchronization  
• Broader resynchronization or reload  

Validation Performed: What checks were completed? For each important check, record:

• What was examined  
• What period or scope was covered  
• What the check was intended to establish  

Findings: What did the team observe? Were discrepancies found?

Remediation: What was corrected, reprocessed, or resynchronized?

Exceptions: What remains unresolved or outside the validated scope?

Conclusion: What evidence supports the decision that the relevant downstream data is ready for use?

That record can support:

• Incident review  
• Operations handoff  
• Downstream communication  
• Governance  
• Audit  
• Future troubleshooting  
• Improvement of the recovery runbook  

Who Gives the All-Clear?

This is an operational question teams often leave undefined until the pressure is highest. Someone has to decide if the data is ready to use again. That decision may involve more than the person who restarted the pipeline. Depending on the incident, it may require input from:

• The replication or data engineering team  
• The source-system owner  
• The downstream platform or application owner  
• data operations  
• A business owner  
• Governance, risk, or another control function  

The exact sign-off path should reflect the importance of the affected data.

The Key Principle: The technical team should be able to explain:

• What was affected  
• What was validated  
• What the checks established  
• What remains uncertain  
• What exceptions still exist  

Then the people responsible for downstream use can make an informed decision.

How Do You Tell Downstream Teams the Data Is Ready Again?

Avoid both extremes. Don’t say everything is fine if your evidence does not support that statement.

But don’t dump a technical incident log on a business stakeholder either. Communicate four things clearly:

• What happened: Replication was interrupted or fell behind during a defined period.
• What was affected: Identify the relevant downstream data or systems.
• What was validated: Explain the scope of the checks performed.
• What Remains: State any open exceptions, limitations, or follow-up work.

The most useful message sounds like:

“Replication has caught up. We validated the affected data for the incident window using the checks described here. We found [no unexplained differences / these identified differences, which were remediated]. These downstream datasets are cleared for use. These exceptions, if any, remain open.”

The exact language should match the evidence. That’s how you rebuild trust without pretending uncertainty does not exist.

A green pipeline is not a validation strategy

The recurring mistake after a recovery is to stop at operational status: The process is running. Lag is back to normal. The dashboard is green.

Everyone wants to close the bridge call. But none of those signals, by themselves, establishes that the affected target data is correct.

The critical question to answer is: “What evidence are we relying on when we tell people they can trust this data again?”

If the answer is unclear, the incident is not finished. 

How to evaluate post-recovery validation capabilities

Not all CDC platforms provide the same level of evidence after an interruption. Some capabilities are basic expectations. Others can significantly reduce the effort, uncertainty, and risk involved in proving a recovery succeeded.

Table Stakes

All enterprise CDC platforms should provide:

• Restart or process-state information
• Operational status and monitoring
• Error and diagnostic information
• Visibility into catch-up activity
• Basic recovery procedures for supported environments

These capabilities help teams answer: Did replication recover and resume processing?  

The harder question is: Can we prove the recovered data is trustworthy?

Important capabilities that aren’t always included

• Source-to-target comparison functionality
• Visibility into recovery position and processing history
• Support for historical catch-up where retained history is available
• Meaningful diagnostics that support root-cause investigation
• Recovery-state inspection and validation evidence
• Visibility into discrepancies discovered during reconciliation
• Support for validating large datasets efficiently
• Documentation and reporting that support audit, governance, or downstream sign-off requirements

These capabilities determine how much manual investigation is required before teams can confidently declare recovery complete.  No single feature should be treated as universal proof that recovery succeeded. The critical question is: “What evidence will this platform give our team after a recovery event, and what additional validation will we still need?" 

Questions to ask every CDC vendor

Ask the vendor to demonstrate:

• What restart or process state is visible after an interruption?
• What monitoring and diagnostic information can operators inspect?
• How does the platform support catch-up after a disruption?
• What source-to-target comparison capabilities are available?
• What differences can those comparison capabilities actually identify?
• What are the limits of those capabilities?
• How do those capabilities fit into a broader validation process?
• What evidence can teams retain to support an all-clear decision?

The standard should be the same for every CDC platform: Don't accept "the process restarted" as proof. Ask what operators can inspect, compare, test, and defend afterward. 

How Rocket Data Replicate & Sync fits

Rocket Data Replicate & Sync (RDRS) includes capabilities that can contribute to post-recovery investigation and validation for supported configurations.

Documented RDRS capabilities include:

• Saved process and restart state
• Monitoring and diagnostic information
• Batch Compare functionality for supported sources
• Archived-log processing support for supported log-based environments where required history remains available

These capabilities can provide operational and validation evidence during a recovery investigation. As with any CDC platform, organizations should evaluate RDRS against their specific sources, targets, configurations, recovery requirements, and operating procedures. The most important question is: "What evidence can RDRS provide in the recovery scenarios our environment is most likely to face?"

Post-Recovery Validation Checklist

Before telling downstream users that the data is ready again, confirm:

• The recovery window is clearly defined.  
• The affected sources, targets, and downstream consumers are understood.  
• The backlog has cleared or the remaining state is understood.  
• Relevant source, replication-platform, and target evidence has been reviewed.  
• Validation went beyond process status alone.  
• Broad checks such as counts or totals were not treated as universal proof.  
• Important records, values, or business keys were compared where appropriate.  
• Missing changes were considered.  
• Duplicate or repeated changes were considered where relevant.  
• Deletes that occurred during the recovery window were considered.  
• Stale or incorrect values were considered.  
• Ordering was evaluated where sequence matters.  
• Unexplained discrepancies were investigated.  
• Any remediation or resynchronization was documented.  
• Remaining exceptions are explicit.  
• Downstream owners understand what is safe to use.  
• The evidence supporting the all-clear has been recorded.  

The Bottom Line

Getting the pipeline running again is only half the recovery. The harder question is: “How do we prove the data is right?”

Start by defining the recovery window. Understand what was affected. Use evidence from the source, replication platform, target, and downstream environment. Look deliberately for missing changes, duplicates, missing deletes, stale values, and ordering problems where they matter. Investigate discrepancies instead of assuming they are harmless. And leave behind enough evidence that someone who was not on the incident call can understand why the target was cleared for use.

Because:

Running again is not the same as recovered.  
And a green pipeline does not mean the data is trustworthy.

Recovery is complete when the process is stable and the organization has enough evidence to trust the data again.

*NIST SP 1800-26: Data Integrity, Detecting and Responding to Destructive Events 

Next: Is Your Current Platform Good Enough at Recovery?

If recovery repeatedly depends on manual validation, tribal knowledge, cross-team investigation, or uncertainty about what happened during the gap, the next question may be bigger than the incident itself.

How to Evaluate Enterprise CDC Recovery Capabilities When Data Loss Is Not Acceptable

Use the evaluation framework to define what your platform should make visible, testable, and provable before the next failure happens.