Your CDC Pipeline Isn't Down. Are You Sure It's Healthy?

A CDC pipeline doesn't need to fail to create risk.

Sometimes replication is still running. The process status is green. The target is still available. But the backlog is growing.  

The target is becoming increasingly stale. And every minute the pipeline falls further behind, the recovery options may become more constrained.  The question isn't simply: "Is replication running?" It's: "Is replication keeping up?"

And if it isn't: "How much time do we have before this becomes a recovery problem?"

The most effective CDC monitoring doesn't just tell operators whether a process is up or down. It helps them recognize when a seemingly healthy pipeline is quietly drifting toward a situation that will be harder to recover from later.  

Good monitoring helps teams answer:

• How far behind are we?
• Is the backlog growing or shrinking?
• Is the pipeline actually making progress?
• Are downstream teams already using stale data?
• Do we still understand our recovery options if the situation gets worse?

The goal isn't to predict every failure. It's to recognize developing problems early enough that the team still has choices.

Depending on the source and replication platforms, effective CDC monitoring may involve metrics and concepts such as replication lag, backlog growth, source and target apply lag, restart state, log position, retained change history, archived logs, RBA/LRSN progression, catch-up behavior, and downstream data exposure. The exact terminology varies, but the goal is always the same: understand whether replication is keeping up and whether recovery remains straightforward if conditions deteriorate.

How do I detect that a CDC pipeline is falling behind before it fails?

A pipeline can be running while lag increases, backlog grows, and downstream data becomes stale.  

Quick Answer: Start by answering five operational questions:

1. How far behind are we?
2. Is the backlog growing, stable, or shrinking?
3. Do we still have the source history we may need if recovery becomes necessary?
4. Is the pipeline actually making progress?
5. Who is already relying on the downstream data, and is anything telling us the situation is getting worse?  

The exact metrics and alert thresholds depend on the source, target, replication platform, workload, recovery requirements, and downstream use case. There is no universal number of minutes, records, or log space that defines a CDC emergency.

A pipeline can say “Running” and still be heading toward a recovery problem.

Monitoring needs to answer two different questions

1. Is the Pipeline Healthy Right Now?

This is the immediate operational question. Is replication:

• Keeping up?  
• Falling behind?  
• Stalled?  
• Recovering?  
• Operating in a degraded state?  

Process status is part of that answer. It isn’t the whole answer.

2. Are we losing recovery options as lag grows?

This is the harder question. A pipeline may still be active while:

• The backlog continues to grow  
• The target becomes increasingly stale  
• Catch-up takes longer  
• Downstream teams continue using old data  
• Available source history changes  
• A relatively contained operational issue becomes a larger recovery problem  

The more important question is: “If this gets worse from here, do we still understand how we would recover?”

1. How far behind are we?

Replication lag tells you how far the target has fallen behind current source activity. But a raw number is not always enough. A backlog of 100,000 changes may be trivial in one environment and significant in another. What operators usually need to understand is: “What does this lag mean in time and business impact?”

CDC replication lag metrics: what to measure and how to interpret them

Different platforms expose lag and progress differently. Depending on the source, target, and replication technology, useful monitoring may include:

• elapsed replication lag  
• source-side lag  
• target or apply lag  
• change or transaction backlog  
• source position versus current processing position  
• queue or task depth  
• throughput  
• last successful processing timestamp  
• catch-up trend  
• platform-specific status and diagnostic information  

The exact metric names matter less than whether operators can answer:

• How far behind are we?
• Where is the delay?
• Is the situation improving?

Ask

• How far behind is the target?  
• When did the lag begin?  
• Is this normal for this workload period?  
• What downstream systems depend on this data?  
• At what point does the lag matter to them?  

Why it matters

A target that is five minutes behind may be acceptable for one workload and unacceptable for another. The technical metric has to be interpreted in the context of the data being used.

What good looks like

The operating team can say: “We are approximately this far behind, and this is what that means for the systems using the data.” 

Not simply: “The task is still running.”

2. Is the backlog growing, stable, or shrinking?

The direction of lag can matter as much as its size. A pipeline that is significantly behind but rapidly catching up presents a different situation from a pipeline with smaller lag that is steadily losing ground. Practitioners often describe this very simply: “Is it burning down?”

Ask

• Is the backlog increasing?  
• Has it stabilized?  
• Is it shrinking?  
• Is the pipeline keeping pace with new source activity?  
• Is it processing faster than new work is arriving during catch-up?  
• Has the trend changed since the incident began? 

Capture rate versus processing or apply rate

Where the platform exposes appropriate metrics, comparing changes entering the replication flow with changes being processed at the target can help show whether the environment is gaining or losing ground. The specific measurements differ by platform. The principle is straightforward: If new work continues arriving faster than the pipeline can process it for a sustained period, the backlog will continue to grow even though the process has not stopped.

What good looks like

Operators can quickly distinguish: falling behind → holding steady → catching up

Red flag

The dashboard tells you the current lag but not whether the situation is improving.

4. Is the pipeline actually making progress?

A process can be alive without meaningfully moving data forward.

That makes evidence of successful processing activity an important sanity check.

Depending on the platform, teams may look at:

• most recent successful replication activity  
• most recent processed change  
• most recent target apply  
• progress in a saved position or processing state  
• backlog movement  
• throughput  
• another platform-specific progress indicator  

Ask

Has anything actually advanced?

And then:

Is it advancing at the rate we would expect for this source and workload?

The answer must account for the normal activity pattern of the system.

A high-volume transactional environment may normally change continuously.

Another source may have legitimate periods of little or no activity.

Why it matters

This helps identify the uncomfortable state where:

“It says it’s running, but nothing is moving.”

What good looks like

The monitoring environment helps distinguish:

the source is naturally quiet

from

replication has stopped making progress.

Red flag

The only health check is whether the replication process exists.

5. Who is already using stale data?

Many CDC issues begin as technical problems and become business problems later:

The source may still be healthy. The target may still be available. Replication may still be running. But reports, dashboards, APIs, operational systems, and analytics environments may already be working from increasingly stale data.

That creates one of the most dangerous CDC failure modes: The screens are up, but the data is getting older.

Ask

• Which applications depend on the affected target?  
• Which reports or dashboards use it?  
• Are analytics, APIs, operational systems, or AI workloads still consuming the data?  
• Does anyone downstream know the feed is stale?  
• Is an important reporting, settlement, batch, customer, or operational window approaching?  
• Should any downstream use be paused or qualified while the problem is investigated?  

Why it matters

Several very different things can be true at once:

• The source is healthy.
• The target is available.
• The pipeline is behind.
• People are still using the target.

That can make stale-data exposure harder to detect than a clean outage.

What good looks like

The team can answer:

“This is how stale the data is, these are the consumers affected, and these people know the current state.”

6. Are warnings, errors, or environmental events telling us something is changing?

Not every warning is a crisis.

And alerting on every message can create enough noise that operators stop trusting the monitoring system.

The goal is to identify the events that matter in your environment and interpret them in context.

Useful Signals May Include

• repeated connectivity errors  
• target-apply failures  
• retries  
• authentication or permissions changes  
• unusual changes in lag  
• source or target maintenance  
• platform-specific diagnostic messages

Look at the environment around the pipeline

A change in replication behavior that coincides with:

• a Db2 event  
• an IPL  
• infrastructure maintenance  
• a network change  
• a security change  
• a schema or configuration update  
• a major workload period  

may need to be interpreted differently from an isolated replication event.

This does not mean the source event necessarily caused the CDC problem.

It means operators should be able to see enough context to investigate the relationship.

What good looks like

The team knows:

• what needs immediate action  
• what needs investigation  
• what can be observed  
• when broader escalation is warranted  

Red flag

Either extreme:

Every warning wakes someone up

or

warnings are ignored until the pipeline stops.

How should you set CDC lag alert thresholds?

How Much CDC Lag Is Too Much?

There is no universal threshold that works across every mainframe CDC environment.

A useful alert threshold should reflect factors such as:

• normal transaction volume  
• expected data currency  
• downstream tolerance for stale data  
• source-history availability  
• recovery objectives  
• workload patterns  
• team response time  
• business cutoffs  
• consequences of failing to respond  

Instead of copying a generic rule such as:

“Alert after 30 minutes of lag.”

ask:

“At what point does this condition begin to threaten the data-currency, operational, or recovery requirement we actually have?”

That is the threshold worth designing around.

Build alerts around decisions, not just numbers

A useful alert should help somebody decide what happens next.

One simple model is:

Investigate

Something has moved outside the environment's expected range.

Examples may include:

• lag starting to trend upward  
• processing progress slowing  
• repeated warnings  
• unexplained changes in throughput  

The immediate job is to understand why.

Act

The condition is materially affecting an operational requirement or increasing recovery complexity.

Actions may include:

• investigating a target bottleneck  
• examining source or infrastructure conditions  
• involving an adjacent team  
• reviewing what source information the recovery path depends on  
• protecting downstream users from stale data  
• preparing for a controlled recovery  

Escalate

The issue has exceeded the technical team's normal operating authority or has begun affecting broader business, risk, or service decisions.

That may require involvement from:

• data or platform leadership  
• mainframe operations  
• downstream system owners  
• business stakeholders  
• governance or risk  
• incident management  

The thresholds will differ.

The decision path should already exist before the alert fires.

The two-minute CDC observability test: Can your team answer these five questions?

A monitoring dashboard is not valuable because it displays metrics. It's valuable because it helps operators make decisions. Try this test: Can a normal operator answer the following five questions in less than two minutes without calling a platform specialist?

This aligns with broader operational guidance from Google Site Reliability Engineering, which treats monitoring as a way to judge service health, diagnose problems, identify trends, and compare behavior over time, not simply collect or display metrics.*

1. How far behind are we?

Can the operator identify the current lag or backlog?

2. Is it getting better or worse?

Can they tell whether the backlog is growing, stable, or burning down?

3. Is processing actually moving?

Can they identify recent successful activity rather than relying on a generic Running status?

4. If this becomes a recovery event, do we understand what recovery depends on?

Can they determine what source history, state, or other information would be relevant?

5. Who is already affected?

Can they identify which downstream systems or teams may be relying on stale data?

If the operator cannot answer those five questions in roughly two minutes, ask:

• What information is missing?  
• What is spread across too many tools?  
• What requires specialist interpretation?  
• What context exists only in someone’s head?  
• What monitoring or operating procedure needs to change?  
• The two-minute framing is a practical operational heuristic, not an industry benchmark.
• The principle is what matters:

The person responding to the incident should be able to understand the state quickly enough to act.

Monitoring that only a platform specialist can interpret is not enough for an operationally critical pipeline.

How to monitor CDC catch-up after an interruption

How Do You Know If CDC Is Actually Catching Up? Monitoring does not stop once the process restarts. During catch-up, ask:

• Is the backlog shrinking?  
• How quickly is it shrinking?  
• Is new source activity still arriving?  
• Is processing keeping pace with new activity while reducing the gap?  
• Are errors or retries occurring?  
• What impact is catch-up having on the source and target?  
• Are downstream systems still working with stale data?  
• What evidence will tell us catch-up is actually complete?  

The important distinction is:

running again

versus

caught up

versus

recovered and validated.

Those are not necessarily the same state.

Mainframe workload patterns can change what “normal” looks like

Mainframe environments often have distinct operating periods:

• peak online processing  
• overnight batch  
• end-of-day  
• month-end  
• financial close  
• seasonal peaks  
• scheduled maintenance  

A lag condition that is normal in one period may signal a problem in another.

• The same is true for:
• source workload  
• log or change volume  
• target pressure  
• available catch-up capacity  
• infrastructure contention  

Avoid defining “healthy” from one average number.

Monitor the pipeline in the context of the workload it is supporting.

When monitoring detects a problem: What should happen first?

Monitoring gives teams something every recovery effort eventually runs out of: time.

The earlier a developing problem is identified, the more recovery options remain available.

Confirm what is actually happening

Is the condition transient or sustained?

Where in the replication path is progress slowing or stopped?

Establish the timeline

How long has it been happening?

When was the last confirmed successful activity?

Understand the backlog

How far behind are you?

Is it growing or burning down?

Identify downstream exposure

Who is already relying on the target?

Do they need to know the data may be stale?

Preserve the current state

Capture the relevant status, errors, timestamps, diagnostics, and other evidence before large changes are made.

Decide whether this remains a monitoring issue or has become a recovery decision

If the pipeline stops or the condition deteriorates enough that recovery actions are being considered, move into a structured recovery process rather than improvising.

My CDC pipeline is falling behind. Can it still catch up?

Lag by itself does not mean data has been lost.

And a growing backlog does not automatically mean a full reload will be required.

But lag may be becoming a recovery problem when the team cannot confidently answer questions such as:

• How far behind are we?  
• Is the backlog shrinking?  
• What recovery/process state exists?  
• What source history would recovery depend on?  
• How long is catch-up likely to take?  
• Who is consuming stale data?  
• What happens if the pipeline stops completely?  
• How will we prove the target is trustworthy afterward?  

At that point, the useful question changes from:

“Is replication running?”

to:

“Do we still understand how we would recover from where we are now?”

How Rocket Data Replicate & Sync Supports Operational Visibility

Rocket Data Replicate & Sync (RDRS) provides operational monitoring and diagnostic capabilities that can help teams understand the state of supported replication environments.

RDRS technical validation identifies capabilities including:

• process status information  
• diagnostic information  
• traces  
• latency checks  
• stored restart/process state for supported configurations  

The exact operator-facing metrics, restart coordinates, alerts, and monitoring behavior can vary by source, target, version, topology, and configuration.

Teams evaluating RDRS should therefore ask Rocket to demonstrate the monitoring experience against their specific environment.

Useful questions include:

• What can the operator see when replication starts falling behind?  
• What latency or process-state information is available?  
• How can the team tell whether replication is progressing?  
• What errors and diagnostics are exposed?  
• What saved restart/process state can operators inspect?  
• What can the team see while catch-up or recovery is underway?  
• How does the monitoring integrate into the organization’s broader operating model?  
• What changes in system behavior should be measured during high-volume catch-up?  

Production-impact claims should also be evaluated using source-specific evidence under both normal and catch-up workloads.

The useful standard is not:

“Does RDRS have monitoring?”

It is:

“Can our normal operator tell what is happening quickly enough to make the right decision?”

CDC monitoring readiness checklist

Before relying on a mainframe CDC pipeline for important downstream data, ask:

• Can we see how far behind replication is?  
• Can we tell whether the backlog is growing, stable, or shrinking?  
• Can we tell whether processing is actually advancing?  
• Do we understand what source information the supported recovery path may depend on?  
• Do we know which downstream systems depend on the affected target?  
• Can we identify when those systems may be using stale data?  
• Are meaningful errors and diagnostics visible?  
• Can we relate replication behavior to relevant source, target, or infrastructure events?  
• Are alert thresholds based on our environment rather than generic defaults?  
• Does each meaningful alert map to an action or escalation decision?  
• Can we monitor progress during catch-up?  
• Can the normal operating team answer the core health questions quickly without specialist help?  

If several answers are no, you may have monitoring data.

You may not yet have enough operational visibility.

The bottom line

Most CDC recovery problems do not begin with a complete outage. They begin with a pipeline that is falling behind, a backlog nobody has explained, and a target that is becoming increasingly stale while the process still appears healthy.

Good monitoring helps teams answer:

• How far behind are we?
• Is the backlog growing or shrinking?
• Is replication actually making progress?
• Who is already exposed to stale data?
• If this becomes a recovery event, do we still understand our options?

The goal is not to collect more metrics.

It's to recognize a developing problem early enough that operators can respond while recovery remains straightforward.

*Google SRE: Monitoring Distributed Systems

Next: Before You Restart

If the pipeline stops — or lag has progressed to the point that recovery action is being considered — move from monitoring into a structured recovery decision.

Before You Restart a Failed CDC Pipeline: 7 Questions to Answer

Work through where replication stopped, where it can resume, what happened during the gap, whether catch-up or broader resynchronization may be required, and how you will prove the target is trustworthy afterward.