January 17, 2018

In our last post we covered a couple of significant performance improvements added to UniDatata v8.2.1. In today’s post we’re covering the remaining Replication performance improvements that you benefit from once upgrading to UniData v8.2.1.

DELAYED STANDBY REPLICATION

Customers need Delayed Standby Replication for two reasons. First, it provides a quarantine period to avoid actions being immediately repeated on the subscriber. Second, it provides a reporting server with information a defined period of time behind the publisher.

Delayed Replication is used in conjunction with a hot standby system to ensure the subscriber is always a specified period of time behind the publisher. The updates are sent to the subscriber but not applied until the specified time delay has been reached. The failover process allows a specified recover-to timestamp which can be used to exclude unwanted publisher transactions.

UniData has a new System Clock structure in shared memory which the smm process will update every second. An added benefit is that other internal processes can use this structure and avoid the cost of a system call to get the current time.

HOW DOES DELAYED STANDBY WORK?

When a replication log is created on the publisher, a timestamp is now included in the replication log. The format of the timestamp is the number of seconds since 00:00:00 GMT, January 1, 1970. The new log format changes the UniData Replication protocol so logs from a prior version will not be compatible with the 8.2 release.

When processing a replication log on the subscriber, the replication writer processes will compare the system clock time in UniData shared memory with the timestamp in the replication log.  If the time difference does not exceed the specified delayed time, the replication writer process will put itself on a waiter list.

The subscriber will continue to receive logs and check to see if the delayed time period has been met. When this happens, the replication writer processes on the waiting list will wake up and begin processing the logs which exceed the delayed time period.

When using Delayed Standby Replication, the replication logs will exist on both publisher and subscriber until the delay period has been met and the logs have been applied to the subscriber. Careful consideration should be given to ensure adequate disk space exists on both the publisher and subscriber systems. The duration of the configured delayed standby time can have a significant impact on the amount of disk space required.

CROSS GROUP TRANSACTIONS

A Cross Group Transaction (CGT) is a transactional update which involves files that are defined in different replication groups. Transactional updates occur during:

• SQL INSERT/UPDATE statements
• Triggers
• Using transaction semantics in UniBasic (i.e. BEGIN TRANSACTION)

CROSS GROUP TRANSACTIONS – SYNCHRONOUS

To understand why we added the option for Asynchronous Transactions, we need to look at how Synchronous CGTs work with replication and the performance limitations that they can introduce.

The general recommendation is to avoid the CGT overhead whenever possible and to configure all files involved in transactional updates in the same replication group. On large complex systems, identifying a set of transactional files for inclusion in one replication group may not be possible and configuring too many high-volume files in the same group may not provide acceptable replication performance.

With a Synchronous CGT, the physical update does not occur until every group involved in the CGT has received their update on the subscriber.

When using the default synchronous cross group transaction functionality, customers have seen the following issues at high volume sites:

• Multiple replication groups associated with the transaction may end up waiting for other groups to complete their portion of the transaction
• The waiting groups are not processing other updates during this time
• If the load on the groups is not balanced, the amount of time the groups are waiting can be significant
• The waiting groups are generating a backlog at the same time which tends to compound the overall issue of timely processing of cross group transactions

CROSS GROUP TRANSACTIONS – ASYNCHRONOUS

Asynchronous CGT handling has been introduced to UniData Replication at UniData 8.2 to prevent the slowdowns that can be encountered as part of a Synchronous CGT. When Asynchronous CGT handling is enabled, each replication group will handle the transactional updates as if they were part of a single group. This eliminates replication groups waiting for all transactional updates to be received by all groups before processing their updates.

In the event of a hard failover, an incomplete Cross Group Transaction may exist on the subscriber. To capture and inquire on these events, we introduced recovery features that:

• Log transaction activities at runtime (Transaction Commit Logging)
• Record any incomplete CGT during a hard failover
• Provide the ability to fire off a user-defined Failover Action script

TRANSACTION COMMIT LOGGING

Although this feature does not require replication to be activated, it was primarily designed to be used with Asynchronous CGT. If Transaction Commit Logging is enabled, then details of the transactions will be logged to the TRANS_COMMIT_LOG file in @UDTHOME/sys account. The logging can also be configured to ignore any error in Transaction Commit Logging or rollback the transaction if Transaction Commit Logging fails.

When this feature is used with replication, the TRANS_COMMIT_LOG file should itself be replicated to the subscriber. This will allow inquiries for incomplete transactions to be completed on the subscriber in the event of a hard failover.

FAILOVER

In the event of a failover, any incomplete cross group transactions are logged in the REP_FAILOVER_LOG file. The record contains the transaction id (TID) and the publisher system timestamp. The TID and timestamp can be used to read the transaction information from TRANS_COMMIT_LOG (if replicated to the subscriber).

FAILOVER ACTION SCRIPT

A user-defined failover action script / bat file which can be fired off at the end of the failover event has been added as part of the Asynchronous CGT functionality. If this script exists, it will be executed at the end of the failover process. It can be used to login to the system account and run a user defined failover paragraph.

Typically, we would expect it to use the failover timestamp to search the REP_FAILOVER_LOG File for any incomplete transaction lists. It can then use the TRANS_COMMIT_LOG file for the Transaction Update List and can correct the updates based on the application logic or to automatically start any phantom-based processing you may need after failing over a system.

REPLICATION PACING

Introduced at UniData 8.1.2, Replication Pacing allows Data Replication to gracefully slow down the pace of the publisher database updates when Replication becomes “overflowed” to minimize Replication disablement. In the event of Replication Disablement, a manual refresh of the subscriber from the publisher is needed. As this is normally a time-consuming exercise, disablement needs to be avoided whenever possible.

Although pacing will slow down the pace of the publisher updates, the mechanism is designed to improve the overall speed of the system by reducing the overhead that can be caused by excessive replication overflow. It should be noted that we see the field level replication and the intelligent queue manager enhancements as limiting if not eliminating the need for pacing to occur at 8.2

Pacing helps avoid Replication disablement due to log files exhausting available disk space. Although disablement allows the publisher to continue, the systems are then out of sync, and the subscriber must be manually refreshed.

Pacing slows down the publisher updates to prevent excessive “overflow” conditions. As the publisher updates slow down, the subscriber can catch up. This reduces the pace of updates being sent to the log files and minimizes the possibility of disablement due to a disk capacity. Pacing also reduces the overhead on the publisher of having to deal with multiple overflow conditions.

WHAT IS REPLICATION OVERFLOW?

Each group defined within UniData Replication uses a shared memory structure called the Replication buffer. During normal conditions, if all the updates are kept within the Replication buffer for a group then overflow does not occur. When UniData Replication cannot use the Replication buffer, the overflow is stored in the disk-based Replication log files.

Replication overflow can be broken down into the following categories:

• Log Info Buffer overflow (Log Info Extension File – LIEF)
• Replication Buffer overflow (Log Extension File – LEF)
• The Transaction Control Area overflow (TCA)
• Replication Log Reserve File overflow (LRF)

There are several potential causes for replication overflow and they may be avoided with tuning of replication. To better understand the configuration parameters and monitoring tools that are available within UniData Replication, please contact the Support / Services team at Rocket who can provide more details. They may even be covered in a future blog as it is another large subject.

CONFIGURING AND UNDERSTANDING THE PARAMETERS AND COMPONENTS OF REPLICATION PACING

There are several components to Replication pacing, some of which can be tuned and some of which are set internally within the product. The three elements are:

• Replication Disablement Threshold
• Replication Pacing Criteria
• Degradation Weight

Replication Disablement Threshold

This is controlled via tuning the udtconfig REP_DISABLE_DISK_PCT and it defines the percentage full that UniData Replication will allow the disk volume containing the Replication Logs (defined via REP_LOG_PATH) to become full before Replication is disabled. The default value is 95%.

With a default value of 95%, if the Volume containing the replication logs gets to 95% full (or more) then replication will be disabled. As discussed earlier, replication disablement should be avoided whenever possible. However, without the disablement threshold you could end up in the situation where the disk volume becomes full and that may cause the recovery process to be even more difficult.

Replication Pacing Criteria

The pacing mechanism takes effect when any of pacing criteria are reached.

• The criteria include Replication overflows and events that may lead to a failure of Replication.
• Each of the criteria has an associated degradation weight which cannot be changed
• The criteria are applied per session

Replication Pacing Criteria Table and Degradation Weight

Pacing Criteria Name	Degradation Weight	Description
LIB approaching full	10	Log Info Buffer is 95% full
LIEF overflow	100	Log is added into LIEF rather than LIB
LEF overflow	4 + (1/Gig)	Log is added into LEF rather than the replication buffer
TCA overflow	100	TCR is created in the TCA overflow flow
Approaching Disablement	150	85% of the REP_DISABLE_DISK_PCT is reached for LIEF / LEF or LRF
LRF overflow	1/Gig	LRF overflowed

Degradation Weight and Time

The REP_PACING phrase (in the repconfig file) defines the pacing level at a group level and has allowed values of 0 to 255. The default is 5, and a value of 0 disables the pacing mechanism.

The verb SET.PRIORITY.LEVEL is used to set the session priority level. The allowed values are 0 to 9 (0 is the highest). The default value is 3. This is currently only used by Replication. Used with no option, it displays the currently assigned value. The verb can be used programmatically within a session. For example, when starting a large batch process, you may want to ensure that pacing does not occur.

The formula is:

<Total Degradation Weight> = Sum(<Degradation Weight>* REP_PACING for the group)

The actual degradation time for the session is defined in milliseconds:

<Total Degradation Weight> * SET.PRIORITY.LEVEL.

The total degradation time for a session can be accessed in UniBasic using the SYSTEM(517) function.

If you wanted to disable pacing for a whole group then setting REP_PACING to 0 would have that effect. If you wanted to disable on a session basis then using the SET.PRIORITY.LEVEL to 0 would have that effect.

A session can go in and out of pacing as it enters and leaves the pacing criteria.

The replication monitoring tool in XAdmin can be used to see if a group has undergone pacing or is currently subject to Pacing.

If the color of the connection status for a group is Blue (see above) this indicates at least one session updating files in that replication group is subject to active pacing.  The Disablement Percentage, Maximum Replication Log Size, and Current Replication Log Size values are viewable in the pane to the right of the traffic lights. The group weight and total pacing weight values are viewable in the group pane.

The information can also be viewed via the command line reptool interface.

OTHER CHANGES STARTING AT 8.2

The tuneable MAX_REP_GROUPS has been added to udtconfig with a default value is 512.  It controls the maximum number of replication groups that can be defined in the repconfig file. If this value is exceeded, UniData will not start, or if exceeded during a ‘reconfig’ the new configuration will not be actioned. The error message that will appear in the rm.errlog will be:

 ‘MAX_REP_GROUPS is too small in udtconfig to start UniData, please increase it to at least a value of nnn.’.

For UNIX versions, an example exception action script is provided in the $UDTBIN directory called except_script_example and will be updated as needed in future releases.

For all versions, a new checking program called SCN.REP.CFG is available from Rocket Support and will be added to the product shortly. The program will scan the current repsys and repconfig files and report potential problems or misconfigurations.

Replication Buffers can now be configured to be larger than 2GB.