Скачать или смотреть Understanding the Desync and Resync Behavior in Your Galera Cluster

Discover why your Galera cluster node desyncs and resyncs periodically, and learn how running backups can impact system consistency.
---
This video is based on the question https://stackoverflow.com/q/63500025/ asked by the user 'Amit P' ( https://stackoverflow.com/u/2638053/ ) and on the answer https://stackoverflow.com/a/63664845/ provided by the user 'Amit P' ( https://stackoverflow.com/u/2638053/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Galera cluster one of the node desyncs and resyncs periodically

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding the Desync and Resync Behavior in Your Galera Cluster

If you're managing a Galera cluster with MariaDB, you might encounter periodic messages that indicate a node is desynchronizing (desync) itself from the cluster and then resynchronizing (resyncing) shortly thereafter. This can lead to confusion regarding the stability and reliability of your database environment. In this guide, we'll explore what causes this behavior and how to address it.

The Issue: Desync and Resync Messages

You may have noticed the following warning message occurring regularly for one of your nodes in a three-node Galera cluster setup:

[[See Video to Reveal this Text or Code Snippet]]

These messages occur every hour, which may raise concerns about your cluster's health. Therefore, it’s important to understand what’s happening under the hood and why these notifications are being triggered.

Configuration and Server Details

Before diving into the solution, it's worth noting that all your servers share the same configuration and specifications:

MariaDB Version: 10.3.12

Galera Version: 25.3.23-1

Operating System: RHEL 7.4

State Transfer Mechanism: Default

Investigation: What Causes Desync and Resync?

After thorough investigation, it was determined that the periodic desyncing and resyncing behavior is primarily due to backup operations. The Mariabackup tool was running scheduled backups on the node, leading to temporary de-synchronization for the sake of data consistency.

How Backup Operations Impact the Node

During backups, the node requires a consistent view of the database, which may trigger the desync behavior.

The node momentarily disconnects from the group to ensure that no changes are made while the backup is taking place.

Once the backup process completes, the node re-establishes its connection with the cluster and resyncs itself.

Other Possible Factors

While backups are the most direct cause, it’s worth reviewing other potential issues that could contribute to similar behavior, including:

Network instability or high latency between nodes.

Incorrect cluster configuration settings.

Insufficient system resources (CPU, memory) on the node experiencing desyncs.

Recommendations for a Smooth Operation

To maintain a smooth and efficient operation of your Galera cluster, consider the following recommendations:

Schedule Backups Wisely: Conduct backups during low-traffic hours to minimize impact on cluster performance.

Monitor Network Stability: Ensure that the network connection between nodes remains stable to avoid unplanned desync events.

Review Configuration Settings: Verify that the Galera configuration is optimized for your specific workload and environment.

Allocate Adequate Resources: Make sure each node has adequate resources to handle backup and operational loads without triggering desync behavior.

Conclusion

In summary, consistent messages about desynchronization and resynchronization within a Galera cluster can be associated with running backup processes. By understanding and monitoring your cluster's behavior, you can ensure that your system remains stable and reliable while performing essential maintenance tasks.

If you have any further questions or experiences related to Galera clusters, feel free to share them in the comments below!