Скачать или смотреть How to Use rsync for Safe File Synchronization with Snapshots

Learn how to ensure `rsync` only syncs specific files by taking snapshots, providing a robust solution for dynamic data directories.
---
This video is based on the question https://stackoverflow.com/q/63403600/ asked by the user 'machinery' ( https://stackoverflow.com/u/1684118/ ) and on the answer https://stackoverflow.com/a/63404375/ provided by the user 'Léa Gris' ( https://stackoverflow.com/u/7939871/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Let rsync take a snapshot of files before copying

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Efficient File Synchronization with rsync: Using Snapshots

In the fast-paced world of data management, ensuring data integrity during synchronization processes is vital. If you're using rsync, a powerful utility for copying and synchronizing files remotely as well as locally, you might encounter scenarios where files are written to the source directory while the synchronization is in progress. This raises a critical question: How can you make rsync take a snapshot of files before initiating the copying process?

In this guide, we’ll dive into an effective solution to manage file synchronization using rsync while addressing the issue of files being added to the source directory during the copy process.

The Problem at Hand

When running rsync to synchronize files, particularly from a dynamic source directory (like /local/data), unintentional changes or additions to the files can complicate the process. Your initial script structure is as follows:

[[See Video to Reveal this Text or Code Snippet]]

In this setup, while the first rsync command is running, new files may be added to /local/data, and this could lead to inconsistencies in the data being synchronized.

The Solution

To solve this challenge, we can enhance our bash script by utilizing file snapshots. This can be achieved by creating a null-delimited list of files you want to synchronize before executing the rsync operation. Below is the revised script, which incorporates this solution:

Step-by-Step Breakdown of the Solution

Set Up Your Environment: Start by defining your source and destination directories. This ensures clarity and easier adjustments in the future.

[[See Video to Reveal this Text or Code Snippet]]

Create a Temporary File List: Use the find command to populate a temporary file list with the specific files to sync. Here we leverage a null-terminated format that rsync can read directly.

[[See Video to Reveal this Text or Code Snippet]]

Run the Initial rsync Command: Utilize the --from0 and --files-from options in rsync to refer to your temporary file list. This helps to synchronize only the files that were present at the time of the snapshot.

[[See Video to Reveal this Text or Code Snippet]]

Verify and Remove Source Files: After the files have successfully been copied, execute a second rsync command that checks the files and deletes them from the source only if the checksum is valid.

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

By implementing this script enhancement, you can ensure that rsync safely synchronizes files while taking a snapshot of the relevant files before initiating the copy operation. This method not only maintains data integrity but also cleverly avoids file conflicts during the synchronization process.

Using rsync effectively can dramatically improve data management efficiency. Now, you have a comprehensive strategy to make file synchronization robust and error-free!