"lvmirror" - Mirroring lvm2 volumes incrementally
|
2016
Mirroring lvm2 volumes efficiently by transferring changed chunks only
Motivation
"rsync" is a great tool for mirroring large amounts of data. I use it daily for mirroring several terabytes of data. Unfortunately, when it comes to large disk images of virtual machines it is inefficient. A solution that keeps track of the changes of a disk image and that only transfers these changes to the destination is needed.
Description
The Logical Volume Manager is capable of maintaining snapshots of logical volumes. It employs a copy-on-write mechanism that keeps track of the changes since snapshot creation. Leveraging these mode of operation, one can read out the copy-on-write table to identify the deltas.
This functionality can be used for performing incremental backups of lvm2 volumes. It works as follows: Initially, an lvm2 snapshot is created and its content transferred to the destination image (either into a local image file or via ssh to a remote image file). Afterwards incremental updates can be done. For doing such an incremental update, another lvm2 snapshot is created, the deltas between the two snapshots are read and applied to the destination image, and finally the older snapshot is deleted. This way, only the changed chunks need to be transferred.
Christophe Varoqui wrote a small C tool for reading the copy-on-write table and merging the identified changes into an existing dump. Thus it provides the basic functionality needed but leaves the snapshot handling to the user. "lvmirror" is a wrapper to this dds tool. It automates snapshot creation and deletion, handles the data transfer via ssh to remote destinations, and provides logging and error handling capabilities.
Implementation
"lvmirror" is a Bash script that acts as a wrapper to the "dds" tool which is written in C. The dds tool is provided for Debian as a package named "lvdds" (the name "dds" was already taken); the lvmirror wrapper is provided as Debian package "lvmirror". See download link below. Please write an email if you are a Debian maintainer who can sponsor these packages for inclusion in the official repositories.
Mini-HowTo
Create symbolic links to the _mirror.sh file that are named by the logical volumes to process.
For instance, if you want to mirror a logical volume named "my_volume" do in the /etc/lvmirror/ directory:
ln -s _mirror.sh my_volume
To process a single volume, just call the script via the symbolic link of the volume:
/etc/lvmirror/mv_volume
To process all configured volumes do:
run-parts /etc/lvmirror/
On first usage, the parameter "--initial" needs to be provided to create an initial snapshot and dump it to the destination:
/etc/lvmirror/mv_volume --initial
To check whether source and destination are in sync, you can create and compare checksums:
/etc/lvmirror/mv_volume --check
Configuration is done in the _config.sh file in the directory /etc/lvmirror/. At minimum, you need to provide the volume group on which the volumes reside on.
Edit /etc/cron.d/lvmirror to enable scheduled mirroring.