How to setup a backup system similar to OSX Time-Machine based on BTRFS

[mahesh]

Introduction
Spoiled by the OSX Time Machine solution I've been using for many years, I was looking for a similar backup method after switching completely to Kubuntu.
My first approach was to use "Back-In-Time" which worked great and did exactly what I needed.
"Back-In-Time" is basically a GUI interface to automate rsync commands and provides a good overview over all the snapshots/backups.
Over time, the complexity and volume of data grew to the point where each "back-in-time" (incremental) backup took about an hour.

It was time to investigate in a new solution.

This guide is intended to help set up a performant backup solution that takes advantage of the "copy on write" technology of BTR file systems.
This should be a "simple" tutorial with examples covering the topics of how to set up an automated BTRFS snapshot/backup configuration, similar to OSX Time-Machine with excluded subfolders. (Without GUI)
Please let me know if you have any suggestions or questions about this tutorial.

Quick objective speed comparison
Amount of changes ~100 MB:

• Time needed for snapshots with "Back-In-Time": ~60 Minutes
• Time needed for snapshots with "BTRBK": <1 Minute

Initial setup

• Kubuntu (EFI boot) installed on a SYSTEM SSD on EXT4 LVM2 (250 GB)
• DATA HDD (2 TB)
• BACKUP HDD (3 TB)
• Windows SSD (150 GB)

Names written in "Italic" are used late ron in the tutorial

Goal
Take incremental snapshots of:

• Root system of Kubuntu
• Home folder (Kubuntu)
• DATA HDD (exuding specific folders)

Steps to setup BTRFS on data disks (Essential if you are coming from EXT4 partitions)
Make sure you have a separate backup of all your data on a separate drive in case anything goes wrong
Code:

Code Select Expand

rsync -a --recursive --info=progress2 /media/UserData/ /media/USB_BackupUserData/Add
Code:

Code Select Expand

--exclude=XYZto exclude folders like Downloads etc. which you don't want to back up

Convert your file systems (DATA and BACKUP) to BTRFS if they are not already.
Code:

Code Select Expand

btrfs-convert /dev/sda1will convert an ext4 partition to btrfs

"Move" all your data on the DATA HDD root into a subvolume called @
Code:

Code Select Expand

btrfs subvolume snapshot /media/UserData/ /media/UserData/@Delete original "file references" from the root of the HDD
-xdev is critical! It defines that it only deletes files in the current mounted device (snapshot)
Code:

Code Select Expand

find /media/UserData -xdev -deleteEdit /etc/fstab

• Mount @ as your "normal" data volume
• Mount DATA HDD as a root mounting point
• Mount BACKUP HDD as a root mounting point

Code:

Code Select Expand

UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  /media/UserData_root        btrfs    defaults,subvol=/                     0       1
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  /media/UserData             btrfs    defaults,subvol=@                     0       1
UUID=YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY  /media/Backup               btrfs    defaults                              0       1

• Create a folder on DATA called snapshots
• Create a folder on BACKUP called something like btr_backup_data
• On @ "convert" every folder which should be excluded from the backup into a subvolume.
• Nested Subvolumes will be ignored during a snapshot of the parent subvolume.

Code:

Code Select Expand

cd /media/UserData
mv Downloads/ Downloads_old
btrfs subvolume create Downloads
cp -ax --reflink=always /media/UserData/Downloads_old/. /media/UserData/Downloads
rm -rf /media/UserData/Downloads_oldAdd COW-off attribute to folders which should have COW off (optional)
https://wiki.archlinux.org/title/Btrfs#Disabling_CoW
Code:

Code Select Expand

chattr -R +C /media/UserData/Downloads

Automate backup with BTRBK

Install btrbk

Code:

Code Select Expand

sudo apt-get update -y
sudo apt-get install -y btrbkEdit /etc/btrbk/btrbk.conf
The following configuration will lead to a daily snapshot which will be kept for 200 days. The first snapshot of each month will be kept 30 months.
Each month, a snapshot of the subvolume VirtualBoxVMs will be taken and saved forever.
Don't preserve snapshots on original disk >1 day (latest).
Code:

Code Select Expand

# Enable transaction log
transaction_log            /var/log/btrbk.log

# Use long timestamp format (includes hour/minute)
timestamp_format           long
 
snapshot_preserve_min  latest
snapshot_preserve      no

# Create snapshots only if the backup disk is attached
#snapshot_create ondemand

target_preserve_min    no
target_preserve        200d 30m

snapshot_dir           btrbk_snapshots

volume /media/UserData_root
  target send-receive /media/Backup/btr_backup_UserData
    subvolume @
      target_preserve 200d 30m
    subvolume @/VirtualBoxVMs
      target_preserve    *mThe config can be tested by executing a dry run of btrbk
Code:

Code Select Expand

sudo btrbk -v -n runCreate script /etc/btrbk.daily/btrfs_backup.sh
This file will be executed with root privileges. Make sure it is only accessible by superusers.
Code:

Code Select Expand

#!/bin/sh -e
btrbk -q runInstall anacron
Add following line to /etc/anacrontab
The script will be executed daily 10 minutes after boot.
Code:

Code Select Expand

1       10      btrbk.daily     /bin/bash /etc/btrbk.daily/btrfs_backup.sh

Steps to setup BTRFS for System disks (Optional)
Basically you can follow this guide to convert an existing Ubuntu installation: https://www.howtoforge.com/how-to-co...buntu-12.10-p2
LVM2 systems users need to mount the LVM2 mapper point e.g.
Code:

Code Select Expand

/dev/mapper/vgkubuntu-rootinstead of
Code:

Code Select Expand

/dev/sda1


Add System disks to automated backup (Optional)
Edit /etc/btrbk/btrbk.conf
Add the following lines to enable backups for the system and home subvolume.
In addition, this config will create a snapshot of the root and home subvolume each day. Daily snapshots are kept for 30 days and monthly for 10 months.
Code:

Code Select Expand

volume /media/System_root
  target send-receive /media/Backup/btr_backup_System
    subvolume @
      target_preserve  30d 10m
    subvolume @home
      target_preserve   30d 10m

Steps to set up non BTRFS disk backups (Optional)
If a backup is needed of a non BTRFS partition (e.g. NTFS) then the data needs to be copied (synced) to a BTRFS partition previous to a snapshot.
In the following example, a Windows user folder is backed up on a BTRFS system.
Adding rsync commands will lead to an increase of the overall backup process time.

Mount the NTFS partition by editing /etc/fstab
Code:

Code Select Expand

UUID=ZZZZZZZZZZZZZZZZ                      /media/Windows              ntfs     defaults,nls=utf8,umask=007,gid=46    0       0Create a folder and subvolume on the backup disk where the data is synced to
Code:

Code Select Expand

mkdir /media/Backup/btr_backup_Windows
 btrfs subvolume create /media/Backup/btr_backup_Windows/syncedit /etc/btrbk.daily/btrfs_backup.sh
Code:

Code Select Expand

#!/bin/sh -e
rsync -az --delete \
      --inplace --numeric-ids --acls --xattrs \
      --delete-excluded \
      --exclude=/Downloads \
      /media/Windows/Users/XYZ/ \
      /media/Backup/btr_backup_Windows/sync/
btrbk -q runAdd the following lines to /etc/btrbk/btrbk.conf
Keep daily backups for 14 days and weekly backups for 20 weeks.
Code:

Code Select Expand

volume /media/Backup/btr_backup_Windows
  subvolume sync
    snapshot_preserve_min   latest
    snapshot_preserve       14d 20w

Final /etc/fstab
Code:

Code Select Expand

UUID=DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD  /media/System_root          btrfs    defaults,subvol=/                     0       1
UUID=DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD  /                           btrfs    defaults,subvol=@                     0       1
UUID=DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD  /home                       btrfs    defaults,subvol=@home                 0       2

UUID=CCCC-CCCC                             /boot/efi                   vfat     umask=0077                            0       1
/dev/mapper/vgkubuntu-swap_1               none                        swap     sw                                    0       0

UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  /media/UserData_root        btrfs    defaults,subvol=/                     0       1
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  /media/UserData             btrfs    defaults,subvol=@                     0       1
UUID=YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY  /media/Backup               btrfs    defaults                              0       1
UUID=ZZZZZZZZZZZZZZZZ                      /media/Windows              ntfs     defaults,nls=utf8,umask=007,gid=46    0       0   Final /etc/btrbk.daily/btrfs_backup.sh
Code:

Code Select Expand

#!/bin/sh -e
rsync -az --delete \
      --inplace --numeric-ids --acls --xattrs \
      --delete-excluded \
      --exclude=/Downloads \
      /media/Windows/Users/MrLee/ \
      /media/Backup/btr_backup_Windows/sync/
btrbk -q run
kdialog --passivepopup 'BTRBK executed' 30

Final /etc/btrbk/btrbk.conf
Code:

Code Select Expand

# Enable transaction log
transaction_log            /var/log/btrbk.log

# Use long timestamp format (includes hour/minute)
timestamp_format           long
 
snapshot_preserve_min  latest
snapshot_preserve      no

# Create snapshots only if the backup disk is attached
#snapshot_create ondemand

target_preserve_min    no
target_preserve        200d 30m

snapshot_dir           btrbk_snapshots


volume /media/Backup/btr_backup_Windows
  subvolume sync
    snapshot_preserve_min   latest
    snapshot_preserve       14d 20w

volume /media/System_root
  target send-receive /media/Backup/btr_backup_System
    subvolume @
      target_preserve  30d 10m
volume /media/UserData_root
  target send-receive /media/Backup/btr_backup_UserData
    subvolume @
      target_preserve 200d 30m
    subvolume @/VirtualBoxVMs
      target_preserve    *m