clubmate.fi

Sync large quantities of files or directories between local machine and server with rsync

A guide to syncing files between local and remote environment with rsync.

rsync is an awesome tool, it’s designed to effortlessly move large quantities of files, locally or over the network.

If your rsync is out of date, I have a little post on how to upgrade it.

rsync syntax

rsync options source destination

Commonly used options:

-r

recursive, for moving directories.

-z

compress file data.

-a

archive mode, combo of -rlptgoD, meaning: preserves symbolic links, special and device files, modification times, group, owner, and permissions.

-P

Combo of --progress and --partial, meaning it shows a progress bar and it’s possible to resume interrupted transfers.

--delete

Delete files that don't exists in source system, good when syncing.

The usual set of option is -zaP, which is easy to remember.

Basic example:

$ rsync -zaP --delete ./uploads -p 5555 bob@yourserver.com:/var/www/example.com/public_html

That’ll sync a local directory uploads/ to remote /var/www/example.com/public_html. Local path can be relative.

More examples below.

Noteworthy thing: if you do commands like that a lot, you could put your ssh credentials into the ~/.ssh/config file, then the host can be referred with a short name, here's an article on how to do that.

Moving files around locally

Make some test files:

$ mkdir rsync-test-{1..2}
$ touch rsync-test-1/file{1..50}

Syncing files from rsynctest1/ to rsynctest2/ is as follows:

$ rsync -r rsynctest1/ rsynctest2

Notice the forward slash / after the source directory. This means that grab the contents of the directory, not the directory itself. Test the difference if you like:

$ rsync -r rsynctest1 rsynctest2

But really, the -a option would fit here like fist in eye:

$ rsync -a rsynctest1/ rsynctest2

It preserves all the setting the directory has, symlinks, permissions etc.

Sync directory from local to remote

Example syntax for moving a file to a server (assumes you set server shortname in ssh config):

$ rsync -zaP --delete uploads/ server_shortname:/path/to/uploads

Note: the --delete option removes the files from the remote that are not present in the local, it's truly synchronizing the directories.

Btw: tab completion in the remote server is possible, see this post for more info.

Sync directory from remote to local

Just "flip it and reverse it":

$ rsync -zaP --delete server_shortname:/path/to/uploads uploads/

Sync both ways

Run it twice using the -u flag.

-u, --update

Skip files that are newer on the receiver

$ rsync -ur uploads/ server_shortname:/path/to/uploads
$ rsync -ur server_shortname:/path/to/uploads uploads/

With more options:

$ rsync -zuaP uploads/ server_shortname:/path/to/uploads
$ rsync -zuaP server_shortname:/path/to/uploads uploads/

Or use Unison, it’s a tool made just for that, but I’ve never used it myself.

Moving a shit ton of files

The -z option enables compression, but only for individual files, it wont compress the whole directory, which would be ideal if moving thousands of files for the first time.

To really bulldoze files over to a server, you could go like:

$ tar zcvf - ./source | ssh bob@example.com "cd /target/dir; tar xvzf -"

Here’s what that does:

1. Compresses the source directory
2. Uploads it to the target path
3. Decompresses the tarball in the remote server

If you want it shorter and more usable, wrap it to a function and put it to your .bashrc or .bash_profile, then you got it at your disposal anywhere:

function zipSync() {
  local source=$1
  local target=$2

  # Get the host and path from the $target string
  IFS=':' read -a array <<< "$target"
  local host=${array[0]}
  local destination=${array[1]}

  tar zcvf - $source | ssh $host "cd $destination; tar xvzf -"
}

Example:

$ zipsync uploads my_server:/var/www/example.com/public_html

Problems with that function:

• Can’t tab complete.
• Can’t specify port in the target path, this won’t work: zipsync uploads -p 5555 bob@xmpl.com:/path/ cause the -p is read as the second parameter.

But: