Rsync Command in Linux.

Primary data failures can be the result of hardware or software failure, data corruption, or a human-caused event, such as a malicious attack (virus or malware), or accidental deletion of data.
Manually syncing folders or copying files from one location to another can be incredibly time-consuming to do. That's where rsync comes in super, hyper handy.

Rsync is a fast and extraordinarily versatile files and folders synchronizing tool. It can copy locally, to/from another host over any remote shell, or to/from a remote Rsync daemon. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. After that first sync, rsync will only move the changes, keeping the mirror identical to the primary server and minimizing network traffic. Scared of losing connection during transfer – this tool will start exactly where it left off, once the connection is re-established.

Installing Rsync

Most Linux distributions and macOS come with the rsync tool pre-installed. If you don't already have rsync on your machine, you can easily install it using your distribution's package manager.

$ sudo apt install rsync

Rsync Command Syntax and Common Options

The rsync utility expressions are as follows:

Local to Local:  rsync [OPTION]... [SRC]... DEST
Local to Remote: rsync [OPTION]... [SRC]... [USER@]HOST:DEST
Remote to Local: rsync [OPTION]... [USER@]HOST:SRC... [DEST]

where:

• OPTION - The rsync options.
• SRC - Source directory.
• DEST - Destination directory.
• USER - Remote username.
• HOST - Remote hostname or IP Address.

rsync has a ton of options that control how the command behaves. The following are the most commonly used options:

• -a, --archive, archive mode, equivalent to combining -rlptgoD. This option tells rsync to syncs directories recursively, transfer special and block devices, preserve symbolic links, modification times, groups, ownership, and permissions.
• -r, --recursive recurses into directories
• -R, --relative preserves the full path, it creates subdirectories in the target location.
• -v, --verbose increases verbosity.
• -z, --compress option forces rsync to compress the data as it is sent to the destination machine. Use this option only if the connection to the remote machine is slow.
• -P, is equivalent to --partial --progress. When this option is used, rsync shows a progress bar during the transfer and keeps the partially transferred files. It is useful when transferring large files over slow or unstable network connections.
• --delete, option is used, rsync deletes extraneous files from the destination location. It is useful for mirroring.
• -q, --quiet use this option if you want to suppress non-error messages.
• -e, option allows you to choose a different remote shell. By default, rsync is configured to use ssh.
• -h, --human-readable outputs numbers in a human-readable format.
  

Copy/Sync Files and Directory Locally

The following command will sync files and directories on a local machine from one location to another location. The user running the command must have read permissions on the source location and write permissions on the destination.

$ sudo rsync -avh /var/log/ ~/Desktop/log-backups/
password for snubmonkey: 
sending incremental file list

created directory /home/snubmonkey/Desktop/log-backups

log/
log/auth.log
log/btmp
log/btmp.1
log/kern.log
log/syslog

...
...

sent 17.18M bytes  received 2.80K bytes  3.82M bytes/sec
total size is 17.17M  speedup is 1.00

Omitting the filename from the destination location copies the file with the later current name. If you want to save the file under a different name, you need to specify the new name on the destination part. Here we specifically chose log-backups. To cut it short, if the destination directory doesn’t exist, rsync will create it.

Trailing slash (/)

It's worth mentioning that rsync treats source folders with a trailing slash (/) differently. If the source directory contains a trailing slash, the command will only transfer the contents of that directory to the destination directory; if not, rsync transfers the source directory to the destination directory.

Here are some examples:

with a trailing slash (/)

$ rsync -avgrtHP ~/Desktop/Out/ ~/Desktop/public

OUPUT

$ ls

Out

without a trailing slash (/)

$ rsync -avgrtHP ~/Desktop/Out ~/Desktop/private

OUPUT

$ ls

 TopG
 leo_caillard_hipster_in_stone-19.jpg
 leo_caillard_hipster_in_stone-3.jpg
 Popsicle-gif.gif
 Prints
'Screenshot-2017-12-2 Kiss Mix.png'
'Screenshot-2017-12-2 Mojito - Kiss Mix.png'

Copy\Sync Files and Directory from a Local to a Remote Server

In other to use rsync to synchronize data remotely, it must be installed on both the source and the destination machine. rsync is configured to use SSH as default remote shell.

In the following example, we are transferring a directory from a local to a remote server:

$ rsync -avgtrHP /Users/probe/www/snubmonkey-Blog/ luna@192.168.1.123:/Users/luna/Backups/Blog-backupX/

Password:
building file list ... 
172 files to consider
created directory /Users/luna/Backups/Blog-backupX

Copy\Sync Files and Directory from a Remote to a Local Server

To transfer data from a remote to a local machine, use the remote location as a source:

$ rsync -avgtrHP luna@192.168.1.123:/Users/luna/Backups/Blog-backupX/ ~/desktop/desktop-backups/

Password:
receiving file list ... 
642 files to consider
created directory /Users/probe/desktop/desktop-backups

Notice the -avgtrHP options, they will tell you a lot.

Rsync over SSH

With all that said, Rsync communication is plain clear text by default.
This makes Rsync is vulnerable to the Man-in-the-middle attacks (MITM). It doesn’t natively provide encryption of any of the data transferred over.
The usual and smart workaround is to use the Secure Shell (SSH) protocol to tunnel Rsync communication.
-e option specify the remote shell to use; here SSH.

$ rsync -avgtrHP -e ssh ~/desktop/expenses/ moon@192.168.1.112:~/Desktop/

moon@192.168.1.112's password: 
building file list ... 
202 files to consider

Rsync with Non-standard SSH Port

Rsync uses the default port 22 over SSH to communicate with the remote host. Most Linux system users set the default SSH port to a non-standard port for security reasons. If such a case, you must specify a SSH port to connect rsync command on a non-standard port.

If SSH on the remote host is listening on a port other than the default 22, specify the port using the -e option to.

Local to a Remote Server with Non-standard SSH Port

$ rsync -avgtrHP -e "ssh -p 6656" ~/desktop/desktop-backups/ venera@192.168.1.212:~/Desktop/

venera@192.168.1.212's password: 
building file list ... 
619 files to consider

Remote to a Local Server with Non-standard SSH Port

$ rsync -avgtrHP -e "ssh -p 6656" venera@192.168.1.212:~/desktop/ ~/desktop/desktop-backups/

venera@192.168.1.212's password: 
building file list ... 
619 files to consider

Use of –-include and --exclude Options

These options allow us to define which files or directories we want to include in our synchronization and which we do not.

In the following example, we will include files and directories only if they begin with the capitalized letter 'P' and exclude everything else.

Let's ls in our current directory on our local end:

$ ls

Popsicle-gif.gif
Prints
Screenshot-2017-12-2 Kiss Mix.png
Screenshot-2017-12-2 Mojito - Kiss Mix.png
Screenshot-2017-12-2 Our Story- Kiss Mix(1).png
Screenshot-2017-12-2 Our Story- Kiss Mix(2).png
Screenshot-2017-12-2 Our Story- Kiss Mix.png
instG
leo_caillard_hipster_in_stone-19.jpg
leo_caillard_hipster_in_stone-3.jpg
TopG

Let's use –include and –exclude options:

rsync -avgrtHP -e "ssh -p 6656" --include 'P*' --exclude '*' ~/desktop/Out/ venera@192.168.1.212:~/desktop/include

venera@192.168.1.212's password: 

Now, let's check on the remote side:

$ cd ~/desktop/include
$ls
Popsicle-gif.gif  Prints

--max-size Option

We can also specify the Max file size to be synced.
Here, we will synchronize only those files which are equal to or smaller than 300k.

rsync -avgrtHP -e "ssh -p 6656" --max-size '300k' ~/desktop/Out/ venera@192.168.1.212:~/desktop/max300k

venera@192.168.1.212's password: 

Remove source files after transfer

--remove-source-files option lets you remove the files from the source after the sync. is done.

rsync -avgrtHP -e "ssh -p 6656" --remvoe-source-files ~/desktop/mp3/venera@192.168.1.212:~/desktop/music

venera@192.168.1.212's password: 

After executing the above command, everything inside the mp3 directory will be removed after successful transfer.

As we've seen, rsync could be is a great replacement for scp , sftp , and cp commands. For best results, backup copies are made on a consistent, regular basis to minimize the amount of data lost between backups. The more time passes between backup copies, the more potential for data loss when recovering from a backup.

That’s all with rsync now, stay connected with Snubmonkey for more exciting and interesting future posts.