in English

Transferring data with On-The-Fly Compression Using SSH and SCP

If you need to transfer large size files over very slow network, and if these files are not yet compressed, the following techniques effectively save your time and the bandwidth needed.

1. SCP Compress Option

SCP‘s -C option enables on-the-fly compression. Transfering an achive.log(/var/tmp/archive.log) using scp with compress option from a remote server to local:

scp -C -p :/var/tmp/archive.log /var/tmp

2. On-the-fly compression using SSH and TAR

Compressing ‘archive.log’ with tar, sending its output to STDOUT, and then piping it from remote server to STDIN of local:

ssh "tar zcpf - -C /var/tmp archive.log" | tar zxpf - -C /var/tmp

Speed Comparison

I tested them with and without on-the-fly compression and measured their real time. I transferred a large size file, archive.log which takes up about 1.1G disk usage from a remote server named “sakura” to local machine.

transfer without on-the-fly compression
executed: scp -p sakura:/var/tmp/archive.log /var/tmp
elapsed time: 9min 23sec
transfer with on-the-fly compression using SCP's -C option
executed: scp -C -p sakura:/var/tmp/archive.log /var/tmp
elapsed timed: 43 sec
transfer with on-the-fly compression using SSH and TAR
executed: ssh sakura "tar zcpf - -C /var/tmp archive.log" | tar zxpf - -C /var/tmp
elapsed time: 36 sec

Do compress and make the most of your CPU power, then you can finish it In a shorter period of time!