Backing up your files on Windows using rsync By John Bent, © 2012 Many of you will be familiar with a variety of Windows copying tools - such as xcopy, robocopy and SyncToy. These tools are great at copying data from one place to another - either on the same PC or to different PCs on the local network. However, in today's Internet-connected world, it is often desirable to copy files to a remote computer over the Internet, which is where a lot of the standard copying tools fall short: Mapping a network drive to another computer across the Internet is not recommended due to the security implications of opening up the NetBIOS/SMB ports (135-139 and 445). To securely map a drive over the Internet you should consider setting up a VPN (virtual private network) connection to the remote computer. This keeps data secure by encrypting it using a secure tunnel. Once you've established a secure connection, you can safely use xcopy, robocopy and similar tools to transfer your files to the remote computer. The next problem you will encounter however is that tools such as xcopy and robocopy perform no file compression, and always copy the entire contents of a file - even if only a single byte has changed. Introducing rsync The rsync tool has been commonplace on Unix and Unix-style systems for many years. Its purpose is to efficiently copy files from one location to another. It provides a number of key benefits that aren't found in any of the standard Windows tools: - The delta copying algorithm means that only the portions of the file that have changed are copied, significantly speeding up the backup time and reducing the required bandwidth. - When used with SSH, rsync provides a totally secure means of transferring files - without the need for a VPN connection. - rsync is totally platform-agnostic; you can use it on Unix, Linux, Mac and Windows. - It's free, open-source and has an extensive history of safe and reliable operation. In this article I will look at using the rsync client on Windows to back up to a rsync server. Any Unix or Linux system can act as a rsync server out of the box. Getting a rsync server running on Windows requires a little more work – this is something I will cover in a future article. Where to get rsync for Windows I spent a lot of time looking for the best port of rsync for Windows, and eventually settled on CWRsync. Although cwRsync is a commercial product, they make their older versions available for download free-of-charge. It is this free version that I will refer to for the remainder of this article. The only issue I ran into with the downloaded version was that it seemed to use up a lot of my CPU while copying. I tracked this down to an outdated Cygwin DLL - and once I'd updated the Cygwin DLLs to the latest version, the problem went away. For convenience, I have produced a ZIP package containing cwRsync together with the updated Cygwin libraries. You can download it here. Installing CWRsync Once you have downloaded the cwRsync package, you should extract the files to C:\Program Files\cwRsync. I strongly recommend installing to this location, otherwise you will need to modify some of the path settings. Setting up your public/private key pair You need to create, and upload, an SSH public key so that your rsync backups can occur without you being prompted for a password each time you perform your backup. It is VERY IMPORTANT that the user who creates the key is the SAME user as the one who runs the automated backup. The easiest way to ensure this is to log in as the local "administrator" to perform the following steps: Click the start button, and choose "run". In the run dialog box, type "cmd" and hit enter. The command prompt window (DOS prompt) will open. In this command prompt, type: cd "c:\program files\cwrsync\bin" and hit enter. Then type: ssh-keygen -t rsa -N '' and hit enter. (Note that the line ends with TWO single quote characters, one after another) You will be asked to "enter file in which to save the key" - just hit enter to accept the default that is given. Now, upload the newly created key to your remote rsync server using this command: rsync -avs "/cygdrive/c/documents and settings/administrator/.ssh/id_rsa.pub" username@server- name:.ssh/authorized_keys Note: Substitute username@server-name for your own user ID and rsync server, e.g. [email protected] On Windows Vista and Windows 7, the Documents and Settings folder is renamed to Users, so you will need to amend the above path accordingly. When you run this rsync command, you will be asked to confirm the first connection - enter "yes", and when prompted for your password, enter the password for your user account on the remote server. Your RSA public key is now uploaded to your remote rsync server and any further logins over ssh that this Windows user (in this case, administrator) performs will not require a password. Choosing What to Back Up The next step is to add one or more rsync commands to the end of the rsync script, telling rsync exactly what to back up. Locate the rsync.cmd file in C:\Program Files\CWRsync. Right-click the file and choose Edit to open the file in Notepad. Leave the file exactly as it is - the settings in place are very important. You are simply going to add one or more rsync commands to the end of it, one for each folder or file you want to back up. Note: If you decided to install CWRsync into a different folder, you should modify this file to change any references to C:\Program Files\CWRSync so that they point to your chosen installation directory. In this example we are adding a single line to the end of the file: rsync -avs --chmod u+rwx "/cygdrive/c/Documents and Settings/administrator" username@server-name: You can see how the source directory: C:\Documents and Settings\administrator is entered as /cygdrive/c/Documents and Settings/administrator this transformation is required for all Windows source folders. You can add as many rsync lines to the end of this script as you like - one for each folder you wish to back up. Once the rsync lines have been added to the script, you should save the file and exit Notepad. Scheduling the Backup All that is left now is to schedule the backup, which is done through Task Scheduler, which is a standard Windows feature. Open Task Scheduler by choosing: Start Menu, Programs, Accessories, System Tools, Scheduled Tasks. The Scheduled Tasks explorer will open. Double click the item named "Add Scheduled Task". In the first Window, simply click "Next". In the second Window, you are asked to "click the program you want windows to run". Do not choose a program from the list. Instead, click the "browse" button. Choose the program c:\program files\cwrsync\cwrsync.cmd In the third window, you are asked to "type a name for this task" - you can name it whatever you like. Perhaps "rsync Backup". You are also asked, in this third window, how often to perform the task. In this example we will choose to perform the backup weekly. In the fourth window, you are asked to choose a start time. We suggest sometime between 10pm and 6am. You should not perform these backups during the business day, as the network traffic they consume could affect your work. In the fifth window, you are asked to "Enter the name and password of a user" - this is the Windows login that the backup will run under, and it is important that you choose the same Windows login that you used to create your RSA key, above. The username is entered in the form computername\username ... for instance: server01\administrator The password should be entered as well. In the sixth and final window, click "finish". Testing your Backup It is extremely important that you test your backup. You are testing not only that the backup works properly and transfers the files you think it will, but also that it will properly log into your remote rsync server without prompting you for a password. The first test, making sure the automated backup is not prompted for a password, is performed simply by running the backup yourself. From the start menu, choose Run, and type in the location of your backup script: c:\progra~1\cwrsync\cwrsync.cmd You should see the backup run in the command prompt that appears, and it should complete without asking you for a password. After you have completed a successful backup, and periodically as time goes on, you should log into your remote rsync server and browse your file system to ensure the files you expect to be there are indeed present. Hints and Tips - When testing your backup, be sure that Administrator is NOT logged onto the system twice - otherwise the DOS window that appears when your backup runs automatically from the Windows scheduler may appear in the other session (perhaps a remote access login). Having the administrator account logged into the system more than once can cause you not to see error messages that will help you diagnose the batch file if it is not running properly from the Windows scheduler. - Alternatively, consider creating a new local user account called rsync, used entirely for performing your rsync backups.