Table of Contents Transferring Files & Data.......................................................................................1 File Transfer: Overview......................................................................................................1 Verifying Files Transferred to the Lou Mass Storage System.............................................4 Using Shift for Local and Remote Transfers (Recommended)....................................5 Shift Transfer Tool Overview..............................................................................................5 Checking Shift Transfer Status and Restarting Transfers..................................................7 Shift Command Options.....................................................................................................9 Using Shift for Local Transfers and Tar Operations..........................................................19 Using Shift for Transfers and Tar Operations Between Two NAS Hosts...........................22 Using Shift for Remote Transfers and Tar Operations......................................................24 Local Transfers....................................................................................................27 Checking File Integrity.....................................................................................................27 Local File Transfer Commands.........................................................................................29 Shift Transfer Tool Overview............................................................................................31 Remote Transfers.................................................................................................32 Remote File Transfer Commands.....................................................................................32 Checking File Integrity.....................................................................................................36 Using GPG to Encrypt Your Data......................................................................................38 Shift Transfer Tool Overview............................................................................................41 Using bbFTP and bbSCP for Remote Transfers.................................................................42 The bbSCP Script........................................................................................................42 Using bbSCP for Test and Verification........................................................................45 Using bbFTP for Remote File Transfers......................................................................47 Using SUP for Remote Transfers......................................................................................52 Using the Secure Unattended Proxy (SUP).................................................................52 Advanced SUP Use.....................................................................................................58 Using the SUP Virtual File System........................................................................58 Using the SUP without the SUP Client...................................................................62 Examples.........................................................................................................................68 Inbound File Transfer Through SFEs: Examples.........................................................68 Outbound File Transfer Examples..............................................................................71 Optimizing/Troubleshooting..................................................................................73 Increasing File Transfer Rates..........................................................................................73 Dealing with Slow File Retrieval.......................................................................................74 TCP Performance Tuning for WAN Transfers....................................................................77 Optional Advanced Tuning for Linux................................................................................79 Streamlining PBS Job File Transfers to Lou......................................................................80 File Transfers Tips............................................................................................................81 Troubleshooting SCP File Transfer Failure with Protocol Error.........................................82 Transferring Files & Data File Transfer: Overview Here is a general overview of the various file transfer scenarios within the NAS environment, with pointers to related articles. File Transfers Between NAS Hosts For basic information about transferring files within the NAS secure enclave, see the following articles: • Local File Transfer Commands - cp, cxfscp, mcp, shiftc, rsync • Remote File Transfer Commands - scp, bbftp/bbscp, shiftc • Pleiades Front-End Usage Guidelines - file transfer between the compute systems or mass storage systems • Streamlining File Transfers from Pleiades Compute Nodes to Lou - within a PBS job • Checking File Integrity - to ensure the integrity of the data before and after the transfer File Transfers Between a NAS Host and Your Local Host Transferring files between a NAS host (such as Pleiades, Endeavour, or Lou) and a remote host, such as your local desktop, is more complex. There are multiple factors that you should be aware of: Which Commands to Use Remote file transfer commands such as scp, bbftp, bbscp, and sup/shiftc are supported on most NAS systems. Depending on the way the transfers are performed, you may need one or both of the client and server software for SCP, bbFTP, the bbSCP script, and/or the SUP/Shift client installed on your local host. Transfer Rates File transfer rates with the scp command, especially using SCP from versions of OpenSSH older than 4.7, can be as slow as 2 megabytes per second (MB/sec). For transferring large files over a long distance, consider doing the following: • Upgrade to the the latest version of OpenSSH. • Enable compression by adding -C to the scp command line if the data will compress well. • Use bbftp/bbscp and/or sup/shiftc. Security Issues • With SCP, your authentication information (such as password or passcode) and data are encrypted. • With bbFTP and bbSCP, only the authentication information is encrypted, while data is not. Transferring Files & Data 1 • You can use the SFEs or Secure Unattended Proxy (SUP). SUP is the easiest way to transfer files from and/or to your site if your local system is configured to allow the transfer. Inbound File Transfers When a file transfer command is initiated on a remote host, such as your local desktop, the transfer must go through either the SFEs or SUP. Using the Secure Front Ends Going through the SFEs requires authentication via RSA SecurID at the time of the transfer. You will be prompted for your RSA SecurID passcode when you issue the file transfer commands (such as scp, bbftp, or bbscp). You can use one of the following approaches: 1. One step via SSH Passthrough: Initiate scp, bbftp/bbscp from your local system to Pleiades, Endeavour, or Lou if SSH Passthrough has been set up. 2. One step via SSH ProxyCommand: Initiate scp from your local system through an SFE using the ssh ‑oProxyCommand option. To learn more, see Inbound File Transfers through SFEs Examples. Using the Secure Unattended Proxy Going through the SUP does not require RSA SecurID token authentication at the time of transfer. Instead, special "SUP keys" using RSA SecurID authentication must be obtained ahead of time. The SUP keys are good for one week and are used automatically to authenticate your file transfers using scp, bbftp, or bbscp issued on a command line or in a job script. TIP: We highly recommended that you learn about and use the Shift tool, which can be used together with the SUP to provide automated, reliable, and fast file transfers. WARNING: Although users have accounts on the SUP servers, no login session is allowed. File transfers going through SUP offers multiple benefits over going through the SFEs: • SUP allows transfers to be unattended; that is, you do not have to type in your password, passphrase, or passcode when the file transfer command is issued. So, file transfers can be done within a script that can be scheduled to run ahead of time. File transfers through the SFEs cannot be done in a script. • File transfers through SUP are done in one step, and setting up SSH passthrough is not needed since the SFEs are not involved. • SUP automatically sets some options, such as the port range allowed for bbFTP transfers, so that you don't have to set them explicitly. So, the syntax for bbFTP over SUP is greatly simplified compared to bbFTP without SUP. TIP: Some sites only allow specific outbound ports; SUP allows setting custom ports manually if needed. For example: sup bbftp -E 'bbftpd -e50000:51000' -e 'put foobar /tmp/foobar' pfe21.nas.nasa.gov See Using the Secure Unattended Proxy (SUP) and Shift Transfer Tool Overview for more information. File Transfer: Overview 2 NAS Username and Your Local Username If your NAS username and local username are different, you may have to add the appropriate username in the scp, bbftp, bbscp, or sup/shiftc command line. • If you issue the command on your local host, then username is your NAS username. • If you issue the command on a NAS host, then username is your local username. In the examples shown in Outbound File Transfer Examples and Inbound