How Do I Get My Files Onto (or Off) the Cluster?
One of the most frequently asked questions is how to get files to Adroit or any other cluster. Again, Linux/MacOS has an answer out of the box: the 'scp' command. For Windows, clients like PuTTY and Mobaxterm or FTP clients (like WS_FTP–paid, sadly–or Filezilla) are needed. In both cases, make sure you're using interactive mode. Filezilla especially can be a pain with Duo Authentication for Nobel, but we have some tips here.
If you're transferring a lot of files, consider zipping them.
The default client to transfer files is 'scp.' This is a copy command that uses SSH to copy files.
The general structure of the scp command is:
scp [options] [netid]@source-host:file/location [netid@]destination-host:file/location
It's generally more straightforward to transfer files from your personal computer to the clusters, since you don't need to specify the user and host for the system you're already in.
Example - Using scp Command
A Princeton student with the netid jessedoe wants to transfer a 'data.csv' file from their personal laptop to the '/scratch/network/jessedoe' folder on the Adroit cluster. To do this, they would use the following command:
scp ~/mydatafiles/data.csv jessedoe@adroit:/scratch/network/jessedoe
Transferring a folder requires the -r option, and would therefore look like this:
scp -r ~/mydatafiles jessedoe@adroit:/scratch/network/jessedoe
If you need to transfer particularly large files, you may need to use Globus. We have additional information on Globus on our website, and run a regular workshop on data transfers with Globus.
Learn More About File Transfer Options
Additional information on learning how to transfer data–with tools such as scp, ftp, rsync, and Globus–can be found on Research Computing's Learning Resources: Data Transfer page.
SSH Keys: scp without typing passwords
Typing passwords every time you want to connect to a machine or, more annoying, every time you want to copy a file to/from a remote machine gets annoying quickly. One solution is to enable passwordless login/remote operations by generating a public/private pair of ssh keys and using them to negotiate the connection. The procedure is explained in this guide.