H2-For-The-Arts Documentation Release 2020 Raffaella D'auria

Home , Box (company), GLX, Ncdu, Rclone, Xming

H2-for-the-Arts Documentation Release 2020

Raffaella D’Auria

Apr 22, 2020

Using the Hoffman2 cluster:

1 Connecting/Logging in 3 1.1 Connecting via a terminal...... 3 1.2 Connecting via NX clients...... 6

2 Unix command line 101 11 2.1 Navigation...... 11 2.2 Environmental variables...... 13 2.3 Working with ﬁles...... 14 2.4 Miscellaneous commands...... 15

3 Data transfer 17 3.1 Data transfer nodes...... 17 3.2 Tools...... 17 3.3 Cloud storage services...... 18 3.4 Globus...... 19 3.5 rclone...... 22 3.6 scp...... 35 3.7 sftp...... 36 3.8 rsync...... 37

4 Rendering 39 4.1 Getting an interactive-session...... 39 4.2 Submitting batch jobs...... 44 4.3 GPU-access...... 49

5 Indices and tables 51

i ii H2-for-the-Arts Documentation, Release 2020

This page will guide you on how to get started on H2. How-to use this documentation Please use the table of contents contained in the menu to navigate to the needed page, or use the search box, also in the page menu, to look up a topic in the documentation by keywords. About the Hoffman2 Cluster The Hoffman2 Cluster is a project of the Institute for Digital Research and Education (IDRE). It opened to users on January 28, 2008. The Hoffman2 Cluster is managed and operated by the IDRE Research Technology Group under the direction of Lisa Snyder.

Using the Hoffman2 cluster: 1 H2-for-the-Arts Documentation, Release 2020

2 Using the Hoffman2 cluster: CHAPTER 1

Connecting/Logging in

1.1 Connecting via a terminal

In order to be able to connect to the Hoffman2 Cluster and open graphical user interfaces such as Maya you need to have an SSH-client and an X Window System installed on your computer. Step by step instructions on how to acquire and use these tools are given below for Linux, Mac and Windows operating systems. All the connections to the cluster are based on the SSH secure protocol that requires authentication. Depending on the operating system that your computer is running, you have several choices for which command line application to use when connecting with the cluster.

1.1.1 Connectiong from Linux

An X Window System server is generally available and running with the default installation (or can be readily installed via the OS package manager). If you are connecting from a Linux box, you can use the standard SSH-client generally installed with the OS and available via any terminal application. Open a terminal window and type the following: ssh-X login_id @hoffman2.idre.ucla.edu where login_id is replaced by your cluster user name.

1.1.2 Connecting from a MacOS

On Mac OS X, the X windows system is called XQuartz. Mac OS X 10.5 10.6 and 10.7 installed it by default, but as of 10.8 Apple has dropped dedicated support and directs users to the open source XQuartz page. You can install XQuartz from the OS distribution media or download it from XQuartz page. MacOS, the operating system running on a Mac computer, comes equipped with a fully functional Termainal application and an SSH-client. The Terminal application can be located using the Spotlight Search or searching the Applications folder in Finder.

3 H2-for-the-Arts Documentation, Release 2020

Note: To enable indirect GLX and to allow remote visualization on the Hoffman2 Cluster, open the MacOS Termianl application and issue at its command prompt: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

Warning: You will need to reboot your machine before being able to open GUI applications on Hoffman2.

Note: You will need to perform the operation above only once

To connect to the Hoffman2 Cluster, open the MacOS Terminal application and issue at its command prompt: ssh-Y login_id @hoffman2.idre.ucla.edu where login_id is replaced by your cluster user name.

1.1.3 Connecting from Windows

To connect to the Hoffman2 Cluster from your Windows computer Install one of any of the following free ssh client programs.

Note: Only one of the following options is needed to connect to the cluster via SSH.

• MobaXterm a free X server for Windows with SSH terminal, SFTP and more. • XMing and Xming fonts provides a X Window System Server for Microsoft to be used in conjunction with GitBash an emulation shell for Windows. • Cygwin a free Linux-like environment for Windows. To add Cygwin/X server, select the xinit package from the X11 category. • Xshell a commercial option. Instructions are given for the ﬁrst two of the options described above.

Instructions for MobaXterm

Download and install the MobaXterm Home Edition, either the Portable or Installer edition will work. To log into the Hoffman2 Cluster open MobaXterm and click on the Session button (the Sessions button is circled in green in MobaXterm application with the Sessions button circled in green.). A new window will pop-up to allow you to choose a type of session. Choose SSH by clicking on the SSH button (upper left corner) and fill the Remote host field with the Hoffman2 Cluster address, that is: hoffman2.idre.ucla.edu save the session by pressing the OK button. You can also create a new session type which you will use to transfer files back and from the cluster to your local machine, by selecting the Sessions button again and choosing SFTP as the session type and fill the Remote host field with the Hoffman2 Cluster data transfer node address, that is:

4 Chapter 1. Connecting/Logging in H2-for-the-Arts Documentation, Release 2020

Fig. 1: MobaXterm application with the Sessions button circled in green.

Fig. 2: MobaXterm application with the SSH session set for Hoffman2.

1.1. Connecting via a terminal 5 H2-for-the-Arts Documentation, Release 2020

dtn1.hoffman2.idre.ucla.edu save the session by pressing the OK button. To access the cluster double click on the save SSH session.

Instructions for XMing and GitBash

Download and install XMing and Xming fonts following the installer instructions. Download and install GitBash following the installer instructions. Start Xming from the programs, open GibBAsh from the Start Menu, at its command promopt issue: export DISPLAY-localhost:0

Connect to the Hoffman2 Cluster by typing: ssh-XY login_id @hoffman2.idre.ucla.edu where login_id is replaced by your cluster user name.

1.2 Connecting via NX clients

NX is a free, secure, compressed protocol for remote X Window System connections for Windows, Linux, Mac OSX, and Solaris. We currently support connecting to the Hoffman2 cluster via the NoMachine client as well as the X2Go client.

1.2.1 NoMachine client

Download NoMachine Client for your operating system

A free NoMachine client is available from NoMachine for Windows, Mac OSX, and most Linux distributions. • Download NoMachine from the website. • Use the download link for your operating system (Windows, Mac OSX, or Linux Distributions).

Install

• Once downloaded, double click on the downloaded package ﬁle. • Click the Run/Install button. • When the NoMachine Client Setup Wizard starts, accept all the setup defaults.

Conﬁgure

• Open the NoMachine client and continue to the connection page. • Click on the New link to create a connection to the Hoffman2 Cluster. • Specify the protocol as SSH and press Continue. • Provide the host name (nx.hoffman2.idre.ucla.edu).

6 Chapter 1. Connecting/Logging in H2-for-the-Arts Documentation, Release 2020

• Leave the port ﬁeld set to 22 and press the Continue button. • Select Use the system login and press the Continue button. • Select Password and press the Continue button. • Select Don’t use a proxy and press the Continue button. • Name your connection and press the Done button. Please notice that if you are planning to suspend and reconnect to an NX session, you will need to repeat the conﬁgu- ration steps described above for the hosts:

nx1.hoffman2.idre.ucla.edu nx2.hoffman2.idre.ucla.edu

Run

• Open the NoMachine client and continue to the connection page. • Click on the saved connection that you previously conﬁgured. • Enter your username and password and your NX session with a virtual desktop on the cluster will start.

How to get a terminal window

After you have logged into the Hoffman2 remote desktop, click on the Applications menu. Then select System Tools and Terminal.

Note: To pin the terminal to the panel, right click on the Terminal menu item and select Add this launcher to panel.

How to terminate your NX session

From the gnome desktop, select: System -> Log Out login_id . . . A small window will open. Click the Log Out button.

How to suspend an NX session

Make note of the actual NX node where your session is running (typically NX1 or NX2) to allow for future reconnec- tions. To terminate the NX client on your computer, close the application or close the virtual desktop window.

How to reconnect to a suspended NX session

If you are planning to reconnect to a session, make note of the actual NX node where your session is running (typically NX1 or NX2). Make sure to configure your NX client for the NX node where your session is running. You will need to create a configuration using either nx1.hoffman2.idre.ucla.edu or nx2.hoffman2.idre.ucla.edu. See the Configuration section for more detailed instructions.

1.2. Connecting via NX clients 7 H2-for-the-Arts Documentation, Release 2020

Note: Reconnecting to a suspended NX session is not always possible. Lengthy computations should be run as batch jobs. We recommend terminating NX sessions (logout) instead of suspending them.

On fast networks, a preferred alternative to NX is to run an X server on your platform (e.g., X11 packages for linux platforms, XQuartz for mac OSX, XMing and Xming fonts for Windows), and connect to the cluster via a terminal (or via PuTTY on Windows platforms) by setting up X11 forwarding.

1.2.2 X2Go client

Install X2Go client

Follow the installation guide at X2Go site. Download the X2Go client, select the appropriate client for your operating system (Windows, Mac OS, Linux) and the correct version onto your computer. Double click on the downloaded package, accept the licensing terms, and follow the instruction on the screen.

Note: MacOS users will need to install the XQuartz dependency prior to installing X2Go. It may be obtained on the XQuartz website. You may also encounter a security warning that “x2goclient.app” is from an unidentiﬁed developer. Simply right click on the app again and select Open.

Conﬁgure the X2Go client to connect to Hoffman2

Launch the application by double clicking on the X2Go Client desktop icon, if one was created, or by searching for the application in your App folder and double clicking on the application ﬁle. If prompted by the Windows Firewall, select Allow access so that sshd.exe and other X2Go related programs will be able to run.

Create a new session

From the Session tab in the X2Go client select New session. You can name your session with any name (e.g., x2go@hoffman2). Leave the Path ﬁeld unchanged and in the Host ﬁeld enter:

x2go.hoffman2.idre.ucla.edu

Leave the other fields unchanged and choose the Session type to be either the default KDE or GNOME. Save the new session by pressing the OK button. You can also create sessions for the specific nodes x2go1 and x2go2 if you will need to reconnect to a suspended session. However, we strongly suggest you start new sessions by using the session associated with the x2go. hoffman2.idre.ucla.edu address. Sessions on the x2go1 and x2go2 nodes can be created by using in the “Host” field respectively:

x2go1.hoffman2.idre.ucla.edu x2go2.hoffman2.idre.ucla.edu

Name the sessions accordingly (for example: x2go1@hoffman2, x2go2@hoffman2).

Start the X2Go client

To connect to the Hoffman2 cluster open the X2Go client, double click on an already created session, and enter your Hoffman2 username and password. The ﬁrst time you connect, you will be prompted to trust the host key. Do so

8 Chapter 1. Connecting/Logging in H2-for-the-Arts Documentation, Release 2020

by pressing the Yes button. On Windows, a few more ﬁrewalls will pop up and you will have to allow access to the X2Go Client and associated programs. You should now be connected to a remote desktop on the Hoffman2 cluster.

1.2. Connecting via NX clients 9 H2-for-the-Arts Documentation, Release 2020

10 Chapter 1. Connecting/Logging in CHAPTER 2

Unix command line 101

The Hoffman2 Cluster is a Linux-based computing platform. One of the most powerful ways to interact with the Hoffman2 Cluster is via the Unix-command line. You can access the Hoffman2 command line by logging into the system either via a terminal application (see: Connecting-via-SSH) or by opening a remote desktop on Hoffman2 (see: Connecting via NX clients) and opening a terminal window. Any of these methods will open a command-line interpreter or unix shell (or most simply shell). This page presents a review of common tasks you may need to execute or script in your shell.

2.1 Navigation

2.1.1 Listing ﬁles

To list what ﬁles are present in the current directory type at the Hoffman2 command prompt:

to list hidden ﬁles or folders:

ls-a

to get extra information on the ﬁles or folders in the current working directory use:

ls-la

-l adds a long listing format with a total of nine ﬁelds, a typical line of output from ls -la could look, for example, as the following:

-rwxr-xr--1 joebruin gobruins 3974052 Apr 17 12:53 myexecutable

where: the first character of the first field indicates the file type (-, for a regular file, d for a directory, l, for a link, etc.); the remiaining 9 characters of the first field indicate the file permissions (for example: rwxr-xr-- means that the file can be read, written and executed by its owner, as denoted by the first three fields rwx; read and exectuted by

11 H2-for-the-Arts Documentation, Release 2020

the owner’s group: r-x - if the path to the current directory is also accessible by the group; and read by any other user on the system: r-- - again assuming that the path to the current directory is also accessible by any user on the system)

Note: By default on the Hoffman2 Cluster $HOME directories are accessible only by the owner and therefore no ﬁle or folder within them, despite their permissions, can be accessed by any other user.

the second field indicates the number of hard links to the file; the third field indicates the file owner (for example joebruin); the fourth file indicates the file group (gobruins in the example above); the first field indicates the file size - in blocks; the sixth, seventh and eight fields indicate the data and time of last modification; and ninth and last field shows the file name (myexecutable in the example above). to display the size of the files in units of bytes, MB or GB:

ls-lah

for example:

-rwxr-xr--1 joebruin gobruins 3.8M Apr 17 12:53 myexecutable

to order the list of ﬁles by time (newest on top):

ls-laht

to order the list of ﬁles by time oldest on top):

ls-lahtr

2.1.2 Files and directories management

2.1.3 Copying, moving,removing ﬁles

To copy a ﬁle from a location to another:

cp myfile.txt myfile1.txt

to move a ﬁle from a location to another:

mv myfile.txt myfile1.txt

to remove a ﬁle:

rm myfile.txt

Changing directory

Upon logging onto the Hoffman2 cluster you land in your $HOME directory which is a space with up to 20 GB of capacity fully backed-up (see: Your-home-directory). To navigate to your temporary storage directory (see: File- system-for-temporary-use) issue:

cd $SCRATCH

to check in which directory you currently are issue:

12 Chapter 2. Unix command line 101 H2-for-the-Arts Documentation, Release 2020

pwd user joebruin would get:

/u/scratch/j/joebruin to move up two directories: cd../.. to move back to the `$HOME directory: cd~

Creating/removing directories

To create the directory test in your $HOME directory issue: mkdir $HOME/test to navigate to this newly created directory issue: cd $HOME/test you check the location of this new directory, issue: pwd user joebruin would get:

/u/home/j/joebruin/test

To remove an empty directory, issue: rmdir test to force removal of a ﬁle or a directory: rm-rf test

2.2 Environmental variables

On a unix-like system any string that is preceded by a $ is a variable, which means that the operating system will interpret the string according to what it has been set to. Environmental variables are a set of variables that inﬂuences the behavior of the command-line or shell interpreter in use (typically Bash). Example of environmental variables that are set for you are:

$HOME $SCRATCH $PATH $LD_LIBRARY_PATH to check the content of a variable, for example $HOME, issue:

2.2. Environmental variables 13 H2-for-the-Arts Documentation, Release 2020

echo $HOME the content of your $PATH and $LD_LIBRARAY_PATH inﬂuences the kind of executables and libraries that you can access at the command line (without specifying their full path).

2.3 Working with ﬁles

Unix-like systems have more than one way to display the content of a ﬁle, or selecgted snippet of it, on your unix shell. Among some of the most usefule commands are: cut, less, more, head and tail. To see how they work, issue at Hoffman2 command promt, for example: cut $HOME/.bashrc less $HOME/.bashrc more $HOME/.bashrc to exit from displaying a ﬁle with more or less issue: q

To see just the first few lines of a file: head $HOME/.bashrc or, for the first two lines: head -n 2 $HOME/.bashrc to see the last few lines of a file: tail $HOME/.bashrc or, for the last two lines: tail -n 2 $HOME/.bashrc to look at the last few lines of a file that is being written on the fly: tail -f /u/local/licenses/LOGS/logit.matlab use: Control-C (to exit) Unix-like system have very powerful editors packed with shortcuts that largely simplify the editing task. The learning curve on some of these editors, such as vi, can be pretty steep. However, there are several simpler, if less powerful, alternatives.

2.3.1 Editors that do not require X11 Forwarding

Here are some of the ﬁle editors available on the Hoffman2 Cluster presented in order of the complexity of their use, from less to more complex: nano emacs vi

14 Chapter 2. Unix command line 101 H2-for-the-Arts Documentation, Release 2020

2.3.2 Editors that require X11 Forwarding

Here are some of the ﬁle editors available on the Hoffman2 Cluster when X11 Forwarding is enabled (see: Connecting via NX clients) presented in order of the complexity of their use, from less to more complex: nano gedit& emacs& vi gvim&

2.4 Miscellaneous commands

Most unix commands have manual pages that can be accessed via the man command, for example to learn more about the ls command, issue: man ls or: man vi

2.4. Miscellaneous commands 15 H2-for-the-Arts Documentation, Release 2020

16 Chapter 2. Unix command line 101 CHAPTER 3

Data transfer

Note: Personal information and other sensitive data, including statutory, regulatory, and contractually protected data — for example, human subjects research, restricted research, student and educational data, and PHI — are prohibited on Hoffman2.

3.1 Data transfer nodes

Hoffman2 has two dedicated and performance tuned data transfer nodes with advanced parallel transfer tools1 to support your research workﬂows. For transferring large ﬁles and/or large datasets, you will need to use the DTN nodes.2 dtn1.hoffman2.idre.ucla.edu dtn2.hoffman2.idre.ucla.edu

DTN SSH key ﬁngerprints

MD5:3c:9c:67:d8:c5:a4:ae:77:07:5f:10:2f:20:4a:75:0f

SHA256:kah9BJwSzrlFnVp9Tg+El2IdcCN7JgN5+Ur2RyIdvwM

3.2 Tools

There are several methods to transfer data between a local computer and Hoffman2 and we will cover some on this page.

1 If your research group requires additional transfer tools, please submit a request via our helpdesk 2 See Role-of-the-login-nodes

17 H2-for-the-Arts Documentation, Release 2020

Depending on your network connection and the amount of data to transfer between your local computer and Hoffman2, you have options under graphical or command-line interace utilities. To move large ﬁles and/or large datasets, we recommend using other parallel transfer tools, e.g. Globus_, rclone, Aspera. Possibly, combine ﬁles into archive. . . Cloud resources. . . .

Note: Allowed Transfer Protocols

3.2.1 Graphical utilities

Many of the graphical utilities support a ﬁle manager with “drag and drop” functionality between your local and remote computers. You may use whichever tool you prefer. . .

Table 1: Graphical Transfer Utilities Application Transfer Protocols Platform CyberDuck SFTP Windows and Ma- cOS FileZilla SFTP Windows, MacOS, Linux MobaXterm SFTP Windows WinSCP SFTP Windows

3.2.2 Command-line utilities

Table 2: Command-line interface Transfer Utilities Application Platform Features scp MacOS, Linux, Windows3 secure copy sftp MacOS, Linux, Windows3 secure ﬁle transfer rsync MacOS and Linux sync ﬁles to and from curl MacOS and Linux xfr data using various protocols wget MacOS and Linux download via HTTP/HTTPS rclone Windows, MacOS, Linux rsync for cloud storage

3.3 Cloud storage services

Warning: Please review UCLA Allowable Data Use - Cloud Storage Services

3 Git for Windows provides a BASH emulation with SSH/SCP/SFTP clients. Git for Windows

18 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

If you have data use questions, please contact IT Services Client Support

Email: [email protected]

Faculty and staff use of cloud storage services must comply with applicable University policies, notably policies relating to the protection of University data and the UC Electronic Communications Policy. This includes the data use requirements in the table below [see here], which are based on University-negotiated agreements established to help safeguard information about individuals and other conﬁdential information for which the campus is a steward. Always employ due care when processing, transmitting, or storing sensitive information. Violation of these data use requirements or other campus policies may result in disciplinary action up to and including termination.

3.3.1 Box

Box is an online cloud storage and collaboration tool that provides users with the ability to easily store, access, and share ﬁles and folders anywhere on any device. UCLA provides a free enterprise Box account to all faculty, staff and students. Currently, a UCLA enterprise Box account comes with unlimited storage space for faculty, staff and students. All accounts offer a 15 gigabyte per ﬁle upload limit and other enterprise features such as version history. NOTE If you need assistance with your Box account, please contact the IT Support Center at [email protected] or by phone at (310) 267-HELP (4357). ** To transfer data between Hoffman2 and your Box account, please use the rclone application on our DTN nodes. **

3.3.2 Google Drive

Google Drive is a file storage and synchronization service developed by Google. Google Drive allows users to store files on their servers, synchronize files across devices, and share files. Google Apps is made available to UCLA as part of the UC Office of the President agreement. Google Apps is not appropriate for storing or sharing any sensitive data, including but not limited to: HIPAA regulated data, credit card information, social security numbers, and driver’s license numbers. ** To transfer data between Hoffman2 and your Google Drive account, please use the rclone application on our DTN nodes. **

3.3.3 Dropbox

Not permitted, all data use prohibited.

3.4 Globus

Note: For more information about Globus please refer to their website.

Globus is a software tool to transfer files (from kilobytes to petabytes) across the web in a reliable, high-performance and secure way. It provides fault-tolerant, fire-and-forget data transfer using simple web or command line interfaces. It is approriate for transferring very large files either between your desktop machine and a remote machine like the Hoffman2 Cluster, or between two remote machines on which you have accounts; both remote machines need to be part of the Globus project. All XSEDE resources are configured as Globus endpoints.

3.4. Globus 19 H2-for-the-Arts Documentation, Release 2020

3.4.1 Installation

If you want to transfer ﬁles to or from your personal computer, you will need to download and install the Globus Connect Personal software. In order to download the Globus Connect Personal software, you will need to create a Globus Connect Personal endpoint. The term endpoint describes the different locations where data can be moved to or from using the Globus transfer, sync and sharing service. Endpoints can either be personal (on a user’s personal computer) or multiuser (located on a server, for use by multiple people. Globus Connect Personal is available, • for Mac OS 10.7 or higher (Intel only) • for common x86-based Linux distributions • for Windows 7, Windows 8, and Windows 10 1. Open your web browser and click on one of the detailed installation instruction links for the platform running on your personal computer - macOS, Linux, Windows. 2. The next step will ask you to create a personal endpoint using the Globus web app. This will require your login to the Globus web app and you can do so using your UCLA Login ID. From the pull-down list of organizations, please select “Univeristy of California-Los Angeles”. 3. On the left-side of the page, you can see the navigation menu, click on endpoints 4. At the top of the page, click on create a personal endpoint 5. You will be asked to name your endpoint 6. Click, “Generate a setup key” and copy it to your paste buffer/clipboard (you will be asked for this unique setup key when you install the software on your personal computer) 7. Download and Install Globus Connect Personal for the running platform (macOS, Linux, Windows) on your personal computer

Important: Windows users: To run the installation with administrator permissions, Hold CTRL + SHIFT and click on the Globus Connect Personal installer. Installing as non-administrator: By default, Globus Connect Personal prompts to be installed in C:Program Files. Regular users can not write to this folder. Instead, browse to a place you have write access to (e.g. your Desktop folder).

3.4.2 Conﬁguration

Windows

1. Right-click the Globus Connect Personal icon in the taskbar and select, Options. . . to configure Globus Connect Personal. Configuration options are divided into four groups; the most important (and commonly used) are the Access and General options. The Access tab lists folders that will be accessible via Globus for file transfer and sharing. You can add folders by clicking the “+” sign and selecting the folder you wish to make accessible.

20 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

Important: By default, the only folder listed is your home directory

To share a folder, add it to the accessible list and check the “Shareable” box. Note: You must be a Globus Plus user to share ﬁles and folders. UCLA does not have an active subscription and therefore is unable to offer sharing on the Hoffman2 Globus multiuser endpoints - Data transfer nodes. The General tab allows you to specify whether you want Globus Connect Personal to run when Windows starts and whether the software should automatically check for updates.

Note: Drive Mapping: Globus Connect Personal on Windows will translate a path beginning with /~/ into your home directory, e.g. C:\Users\'login_id'\. To access paths and drives outside of your home directory, use the syntax /drive_letter/path, for example /C/xinfo lists the C:\xinfo directory. Also, as discussed above, it would be necessary for the C:\xinfo directory to be permitted in the Accessible Folders conﬁguration as well. If the C:\xinfo directory is not permitted in the Accessible Folders conﬁguration, then that folder will not be accessible via your endpoint.

macOS

1. Click the Globus Connect Personal icon in the main menu bar and select Preferences. . . to configure Globus Connect Personal. The Access preferences tab lists accessible directories for file transfer and sharing and provides more control over what information is accessible on your Globus Connect Personal endpoint. By default, your home directory (e.g.: /Users/'login_id') is read/write accessible. The check box Deny access to hidden (e.g. security) files in your home directory option controls whether or not you can access hidden files (i.e. filenames beginning with “.”“) in your home directory. By default, Globus Connect Personal does not allow access to files like: ~/.globusonline and ~/.ssh. Click the “+” icon and select a folder to make it accessible for transfers. To allow a folder to be shared with others, add it to the accessible list and check the “Sharable” box.

Note: If you remove everything from the access list, no files will be accessible on your Globus Connect Personal endpoint and you will be prompted to add accessible paths. You can either click “+” and add directories and files, or click “Reset to Defaults”. You must be a Globus Plus user to share files and folders. If you are not a Globus Plus user, click the “Enable sharing” icon and follow the instructions. UCLA does not have an active subscription and therefore is unable to offer sharing on the Hoffman2 Globus multiuser endpoints - Data transfer nodes.

3.4.3 Using Globus

This section describes using the Globus web app to transfer ﬁles to or from a personal computer and the Hoffman2 Globus endpoints - Data transfer nodes. 1. Be sure the Globus Connect Personal application is running on your personal computer 2. Open your web browser and connect to the Globus web app 3. Login to the Globus web app with your UCLA Login ID credentials. To do so, from the pull-down list of organizations, please select “Univeristy of California-Los Angeles” 4. On the left-side of the page, you can see the navigation menu, click on “File manager”

3.4. Globus 21 H2-for-the-Arts Documentation, Release 2020

5. Click on the text field, “Collection” and begin to type the name for your personal endpoint and click on it to select the endpoint 6. You should see a directory listing for the path, /~/ (which should list the contents of your home directory) 7. On the right-side of the page, there are some options - click “Transfer or Sync to. . . ” 8. The page should now be divided with your personal endpoint on the left and now click on “Collection” text field in the right pane and search, “Official UCLA Hoffman2 Data Transfer Node 1” or “Official UCLA Hoffman2 Data Transfer Node 2” 9. When you select a Hoffman2 Cluster endpoint, you will be prompted to enter your username and password. Enter your Hoffman2 username and password. Your Hoffman2 Cluster home directory and its contents will display. 10. To transfer a file or directory, select it and near the bottom of the window pane, click the “Start” button

Note: You will receive an email from Globus Notification ([email protected]) when the file transfer has completed. To have Globus show you the status and history of your file transfers, click “Manage Data” from the menu and select “Activity”.

3.5 rclone

What is it? rclone is a command line program to sync ﬁles and directories to and from cloud storage - https://rclone.org

3.5.1 Installation

1. SSH to one of our data transfer nodes, either dtn1 or dtn2, e.g

$ ssh -Y`login_id`@dtn1.hoffman2.idre.ucla.edu

Where login_id is replace by your cluster user name. The ﬂag, -Y is to enable trusted X11 forwarding 2. Download and unzip the most recent version of rclone

$ wget https://downloads.rclone.org/rclone-current-linux-amd64.zip $ unzip rclone-current-linux-amd64.zip

At the time of this document, rclone-v1.51.0 was the current version. Please replace the version number below with the version you downloaded. . . 3. You can now copy the rclone executable to your $HOME/bin directory. If the copy fails, you need to create $HOME/bin subdirectory, e.g. “mkdir $HOME/bin”

$ cp rclone-v1.51.0-linux-amd64/rclone $HOME/bin/.

To run the software, type:

$ rclone

3.5.2 Conﬁguration

22 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

Set-up rclone to sync with Box

Tip: More detailed instructions can be found on rclone’s website

1. Connect to our DTN nodes, either dtn1 or dtn2 and enable trusted X11 forwarding, e.g.

$ ssh -Y`login_id`@dtn1.hoffman2.idre.ucla.edu

Where login_id is replaced by your cluster user name 2. Type, rclone conﬁg

$ rclone config No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q>

3. Type, “n” for new remote [connection] 4. Enter a name for this connection, e.g. “box” 5. Enter the type of storage from the menu - Box. Type, “box”

Type of storage to configure. Enter a string value. Press Enter for the default(""). Choose a number from below, or type in your own value 1 / 1Fichier \ "fichier" 2 / Alias for an existing remote \ "alias" 3 / Amazon Drive \ "amazon cloud drive" 4 / Amazon S3 Compliant Storage Provider(AWS, Alibaba, Ceph, Digital Ocean,

˓→Dreamhost, IBM COS, Minio, etc) \ "s3" 5 / Backblaze B2 \ "b2" 6 / Box \ "box" 7 / Cache a remote \ "cache" 8 / Citrix Sharefile \ "sharefile" 9 / Dropbox \ "dropbox" 10 / Encrypt/Decrypt a remote \ "crypt" 11 / FTP Connection \ "ftp" 12 / Google Cloud Storage(this is not Google Drive) \ "google cloud storage" 13 / Google Drive \ "drive" 14 / Google Photos \ "google photos" (continues on next page)

3.5. rclone 23 H2-for-the-Arts Documentation, Release 2020

(continued from previous page) 15 / Hubic \ "hubic" 16 / In memory object storage system. \ "memory" 17 / JottaCloud \ "jottacloud" 18 / Koofr \ "koofr" 19 / Local Disk \ "local" 20 / Mail.ru Cloud \ "mailru" 21 / Mega \ "mega" 22 / Microsoft Azure Blob Storage \ "azureblob" 23 / Microsoft OneDrive \ "onedrive" 24 / OpenDrive \ "opendrive" 25 / Openstack Swift(Rackspace Cloud Files, Memset Memstore, OVH) \ "swift" 26 / Pcloud \ "pcloud" 27 / Put.io \ "putio" 28 / QingCloud Object Storage \ "qingstor" 29 / SSH/SFTP Connection \ "sftp" 30 / Sugarsync \ "sugarsync" 31 / Transparently chunk/split large files \ "chunker" 32 / Union merges the contents of several remotes \ "union" 33 / Webdav \ "webdav" 34 / Yandex Disk \ "yandex" 35 / http Connection \ "http" 36 / premiumize.me \ "premiumizeme" Storage>

6. At the prompt for a “Box App Client Id”, just hit “Enter” to accept the default 7. At the prompt for a “Box App Client Secret”, just hit “Enter” to accept the default 8. At the prompt for a “Box App conﬁg.json location”, just hit “Enter” to accept the default 9. Type, “1” for the box_sub_type; Rclone should act on behalf of a user

Enter a string value. Press Enter for the default("user"). Choose a number from below, or type in your own value 1 / Rclone should act on behalf of a user \ "user" (continues on next page)

24 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

(continued from previous page) 2 / Rclone should act on behalf of a service account \ "enterprise"

10. Edit Advanced Conﬁg? Up to you; In this example I said, “No”

Edit advanced config?(y/n) y) Yes n) No(default) y/n> n

11. Use Auto Conﬁg? Say ‘Yes’ and wait for ﬁrefox to launch. You will need to authenticate with your box password and UCLA Shibboleth to authorize the application rclone’s access to your UCLA Box account

YourMagicTokenHerYourMagicTokenHere your browser doesn't open automatically go to the following link: http://

˓→[FollowTheLinkInYourTerminal Log in and authorize rclone for access Waiting for code... Got code ------[box] type = box box_sub_type = user token = YourMagicTokenHere ------

12. Type, “Y” to accept the new settings and save the conﬁguration

Important: At this point you’re done, unless you want to password protect your rclone conﬁguration (recommended). If you want to password protect your conﬁguration, hit ‘p’; then ‘q’ to quit.

Set-up rclone to sync with Google Drive

In the following example we will: • conﬁgure rclone for a remote connection to your UCLA Google Drive • copy a ﬁle from Hoffman2 to a new folder on Google Drive

STEP 1: Create a folder on Google Drive

Note: We will be creating a new folder on your UCLA Google Drive to test a transfer later. . .

1. Connect your web browser to Google Drive 2. Authenticate with your @ucla mailbox, e.g. login_id‘@g.ucla.edu; where ‘‘login_id‘ is replaced by your UCLA Logon ID 3. Click on NEW and select FOLDER and give it a name, e.g. “h2xfr”

3.5. rclone 25 H2-for-the-Arts Documentation, Release 2020

STEP 2: Conﬁguring an rclone connection to your Google Drive

Tip: More detailed instructions can be found on rclone’s website

1. Connect to our DTN nodes, either dtn1 or dtn2 and enable trusted X11 forwarding, e.g.

$ ssh -Y`login_id`@dtn1.hoffman2.idre.ucla.edu

Where login_id is replaced by your cluster user name 2. Type, rclone conﬁg

$ rclone config No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q>

3. Type, “n” for new remote [connection] 4. Enter a name for this connection, e.g. “gdrive” 5. Enter the type of storage from the menu - Google Drive. Type, “drive”

26 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

6. Next, you will need to either create a Google Application ID [for best performance] or use the default internal key. Should you choose the default internal key, just press, ‘enter.’

Important: For best performance, you will need to create a Google Application ID. If you choose to do so, please refer to the steps outlined in, https://rclone.org/drive/#making-your-own-client-id

Your terminal should be here . . .

Google Application Client Id (continues on next page)

3.5. rclone 27 H2-for-the-Arts Documentation, Release 2020

(continued from previous page) Setting your own is recommended. See https://rclone.org/drive/#making-your-own-client-id for how to create your own. If you leave this blank, it will use an internal key which is low performance. Enter a string value. Press Enter for the default(""). client_id>

Google Application Client Secret Setting your own is recommended. Enter a string value. Press Enter for the default(""). client_secret>

Question: What level of access do you want to give rclone? In this example, I’ve set it to ‘1’

Scope that rclone should use when requesting access from drive. Enter a string value. Press Enter for the default(""). Choose a number from below, or type in your own value 1 / Full access all files, excluding Application Data Folder. \ "drive" 2 / Read-only access to file metadata and file contents. \ "drive.readonly" / Access to files created by rclone only. 3 | These are visible in the drive website. | File authorization is revoked when the user deauthorizes the app. \ "drive.file" / Allows read and write access to the Application Data folder. 4 | This is not visible in the drive website. \ "drive.appfolder" / Allows read-only access to file metadata but 5 | does not allow any access to read or download file content. \ "drive.metadata.readonly" scope>1

In this example, I just hit ‘enter’ to accept the default

ID of the root folder Leave blank normally.

Fill in to access "Computers" folders(see docs), or for rclone to use a non root folder as its starting point.

Note that if this is blank, the first time rclone runs it will fill it in with the ID of the root folder.

Enter a string value. Press Enter for the default(""). root_folder_id>

In this example, I just hit ‘enter’ to accept the default

Service Account Credentials JSON file path Leave blank normally.

Needed only if you want use SA instead of interactive login.

(continues on next page)

28 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

(continued from previous page) Enter a string value. Press Enter for the default(""). service_account_file>

You can conﬁgure the advanced settings, in this example, I did not. . .

Edit advanced config?(y/n) y) Yes n) No(default) y/n> n

In this example, I’m saying ‘no’ to auto conﬁg and just copy and paste the link in my web browser

Remote config Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine y) Yes(default) n) No y/n> n

Now copy the link provided in your conﬁguration and paste in your web browser to give rclone access to your UCLA Google Drive

Please go to the following link: https://PleaseFollowTheLinkOnYourConsole/

DO you approve?

../_static/rclone-gdrive-auth.png

Copy and paste the verﬁcation code from your browser window

Enter verification code>

Configure this as a team drive? y) Yes n) No(default) y/n> n

REVIEW THE REMOTE SETTINGS and type “y” to save the connection

------[gdrive] type= drive client_id=[This will list your client_id] client_secret=[This will list your client_secret] scope= drive token=[This will list your token] ------y) Yes this is OK(default) e) Edit this remote (continues on next page)

3.5. rclone 29 H2-for-the-Arts Documentation, Release 2020

(continued from previous page) d) Delete this remote y/e/d> y Current remotes:

Name Type ======gdrive drive e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q>

Important: At this point you’re done and have the option to password protect access to rclone. If you choose to set a password, you will need it every time you use rclone.

Set a password (s) or quit (q) rclone conﬁg

Current remotes:

Name Type ======gdrive drive e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q>

3.5.3 Using rclone rclone command list

Table 3: rclone commands command description rclone about Get quota information from the remote.

rclone authorize remote authorization.

Continued on next page

30 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

Table 3 – continued from previous page command description rclone cachestats Print cache stats for a remote

rclone cat Concatenates any ﬁles and sends them to stdout.

rclone check Checks the ﬁles in the source and destination match.

rclone cleanup Clean up the remote if possible

rclone conﬁg Enter an interactive conﬁguration session.

rclone copy Copy ﬁles from source to dest, skipping already copied

rclone copyto Copy ﬁles from source to dest, skipping already copied

rclone copyurl Copy url content to dest.

rclone cryptcheck Cryptcheck checks the integrity of a crypted remote.

rclone cryptdecode Cryptdecode returns unencrypted ﬁle names.

rclone dbhashsum Produces a Dropbox hash ﬁle for all the objects in the path.

rclone dedupe Interactively ﬁnd duplicate ﬁles and delete/rename them.

rclone delete Remove the contents of path.

rclone deleteﬁle Remove a single ﬁle from remote.

rclone genautocomplete Output completion script for a given shell.

rclone gendocs Output markdown docs for rclone to the directory sup- plied.

Continued on next page

3.5. rclone 31 H2-for-the-Arts Documentation, Release 2020

Table 3 – continued from previous page command description rclone hashsum Produces an hashsum ﬁle for all the objects in the path.

rclone link Generate public link to ﬁle/folder.

rclone listremotes List all the remotes in the conﬁg ﬁle.

rclone ls List the objects in the path with size and path.

rclone lsd List all directories/containers/buckets in the path.

rclone lsf List directories and objects in remote:path formatted for parsing

rclone lsjson List directories and objects in the path in JSON format.

rclone lsl List the objects in path with modiﬁcation time, size and path.

rclone md5sum Produces an md5sum ﬁle for all the objects in the path.

rclone mkdir Make the path if it doesn’t already exist.

rclone mount Mount the remote as ﬁle system on a mountpoint.

rclone move Move ﬁles from source to dest.

rclone moveto Move ﬁle or directory from source to dest.

rclone ncdu Explore a remote with a text based user interface.

rclone obscure Obscure password for use in the rclone.conf

rclone purge Remove the path and all of its contents.

Continued on next page

32 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

Table 3 – continued from previous page command description rclone rc Run a command against a running rclone.

rclone rcat Copies standard input to ﬁle on remote.

rclone rcd Run rclone listening to remote control commands only.

rclone rmdir Remove the path if empty.

rclone rmdirs Remove empty directories under the path.

rclone serve Serve a remote over a protocol.

rclone settier Changes storage class/tier of objects in remote.

rclone sha1sum Produces an sha1sum ﬁle for all the objects in the path.

rclone size Prints the total size and number of objects in remote:path.

rclone sync Make source and dest identical, modifying destination only.

rclone touch Create new file or change file modification time.

rclone tree List the contents of the remote in a tree like fashion.

rclone version Show the version number.

rclone ﬂag list

rclone has a number of options to control its behavior. Options that take parameters can have the values passed in two ways, --option=value or --option value. However boolean (true/false) options behave slightly differently to the other options in that --boolean sets the option to true and the absence of the ﬂag sets it to false. It is also possible to specify --boolean=false or --boolean=true. Note that --boolean false is not valid - this is parsed as --boolean and the false is parsed as an extra command line argument for rclone. Options which use TIME use the go time parser. A duration string is a possibly signed sequence of decimal numbers,

3.5. rclone 33 H2-for-the-Arts Documentation, Release 2020 each with optional fraction and a unit suffix, such as “300ms”, “-1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. Options which use SIZE use kByte by default. However, a suffix of b for bytes, k for kBytes, M for MBytes, G for GBytes, T for TBytes and P for PBytes may be used. These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively. The rclone global flag list is available to every rclone command and is split into two groups, non backend and backend flags. rclone copy

Note: For more detailed information, please refer to the rclone copy page on their website. rclone copy - copies the source to the destination, skipping already copied Synopsis Copy the source to the destination. Doesn’t transfer unchanged files, testing by size and modification time or MDSUM. Doesn’t delete files from the destination. Note that it is always the contents of the directory that is synced, not the directory. SO when source:path is a directory, it’s the contents of source:path that are copied, not the directory name and contents.

$ rclone copy source:path destination:path[flags]

Note: Use the -P/--progress ﬂag to view real-time transfer statistics

Hint: See the –no-traverse option for controlling whether rclone lists the destination directory or not. Supplying this option when copying a small number of ﬁles into a large destination can speed transfers up greatly.

Example: Using rclone to copy a ﬁle to Google Drive Let’s copy the rclone zip ﬁle from Hoffman2 to your Google Drive, h2xfr folder dtn1:~$ rclone copy rclone-current-linux-amd64.zip gdrive:h2xfr Enter configuration password: password> 2020/04/06 16:40:20 INFO : rclone-current-linux-amd64.zip: Copied(new) 2020/04/06 16:40:20 INFO : Transferred: 11.177M / 11.177 MBytes, 100%,3.671 MBytes/s, ETA 0s Transferred:1/1, 100% Elapsed time:3.0s dtn1:~$

‘rclone-current-linux-amd64.zip’ is the file in your current working directory that you want to transfer from Hoffman2 to Google Drive. ‘gdrive’ is the name of the connection you gave when you configured your rclone connection and ‘h2xfr’ is the name of the folder you created in Google Drive. If you configured a password for rclone, you will be prompted for it before the file is sent. That’s it, the file has been uploaded. You can view the remote end with the ls command.

34 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

dtn1:~$ rclone ls gdrive:h2xfr password: 11913756 rclone-v1.51.0-linux-amd64.zip

• It may be useful to view the contents of your remote connection before uploading or downloading ﬁles. To do so without having to use a browser, use the following commands: rclone lsd[remote]:

** Replace remote: with the name of your remote connection, e.g. gdrive or box • To view the contents of a speciﬁc directory, e.g. ‘h2xfr’ in your Google Drive, use the command: rclone ls gdrive:h2xfr

• If you want to test a command, use the --dry-run ﬂag. Below assumes the name of your rclone remote connection to Google Drive is named, ‘gdrive’ and the directory you’re syncing to is named, ‘h2xfr’ rclone[command] --dry-run gdrive:h2xfr

3.6 scp

For security reasons, Hoffman2 Cluster allows ﬁle transfer only with scp or sftp or grid-ftp. For the same reason, you should use an scp or sftp client on your local machine. You should not use the scp command on the cluster. The scp and sftp commands transfer ﬁles using the secure shell protocol (ssh) in which data is encrypted during transfer. The use of scp requires that an scp client be run on the machine that you use to initiate the transfer and that it communicate with a server run on any other machines which participate in the transfer. The Hoffman2 Cluster, like most Linux and Unix systems, runs both a client and a server. There is an scp client command on desktop Linux/Unix systems and on Macs (use Terminal). On Windows, you usually have to install an ssh client which comes with an scp program. The syntax of the Linux/Unix scp command is very similar to the cp command. For complete scp syntax, enter: man scp

Here is a simplified scp syntax that accomplishes most transfers: scp [-r] source target where source is the name of the file on your local machine, and target will be the name of the file on the cluster. For the source on your local machine, specify an absolute or relative file name or directory name. You can use wild cards to transfer multiple files to an existing target directory. Specify -r to transfer a whole source directory and its files. For the target on the cluster, specify your login_id and the Hoffman2 address, followed by a colon (:), followed by the file specification. You can specify the directory where the file is to be saved, or a dot “.” meaning the same name in your home directory, or an absolute or relative path including a new file name. For large files or large amounts of data, use the Hoffman2 data transfer node dtn2.hoffman2.idre.ucla.edu

`login_id`@dtn2.hoffman2.idre.ucla.edu:filespec

For example:

3.6. scp 35 H2-for-the-Arts Documentation, Release 2020

scp myfile `login_id`@dtn2.hoffman2.idre.ucla.edu:. will transfer the file named myfile from your current directory on your local machine to your home directory on the Hoffman2 Cluster. Its name on the cluster will be $HOME/myfile

3.7 sftp

If you would prefer to use a graphical SFTP client. . .

Please look under Tools for a list of SFTP GUI applications. secure ﬁle transfer program sftp is a ﬁle transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. It may also use many features of ssh, such as public key authentication and compression.

Table 4: SFTP Interactive Commands Command Function Example cd Change remote directory cd [path] to path lcd Change local directory to lcd [path] path ls Display remote directory ls listing lls Display local directory lls listing pwd Display remote working pwd directory lpwd Print local working direc- lpwd tory mkdir Create remote directory mkdir [path] speciﬁed by path get Retrieve the remote path get remote_path [local_path] and store it on the local machine put Upload local-path and put local_path [remote_path] store it on the remote machine exit Quit SFTP exit quit Quit SFTP quit help Display help text sftp help

For complete syntax, please refer to the man page.

$ man scp

Let’s establish an SFTP connection Replace login_id with your cluster user name below. This is an example of using the sftp client on macOS Terminal:

36 Chapter 3. Data transfer H2-for-the-Arts Documentation, Release 2020

$ sftp `login_id`@hoffman2.idre.ucla.edu `login_id`@hoffman2.idre.ucla.edu's password: Connected to `login_id`@hoffman2.idre.ucla.edu. sftp>

Now let’s move a ﬁle from our local computer to Hoffman2 Replace login_id with the user name on your local computer. What is my current local working directory? and what ﬁles are listed: sftp> lpwd Local working directory: /Users/`login_id`/share/ sftp> lls a.out index.html

What is my remote working directory? sftp> pwd Remote working directory: /u/home/l/`login_id`

Let’s create a new directory on the remote computer and change our working directory to it sftp> mkdir uploads sftp> cd uploads

Copy ﬁle, “a.out” from local computer to Hoffman2 sftp> put a.out Uploading a.out to /u/home/l/login_id`/uploads/a.out a.out

˓→ 100% 3125 703.0KB/s 00:00 sftp> ls a.out

3.8 rsync

The rsync command uses the SSH2 protocol to efficiently transfer files. It is perhaps most useful in keeping groups of files on different computers up to date with each other. Here is a 2-part example of discovering the status of files in a common directory named mydir. It is comparing files in your Hoffman2 $HOME/mydir directory with those on your local machine mydir directory. You need both parts to ensure any new files from either source are synchronized. Part 1: Run this on your local machine:

$ rsync -an --itemize-changes `login_id`@dtn2.hoffman2.idre.ucla.edu:mydir .

Any ﬁles preﬁxed with > in the ouput are different on Hoffman2 and you may want to download them from Hoffman2 (get):

$ rsync -av `login_id`@dtn2.hoffman2.idre.ucla.edu:mydir .

Part 2: Run this on your local machine:

3.8. rsync 37 H2-for-the-Arts Documentation, Release 2020

$ rsync -an --itemize-changes mydir `login_id`@dtn2.hoffman2.idre.ucla.edu:

Any ﬁles preﬁxed with < in the output are different on your local machine and you may want to upload them to Hoffman2 (put):

$ rsync -av mydir `login_id`@dtn2.hoffman2.idre.ucla.edu:

For more information about the rsync command and additional options, enter man rsync at the shell prompt:

$ man rsync

38 Chapter 3. Data transfer CHAPTER 4

Rendering

Computational resources on the Hoffman2 cluster are managed by a scheduler. Any CPU or memory intensive computing task should be performed within either a scheduled interactive session or a batch. This section addresses how to request an interactive session and how to submit a non interactive job for batch execution. Upon connecting to the cluster you are on a login node (unless you are connecting to the cluster via Jupyter Noteboks or Jupyter Lab). Login nodes are meant as a gateway to the computational resources on the cluster and should be used only for light editing, interactive session requests, or batch job submission. (For more information, see Role of the login nodes).

4.1 Getting an interactive-session

4.1.1 Basic usage

To request an interactive session, open your terminal/GitBash window and connect to the Cluster (see: Connect- ing/Logging in). You can request an interactive session on one or more compute nodes. To get an interactive session with a runtime longer than the default 2 hours, you will need to specify a value for the scheduler complex h_rt. To request more memory than the default value of 1GB, you should specify a value for the scheduler complex h_data. For example, to request an interactive session with a runtime of 3 hours and a total of 4GB of mememory, issue at the Hoffman2 command prompt: qrsh-l h_rt=3:00:00,h_data=4G

If you are planning to run an application that will use more than one CPU core, you should request cores using the -pe directive (where is the name of the parallel environment and is the number of cores that you are plannig to use). If your application can run multiple threads, or, more generally, uses multiple cores in a shared memory paradigm, you will need to reqeust the number of cores you are planning to use with the -pe shared directive. For example, to request 4 CPU core, a runtime of 8 hours, and 2GB of memory per core, issue:

39 H2-for-the-Arts Documentation, Release 2020

qrsh-l h_rt=8:00:00,h_data=2G-pe shared4

Note: Notice that h_data is memory per core. Since you are requesting 2GB of memory per core and 4 cores, your total memory request is: 2GB x 4 cores = 8GB.

If your job/program can run across multiple nodes (e.g. MPI), use:

qrsh-l h_rt=8:00:00,h_data=2G-pe dc\ * 4

A word about h_data: When invoking an interactive session with qrsh, the proper memory size needs to be speciﬁed via h_data. If you are unsure of what amount of memory is appropriate for your interactive session, you could add -l exclusive to your qrsh command. In that case, the h_data value is used by the scheduler to select a compute node having a total amount of memory equal or greater than what speciﬁed with h_data. In this case, the memory limit for the job is the compute node’s physical memory size. For example, the command:

qrsh-l h_rt=8:00:00,h_data=32G,exclusive

will start an interactive session on a compute node equipped with at least 32G of physical memory.

Note: You can only request as much memory as is available on nodes on the cluster. Interactive session requested via qrsh without specifying an h_data value are automatically assigned an h_data=1G, which may or may not be too small for your application.

4.1.2 Resource limitation

You can specify up to 24 hours in your qrsh request using:

qrsh-l h_rt=24:00:00

Hoffman2 cluster’s compute nodes have different memory sizes. When you request more then one core (using: -pe shared ), the total memory requested on the node will be the product of the number of cores time the memory per core (h_data). In general, the larger the total memory requested, the longer the wait. Please refer to the output of the command:

qhost

to see what total memory is available on the various nodes on the cluster, keeping in mind that not all hosts may be accessible to you. When you request multiple cores, or a large amount of total memory, you may or may not get the interactive session immediately, depending on how busy the cluster is and the permission level of your account. To see to which class of nodes (memory, number of cores, etc.) you have access to, you can enter the following at the Hoffman2 command prompt:

mygroups

40 Chapter 4. Rendering H2-for-the-Arts Documentation, Release 2020

4.1.3 What is a node?

A node is a physical server comprised of mutliples cores. Hoffman2 has a number of nodes available to the entire UCLA community. Additionally, research groups can purchase dedicated compute nodes. Group owned nodes, allow users a runtime up to fourteen days, and jobs from the group are garanteed to start within twenty-four hours from submission (and most typically less). For more information on running jobs on owned please see: Job scheduling policy. If your group is interested in purchasing nodes please visit: Purchasing nodes

4.1.4 Running on your group’s nodes

The following section does not apply to you if your research group has not purchased Hoffman2 compute nodes To run on your group nodes, add the -l highp switch to your qrsh command. For example, to request an interacrive session with a duration of two days (48 hours), 4GB of memory (and one core), issue the command: qrsh-l highp,h_rt=48:00:00,h_data=4G

You could also request multiple cores using the -pe dc\* or -pe shared as described above. When combining with -l highp, the amount of cores or memory size need to be compatible with your group compute nodes. Contact user support if you are not sure. Although you are allowed to specify h_rt as high as 336 hours (14 days) for an qrsh session, it is not recommended. For example, if the network connection is interrupted (e.g. your laptop or desktop computer goes into sleep mode), the qrsh session may be lost, possibly terminating all running programs within it.

4.1.5 qrsh examples

Note: The parameters associated with the -l directive are separated by commas without any white space in between.

Request a single processor for 2 hours. The default memory size depends on the queue in which your session starts. For campus users, the default memory size is 1GB. Use the -l directive with the h_data parameter to request more memory. To request a single processor for 24 hours from the interactive queues, issue the command: qrsh-l h_rt=24:00:00,h_data=1G

To request 8 processors for 4 hours (total 8*1G=8GB memory) on a single node from the interactive queues, issue the command: qrsh-l h_data=1G,h_rt=4:00:00,h_data=1G-pe shared8

To request 4 processors for 3 hours (total 4*1G=4GB memory) on a single node, issue the command: qrsh-l h_data=1G,h_rt=3:00:00,h_data=1G-pe shared4

To request 12 processors, 1GB of memory per processor, for 2 hours, issue the command: qrsh-l h_data=1G,h_rt=2:00:00-pe dc\ * 12

4.1. Getting an interactive-session 41 H2-for-the-Arts Documentation, Release 2020

Note: The 12 CPUs are distributed across multiple compute nodes. The backslash \ in dc\* is signiﬁcant when you issue this command in an interactive unix shell.

4.1.6 qrsh startup time

A qrsh session is scheduled along with all other jobs managed by the scheduler software. The shorter time (the -l h_rt option), and the fewer number of processors (the -pe option), the better chance you have of getting a session. Request just what you need for the best use of computing resources. Be considerate to other users by exiting your qrsh session when you are done to release the computing resources to other users.

4.1.7 Interpreting error messages

Occasionally, you may encounter one of the following messages: error: no suitable queues or, qrsh: No match. If you see the no suitable queues message and you are requesting the interactive queues, be sure you have not requested more than 24 hours. This message may mean there is something incompatible with the various parameters you have speciﬁed and your qrsh session can never start. For example, if you have requested -l h_rt=25:00:00 but your userid is not authorized to run sessions or jobs for more than 24 hours. If your session could not be scheduled, ﬁrst try your qrsh command again in case it was a momentary problem with the scheduler. If your session still cannot be scheduled, try lowering either the value of h_rt, the number of processors requested, or both, if possible.

4.1.8 Running MPI with qrsh

The following instructions are speciﬁc to IntelMPI library. They may not apply to other MPI implementations (see: How to run MPI for more information). There are 2 main steps to run an MPI program in a qrsh session. You need to do step #1 only once per qrsh session. You can repeatedly execute step #2 within the same qrsh session. The executable MPI program is named foo in the following example. Set up the environment In the qrsh session at the shell prompt, enter one of the following commands: If you are in bash shell: source/u/local/bin/set_qrsh_env.sh

If you are in csh/tcsh shell: source/u/local/bin/set_qrsh_env.csh

Launch your MPI program Assume your MPI program is named foo and is located in the current directory. Run the program using all allocated processors with the command:

42 Chapter 4. Rendering H2-for-the-Arts Documentation, Release 2020

mpiexec.hydra -n $NSLOTS ./foo

You could replace $NSLOTS with an integer, which is less than the number of processors you requested on your qrsh command. For example:

mpiexec.hydra-n4./foo

The command to see mpiexec options is:

mpiexec.hydra-help

You do not have to create a hostﬁle and pass it to mpiexec.hydra with its -machinefile or -hostfile option because mpiexec.hydra automatically retrieves that information from UGE.

4.1.9 Additional tools

Additional scripts are available that may help you run other parallel distributed memory software. You can enter these commands at the compute node’s shell prompt.

get_pe_hostfile

Returns the contents of the UGE pe_hostfile ﬁle for the current qrsh session. If you have used the -pe directive to request multiple processors on multiple nodes, you will probably need to tell your program the names of those nodes and how many processors have been allocated on each node. This information is unique to your current qrsh session. To create an MPI-style hostﬁle named hfile in the current directory:

get_pe_hostfile| awk'{print $1" slots="$2}'> hfile

The UGE pe_hostfile is located:

$SGE_ROOT/$SGE_CELL/spool/node/active_jobs/sge_jobid.1/pe_hostfile

or,

$SGE_ROOT/$SGE_CELL/spool/node/active_jobs/sge_jobid.sge_taskid/pe_hostfile

where node and sge_jobid are the hostname and UGE $JOB_ID, respectively, of the current qrsh session. sge_taskid is the task number of an array job $SGE_TASK_ID. To return the value of UGE JOB_ID for the current qrsh session, issue the command:

get_sge_jobid

To return the contents of the UGE environment ﬁle for the current qrsh session, issue:

get_sge_env

which is used by the set_qrsh_env scripts. UGE-specific environment variables are defined in the file:

$SGE_ROOT/$SGE_CELL/spool/node/active_jobs/sge_jobid.1/environment

or,

4.1. Getting an interactive-session 43 H2-for-the-Arts Documentation, Release 2020

$SGE_ROOT/$SGE_CELL/spool/node/active_jobs/sge_jobid.sge_taskid/environment where node and sge_jobid are the hostname and UGE $JOB_ID, respectively, of the current qrsh session. sge_taskid is the task number of a array job $SGE_TASK_ID.

4.2 Submitting batch jobs

4.2.1 Submit batch jobs from the cluster nodes

In order to run a job under the Univa Grid Engine, you need to supply UGE directives and their arguements to the job scheduler. The easiest way to do this is to create a UGE command file that consists of a set of UGE directives along with the commands required to execute the actual job. The command file for submitting a job can either be built using queue scripts provided by IDRE, or by building an UGE command file yourself.

4.2.2 Queue scripts

Each IDRE-provided queue script is named for a type of job or application. The queue script builds a UGE command file for that particular type of job or application. A queue script can be run either as a single command to which you provide appropriate options, or as an interactive application, which presents you with a menu of choices and prompts you for the values of options. For example, if you simply enter a queue script command such as: job.q without any command-line arguments, the queue script will enter its interactive mode and present you with a menu of tasks you can perform. One of these tasks is to build the command file, another is to submit a command file that has already been built, and another is to show the status of jobs you have already submitted. See queue scripts for details, or select Info from any queue script menu, or enter man queue at a shell prompt. You can also enter myjobs at the shell prompt to show the status of jobs you have submitted and which have not already completed. You can also enter groupjobs at the shell prompt to show the status of pending jobs everyone in your group has submitted. Enter groupjobs -help for options. IDRE-provided queue scripts can be used to run the following types of jobs: • Serial Jobs • Serial Array Jobs • Multi-threaded Jobs • MPI Parallel Jobs • Application Jobs

4.2.3 Serial jobs

A serial job runs on a single thread on a single node. It does not take advantage of multi-processor nodes or the multiple compute nodes available with a cluster. To build or submit an UGE command ﬁle for a serial job, you can either enter: job.q [queue-script-options]

44 Chapter 4. Rendering H2-for-the-Arts Documentation, Release 2020

or, you can provide the name of your executable on the command line:

job.q [queue-script-options] name_of_executable [executable-arguments]

When you enter job.q without the name of your executable, it will interactively ask you to enter any needed memory, wall-clock time limit, and other options, and ask you if you want to submit the job. You can quit out of the queue script menu and edit the UGE command file, which the script built, if you want to change or add other Univa Grid Engine options before you submit your job. If you did not submit the command file at the end of the menu dialog and decided to edit the file before submitting it, you can submit your command file using either a queue script Submit menu item, or the qsub command:

qsub executable.cmd

When you enter job.q with the name of your executable, it will by default build the command ﬁle using defaults for any queue script options that you did not specify, submit it to the job scheduler, and delete the command ﬁle that it built.

4.2.4 Serial array jobs

Array jobs are serial jobs or multi-threaded jobs that use the same executable but different input variables or input files, as in parametric studies. Users typically run thousands of jobs with one submission. The UGE command file for a serial array job will, at the minimum, contain the UGE keyword statement for a lower index value and an upper index value. By default, the index interval is one. UGE keeps track of the jobs using the environment variable SGE_TASK_ID, which varies from the lower index value to the upper index value for each job. Your program can use SGE_TASK_ID to select the input files to read or the options to be used for that particular run. If your program is multi-threaded, you must edit the UGE command file built by the jobarray.q script and add an UGE keyword statement that specifies the shared parallel environment and the number of processors your job requires. In most cases you should request no more than 8 processors because the maximum number of processors on most nodes is 8. See the For a multi-threaded OpenMP job section for more infromation. To build or submit an UGE command file for a serial array job, enter:

jobarray.q

For details, see the section Running an Array of Jobs Using UGE.

4.2.5 Multi-threaded jobs

Multi-threaded jobs are jobs which will run on more than one thread on the same node. Programs using the OpenMP- based threaded library are a typical example of those that can take advantage of multi-core nodes. If you know your program is multi-threaded, you need to request that UGE allocate multiple processors. Otherwise your job will contend for resources with other jobs that are running on the same node, and all jobs on that node may be adversely affected. The queue script will prompt you to enter the number of tasks for your job. The queue script default is 4 tasks. You should request at least as many tasks as your program has threads, but usually no more than 8 tasks because the maximum number of processors on most nodes is 8. See the Scalability Benchmark section for information on how to determine the optimal number of tasks. To build or submit an UGE command ﬁle for a multi-threaded job, enter:

openmp.q

For details, see OpenMP programs and Multi threaded programs.

4.2. Submitting batch jobs 45 H2-for-the-Arts Documentation, Release 2020

4.2.6 MPI parallel jobs

MPI parallel jobs are those executable programs that are linked with one of the message passing libraries like Open- MPI. These applications explictly send messages from one node to another using either a Gigabit Ethernet (GE) interface or Infiniband (IB) interface. IDRE recommends that everyone use the Infiniband interface because latency for message passing is short with the IB interface compared to the GE interface. When MPI jobs are submitted to the cluster, one needs to tell the UGE scheduler how many processors are needed to run the jobs. The queue script will prompt you to enter the number of tasks for your job. The queue script default for generic jobs is 4 parallel tasks. Please see the Scalability Benchmark below for information on how to determine the optimal number of tasks. To build or submit an UGE command file for a parallel job, enter: mpi.q

For details, see the How to Run MPI section.

4.2.7 Application jobs

An application job is one which runs software provided by a commercial vendor or is open source. It is usually installed in system directories (e.g., MATLAB). To build or submit an UGE command ﬁle for an application job, enter: application.q where application is replaced with the name of the application. For example, use matlab.q to run MATLAB batch jobs. For details, see Software and its subsequent links for each package or program on to how to run them.

4.2.8 Batch job output ﬁles

When a job has completed, UGE messages will be available in the stdout and stderr files that were were defined in your UGE command file with the -o and -e or -j keywords. Program output will be available in any files that your program has written. If your UGE command file was built using a queue script, stdout and stderr from UGE will be found in one of: jobname.joblog jobname.joblog.$JOB_ID jobname.joblog.$JOB_ID.$SGE_TASK_ID (for array jobs)

Output from your program will be found in one of: jobname.out jobname.out.$JOB_ID jobname.output.$JOB_ID jobname.output.$JOB_ID.$SGE_TASK_ID (for array jobs)

4.2.9 Build a UGE command ﬁle for your job and use UGE commands directly

This section describes building an UGE command ﬁle yourself, instead of letting a queue script build it for you. Or you may modify an UGE command ﬁle that a queue script has built, according to the information presented here.

46 Chapter 4. Rendering H2-for-the-Arts Documentation, Release 2020

The UGE keyword statements in a command file are called active comments because they begin with #$ and comments in a script file normally begin with #. Any qsub command line option can be used in the command file as an active comment. The qsub command line options are listed on the submit man page Each UGE keyword statement begins with #$ followed by the UGE keyword and its value, if any. For example:

#$ -cwd #$ -o jobname.joblog #$ -j y where jobname is the name of your job. Here the first UGE statement #$ -cwd specifies that the current working directory is to be used for the job. The second UGE statement #$ -o jobname.joblog names the output file in which the UGE command file will write its standard out messages. The third #$ -j y specifies that any messages that UGE may write to standard error are to be merged with those it writes to standard out. After you have created the UGE command file, issue the appropriate UGE commands from a login node to submit and monitor the job. See Commonly-Used UGE Commands

4.2.10 Using job arrays

For job arrays you need to use an UGE keyword statement of the form:

#$ -t lower-upper:interval

Please see Running an Array of Jobs Using UGE for more information.

4.2.11 For a parallel MPI job

For a parallel MPI job you need to have a line that speciﬁes a parallel environment:

#$ -pe dc* number_of_slots_requested

The maximum number_of_slots_requested value that you should use depends on your account’s access level.

4.2.12 For a multi-threaded/OpenMP job

For a multi-threaded OpenMP job you need to request that all processors be on the same node by using the shared parallel environment.

#$ -pe shared number_of_slots_requested where the maximum number_of_slots_requested no larger than the number of CPU/cores of a compute node.

4.2. Submitting batch jobs 47 H2-for-the-Arts Documentation, Release 2020

4.2.13 Parallel environments (PE)

For Threaded Programs (e.g. OpenMP) PE Description shared p p processors on a single node For MPI Programs PE Description dc* p p processors across multiple nodes. The * is signiﬁcant. signiﬁcant. There is no space between dc and *. There is space between * and the value of p. Requesting whole nodes PE Description node* n n nodes (normally used with -l exclusive)

4.2.14 How to reserve an entire node

One or more whole nodes for parallel jobs To get one or more entire nodes for parallel jobs, use -pe node* n -l exclusive in your qsub or qrsh command or add them to your UGE command ﬁle, where n is the number of nodes you are requesting. Example of requesting 2 whole nodes with qsub: qsub-pe node2-l exclusive [other options]

Example of requesting 3 whole nodes in a UGE command ﬁle:

#$ -l exclusive #$ -pe node 3

Submit a batch job from the UCLA Grid Portal

Warning: UCLA Grid Portal will be taken down in near future.

The UCLA Grid Portal provides a web portal interface to the Hoffman2 Cluster. Every Hoffman2 Cluster user can access the UCLA Grid Portal. To submit a batch job from the UCLA Grid Portal, click the Job Services tab then click one of the following: Generic Jobs, Applications, or Multi-Jobs. Generic Jobs: Use this page to submit a job that runs a program or script that either you or a colleague have written and is usually installed in your home directory. In the fill-in form provided, supply the name of the executable, any job parameters, time request, number of processors. Applications: Use this page to submit a commonly used application. Normally, you are required to know less about an application than a generic job, as the UCLA Grid Portal keeps track of the location of the executable and other information about the application. You normally must prepare an input file that the application will read or run. Some applications can present forms to you on the UCLA Grid Portal that you can fill in to create the input file if you are not familiar with application requirements. Multi-Jobs: Use this page to submit multiple jobs that run a program or script that either you or a colleague have written. For details, see Running an Array of Jobs Using UGE. After you submit a job, click Job Status where you can monitor its progress, and view and download its output after your job completes.

48 Chapter 4. Rendering H2-for-the-Arts Documentation, Release 2020

4.3 GPU-access

4.3.1 How to access GPU nodes

In order to use a node that has a GPU, you need to request it from the job scheduler. Nodes may have two GPUs (Tesla T10) or three GPUs (Tesla M2070 nodes). To begin an interactive session, at the shell prompt, enter:

qrsh-l gpu

On Hoffman2 there are currently four publicly available GPU nodes cuda capability of 6.1 (all other publicly available nodes have cuda capability less than 3), each of this node is equipped with one P4 card. To request one of these nodes, please use:

qrsh-l gpu,P4

The above qrsh command will reserve an entire gpu node with its 2 or 3 gpu processors. An interactive session made with the above ‘‘qrsh ‘‘command will expire in 2 hours. To specify a different time limit for your session, use the h_rt or time parameter. Example for requesting 9 hours:

qrsh-l gpu,h_rt=9:00:00

To reserve two gpu-nodes use:

qrsh-l gpu-pe dc_gpu2

To see which node(s) were reserved, at a gpu-node shell prompt enter:

get_pe_hostfile

To see if the gpu nodes are up and/or in use, at any shell prompt enter:

qhost_gpu_nodes

To see the speciﬁcs for a particular gpu node, at a g-node shell prompt enter:

gpu-device-query.sh

To get a quick session for compiling or testing your code:

qrsh-l i,gpu

4.3.2 How to specify GPU types

There are multiple GPU types available in the cluster. Each type of GPU has a different compute capability, memory size and clock speed, among other things. If your GPU program requires a speciﬁc GPU type to run, you need to specify it explicitly. Without specifying GPU type allows UGE to arbitrarily pick any available GPU for your job. You may need to compile your code on the machine that has the required type of GPU. Currently, the following GPU types are available:

4.3. GPU-access 49 H2-for-the-Arts Documentation, Release 2020

GPU type Compute Capability Number of cores Global Memory Size UGE option Tesla V100 7.0 5120 32 GB -l gpu,V100 Tesla P4 6.1 2560 8 GB -l gpu,P4 Tesla T10 1.3 240 4.3 GB -l gpu,T10 Tesla M2070 2.0 448 5.6 GB -l gpu,fermi Tesla M2090 2.0 512 6 GB -l gpu,fermi

References: http://developer.nvidia.com/cuda-gpus http://www.nvidia.com/object/preconﬁgured-clusters.html The UGE options in the table above can be combined with other UGE options, for example: qrsh-l gpu,fermi,h_rt=3:00:00

[1] If you specify -l fermi the job will go to either M2070 or M2090 GPU nodes. If you specify -l M2070 the job will only go to M2070 and will not go to M2090 even when the later is available. If you specify -l M2090 the job will only go to M2090 and will not go to M2070 even when the later is available. This implies potentially longer wait time. For most users, we recommend using -l fermi instead of -l M2070 or -l M2090 unless you speciﬁcally want to use either one of them (e.g., benchmarking the differences between M2070 and M2090)

4.3.3 CUDA

CUDA is installed in /u/local/cuda/ on the Hoffman2 Cluster. There are several versions available. To see which versions of cuda are available please issue: module av cuda to load version 10.), use: module load cuda/10.0

Note: You will be able to load a cuda module only when on a GPU node (either in an interactive session, requested with: qrsh -l gpu, or within batch jobs in which you have requested one or more GPU nodes with: -l gpu).

Already compiled samples of NVIDIA GPU Computing Software Development Kit are available, for example for cuda 10.0, in:

/u/local/cuda/10.0/NVIDIA_CUDA-10.0_Samples/bin/x86_64/linux/releasemodule load cuda/

˓→10.0

To install CUDA in your home directory, please see the instructions in the /u/local/cuda/README_ATS ﬁle. To install the NVIDIA GPU Computing Software Development Kit in your home directory, please see the instructions in the

50 Chapter 4. Rendering CHAPTER 5

Indices and tables

• genindex • modindex • search