VSC documentation Documentation Release 1.0 VSC (Vlaams Supercomputing Center) Sep 03, 2021 Contents 1 Getting access 1 1.1 Required steps to get access.......................................1 1.2 VSC accounts..............................................1 1.3 How to request an account?.......................................1 1.4 Next steps................................................4 1.5 Additional information..........................................4 2 Access and data transfer 7 2.1 Logging in to a cluster..........................................7 2.2 Data storage............................................... 13 2.3 Transferring data............................................. 16 2.4 GUI applications on the clusters..................................... 24 2.5 VPN................................................... 24 3 Software stack 25 3.1 Using the module system......................................... 25 4 Running jobs 31 4.1 Job script................................................. 31 4.2 Submitting and monitoring a job..................................... 32 4.3 Job output................................................ 37 4.4 Troubleshooting............................................. 38 4.5 Advanced topics............................................. 40 5 Software development 43 5.1 Programming paradigms......................................... 43 5.2 Development tools............................................ 50 5.3 Libraries................................................. 72 5.4 Integrating code with software packages................................. 82 6 VSC hardware 87 6.1 Tier-2 hardware............................................. 87 6.2 Tier-1 hardware............................................. 103 7 Globus file and data sharing platform 107 7.1 What is Globus.............................................. 107 7.2 Access.................................................. 107 i 7.3 Managing and transferring files..................................... 109 7.4 Local endpoints............................................. 112 7.5 Data sharing............................................... 113 7.6 Manage Globus groups.......................................... 114 7.7 Command Line Interface (CLI)..................................... 116 7.8 Glossary................................................. 117 8 Frequently asked questions (FAQs) 119 8.1 General questions............................................ 119 8.2 Access to the infrastructure....................................... 122 8.3 Running jobs............................................... 123 8.4 Software................................................. 129 Index 137 ii CHAPTER 1 Getting access 1.1 Required steps to get access New users of the VSC clusters should take the following three steps to get access: 1. create a public/private key pair 2. apply for a VSC account 3. login to the cluster 1.2 VSC accounts In order to use the infrastructure of the VSC, you need a VSC-user ID, also called a VSC account. Check the VSC website for conditions. All VSC-accounts start with the letters "vsc” followed by a five-digit number. The first digit gives information about your home institution. There is no relationship with your name, nor is the information about the link between VSC- accounts and your name publicly accessible. Your VSC account gives you access to most of the VSC Tier-2 infrastructure, though for some more specialized hardware you may have to ask access separately. The rationale is that we want to ensure that that specialized (usually rather expensive) hardware is used efficiently for the type of applications it was purchased for. Contact your local VSC coordinator to arrange access when required. For the main Tier-1 compute cluster you need to submit a project application (or you should be covered by a project application within your research group). 1.3 How to request an account? Unlike your institute account, VSC accounts don’t use regular fixed passwords but a key pair consisting of a public an private key because that is a more secure authentication technique. To apply for a VSC account, you need a 1 VSC documentation Documentation, Release 1.0 public/private key pair. 1.3.1 Create a public/private key pair A key pair consists of a private and a public key. 1. The private key is stored on the computer(s) you use to access the VSC infrastructure and always stays there. 2. The public key is stored on the VSC systems you want to access, allowing to prove your identity through the corresponding private key. How to generate such a key pair depends on your operating system. We describe the generation of key pairs in the client sections for • Linux, • Windows and • macOS (formerly OS X). Without a key pair, you won’t be able to apply for a VSC account. Warning: It is clear from the above that it is very important to protect your private key well. Therefore: • You should choose a strong passphrase to protect your private key. • You should not share your key pair with other users. • If you have accounts at multiple supercomputer centers (or on other systems that use SSH), you should seriously consider using a different key pair for each of those accounts. In that way, if a key would get compromised, the damage can be controlled. • For added security, you may also consider to use a different key pair for each computer you use to access your VSC account. If your computer is stolen, it is then easy to disable access from that computer while you can still access your VSC account from all your other computers. The procedure is explained on a separate web page “access from multiple machines”. Your VSC account is currently managed through your institute account. 1.3.2 Applying for the account Once you have a valid public/private key pair, you can submit an account request. The following algorithm guides you to the appropriate approach. Users from the KU Leuven and UHasselt association UHasselt has an agreement with KU Leuven to run a shared infrastructure. Therefore the procedure is the same for both institutions. Who? Access is available for faculty, students (under faculty supervision), and researchers of the KU Leuven, UHasselt and their associations. How? • Researchers with a regular personnel account (u-number) can use the generic procedure. 2 Chapter 1. Getting access VSC documentation Documentation, Release 1.0 • If you are in one of the higher education institutions associated with KU Leuven, the generic procedure may not work. In that case, please e-mail [email protected] to get an account. You will have to provide a public ssh key generated as described above. • Lecturers of KU Leuven and UHasselt that need HPC access for giving their courses: The procedure requires action both from the lecturers and from the students. Lecturers should follow the specific procedure for lecturers, while the students should simply apply for the account through the generic procedure. Users of Ghent University Association All information about the access policy is available in English at the UGent HPC web pages. • Researchers can use the generic procedure. • Master students can also use the infrastructure for their master thesis work. The promotor of the thesis should first send a motivation to [email protected] and then the generic procedure should be followed (using your student UGent id) to request the account. Users of the Antwerp University Association (AUHA) Who? Access is available for faculty, students (master’s projects under faculty supervision), and researchers of the AUHA. How? • Researchers can use the generic procedure. • Master students can also use the infrastructure for their master thesis work. The promotor of the thesis should first send a motivation to [email protected] and then the generic procedure should be followed (using your student UAntwerpen id) to request the account. Users of Brussels University Association All information about the access policy is available on the VUB HPC documentation website. • Researchers can use the generic procedure. • Master students can also use the infrastructure for their master thesis work. The promotor of the thesis should first send a motivation to [email protected] and then the generic procedure should be followed to request the account. Everyone else Who? Check that you are eligible to use VSC infrastructure. How? Ask your VSC contact for help. If you don’t have a VSC contact yet, and please get in touch with us. 1.3. How to request an account? 3 VSC documentation Documentation, Release 1.0 Generic procedure for academic researchers For most researchers from the Flemish universities, the procedure has been fully automated and works by using your institute account to request a VSC account. Check below for exceptions or if the generic procedure does not work. 1. Open the VSC account page. 2. Select your “home” institution from the drop-down menu and click the “confirm” button. 3. Log in using your institution login and password. 4. You will be asked to upload the public key you created earlier. 5. You will get an e-mail to confirm your application, click the included link to do so. 6. After the account has been approved by the VSC, your account will be created and you will get a confirmation e-mail. Warning: Allow for at least half an hour for your account to be properly created after receiving the confirmation email! Note: If you can’t connect to the VSC account page , some browser extensions have caused problems (and in particular some security-related extensions), so you might try with browser extensions disabled. 1.4