COMPSs Manual Workflows and Distributed Computing Group Last updated : June, 2020 Online version available at COMPSs - ReadTheDocs Table of contents Table of contents i List of figures v List of tables vii 1 What is COMPSs? 3 2 Quickstart 5 2.1 Install COMPSs..............................................5 2.2 Write your first app............................................8 2.3 Useful information............................................. 19 3 Installation and Administration 21 3.1 Dependencies................................................ 21 3.1.1 Build Dependencies........................................ 25 3.1.2 Optional Dependencies...................................... 25 3.2 Building from sources........................................... 25 3.2.1 Post installation.......................................... 27 3.3 Pip..................................................... 27 3.3.1 Pre-requisites........................................... 27 3.3.2 Installation............................................ 27 3.3.3 Post installation.......................................... 28 3.4 Supercomputers.............................................. 28 3.4.1 Prerequisites............................................ 28 3.4.2 Installation............................................ 29 3.4.3 Configuration........................................... 29 3.4.4 Post installation.......................................... 32 3.5 Additional Configuration......................................... 37 3.5.1 Configure SSH passwordless................................... 37 3.5.2 Configure the COMPSs Cloud Connectors........................... 38 3.6 Configuration Files............................................ 39 3.6.1 Resources file........................................... 39 3.6.2 Project file............................................. 40 3.6.3 Configuration examples..................................... 41 4 Application development 51 4.1 Java..................................................... 51 4.1.1 Programming Model....................................... 51 4.1.2 Application Compilation..................................... 58 4.1.3 Application Execution...................................... 59 4.2 Python Binding.............................................. 59 4.2.1 Programming Model....................................... 59 i 4.2.2 Application Execution...................................... 79 4.2.3 Integration with Jupyter notebook............................... 80 4.2.4 Integration with Numba..................................... 82 4.3 C/C++ Binding.............................................. 84 4.3.1 Programming Model....................................... 84 4.3.2 Use of programming models inside tasks............................ 89 4.3.3 Application Compilation..................................... 90 4.3.4 Application Execution...................................... 94 4.3.5 Task Dependency Graph..................................... 94 4.4 Constraints................................................. 94 5 Execution Environments 99 5.1 Local.................................................... 99 5.1.1 Executing COMPSs applications................................ 99 5.1.2 Results and logs.......................................... 105 5.1.3 COMPSs Tools.......................................... 107 5.2 Supercomputers.............................................. 115 5.2.1 Common usage.......................................... 115 5.2.2 MareNostrum 4.......................................... 121 5.2.3 MinoTauro............................................. 123 5.2.4 Nord 3............................................... 124 5.2.5 Enabling COMPSs Monitor................................... 125 5.3 Docker................................................... 127 5.3.1 What is Docker?......................................... 127 5.3.2 Requirements........................................... 127 5.3.3 Execution in Docker....................................... 127 5.3.4 Execution with TLS....................................... 129 5.3.5 Execution results......................................... 129 5.3.6 Execution examples........................................ 130 5.4 Chameleon................................................. 131 5.4.1 What is Chameleon?....................................... 131 5.4.2 Execution in Chameleon..................................... 131 5.5 Dynamic infrastructures.......................................... 132 5.5.1 What are COMPSs Agents?................................... 132 5.5.2 Deploying a COMPSs Agent................................... 132 5.5.3 Executing an operation...................................... 135 5.5.4 Modifying the available resources................................ 136 6 Tracing 139 6.1 COMPSs applications tracing....................................... 139 6.1.1 Basic Mode............................................ 139 6.1.2 Advanced Mode.......................................... 142 6.1.3 Custom Installation and Configuration............................. 145 6.2 Visualization................................................ 145 6.2.1 Trace Loading........................................... 145 6.2.2 Configurations........................................... 145 6.2.3 View Adjustment......................................... 147 6.3 Interpretation............................................... 148 6.4 Analysis................................................... 148 6.4.1 Graphical Analysis........................................ 148 6.4.2 Numerical Analysis........................................ 150 6.5 PAPI: Hardware Counters........................................ 151 6.6 Paraver: configurations.......................................... 152 6.7 User Events in Python.......................................... 153 6.7.1 Events in main code....................................... 154 6.7.2 Events in task code........................................ 154 6.7.3 Result trace............................................ 155 6.7.4 Practical example......................................... 155 7 Persistent Storage 157 7.1 First steps................................................. 157 7.1.1 Defining the data model..................................... 158 7.1.2 Interacting with the persistent storage............................. 161 7.1.3 Running with persistent storage................................. 162 7.2 COMPSs + dataClay........................................... 163 7.2.1 COMPSs + dataClay Dependencies............................... 163 7.2.2 Enabling COMPSs applications with dataClay......................... 163 7.2.3 Executing a COMPSs application with dataClay....................... 164 7.3 COMPSs + Hecuba............................................ 164 7.3.1 COMPSs + Hecuba Dependencies............................... 164 7.3.2 Enabling COMPSs applications with Hecuba......................... 164 7.3.3 Executing a COMPSs application with Hecuba........................ 164 7.4 COMPSs + Redis............................................. 164 7.4.1 COMPSs + Redis Dependencies................................. 165 7.4.2 Enabling COMPSs applications with Redis.......................... 166 7.4.3 Executing a COMPSs application with Redis......................... 168 7.5 Implement your own Storage interface for COMPSs.......................... 169 7.5.1 Generic Storage Object Interface................................ 170 7.5.2 Generic Storage Runtime Interfaces............................... 170 7.5.3 Storage Interface usage...................................... 174 8 Sample Applications 175 8.1 Java Sample applications......................................... 175 8.1.1 Hello World............................................ 175 8.1.2 Simple............................................... 177 8.1.3 Increment............................................. 178 8.1.4 Matrix multiplication....................................... 181 8.1.5 Sparse LU decomposition.................................... 182 8.1.6 BLAST Workflow......................................... 184 8.2 Python Sample applications....................................... 185 8.2.1 Simple............................................... 185 8.2.2 Increment............................................. 187 8.2.3 Kmeans.............................................. 188 8.2.4 Kmeans with Persistent Storage................................. 194 8.2.5 Matmul.............................................. 201 8.2.6 Lysozyme in water........................................ 204 8.3 C/C++ Sample applications....................................... 211 8.3.1 Simple............................................... 212 8.3.2 Increment............................................. 214 9 PyCOMPSs Player 221 9.1 Requirements and Installation...................................... 221 9.1.1 Requirements........................................... 221 9.1.2 Installation............................................ 221 9.2 Usage.................................................... 222 9.2.1 Start COMPSs infrastructure in your development directory................. 222 9.2.2 Running applications....................................... 223 9.2.3 Running the COMPSs monitor................................. 223 9.2.4 Running Jupyter notebooks................................... 224 9.2.5 Generating the task graph.................................... 224 9.2.6 Tracing applications or notebooks................................ 224 9.2.7 Adding more nodes.......................................