COMPSs Manual Workflows and Distributed Computing Group Last updated : July, 2021 Online version available at COMPSs - ReadTheDocs Table of contents Table of contents i List of figures v List of tables vii 1 What is COMPSs? 3 1.1 More information:.............................................4 2 Quickstart 5 2.1 Install COMPSs..............................................5 2.2 Write your first app............................................8 2.3 Useful information............................................. 20 3 Installation and Administration 23 3.1 Dependencies................................................ 23 3.1.1 Build Dependencies........................................ 28 3.1.2 Optional Dependencies....................................... 28 3.2 Building from sources........................................... 28 3.2.1 Post installation.......................................... 30 3.3 Pip...................................................... 30 3.3.1 Pre-requisites............................................ 30 3.3.2 Installation............................................. 30 3.3.3 Post installation.......................................... 31 3.4 Supercomputers............................................... 31 3.4.1 Prerequisites............................................ 31 3.4.2 Installation............................................. 31 3.4.3 Configuration............................................ 32 3.4.4 Post installation.......................................... 36 3.5 Additional Configuration.......................................... 43 3.5.1 Configure SSH passwordless.................................... 43 3.5.2 Configure the COMPSs Cloud Connectors............................ 44 3.6 Configuration Files............................................. 45 3.6.1 Resources file............................................ 45 3.6.2 Project file............................................. 46 3.6.3 Configuration examples...................................... 47 4 Application development 57 4.1 Java..................................................... 57 4.1.1 Programming Model........................................ 57 4.1.2 Application Compilation...................................... 65 4.1.3 Application Execution....................................... 66 4.2 Python Binding............................................... 67 i 4.2.1 Programming Model........................................ 67 4.2.2 Application Execution....................................... 100 4.2.3 Integration with Jupyter notebook................................ 101 4.2.4 Integration with Numba...................................... 103 4.3 C/C++ Binding.............................................. 105 4.3.1 Programming Model........................................ 105 4.3.2 Use of programming models inside tasks............................. 111 4.3.3 Application Compilation...................................... 112 4.3.4 Application Execution....................................... 116 4.3.5 Task Dependency Graph...................................... 116 4.4 Constraints................................................. 117 5 Execution Environments 121 5.1 Master-Worker Deployments........................................ 121 5.1.1 Local................................................ 121 5.1.2 Supercomputers.......................................... 139 5.1.3 Docker................................................ 154 5.1.4 Chameleon............................................. 158 5.1.5 Jupyter Notebook......................................... 159 5.2 Agents Deployments............................................ 162 5.2.1 Local................................................ 164 5.2.2 Supercomputers.......................................... 171 5.3 Schedulers.................................................. 172 6 Tracing 173 6.1 COMPSs applications tracing....................................... 173 6.1.1 Basic Mode............................................. 174 6.1.2 Advanced Mode.......................................... 179 6.1.3 Trace for Agents.......................................... 182 6.1.4 Custom Installation and Configuration.............................. 184 6.2 Visualization................................................ 184 6.2.1 Trace Loading........................................... 184 6.2.2 Configurations........................................... 185 6.2.3 View Adjustment.......................................... 186 6.3 Interpretation................................................ 188 6.4 Analysis................................................... 188 6.4.1 Graphical Analysis......................................... 188 6.4.2 Numerical Analysis......................................... 190 6.5 PAPI: Hardware Counters......................................... 194 6.6 Paraver: configurations........................................... 195 6.7 User Events in Python........................................... 196 6.7.1 Events in main code........................................ 197 6.7.2 Events in task code......................................... 197 6.7.3 Result trace............................................. 198 6.7.4 Practical example......................................... 198 7 Persistent Storage 201 7.1 First steps.................................................. 201 7.1.1 Defining the data model...................................... 202 7.1.2 Interacting with the persistent storage.............................. 206 7.1.3 Running with persistent storage................................. 207 7.2 COMPSs + dataClay........................................... 208 7.2.1 COMPSs + dataClay Dependencies............................... 208 7.2.2 Enabling COMPSs applications with dataClay......................... 208 7.2.3 Executing a COMPSs application with dataClay........................ 208 7.3 COMPSs + Hecuba............................................ 208 7.3.1 COMPSs + Hecuba Dependencies................................ 208 7.3.2 Enabling COMPSs applications with Hecuba.......................... 208 7.3.3 Executing a COMPSs application with Hecuba......................... 209 7.4 COMPSs + Redis............................................. 209 7.4.1 COMPSs + Redis Dependencies................................. 209 7.4.2 Enabling COMPSs applications with Redis........................... 210 7.4.3 Executing a COMPSs application with Redis.......................... 213 7.5 Implement your own Storage interface for COMPSs........................... 214 7.5.1 Generic Storage Object Interface................................. 214 7.5.2 Generic Storage Runtime Interfaces............................... 215 7.5.3 Storage Interface usage...................................... 219 8 Sample Applications 221 8.1 Java Sample applications......................................... 221 8.1.1 Hello World............................................. 221 8.1.2 Simple................................................ 223 8.1.3 Increment.............................................. 225 8.1.4 Matrix multiplication....................................... 226 8.1.5 Sparse LU decomposition..................................... 229 8.1.6 BLAST Workflow......................................... 231 8.2 Python Sample applications........................................ 232 8.2.1 Simple................................................ 232 8.2.2 Increment.............................................. 234 8.2.3 Kmeans............................................... 235 8.2.4 Kmeans with Persistent Storage................................. 241 8.2.5 Matmul............................................... 249 8.2.6 Lysozyme in water......................................... 252 8.3 C/C++ Sample applications....................................... 260 8.3.1 Simple................................................ 261 8.3.2 Increment.............................................. 263 9 PyCOMPSs Player 271 9.1 Requirements and Installation....................................... 271 9.1.1 Requirements............................................ 271 9.1.2 Installation............................................. 271 9.2 Usage.................................................... 272 9.2.1 Start COMPSs infrastructure in your development directory.................. 273 9.2.2 Running applications........................................ 273 9.2.3 Running the COMPSs monitor.................................. 274 9.2.4 Running Jupyter notebooks.................................... 274 9.2.5 Generating the task graph..................................... 274 9.2.6 Tracing applications or notebooks................................ 275 9.2.7 Adding more nodes......................................... 275 9.2.8 Removing existing nodes...................................... 276 9.2.9 Stop pycompss ........................................... 276 10 PyCOMPSs Notebooks 277 10.1 Syntax.................................................... 277 10.1.1 Basics of programming with PyCOMPSs............................ 277 10.1.2 PyCOMPSs: Synchronization................................... 279 10.1.3 PyCOMPSs: Using objects, lists, and synchronization..................... 282 10.1.4 PyCOMPSs: Using objects, lists, and synchronization..................... 285 10.1.5 PyCOMPSs: Using objects, lists, and synchronization. Using collections........... 288 10.1.6 PyCOMPSs: Using objects, lists, and synchronization. Using dictionary............ 292 10.1.7 PyCOMPSs: Using objects, lists, and synchronization. Managing