Hyperworks 2017 Licensing

Learn more at altairhyperworks.com Copyrights, Trademarks, Patents and Third Party Licenses

Intellectual Property Rights Notice: Copyrights, Trademarks, Trade Secrets, Patents & Third Party Software Licenses Altair® HyperWorks® v.2017 A Platform for Innovation® Copyright© 1986-2016 Altair Engineering Inc. All Rights Reserved.

Special Notice: Pre-release versions of Altair software are provided ‘as is’, without warranty of any kind. Usage is strictly limited to non-production purposes.

HyperWorks® 2017 Products HyperMesh® 1990-2016; HyperCrash® 2001-2016; OptiStruct® 1996-2016; RADIOSS® 1986-2016; HyperView® 1999-2016; HyperView Player® 2001-2016; HyperMath® 2007-2016; HyperStudy® 1999-2016; HyperGraph® 1995-2016; MotionView® 1993-2016; MotionSolve® 2002-2016; HyperForm® 1998-2016; HyperXtrude® 1999- 2016; Process Manager™ 2003-2016; Templex™ 1990-2016; TextView™ 1996-2016; MediaView™ 1999-2016; TableView™ 2013-2016; BatchMesher™ 2003-2016; HyperWeld® 2009-2016; HyperMold® 2009-2016; Manufacturing Solutions™ 2005-2016; solidThinking Inspire® 2017 2009-2016; solidThinking Evolve® 2017 1993-2016; Durability Director™ 2009-2016; Suspension Director™ 2009-2016; AcuSolve® 1997-2016; AcuConsole® 2006-2016; SimLab® 2004-2016; Virtual Wind Tunnel™ 2012-2016; FEKO® (©1999-2014 Altair Development S.A. (Pty) Ltd.; ©2014-2016 Altair Engineering, Inc.); ConnectMe™ 2014-2016.

Additional Altair Products: Multiscale Designer™ 2011-2016; Flux™ v.12.2 1983-2016; InCa3D v.3.1 1996-2016; CDE v.2 2012-2016; Got-It v.3 2002-2016; WinProp v.14 2000-2016

Altair Packaged Solution Offerings (PSOs) Copyright© 2008-2016 Automated Reporting Director™ 2008-2016; GeoMechanics Director 2011-2016; Impact Simulation Director™ 2010-2016; Model Mesher Director™ 2010-2016; Model Verification Director™ 2013-2016; NVH Director™ 2010-2016; Squeak and Rattle Director™ 2012-2016; Virtual Gauge Director™ 2012-2016; Weight Analytics™ 2013-2016; Weld Certification Director™ 2014-2016

Altair Simulation Cloud Suite: Simulation Manager™ 2003-2016; Compute Manager™ 2003-2016; Display Manager™ 2003–2016; and Process Manager™ 2003-2016.

Altair PBS Works™: Compute Manager™ 2012-2016; Display Manager™ 2013-2016; PBS™ 1994-2016; PBS Pro™ 1994-2016; PBS Professional® 1994-2016; PBS Application Services™ 2008-2016; PBS Analytics™ 2008-2016; and PBS Desktop™ 2008-2012; e-Compute™ 2000-2010; OpenPBS® 1994-2003 and Personal PBS® 2008-2012.

Software products of solidThinking, Inc., a wholly owned subsidiary of Altair Engineering: solidThinking Inspire® 2017 2009-2016; solidThinking Evolve® 2017 1993-2016; solidThinking Compose™ 2017 2007-2016; solidThinking Activate™ 2017 1989-2016; solidThinking Embed™ 2017

1989-2016; solidThinking Embed™ SE 2017 1989-2016; Click2Extrude™ Metal 2017 1996-2016; Click2Extrude™ Polymer 2017 1996-2016; Click2Cast® 14.0 2011-2016; Click2Form™ 2017 1998-2016; Envision™ 2013-2016.

ALTAIR ENGINEERING INC. Proprietary and Confidential. Contains Trade Secret Information. Not for use or disclosure outside of Altair and its licensed clients. Information contained in Altair software shall not be decompiled, disassembled, “unlocked”, reverse translated, reverse engineered, or publicly displayed or publicly performed in any manner. Usage of the software is only as explicitly permitted in the end user software license agreement. Copyright notice does not imply publication.

Third party software licenses AcuConsole contains material licensed from Intelligent Light (www.ilight.com) and used by permission.

Software Security Measures: Altair Engineering Inc. and its subsidiaries and affiliates reserve the right to embed software security mechanisms in the Software for the purpose of detecting the installation and/or use of illegal copies of the Software. The Software may collect and transmit non-proprietary data about those illegal copies. Data collected will not include any customer data created by or used in connection with the Software and will not be provided to any third party, except as may be required by law or legal process or to enforce our rights with respect to the use of any illegal copies of the Software. By using the Software, each user consents to such detection and collection of data, as well as its transmission and use if an illegal copy of the Software is detected. No steps may be taken to avoid or detect the purpose of any such security mechanisms.

Technical Support

Altair provides comprehensive software support via web FAQs, tutorials, training classes, telephone, and e-mail.

Altair Support on the World Wide Web

The Altair web site is a valuable online companion to Altair software. Visit www.altairhyperworks.com for tips and tricks, training course schedules, training/tutorial videos, and other useful information.

Altair Training Classes

Altair training courses provide a hands-on introduction to our products, focusing on overall functionality. Courses are conducted at our main and regional offices or at your facility. If you are interested in training at your facility, please contact your account manager for more details. If you do not know who your account manager is, please send an e-mail to [email protected] and your account manager will contact you.

Telephone and e-mail

When contacting Altair support, please specify the product and version number you are using along with a detailed description of the problem. Many times, it is very beneficial for the support engineer to know what type of workstation, operating system, RAM, and graphics board you have, so please have that information ready. If you send an e-mail, please specify the workstation type, operating system, RAM, and graphics board information in the e-mail. To contact an Altair support representative, reference the following table or the information available on the HyperWorks website: http://www.altairhyperworks.com/ClientCenterHWSupportProduct.aspx

Location Telephone e-mail

Australia 64.9.413.7981 [email protected]

Brazil 55.11.3884.0414 [email protected]

Canada 416.447.6463 [email protected]

China 86.400.619.6186 [email protected]

France 33.1.4133.0992 [email protected]

Germany 49.7031.6208.22 [email protected]

India 91.80.6629.4500 [email protected] 1.800.425.0234 (toll free)

Italy 39.800.905.595 [email protected]

Japan 81.3.5396.2881 [email protected]

Korea 82.70.4050.9200 [email protected]

Mexico 55.56.58.68.08 [email protected]

New Zealand 64.9.413.7981 [email protected]

North America 248.614.2425 [email protected]

Scandinavia 46.46.460.2828 [email protected]

Spain 34.910.810.080 [email protected]

South Africa 27.21.8311500 [email protected]

United Kingdom 01926.468.600 [email protected]

For questions or comments about this help system, send an email to [email protected]. In addition, the following countries have resellers for Altair Engineering: Colombia, Czech Republic, Ecuador, Israel, Russia, Netherlands, Turkey, Poland, Singapore, Vietnam, Indonesia Official offices with resellers: Canada, China, France, Germany, India, Malaysia, Italy, Japan, Korea, Spain, Taiwan, United Kingdom, USA See www.altair.com for complete contact information.

Contents

1 Introduction ...... 4

1.1 About HyperWorks ...... 5 1.2 HyperWorks 2017 Licensing ...... 6 1.3 HyperWorks 2017 Software and Hardware Release Notes ...... 7

2 RADIOSS HMPP Installation Overview ...... 11

2.1 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI ...... 17 2.1.1 System Requirements ...... 17 2.1.2 Installing the Software and Activating the License ...... 17 2.1.3 Configuring the Machine ...... 17 2.1.4 Environment Variable Settings ...... 18 2.1.5 Launching a RADIOSS Computation ...... 19 2.1.6 Troubleshooting ...... 21 2.2 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI and NVIDIA CUDA ...... 22 2.2.1 System Requirements ...... 22 2.2.2 Installing the Software and Activating the License ...... 22 2.2.3 Configuring the Machine ...... 22 2.2.4 Environment Variable Settings ...... 23 2.2.5 Launching a RADIOSS Computation ...... 25 2.2.6 Troubleshooting ...... 26 2.3 Installing RADIOSS HMPP on x86-64 Linux Cluster using Platform MPI 9.1.2 ...... 27 2.3.1 System Requirements ...... 27 2.3.2 Installing the Software and Activating the License ...... 27 2.3.3 Configuring the Machine ...... 27 2.3.4 Launching a RADIOSS Computation ...... 28 2.3.5 Troubleshooting ...... 30 2.4 Installing RADIOSS HMPP on Windows 64 Cluster using Intel MPI ...... 32 2.4.1 System Requirements ...... 32 2.4.2 Installing the Software and Activating the License ...... 32 2.4.3 Configuring the Machine ...... 32 2.4.4 Installing the Intel MPI Library ...... 32 2.4.5 Installing on a Cluster ...... 34 2.4.6 Launching a RADIOSS Computation ...... 35 2.4.7 Troubleshooting ...... 37

1 2.5 Installing RADIOSS HMPP on Windows Server 2008 R2 Computer Cluster using MS MPI ...... 38 2.5.1 Installing the Software and Activating the License ...... 38 2.5.2 Configuring the Machine ...... 38 2.5.3 Launching a RADIOSS Computation ...... 38 2.5.4 The Compute Clusters Pack Job ...... 39 2.5.5 Launching the Compute Cluster Job Manager ...... 41 2.6 Installing RADIOSS HMPP on Windows 64 Cluster using Platform MPI 9.1.2 ...... 45 2.6.1 System Requirements ...... 45 2.6.2 Installing the Software and Activating the License ...... 45 2.6.3 Configuring the Machine ...... 45 2.6.4 Launching a RADIOSS Computation ...... 46 2.6.5 Troubleshooting ...... 47

3 OptiStruct SPMD Installation Overview ...... 49

3.1 Load Decomposition Method ...... 50 3.1.1 Supported Solution Sequences ...... 51 3.1.2 Number and Type of Nodes Available for Parallelization ...... 52 3.1.3 Frequently Asked Questions ...... 53 3.2 Domain Decomposition Method ...... 55 3.2.1 Supported Solution Sequences ...... 55 3.2.2 Frequently Asked Questions ...... 56 3.3 Multi-Model Optimization ...... 58 3.3.1 Supported Solution Sequences ...... 58 3.4 Failsafe Topology Optimization ...... 60 3.5 Supported Platforms ...... 62 3.6 Frequently Asked Questions ...... 63 3.7 OptiStruct SPMD Setup ...... 66 3.7.1 Installing OptiStruct SPMD on Linux Machines ...... 66 3.7.2 Installing OptiStruct SPMD on Windows Machines ...... 72

4 AcuSolve Installation Overview ...... 81

4.1 Configuring an AcuSolve Installation ...... 83 4.1.1 Configuring AcuSolve to Use Legacy ACUSIM Licensing ...... 84 4.1.2 Configuring AcuSolve for Distributed Memory Parallel Computing ...... 85 4.1.3 Configuring AcuSolve for Parallel Processing on Windows Platforms ...... 86 4.1.4 Configuring AcuSolve for Parallel Processing on a Single Windows Host ...... 86 4.1.5 Configuring the AcuSolve Environment on Linux Platforms ...... 93 4.2 Installing AcuSolve 2017 Alongside Older Distributions on Linux ...... 95

2 4.3 Testing the AcuSolve Installation ...... 96 4.3.1 Channel Example ...... 96 4.3.2 Pipe Example ...... 97

5 FEKO Installation Overview ...... 99

6 RADIOSS HMPP to MADYMO 7.1 MPP Coupling ...... 100

6.1 System Requirements ...... 101 6.2 Installing RADIOSS and Activating the License ...... 102 6.3 Installing MADYMO and Activating the License ...... 103 6.4 Configuring the Machine ...... 104 6.5 Launching a RADIOSS to MADYMO Computation ...... 105

3 Introduction 1

Introduction This guide provides instructions for the advanced installation and operation of HyperWorks software on supported platforms. This includes installing RADIOSS HMPP, OptiStruct SPMD, and installing and configuring AcuSolve.

HyperWorks 2017 uses InstallAnywhere 2015 SP1 as its installation tool. The HyperWorks 2017 installers are InstallAnywhere web installers. InstallAnywhere provides a consistent look and feel of the HyperWorks installation wizard across all platforms supported by Altair HyperWorks products.

In order to run HyperWorks 2017 applications, you need to connect the applications to the Altair License Management System 13.0 (or higher, using the latest version is recommended). Details of the installation and how to start the Altair License Manager can be found in the Altair License Management System 13.0 Installation Guide. The license packages are available on the Altair web site (altairhyperworks.com) or Altair Connect (connect.altair.com).

This chapter covers the following:

• 1.1 About HyperWorks (p. 5) • 1.2 HyperWorks 2017 Licensing (p. 6) • 1.3 HyperWorks 2017 Software and Hardware Release Notes (p. 7) HyperWorks 2017 Advanced Installation Guide 5

1.1 About HyperWorks

HyperWorks® is the most comprehensive open-architecture simulation platform, offering best-in-class technologies to design and optimize high performance, efficient and innovative products.

HyperWorks includes finite element modeling, analysis and optimization for structures, fluids, multibody dynamics, electromagnetics and antenna placement, model-based development, and multiphysics. Users have full access to a wide suite of design, engineering, visualization, and data management solutions from Altair and its technology partners.

The HyperWorks 2017 suite offers many new functionalities in areas such as model-based development, electromagnetism, composites structures analysis, efficient modeling and meshing, multiphysics and multi-disciplinary analysis, structural optimization and design exploration.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 6

1.2 HyperWorks 2017 Licensing

HyperWorks 2017 licensing is based on HyperWorks Units (HWUs). HyperWorks Units licenses are server licenses. Customers can launch any application within the HyperWorks suite as long as there are sufficient HyperWorks Units to grant the license check out.

To invoke HyperWorks 2017 applications, after completing the HyperWorks 2017 installation, you need to point the applications to an activated Altair License Manager or a valid node stand-alone license by setting the proper environment variable.

The HyperWorks 2017 package does not include the license package that sets up the Altair License Manager. Download the Altair license package from the Altair web site and follow the Altair License Management System 13.0 Installation and Operation Guide to install and activate the Altair License Manager properly.

In some cases, Altair provides “feature” based licenses, which allow a particular application to run as long as the feature exists in the license file. A feature license is a standalone license that does not consume HyperWorks Units.

Questions regarding the software and license installation may be directed to Altair Support. See the Contact page for further contact information.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 7

1.3 HyperWorks 2017 Software and Hardware Release Notes

Various changes have been made to the HyperWorks 2017 software package including the HyperWorks products, hardware support, installation, and invoking process.

Note: Changes to the Altair Licensing System are detailed in the Altair HyperWorks Licensing Release Notes.

New Applications

Click2Extrude Metal solidThinking® Click2Extrude Metal™ is a simulation environment designed to help extrusion companies meet the ever increasing demands to produce complex profiles with tight tolerances, quality surface finishes, and high strength properties at reduced cost. Click2Extrude Metal is a virtual press where users can visualize material flow and temperature inside a die during extrusion and make necessary changes to ensure balanced flow, while identifying and eliminating product defects. Click2Extrude Polymer solidThinking® Click2Extrude Polymer™ is a simulation environment designed to help extrusion companies meet the ever increasing demands to produce complex profiles with tight tolerances, quality surface finishes, and high strength properties at reduced cost. Click2Extrude is a virtual press where users can visualize material flow and temperature inside a die during extrusion and make necessary changes to ensure balanced flow, while identifying and eliminating product defects. Click2Form solidThinking® Click2Form™ is a virtual environment for early stamping feasibility analysis based on part or tooling geometry. It allows you to look at thinning and formability issues as well as estimate blank shape and maximize material utilization via blank fitting and nesting utilities.

Altair ConnectMe Altair ConnectMe is a tool that provides a window based launcher, Altair software update alerts, and Altair news. It provides a single place to search, filter, launch, and update Altair products.

Hardware and Platform Changes

Operating Systems Additions Support for the following platforms has been added: • Windows 10

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 8

• Red Hat Enterprise Linux 7.2 •SUSE Linux Enterprise Server 12 Service Pack 1 • CentOS Linux 7.2

Discontinued Hardware Platform Support Systems Support for the following platforms has been dropped across all HyperWorks applications: •Windows Server 2008 • CentOS Linux 5.9 • Red Hat Enterprise Linux 5.9 • Mac OS X (all)

Installation The HyperWorks 2017 suite contains several independent product packages. This gives you the flexibility to download desired packages as needed. Any HyperWorks 2017 standalone (SA) update package can be installed either on top of an existing 2017 installation, or can be installed independently without the prerequisite of a 2017 base. Multiple product SA updates can now be installed concurrently with or without the 2017 base.

HyperWorks 2017 brings with it the robust capabilities and features of HyperWorks 14.0 installers, as well as additional support for HotFix installers. These HotFix installers are quick patch type installers that are used to install on top of a prerequisite base installation, unlike the other full installers. These are marked with the word “hotfix” in the installer name and bring with them the quick fixes and features to better suit your needs.

Software Changes

Installation and Packaging

• Added Altair digital signature to key executable files in HW Desktop and Solvers. • Added Click2Extrude Metal product for Windows. • Added Click2Extrude Polymer product for Windows. • Added Click2Form product for Windows. • Added Linux support for Virtual Wind Tunnel. •Upgraded the HyperWorks installers to InstallAnywhere 2015 SP1 web installers. • Improved the Windows uninstallers. • Improved error logging when the Master Installer is used. •Standardized MPI implementations: The versions of the Message Passing Interface (MPI) have been standardized across FEKO, AcuSolve, RADIOSS, and OptiStruct. The Intel, IBM Platform, Microsoft, and MPICH MPI implementations are now included. End users can still use their own MPI implementation.

Invoking

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 9

• Added file associations for .hm, .mdl, and .mvw to open in HyperWorks Desktop. • Added thumbnail preview for .hm files. • Added better font size management for the HyperWorks Desktop applications, including HyperMesh under Linux user environment where the default size is set to a 9 point font instead of using a pixel font size. • Added Linux launch icons to HyperWorks applications. • Updated Linux Compatibility Tool: A Linux utility for system diagnostics to help troubleshoot Linux issues. • Added better version information to the shortcuts for the Windows 8.1 Start menu and Metro style Start menu.

Resolved Issues • Improved the Start menu folder removal portion of the HyperWorks uninstallers. • Resolved an issue where the Help Configuration panel was skipped when clicking the Previous button from the Pre-Installation Summary panel. • Resolved an issue where Start menu shortcuts were not overwritten on Windows 7 when overwriting an existing HyperWorks installation and Start menu folder. • Improved the Linux console mode installations. • Improved error logging when the Master Installer is used. • When installing using the Master Installer, you can now cancel your installation at any time. • On Linux during a console mode install, the installer will now allow you to go back past the Install Pick List panel. • On Linux, when installing using the console mode, the Help Configuration prompt will no longer add another set of double quotes on the user-defined path to installed help if the back command is used. • On Linux, there are no longer duplicate progress install bars when the back command is used from the Summary Panel. • Corrected Linux “read + execute” permissions of the sub installers are set when using the Master Installer. • Resolved graphics issues on Optimus based video card setups.

Known Limitations in HyperWorks Installer 2017 • Using the Back/Previous option during an installation may not reset the installer variables correctly if changes are made and the install session is continued. • On Linux, installing in a directory with spaces is not supported.

• On Linux, during a console mode install, when using a response file output command -r the Help Configuration in the response file may not write out correctly. • If there is not enough TMP/TEMP space on the main drive (for example, C: under Windows), then the installer will prompt for user input to a new location. Some Java TMP/TEMP space will still try to go to the main drive even when selecting this new location.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 10

• When clicking Cancel to stop an installation, the Master Installer does not clean up the already installed files, but does clean up the files in the TMP/TEMP folder. You will need to uninstall/delete the installed files from the chosen install folder manually if Cancel was used. • When clicking Cancel to stop an installation, the Add/Remove uninstaller icon may no longer function correctly and manual removal of the installed files would be necessary. • The HyperWorks installers do not support the use of the default InstallAnywhere installer.properties files. We request that a unique name is used when using a response file; for example, installer_hw2017.properties, and to use it with the -f / command line option. • For NETSETUP client installs from a server installation of HyperWorks, a UNC path must be used to get to the server installation path through Windows Explorer or Windows Network.

Note: Using mapped drive letters or variables is not currently supported in the client installer.

• On Linux, there is an issue with using the Master Installer with the IATEMPDIR variable, to specify a custom temporary directory location. All sub-installers will try to use /tmp from the machine running the installers for temporary installer files. • With some versions of Gnome, there is a bug where the Linux icons are not placed in a hierarchy. Instead, they are placed in “Other” with a flat layout, meaning that all the icons appear at the same folder level in “Other.” • The installer does not run the Linux icon installation by default; the user must run the icon installer. • On Windows 8, desktop and Start menu icons are blank for server and client installations. This can be resolved by changing the Windows File Explorer policy to “Allow the use of remote paths in file shortcut icons”. • When the Master Installer is used, desktop shortcuts, if any, will not be removed during uninstallation.

Proprietary Information of Altair Engineering RADIOSS HMPP Installation Overview 2

RADIOSS HMPP Installation Overview

Introduction of RADIOSS HMPP The performance criterion in computation is an essential point in the architectural conception of RADIOSS. In the case of analysis of systems with a high number of degrees of freedom, the use of a shared memory parallel machines or large distributed memory clusters are common.

To efficiently adapt and optimize the performance on every type of machines, RADIOSS combines the benefit of two models of parallel programming inside the same executable: • SMP: Shared Memory Parallelism •SPMD: Single Program Multiple Data

This enhanced parallel version is called RADIOSS Hybrid MPP (HMPP).

RADIOSS HMPP increases scalability and performance by having two parallelization levels: • High level: SPMD parallelization based on domain decomposition and MPI • Low level: SMP multi-threaded processes under OpenMP

Customer Benefits This new approach allows Altair to reach an impressive level of scalability. RADIOSS HMPP can scale up to 512 cores and more for a real performance breakthrough.

RADIOSS HMPP is independent of the computer architecture, more flexible and efficiently adapted to any hardware resources. It can run on distributed memory machines, shared memory machines, workstation cluster, or a high performance computation cluster. It can better exploit the inside power of highly multi-core machines and optimize the software according to the hardware.

It decreases installation and maintenance costs by having a unique parallel version instead of 2 types of executables.

It improves the quality of the code in term of numerical results by having a full convergence between SMP and SPMD results. Parith/ON gives a unique answer independent of the number of SPMD domains and the number of SMP threads used. HyperWorks 2017 Advanced Installation Guide 12

Example In the example below, the hardware is composed of n nodes of a cluster. Each node is a dual processor machine and each processor is a dual core. On each node the memory is shared between processors but the memory is distributed between the nodes.

Figure 1: Example of Hybrid version, where each SPMD domain uses 2 SMP threads

With the Hybrid version, it is easy to optimize the software to run on such complex architecture. On the figure above, each SPMD domain is computed by a different processor. The number of threads per domain is set to two, so each core of a processor is running a thread of the associated SPMD domain.

How Does it Work? RADIOSS Starter divides the model into several SPMD domains.

The /SPMD card in RADIOSS Starter input deck is used to set the number of SPMD domains by the value defined in the second field. This value must match the number of processors requested by the mpirun command (-np) when running RADIOSS Engine. In this card it is now possible to define the number of SMP threads used to compute each domain.

Note: It is also possible to set the number of SPMD domains using command line argument –nspmd or –np and the number of SMP threads using command line argument – nthread or –nt. This ability avoids having to modify the input deck.

RADIOSS Starter writes as many RESTART files as the number of SPMD domains which will be used for the run. Each RESTART file is built from the only data that will be needed by the relative program, and is read directly by this program when running RADIOSS Engine.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 13

Then, the mpirun command is used to start all the SPMD programs. Each program computes a SPMD domain using the number of SMP threads set.

The management of computation at the frontiers of the domains remains and it is necessary to communicate some information between the programs. These communications have been implemented through the use of the Message Passing Interface (MPI).

For example, the /SPMD card defines 16 SPMD domains and 2 SMP threads per domain:

#---1----|----2----|----3----|----4----|----5----|----6----| /SPMD # Domdec Nspmd Dkword Nthread 0 16 0 2

As an alternative, this can be set without changing the input, directly from the RADIOSS Starter command line:

./s_2017_linux64 –nspmd 16 –nthread 2 –i ROOTNAME_0000.rad

Then, RADIOSS Engine is run using the mpirun command:

mpirun –np 16 ./e_2017_linux64_impi –i ROOTNAME_0001.rad

Note:

• -np value of mpirun must match –nspmd value (or Nspmd input field) • -nthread can be redefined from the RADIOSS Engine command line • The total number of processes is equal to Nspmd*Nthread (32 in this case)

In order to facilitate transition to the HMPP version, it is possible to bypass MPI installation and use special binaries only SMP capable (see table below). They work the same and give same numerical answers, except that they do not require MPI and do not need mpirun to start but are restricted to Nspmd=1. To avoid confusion, the HMPP Engine executables are suffixed with their MPI libraries (example _impi for Intel MPI). The Starter executables are common. Under Linux and Windows, the Extended Single Precision version also exists. Such executables are suffixed with “_sp”. They offer improving performance by using single precision arithmetic while keeping reasonable accuracy.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 14

Executable Tables In version 2017, names of executables have been reviewed in order to be more meaningful.

Table 1: Double Precision

HW Parallel Starter Engine radflex Platform Platform Platform 2017 2017 Info Release

win64 smp s_2017_ e_2017_ radflex_2017_ Windows win64 win64.exe win64.exe win64.exe 64 bit, Intel compiler

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 impi win64.exe win64_impi win64.exe 64 bit, Intel .exe compiler, Intel MPI

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 msmpi win64.exe win64_msmpi win64.exe 64 bit, Intel .exe compiler, MS MPI

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 plmpi win64.exe win64_ win64.exe 64 bit, Intel plmpi.exe compiler, Platform MPI 9.1.2

linux64 smp s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 linux64 linux64 linux64.exe bit, Intel compiler

linux64 hybrid s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 impi linux64 linux64_impi linux64.exe bit, Intel compiler, Intel MPI

linux64 hybrid s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 cuda linux64 linux64_ linux64.exe bit, Intel impi cuda_impi compiler, Cuda GPU, Intel MPI

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 15

HW Parallel Starter Engine radflex Platform Platform Platform 2017 2017 Info Release

linux64 hybrid s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 plmpi linux64 linux64_plmpi linux64.exe bit, Intel compiler, Platform MPI 9.1.2

Table 2: Single Precision

HW Parallel Starter Engine radflex Platform Platform platform 2017 2017 Info Release

win64 smp s_2017_ e_2017_ radflex_2017_ Windows win64 sp win64_sp.exe win64_sp.exe win64.exe 64 bit, Intel compiler, single precision

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 impi win64_sp.exe win64_impi_ win64.exe 64 bit, Intel sp sp.exe compiler, Intel MPI, single precision

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 msmpi win64_ win64_ win64.exe 64 bit, Intel sp sp.exe msmpi_sp compiler, MS .exe MPI,single precision

win64 hybrid s_2017_ e_2017_ radflex_2017_ Windows win64 plmpi win64.exe win64_plmpi_ win64.exe 64 bit, Intel sp sp.exe compiler, Platform MPI 9.1.2, single precision

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 16

HW Parallel Starter Engine radflex Platform Platform platform 2017 2017 Info Release

linux64 smp s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 sp linux64_sp linux64_sp linux64 bit, Intel compiler, single precision

linux64 hybrid s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 impi linux64_sp linux64_impi_ linux64 bit, Intel sp sp compiler, Intel MPI, single precision

linux64 hybrid s_2017_ e_2017_ radflex_2017_ Linux 64- linux64 plmpi linux64_sp linux64_plmpi_splinux64 bit, Intel sp compiler, Platform MPI 9.1.2, single precision

This chapter covers the following:

• 2.1 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI (p. 17) • 2.2 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI and NVIDIA CUDA (p. 22) • 2.3 Installing RADIOSS HMPP on x86-64 Linux Cluster using Platform MPI 9.1.2 (p. 27) • 2.4 Installing RADIOSS HMPP on Windows 64 Cluster using Intel MPI (p. 32) • 2.5 Installing RADIOSS HMPP on Windows Server 2008 R2 Computer Cluster using MS MPI (p. 38) • 2.6 Installing RADIOSS HMPP on Windows 64 Cluster using Platform MPI 9.1.2 (p. 45)

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 17

2.1 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI

2.1.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD64 systems under Linux: RHEL 6.2, 6.5, 7.0 SLED 11 SP2 SLES 11 SP2 b. The interconnect is Infiniband or INTEL Omni PATH. The support of Infiniband is ensured via DAPL interface. Ensure that libdapl.so is installed with low level network drivers. To support DAPL, the Linux kernel level must be 2.6.18 or above. c. If the cluster network is Infiniband using OFED drivers: the Linux kernel level must be 2.6.18 or higher, the OFED drivers must be OFED 1.3 or higher. d. It is highly recommended to have the same hardware and OS level on all nodes. e. The MPI library is: Intel MPI l_mpi_p_5.1.3 or above. f. RADIOSS supports INTEL Omni PATH Architecture. Ensure that the operating system is RHEL 7.0 or above and the MPI library is Intel MPI Library 2017 or above.

2.1.2 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Linux64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.1.3 Configuring the Machine

The account used to run RADIOSS must exist on every cluster's node.

RADIOSS executables must be reachable from all cluster nodes (same path on all nodes). The computation directory must be the same (same path) on all cluster nodes.

SSH Installation

Intel MPI uses ssh to launch RADIOSS on all different nodes. ssh needs to be configured to permit the connection on all hosts of the cluster without typing passwords. Refer to ssh manual pages to generate and install properly rsa keys (ssh_keygen tool).

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 18

To check the functionality of ssh, the following test can be performed on the different nodes:

[radioss@host1] ssh host1 ls [radioss@host1] ssh host2 ls ... [radioss@host1] ssh host[n] ls

Environment Variables Setting

The radflex_2017_linux64 tool needs the liblmx-altair.13.0.0.so library from the HyperWorks 2017 installation.

Set the LD_LIBRARY_PATH environment variable to $ALTAIR_HOME/hwsolvers/common/bin/linux64:

[radioss@host1] setenv LD_LIBRARY_PATH $ALTAIR_HOME/hwsolvers/common/bin/linux64

Intel MPI Installation Intel MPI needs the python-xml package installed. The installation package of Intel MPI is made of a self extracting script. This package contains an RPM which will be installed. You will need to follow the installation instructions. The install path by default is /opt/intel/ compilers_and_libraries_2017../linux/mpi.

2.1.4 Environment Variable Settings stacksize limit: stacksize limit must be set to unlimited with an unlimited value of more than 400Mbytes.

KMP_STACKSIZE, the stacksize per thread, must also be set to 400Mbytes.

Example under csh:

limit stacksize unlimited setenv KMP_STACKSIZE 400m

I_MPI_FABRICS By default the interconnect used is Gigabit Ethernet. To use other interconnects, you need to set the I_MPI_FABRICS environment variable. The variable syntax is as follows: I_MPI_FABRICS=|:. If you specify only , it is used for both intra-node and inter-node communications. I_MPI_FABRICS can have following values:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 19

shm[:] Shared memory (for intra-node communication only) dapl[:] InfiniBand, iWarp tcp [:] InfiniBand, Ethernet tmi[:] Intel Omni-Path Architecture I_MPI_PIN_DOMAIN I_MPI_PIN_DOMAIN must be set to auto in order to optimize the placement of SPMD programs. On some old systems, setting I_MPI_PIN_DOMAIN to auto may not work properly if the system is not affinity capable (see KMP_AFFINITY). In this case it is advised to set I_MPI_PIN_DOMAIN to omp. I_MPI_ADJUST_BCAST I_MPI_ADJUST_BCAST must be set to 1 in order to optimize collective operation of type broadcast (important for high number of SPMD domains, 64 and more). I_MPI_ADJUST_REDUCE I_MPI_ADJUST_REDUCE must be set to 2 in order to optimize collective operation of type reduce (important for high number of SPMD domains, 64 and more). KMP_AFFINITY In order to check if the system is affinity capable, the user may set KMP_AFFINITY to verbose. If the system is affinity capable, it is advised to set KMP_AFFINITY to compact if Hyper Threading is disabled, or to scatter if Hyper Threading is enabled. OMP_NUM_THREADS

Instead of using the –nt parameter, the user can set the number of threads the using environment variable OMP_NUM_THREADS. Using this environment variable may help the system to better determine affinity (see KMP_AFFINITY=verbose). LD_LIBRARY_PATH

You need to set the LD_LIBRARY_PATH to the path where the vendor libdapl.so is installed. Check the Interconnect vendor for availability of libdapl.so library. The use of libdapl.so requires kernel 2.6.18 or newer to be installed on the machine. For example:

[radioss@host1~]$setenv LD_LIBRARY_PATH /usr/local/ibgd/lib [radioss@host1~]$setenv I_MPI_DEVICE rdssm [radioss@host1~]$setenv I_MPI_PIN_DOMAIN auto [radioss@host1~]$setenv KMP_AFFINITY compact

2.1.5 Launching a RADIOSS Computation

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 20

On a Single Machine

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/radflex_2017_linux64

[radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 -input [ROOTNAME]_0000.rad –np [n]

[radioss@host1~]$ [Intel MPI path]/bin/mpirun -np [n] $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/e_2017_linux64_impi -input [ROOTNAME]_0001.rad –nt [t]

Where

[Intel MPI path]: is the path where Intel MPI is installed

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain. Instead of using –nt it is recommended to use OMP_NUM_THREADS environment variable.

[ROOTNAME]: is the rootname of the dataset

On an Intel EM64T Cluster

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/linux64/radflex_2017_linux64 [radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 –input [ROOTNAME]_0000.rad –np [Nspmd] [radioss@host1~]$ [Intel MPI path]/bin/mpirun -configfile [cgfile]

Where

[Intel MPI path]: is the path where Intel MPI is installed

[ [cgfile]: is a file describing each node of the computation. This file is as follows:

-n [n] -host host1 [Engine path]/e_2017_linux64_impi –input [ROOTNAME]_0001.rad –nt [t] -n [n] -host host2 [Engine path]/e_2017_linux64_impi –input [ROOTNAME]_0001.rad –nt [t] -n [n] -host host3 [Engine path]/e_2017_linux64_impi –input [ROOTNAME]_0001.rad –nt [t] ... -n [n] -host hostp [Engine path]/e_2017_linux64_impi –input [ROOTNAME]_0001.rad –nt [t]

Where host1...hostp: are the hostnames of the different nodes.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 21

[n]: is the number of SPMD domains on host p. The sum of all the SPMD domains given in the config file must match the value given for the Starter –np [Nspmd] option.

[Engine path]: is the path to the RADIOSS Engine.

[ROOTNAME]: is the rootname of the dataset

-nt[t]: is the number of SMP threads per SPMD domain.

2.1.6 Troubleshooting

Stacksize Modification When the stacksize is modified on the cluster, the user needs to re-launch the python daemons. Otherwise the new stacksize value is not taken into account (refer to Environment Variable Settings for the recommended setting of stacksize).

Use of DAPL Settings The use of DAPL modus requires kernel version 2.6.18 or newer. Using an older kernel may cause RADIOSS to hang at startup phase. memlock limit

The file /etc/security/limits.conf should be checked and modified to contain the line below:

* soft memlock unlimited

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 22

2.2 Installing RADIOSS HMPP on x86-64 Linux Cluster using Intel MPI and NVIDIA CUDA

2.2.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD64 systems under Linux: RHEL 6.2, 6.5, 7.0 SLED 11 SP2 SLES 11 SP2 b. The interconnect is Infiniband. The support of Infiniband is ensured via DAPL interface. Ensure that libdapl.so is installed with low level network drivers. To support DAPL, the Linux kernel level must be 2.6.18 or above. c. If the cluster network is Infiniband using OFED drivers: the Linux kernel level must be 2.6.18 or higher, the OFED drivers must be OFED 1.3 or higher. d. It is highly recommended to have the same hardware and OS level on all nodes. e. The MPI library is: Intel MPI l_mpi_p_5.1.3 or above. The Intel MPI runtime kit is available for free at http://software.intel.com/en-us/intel-mpi-library/ f. GPU card: Nvidia Fermi C2070 & M2090 Nvidia Kepler K20 g. Cuda driver v4.2

2.2.2 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Linux64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.2.3 Configuring the Machine

The account used to run RADIOSS must exist on every cluster's node.

RADIOSS executables must be reachable from all cluster nodes (same path on all nodes). The computation directory must be the same (same path) on all cluster nodes.

SSH Installation

Intel MPI uses ssh to launch RADIOSS on all different nodes.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 23 ssh needs to be configured to permit the connection on all hosts of the cluster without typing passwords. Refer to ssh manual pages to generate and install properly rsa keys (ssh_keygen tool).

To check the functionality of ssh, the following test can be performed on the different nodes:

[radioss@host1] ssh host1 ls [radioss@host1] ssh host2 ls ... [radioss@host1] ssh host[n] ls

Environment Variables Setting

The radflex_2017_linux64 tool needs the liblmx-altair.13.0.0.so library from the HyperWorks 2017 installation.

Set the LD_LIBRARY_PATH environment variable to $ALTAIR_HOME/hwsolvers/common/bin/linux64:

[radioss@host1] setenv LD_LIBRARY_PATH $ALTAIR_HOME/hwsolvers/common/bin/linux64

2.2.4 Environment Variable Settings stacksize limit: stacksize limit must be set to unlimited with an unlimited value of more than 400Mbytes.

KMP_STACKSIZE, the stacksize per thread, must also be set to 400Mbytes.

Example under csh:

limit stacksize unlimited setenv KMP_STACKSIZE 400m

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 24

If you specify only , it is used for both intra-node and inter-node communications. I_MPI_FABRICS can have following values:

[radioss@host1~]$setenv LD_LIBRARY_PATH /usr/local/ibgd/lib:/opt/cuda/lib64 [radioss@host1~]$setenv I_MPI_DEVICE rdssm [radioss@host1~]$setenv I_MPI_PIN_DOMAIN auto

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 25

[radioss@host1~]$setenv KMP_AFFINITY compact

2.2.5 Launching a RADIOSS Computation

On a Single Machine

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/linux64/radflex_2017_linux64

[radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 -input [ROOTNAME]_0000.rad –np [n]

[radioss@host1~]$ [Intel MPI path]/bin/mpirun -np [n] $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/e_2017_linux64_cuda_impi -input [ROOTNAME]_0001.rad –nt [t] -gpuid [id]

Where

[Intel MPI path]: is the path where Intel MPI is installed

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain. Instead of using –nt it is recommended to use OMP_NUM_THREADS environment variable.

[id]: is the ID of the Nvidia GPU card to use.

[ROOTNAME]: is the rootname of the dataset

On an Intel EM64T Cluster

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/linux64/radflex_2017_linux64

[radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 –input [ROOTNAME]_0000.rad –np [Nspmd]

[radioss@host1~]$ [Intel MPI path]/bin/mpirun -configfile [cgfile]

Where

[Intel MPI path]: is the path where Intel MPI is installed

[ [cgfile]: is a file describing each node of the computation. This file is as follows:

-n [n] -host host1 [Engine path]/e_2017_linux64_cuda_impi –input [ROOTNAME]_0001.rad –nt [t] -gpuid [id] -n [n] -host host2 [Engine path]/e_2017_linux64_cuda_impi –input [ROOTNAME]_0001.rad –nt [t] -gpuid [id]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 26

-n [n] -host host3 [Engine path]/e_2017_linux64_cuda_impi –input [ROOTNAME]_0001.rad –nt [t] -gpuid [id] ... -n [n] -host hostp [Engine path]/e_2017_linux64_cuda_impi –input [ROOTNAME]_0001.rad –nt [t] -gpuid [id]

Where host1...hostp: are the hostnames of the different nodes.

[n]: is the number of SPMD domains on host p. The sum of all the SPMD domains given in the config file must match the value given for the Starter –np [Nspmd] option.

[Engine path]: is the path to the RADIOSS Engine.

[ROOTNAME]: is the rootname of the dataset

-nt[t]: is the number of SMP threads per SPMD domain.

-gpuid [id]: is the GPU card ID to use. If it is not specified then by default each MPI process will run using a different device (MPI process 1 will use device 0, MPI process 2 will use device 1 and so on).

2.2.6 Troubleshooting

Recommendation to use Nvidia GPU Executable

Nvidia GPU only speeds up Implicit Iterative Solver (/IMPL/SOLVER/1)

Use of DAPL Settings The use of DAPL modus requires kernel version 2.6.18 or newer. Using an older kernel may cause RADIOSS to hang at startup phase. memlock limit

The file /etc/security/limits.conf should be checked and modified to contain the line below:

* soft memlock unlimited

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 27

2.3 Installing RADIOSS HMPP on x86-64 Linux Cluster using Platform MPI 9.1.2

2.3.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD64 systems under Linux: RHEL 5.8 RHEL 6.2 SLED 11 SP2 SLES 11 SP2 b. The interconnect is Infiniband or INTEL Omni PATH. The support of Infiniband is ensured via DAPL interface. Ensure that libdapl.so is installed with low level network drivers. To support DAPL, the Linux kernel level must be 2.6.18 or above. c. If the cluster network is Infiniband using OFED drivers: the Linux kernel level must be 2.6.18 or higher, the OFED drivers must be OFED 1.3 or higher. d. It is highly recommended to have the same hardware and OS level on all nodes. e. The MPI library is PL MPI 9.1.2 or higher, and must be installed on all the cluster nodes. Contact IBM to obtain IBM Platform MPI.

2.3.2 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Linux64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.3.3 Configuring the Machine

The account used to run RADIOSS must exist on every cluster's node.

RADIOSS executables must be reachable from all cluster nodes (same path on all nodes). The computation directory must be the same (same path) on all cluster nodes.

RSH Installation

Intel MPI uses ssh to launch RADIOSS on all different nodes. Computation nodes need to be accessed by all other nodes without typing a password. Refer to the ssh manual to generate and properly install rsa keys (ssh-keygen tool).

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 28

To check the functionality of ssh, the following test can be performed on the different nodes:

[radioss@host1] ssh host1 ls [radioss@host1] ssh host2 ls ... [radioss@host1] ssh host[n] ls

Alternatively rsh can be used. To enable rsh, the MPI_REMSH environment variable must be set to rsh:

[radioss@host1] setenv MPI_REMSH rsh

Environment Variables Setting

The radflex_2017_linux64 tool needs the liblmx-altair.13.0.0.so library from the HyperWorks 2017 installation.

Set the LD_LIBRARY_PATH environment variable to $ALTAIR_HOME/hwsolvers/common/bin/linux64:

[radioss@host1] setenv LD_LIBRARY_PATH $ALTAIR_HOME/hwsolvers/common/bin/linux64

PL MPI Installation PL MPI is distributed by Platform Computing as a package. It must be installed with root privilege on each node running the command (please refer to the PL MPI release notes for more information):

sudo ./platform_mpi-09.01.02.00rx64.bin

2.3.4 Launching a RADIOSS Computation

On a Single Machine

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/linux64/radflex_2017_linux64

[radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 -input [ROOTNAME]_0000.rad –np [n]

[radioss@host1~]$ [LP MPI path]/bin/mpirun -stdio=i0

-cpu_bind_mt=[ll | cyclic] -np [n]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 29

$ALTAIR_HOME/hwsolvers/radioss/bin/linux64/e_2017_linux64_plmpi -input [ROOTNAME]_0001.rad –nt [t]

Where

-stdio=i0: is to permit RADIOSS Engine to properly read standard input.

-cpu_bind_mt=[ll | cyclic]: to improve process placement over processors. This is useful for performance improvement. Use –cpu_bind_mt=ll on uniform memory system like Intel clovertown or Hartertown and –cpu_bind_mt=cyclic on non uniform memory system (ccNUMA) like Intel Nehalem or AMD Barcelona. Use –cpu_bind_mt only when [t] is lesser or equal to the number of cores per socket.

[PL MPI path]: is the path where PL MPI is installed

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain

[ROOTNAME]: is the rootname of the dataset

On a Cluster

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/common/bin/linux64/radflex_2017_linux64

[radioss@host1~]$ $ALTAIR_HOME/hwsolvers/radioss/bin/linux64/s_2017_linux64 –input [ROOTNAME]_0000.rad –np [Nspmd]

[radioss@host1~]$ [HP MPI path]/bin/mpirun -stdio=i0

Where

-np [Nspmd]: Specifies the number of SPMD domains for RADIOSS Starter

-stdio=i0: is to permit RADIOSS Engine to properly read standard input.

-cpu_bind_mt=11: to improve process placement over processors. This is useful for performance improvement. Use –cpu_bind_mt=ll on uniform memory system like Intel clovertown or Hartertown and –cpu_bind_mt=cyclic on non uniform memory system (ccNUMA) like Intel Nehalem or AMD Barcelona.

[PL MPI path]: is the path where PL MPI is installed

[-TCP|-GM|-MX|-ITAPI|-ELAN|-VAPI]: to set the interconnect to use. It is recommended to let the default interconnect. It is possible to use –prot mpirun option to check which interconnect is used. Please refer to PL MPI manual for more information.

[ROOTNAME]: is the rootname of the dataset

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 30

-nt[t]: is the number of SMP threads per SPMD domain

[appfile]: is a file describing each node of the computation. This file is as follows:

-h host1 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_linux64_plmpi -h host2 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_linux64_plmpi -h host3 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_linux64_plmpi ... -h hostp -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_linux64_plmpi

Where host1...hostp: are the hostnames of the different nodes.

[Computation path]: is the path where the dataset is launched.

[n]: is the number of SPMD domains on host p. The sum of all the SPMD domains given in the appfile must match the value given for the Starter –np [Nspmd] option.

[Engine path]: is the path to the RADIOSS Engine.

2.3.5 Troubleshooting

Improving Performance for Multi-Threading For hybrid version, it is important that system efficiently allocates threads to processor core. To allow efficient placement, the KMP_AFFINITY environment variable must not be set to compact. Let it unset.

Stacksize Limit stacksize limit must be set to unlimited with an unlimited value of more than 400Mbytes. When using the KMP_STACKSIZE environment variable (the stacksize per thread), the value must also be set to 400Mbytes.

Example under csh:

limit stacksize unlimited setenv KMP_STACKSIZE 400m

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 31 memlock limit

The file /etc/security/limits.conf should be checked and modified to contain the line below:

* soft memlock unlimited * hard memlock unlimited

This will avoid error in case the memlock limit is not enough.

Usage of –ndd HPMPI option under Infiniband

On some systems, it may be required to add the –ndd option to mpirun command to avoid segmentation violation. This type of error corresponds to a memory management error between system calls and Infiniband.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 32

2.4 Installing RADIOSS HMPP on Windows 64 Cluster using Intel MPI

2.4.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD Opteron systems under Windows 64 systems: Windows XP 64 Windows 7 Windows Server 2003 (64 bit), Windows Server 2005 (64 bit) Windows Server 2008 R1 Windows Server 2008 R2 b. The cluster interconnect is Infiniband. c. It is highly recommended to have the same hardware and OS level on all nodes. d. The MPI library is: Intel MPI 5.1.3 (mpi-rt_p_5.1.031.exe) or higher. The Intel MPI runtime kit is available for free at http://software.intel.com/en-us/intel-mpi-library/

2.4.2 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Windows 64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.4.3 Configuring the Machine

The account used to run RADIOSS must exist on every cluster's node.

RADIOSS executables must be reachable from all cluster nodes (same path on all nodes). The computation directory must be the same (same path) on all cluster nodes.

Intel MPI Installation Intel MPI Runtime Environment Kit for Windows is available on Intel website (www.intel.com) for free. The complete installation guide is available on Intel website.

2.4.4 Installing the Intel MPI Library

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 33

To install the Intel MPI Library, you need to use an account that is a member of the Administrators privilege group. Any normal account with Users, Debugger Users or higher user privilege can use the Intel MPI Library. You need to uninstall the previous version of the Intel MPI Library (if it is installed).

Note: The default installation directory for the Intel MPI Library is C:\Program Files (x86)\Intel\MPI\. for the Intel 64 architecture.

To install the Intel MPI Library on Windows* OS, do the following:

1. Double-click on the distribution file w_mpi-rt_p_..exe. 2. Choose a directory in which the contents of the self-extracting installation file will be placed before the actual installation begins.

After installation, the files will still be located in this directory. C:\Program Files (x86)\Intel \Download is used by default on the Intel 64 architecture. The Welcome dialog appears. 3. Click Next. 4. Read and accept the license agreement and click Next to continue. 5. You may be asked to provide a path to the license file. A license file that allows installation was sent to you by email along with your serial number. Click Next to continue. 6. Select the Complete radio button to install full version of the Intel MPI Library. To choose the desired components manually, select Custom. Click Next to continue. 7. The installation program can add the Intel® MPI Library installation directory to the PATH, LIB, and INCLUDE environment variables. Make your selection and click Next to continue. Alternatively, you may set up environment by running the files \em64t\bin\mpivars.bat. 8. When the installation completes, click Finish.

See the Intel MPI Library for Windows* OS Getting Started Guide for more information.

Launching the smpd Service

The command for launching multi-purpose daemons on Microsoft Windows is called “smpd”, which is an acronym for Simple Multi-Purpose Daemons.

1. To check if smpd service is correctly running, type the command:

smpd –status

Status should indicate that smpd is running.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 34

2. To restart the daemons, type:

smpd –restart

2.4.5 Installing on a Cluster

Microsoft* Windows* CCS 2003 The Intel MPI Library installer uses Microsoft* Compute Cluster Pack (CCP) facilities to determine cluster nodes and to install the Intel MPI Library on all of them. This feature is available on clusters with Microsoft* Windows* Compute Cluster Server 2003 with the Compute Cluster Pack (CCP) installed and configured package. Set the check box at the installation stage to choose single node or full cluster installation.

2.4.5.1 Installing on Microsoft* Windows 7*

The cluster installation feature is not accessible on clusters without the CCP package. Extract the .msi files and use them for silent installation to cluster nodes. Do the following to install the Intel MPI Library on Microsoft* Windows* XP or Microsoft* Windows Vista* based cluster:

1. For silent installations to cluster nodes, extract the .msi files. 2. Install the Intel MPI Library on single node.

3. Find the extracted distributive package at C:\Program Files (x86)\Intel\Download\w_mpi- rt_p_. on the Intel 64 based systems 4. Share this subfolder for read access from other nodes.

5. Place the license file in the same folder as the *.msi files. 6. Login to each cluster node and use the msiexec command to install the Intel MPI Library on them. (SDK) Intel 64 based systems

$> msiexec /i w_mpi_p__em64t.msi /q

Or, from the shared folder Intel 64 based systems

$> msiexec /i \\host_name\shared_folder\setup\w_mpi_p__em64t.msi /q

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 35

2.4.5.2 Installing on Microsoft Windows HPC Server 2008

On Windows HPC Server 2008 disable the User Accounts Control (UAC) during the Intel MPI Library installation.

Note:

• The default settings are used during installation through the .msi files. • Use the Microsoft* Remote Installation Service (RIS) or Microsoft* Windows Deployment Service (WDS) to simplify cluster installation.

PATH Environment Setting

The radflex_2017_win64 tool needs the liblmx-altair.13.0.0.so.dll library from the HyperWorks 2017 installation.

Add to the PATH environment variable : %ALTAIR_HOME%\hwsolvers\common\bin\win64

Stacksize Limit

KMP_STACKSIZE environment variable, the stacksize per thread, must also be set to 400Mbytes.

Example under DOS shell or cywin shell:

set KMP_STACKSIZE=400m

2.4.6 Launching a RADIOSS Computation

(Please see the Intel MPI Library for Windows* OS Reference Manual and Intel MPI Library for Windows* OS Getting Started Guide).

On a Single Machine

D:\Dataset> copy %ALTAIR_HOME%\hwsolvers\bin\win64\radflex2017_win64.exe D:\Dataset> %ALTAIR_HOME%\hwsolvers\bin\win64\s_2017_win64.exe –input [ROOTNAME]_0000.rad –np [n] D:\Dataset> [Intel_MPI_path]\bin\mpiexec –delegate -envall -n [n] %ALTAIR_HOME% \hwsolvers\bin\win64\e_2017_win64_impi.exe –input [ROOTNAME]_0001.rad –nt [t]

Where

[Intel_MPI_path]: is the path where Intel MPI is installed. Default Intel MPI installation path is C: \Program Files (x86)\Intel\MPI\5.1.3\em64t

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 36

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain

[ROOTNAME]: is the rootname of the dataset

-delegate is an option to enable the domain-based authorization with the delegation ability. If this option is not used, you may have to enter the user’s password several times

-envall is an option to pass all environment variables in current environment

On a Windows 64 Cluster

D:\Dataset> copy %ALTAIR_HOME%\hwsolvers\common\bin\win64\radflex2017_win64.exe D:\Dataset> %ALTAIR_HOME%\hwsolvers\radioss\bin\win64\s_2017_win64.exe –input [ROOTNAME]_0000.rad –np [Nspmd] D:\Dataset> [IntelMPI path]\bin\mpiexec –delegate –envall -genv I_MPI_DEVICE [sock|rdssm] -wdir [Computation path] –map [drive:\\host\share] -n[n] -machinefile [machinefile] %ALTAIR_HOME%\hwsolvers\radioss\bin\win64\e_2017_win64_impi.exe -input [ROOTNAME]_0001.rad –nt[t]

Where

[Intel_MPI_path]: is the path where Intel MPI is installed. Default Intel MPI installation path is C: \Program Files (x86)\Intel\MPI\5.1.3\em64t

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain

[ROOTNAME]: is the rootname of the dataset

[Computation path]: is the path where the dataset is launched.

-genv I_MPI_DEVICE is an option to select the particular network fabrics to be used.

[sock|rdssm]: to set the interconnect to use (sock for TCP/IP-capable network, such as Ethernet; rdssm for shared-memory+DAPL such as InfiniBand). For other type of interconnect please consult Intel MPI Library for Windows* OS Reference Manual.

-map is an option to create network mapped drive on nodes before starting executable. Network drive will be automatically removed after the job completion.

[machinefile]: file which describes the number of SPMD programs per node. For example,

host1:n1 host2:n2 host3:n3 . .

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 37

. hostp:np

Where host1...hostp: are the hostnames of the different nodes.

-[np]: is the number of SPMD domains on host p. The sum of all the SPMD domains given in the machinefile must match the value given for the Starter –np [n] option.

2.4.7 Troubleshooting

Use of Standard-in Redirection in Intel MPI for Windows64

Do not redirect the Engine input file, but use the engine –input command line argument. The redirection may cause an RADIOSS Engine to hang.

2.4.7.1 Setting up SMPD Services

The Intel MPI Library uses a Simple Multi-Purpose Daemon (SMPD) job startup mechanism. In order to run programs compiled with Microsoft* Visual Studio* (or related), set up a SMPD service. You should have administrator privileges to start the SMPD service and all users can launch processes with mpiexec. During the Intel MPI Library installation, the SMPD service is started. During installation you can cancel the SMPD service startup.

1. To start, restart, stop or remove the SMPD service manually when the Intel MPI Library is installed, find smpd.exe in the \em64t\bin. 2. To remove any previous SMPD service, use > smpd.exe –remove 3. To install the SMPD service manually, use the following command on each node of the cluster: > smpd.exe –install

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 38

2.5 Installing RADIOSS HMPP on Windows Server 2008 R2 Computer Cluster using MS MPI

2.5.1 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Windows 64.

2. On top of the RADIOSS installation, extract the files from rd_spmd2017_win64.zip into the same directory where RADIOSS is installed. 3. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.5.2 Configuring the Machine

This document assumes that WINDOWS HPC PACK 2008 R2 has been properly installed and configured. All computation nodes are seen in the "Compute Cluster Administrator." Specifically, a head node and one or more compute nodes have to be configured. It is assumed that all jobs are submitted from the Compute Cluster Administrator window on the head node to be run on the compute nodes.

The executable and the RADIOSS data sets need to be reachable by the cluster.

Data sets and RADIOSS executing environment must be accessible from all nodes (computation and master node) using shared file systems.

To set cluster-wide environment variables (example: ALTAIR_LM_LICENSE_FILE) use the cluscfg command. This tool requires cluster administrative rights. Refer to the Compute Clusters User’s Guide for more information.

PATH Environment Settings

The radflex_2017_win64 tool needs the liblmx-altair.13.0.0.so.dll library from the HyperWorks 2017 installation.

Add to the PATH environment variable : %ALTAIR_HOME%\hwsolvers\common\bin\win64

2.5.3 Launching a RADIOSS Computation

Earlier releases of MS MPI have problems properly reading the engine [ROOTNAME]_0001.rad input file. RADIOSS Engine hangs at start up. This problem is corrected in Windows Compute Cluster Pack release 1.0.0676.14.

To bypass this problem, you must add /END and one "#" character on the next line at the end of the Engine input file.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 39

For example

# /VERS/51 /RUN/CRA2V31/1 20.0 /ANIM/DT 0.0 2.0 /TFILE 0.05 /END #

2.5.4 The Compute Clusters Pack Job

The Compute Clusters Pack job is an XML template file. This file is loaded in the Compute Cluster Job Manager:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 40

MinCores="1" MaxCores="1" MinSockets="1" MaxSockets="1" MinNodes="1" MaxNodes="1" NiceId="1" CommandLine="copy %ALTAIR_HOME%\hwsolvers\radioss\bin \win64\radflex_2017_win64.exe ." WorkDirectory="D:\Olivier\Radioss_execution" StdOutFilePath="Radflex.out" StdErrFilePath="Radflex_err.out" HasCustomProps="false" IsParametric="false" GroupId="69" ParentJobState="Configuring" UnitType="Core" ParametricRunningCount="0" ParametricCanceledCount="0" ParametricFailedCount="0" ParametricQueuedCount="0" />

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 41

The amount of needed processors is set in the job settings:

MaximumNumberOfProcessors="4" MinimumNumberOfProcessors="4". Both entries must be equal and set to the total number of cores needed (= n x t, with n number of SPMD domains, and t number of SMP threads per domain)

Replace:

• %WORKDIR% with the path to the working directory. The path must be absolute and include the file server. Example: //fileserver/C$/EXECUTION • [ROOTNAME] with the dataset rootname This template defines four tasks:

• The first and second task is to copy radflex2017_win64.exe and liblmx-altair.13.0.0.so.dll from the installation path to the work directory. Change the WorkDirectory entry according to the place where your data set runs. Use network path syntax. • The third task is to launch RADIOSS Starter. Change the Stdin field to set the Starter input data set and the WorkDirectory entry according to the place where your data set runs. Use network path syntax. • The fourth task launches RADIOSS Engine. Set MaximumNumberOfProcessors and MinimumNumberOfProcessors to the number of needed processors. Both values must be equal. Set the WorkDirectory entry according to the place where your data set runs (use network path syntax) and then set the Stdin field to set the Engine input data set.

2.5.5 Launching the Compute Cluster Job Manager

The Compute Cluster Job Manager program is used to launch a job. This program submits the RADIOSS job with the Windows Compute Cluster Pack Queue manager.

1. To load the XML template job, click File > Create New Job From Description file, and select the .xml file.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 42

Figure 2:

The New Job dialog opens. This dialog allows you to verify the settings stored in the .xml template file. Changes can be made to correct the job. The corrected job can be submitted and stored. 2. In thew New Job dialog, set the following fields: • Job and project name • Type of resources • Minimum and Maximum processors required to run the job. These must be equal.

Figure 3:

3. In the left pane, click on Task List.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 43

Figure 4:

4. Check in the Starter Task by populating the following fields: • dataset ROOTNAME • Numbers of CPUs to execute • Working directory

Figure 5:

5. Check in the Engine Task by populating the following fields: • dataset ROOTNAME • Numbers of CPUs to execute (min and max fields must be equal) • Working directory

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 44

Figure 6:

6. After all changes are complete, save the job as an XML Template file, and submit it.

Tip: You can use the Job Queue Manager to check the progress of your job.

Figure 7:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 45

2.6 Installing RADIOSS HMPP on Windows 64 Cluster using Platform MPI 9.1.2

2.6.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD Opteron systems under Windows 64 systems: Windows XP 64 Windows 7 Windows Server 2003 (64 bit), Windows Server 2005 (64 bit) Windows Server 2008 R1 Windows Server 2008 R2 b. The cluster interconnect is Infiniband. c. It is highly recommended to have the same hardware and OS level on all nodes. d. The MPI library is: Platform MPI 9.1.2 or higher, and must be installed on all the cluster nodes. Contact IBM to obtain IBM Platform MPI 9.1.2.

2.6.2 Installing the Software and Activating the License

1. Install HyperWorks 2017 package for Windows 64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

2.6.3 Configuring the Machine

The account used to run RADIOSS must exist on every cluster's node.

RADIOSS executables must be reachable from all cluster nodes (same path on all nodes). The computation directory must be the same (same path) on all cluster nodes.

PL MPI Installation PL MPI is distributed by HP as a Windows executable. It must be installed having Administrator privilege on each node executing the package:

Platform-MPI-09.1.2.0-rc8.x64.exe

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 46

PATH Environment Setting

The radflex_2017_win64 tool needs the liblmx-altair.13.0.0.so.dll library from the HyperWorks 2017 installation.

Add to the PATH environment variable : %ALTAIR_HOME%\hwsolvers\common\bin\win64

2.6.4 Launching a RADIOSS Computation

On a Single Machine

D:\Dataset> copy %ALTAIR_HOME%\hwsolvers\common\bin\win64\radflex2017_win64.exe

D:\Dataset> %ALTAIR_HOME%\hwsolvers\radioss\bin\win64\s2017_win64.exe –input [ROOTNAME]_0000.rad –np [n]

D:\Dataset> [PL MPI path]\bin\mpirun -cpu_bind=ll -np [n] %ALTAIR_HOME%\hwsolvers \radioss\bin\win64\e2017_win64_plmpi –input [ROOTNAME]_0001.rad –nt [t]

Where

-cpu_bind=ll: to improve process placement over processors. This is useful for performance improvement. Use –cpu_bind_mt=ll on uniform memory system like Intel clovertown or Hartertown and –cpu_bind_mt=cyclic on non uniform memory system (ccNUMA) like Intel Nehalem or AMD Barcelona

[PL MPI path]: is the path where PL MPI is installed. Default PL-MPI installation path is C:\Program Files (x86)\Hewlett-Packard\HP-MPI

[n]: is the number of SPMD domains

[t]: is the number of SMP threads per SPMD domain

[ROOTNAME]: is the rootname of the dataset

On a Windows 64 Cluster

D:\Dataset> copy %ALTAIR_HOME%\hwsolvers\common\bin\win64\radflex2017_win64.exe D:\Dataset> %ALTAIR_HOME%\hwsolvers\radioss\bin\win64\s_2017_win64.exe –input [ROOTNAME]_0000.rad –np [Nspmd] D:\Dataset> [PL MPI path]\bin\mpirun - cpu_bind_mt=[ll | cyclic] [-TCP|-IBAL|-MX] -f appfile -- -input [ROOTNAME]_0001.rad –nt [t]

Where

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 47

-np [Nspmd]: Specifies the number of SPMD domains for RADIOSS Starter.

-stdio=i0: allows RADIOSS Engine to properly read Standard-In

[PL MPI path]: is the path where PL MPI is installed. Default PL-MPI installation path is C:\Program Files (x86)\Platform Computing\Platform-MPI

[-TCP|-IBAL|-MX]: to set the Interconnect to use

[ROOTNAME]: is the rootname of the dataset

-nt[t]: is the number of SMP threads per SPMD domain

[appfile]: file which describes each nodes of the computation. The file is as follows: For example,

-h host1 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_win64_plmpi.exe

-h host2 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_win64_plmpi.exe

-h host3 -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_win64_plmpi.exe

...

-h hostp -e MPI_FLAGS=y,E -e MPI_WORKDIR=[Computation path] -np [n] [Engine path]/ e_2017_win64_plmpi.exe

Where host1...hostp: are the hostnames of the different nodes.

[Computation path]: is the path where the dataset is launched.

[n]: is the number of SPMD domains on host p. The sum of all the SPMD domains given in the appfile must match the value given for the Starter –np [Nspmd] option.

[Engine path]: is the path to the RADIOSS Engine.

2.6.5 Troubleshooting

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 48

Use of Standard-in Redirection in PL MPI for Windows64

Do not redirect the Engine input file, but use the engine –input command line argument. The redirection may cause an RADIOSS Engine to hang.

Proprietary Information of Altair Engineering OptiStruct SPMD Installation Overview 3

OptiStruct SPMD Installation Overview Single Program, Multiple Data (SPMD) is a parallelization technique in computing that is employed to achieve faster results by splitting the program into multiple subsets and running them simultaneously on multiple processors/machines. SPMD typically implies running the same process or program on different machines (nodes) with different input data for each individual task. However, SPMD can sometimes be implemented on a single machine with multiple processors depending upon the program and hardware limitations/requirements. In this section, discussion about SPMD typically includes the application of Shared Memory Parallelization (SMP) in conjunction with the MPI-based parallelization. Typically, this combination can be termed as Hybrid Shared/Distributed Memory Parallelization, and it will henceforth be referred to as SPMD.

This chapter covers the following:

• 3.1 Load Decomposition Method (p. 50) • 3.2 Domain Decomposition Method (p. 55) • 3.3 Multi-Model Optimization (p. 58) • 3.4 Failsafe Topology Optimization (p. 60) • 3.5 Supported Platforms (p. 62) • 3.6 Frequently Asked Questions (p. 63) • 3.7 OptiStruct SPMD Setup (p. 66) HyperWorks 2017 Advanced Installation Guide 50

3.1 Load Decomposition Method

Load Decomposition Method (LDM) in OptiStruct can be used when a run is distributed into parallel tasks, as shown in Figure 8. The schematic shown is applicable to a Load Decomposition Method run on multiple machines. The entire model is divided into parallelizable subcase. Table 2 lists the various supported solution sequences and parallelizable steps.

Figure 8: Overview of the LDM implementation in OptiStruct

In LDM, the model (analysis/optimization) is split into several tasks, as shown in Figure 8. The tasks are assigned to various nodes that run in parallel. Ideally, if the model is split into N parallel tasks, then (N+1) nodes/machines would be required for maximum efficiency. (This is dependent on various other factors like: type of tasks, processing power of the nodes, memory allocation at each node and so on. During a LDM run, using more than (N+1) nodes for N parallelizable tasks would not increase efficiency.) The extra node is known as the manager node. The manager node decides the nature of data assigned to each node and the identity of the master node. The manager node also distributes multiple input decks and tasks to various nodes. It does not require a machine with high processing power, as no analysis or optimization is run on the manager node. The master node, however, requires a higher amount of memory, since it contains the main input deck and it also collects all results and

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 51 performs all processes that cannot be parallelized. Optimization is run on the master node. The platform dependent Message Passing Interface (MPI) helps in the communication between various nodes and also between the master node and the slave nodes.

Note: • A task is a minimum distribution unit used in parallelization. Each buckling analysis subcase is one task. Each Left-Hand Side (LHS) of the static analysis subcases is one task. Typically, the static analysis subcases sharing the same SPC (Single Point Constraint) belong to one task. Note that not all tasks can be run in parallel at the same time. For example, a buckling subcase can not start before the execution of its STATSUB subcase. • The manager can also be included within the master node by specifying np = N+1 for N nodes and repeating the first node in the appfile/hostfile in a cluster setup (-np option, appfile/hostfile are explained in the following sections).

3.1.1 Supported Solution Sequences

OptiStruct can handle a variety of solution sequences as seen in the overview. However, all solution sequences do not lend themselves to parallelization. In general, many steps in a program execution are not parallelized. Steps like pre-processing and matrix assembly are repeated on all nodes, while response recovery, screening, approximation, optimization and output of results are all executed on the master node.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 52

Table 3: Load Decomposition Method - Parallelizable Steps for Various Solution Sequences

Solution Sequences that Parallelizable Steps Non-Parallelizable Steps Support Parallelization

Static Analysis a. Two or more static a. Iterative Solution is not boundary conditions parallelized. (only Direct are parallelized (matrix Solution is parallelized) factorization is the step that is parallelized since it is computationally intensive.) b. Sensitivities are parallelized (even for a single boundary condition as analysis is repeated on all slave nodes)

Buckling Analysis a. Two or more buckling subcases are parallelized b. Sensitivities are parallelized

Direct Frequency Response a. Loading frequencies are Analysis parallelized b. No optimization

Modal Frequency Response a. Loading frequencies are a. Modal FRF pre-processing Analysis parallelized is not parallelized b. Sensitivities are not parallelized

In HyperWorks 11.0 and later, the presence of non-parallelizable subcases WILL NOT make the entire program non-parallelizable. The program execution will continue in parallel and the non-parallelizable subcase will be executed as a serial run.

3.1.2 Number and Type of Nodes Available for Parallelization

The types and functions of the nodes that are used in Load Decomposition Method are indicated in Table 4. The first node is automatically selected as the manager, the second node is the master node and the rest are slave nodes.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 53

Table 4: Types and Functions of the Nodes

Node Type Functions

Master node • Runs all non-parallelizable tasks (1 node) • Optimization is run here

Slave node • Runs all parallelizable tasks (N-2 nodes) • Input deck copies are provided

Manager node • No tasks are run on this node; it manages (1 node) the way nodes are assigned tasks. • Manager makes multiple copies of the input deck and sends them to the slave nodes.

This assignment is based on the sequence of nodes that the user specifies in the .appfile. The .appfile is a text file which contains process counts and the list of programs. Nodes can be repeated in the .appfile, multiple cores of the repeated nodes will be assigned parallel jobs in the same sequence discussed here.

3.1.3 Frequently Asked Questions

How many nodes should I use for LDM? LDM parallelization is based on task distribution. If the maximum number of tasks which can be run at the same time is K, then using (K+1) nodes is ideal (the extra one node for the manager distributing tasks). Using more than (K+1) nodes will not improve the performance.

The .out file suggests the number of nodes you can use, based on your model (Figure 2). When there are only M physical nodes available (M < K), the correct way to start the job is to use M+1 nodes in the hostfile/appfile. The manager node requires only a small amount of resources and can be safely shared with the master. The way to assign such a distribution is repeating the first physical host in the hostfile/appfile.

For example, hostfile for Intel MPI can be:

Node1 Node1 Node2 Node3

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 54

…….

Note: For Frequency Response Analysis any number of nodes may be used (up to the number of loading frequencies in the model) but less than or equal to the number of physical processors (or sockets) in the cluster.

How do I run LDM on a dual/quad core machine?

The ideal number of nodes is min(K+1, M), where K is the maximum number of tasks that can be run at the same time, and M is the number of physical processor (or sockets).

Note: For dual/quad code machines, it may be more efficient to run OptiStruct in serial + SMP mode, depending on the available memory (use –nt argument in the solver script).

Will LDM use less memory on each node than in the serial run? No, memory estimates for serial runs and parallel runs on each node are the same. They are based on the solution of a single (most demanding) subcase.

Will LDM use less disk space on each node than in the serial run? Yes. Disk space usage on each node will be smaller, because, only temporary files related to task(s) solved on this node will be stored. But the total amount of disk space will be larger than that in the serial run and this can be noticed, especially in parallel runs on a shared-memory machine.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 55

3.2 Domain Decomposition Method

In addition to Load Decomposition Method, OptiStruct SPMD includes another approach for parallelization called Domain Decomposition Method (DDM) for static analysis and optimization. DDM allows you to run a single subcase of Static Analysis and/or optimization with multiple processors in either Shared Memory Processing (SMP) or Distributed Memory Processing (DMP) cluster computers. The solution time will be significantly reduced in DDM mode and the scalability is much higher compared to the legacy shared memory processing parallelization approach, especially on machines with a high number of processors (for example, greater than 8).

Figure 9: Example illustrating Graph Partitioning for the DDM implementation in OptiStruct

The DDM process utilizes graph partition algorithms to automatically partition the geometric structure into multiple domains (equal to the number of MPI nodes). During FEA analysis/optimization, an individual domain only processes its domain related calculations. Such procedures include element matrix assembly, linear solution, stress calculations, sensitivity calculations, and so on. The necessary communication across domains is accomplished by OptiStruct and is required to guarantee the accuracy of the final solution. When the solution is complete, result data is collected and output to a single copy of the .out file. From the user’s perspective, there will be no difference between DDM and serial runs in this aspect.

3.2.1 Supported Solution Sequences

Linear and Nonlinear Static Analysis/Optimization Solution Sequences are generally supported. The following solutions, however, are currently not supported: a. Static Analysis (iterative solver) b. Level Set Method (static optimization)

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 56

c. Preloading (static analysis)

Note:

a. The –ddm run option can be used to activate DDM. b. In DDM mode, there is no distinction between node types (for example, manager node, master node, slave node, and so on). All nodes are considered as working nodes. If –np n is specified, OptiStruct partitions n geometric domains and assigns each domain to one MPI node.

c. Hybrid computation is supported. –nt can be used to specify the number of threads in an SMP run. Sometimes, hybrid performance may be better than pure MPI or pure SMP mode, especially for blocky structures. It is also recommended that the total number of cores (n x m) should not exceed the physical cores of the machine.

3.2.2 Frequently Asked Questions

How many MPI processes (-np) and threads (-nt) per process should I use for DDM runs? This depends on whether DDM is run in a cluster with separate machines or in a shared memory machine with multiple processors.

Shared memory machine with multiple processors: • It is important to try avoiding an out-of-core solution in a shared memory machine with multiple processors. Otherwise multiple processors will compete with the system I/O resources and slow down the entire solution. Use –core in and limit the number of MPI processes -np to make sure OptiStruct runs in the in-core mode. The number of MPI processes –np is usually dictated by the memory demand. When -np is set, you can determine the number of threads per MPI process based on the total number of cores available.

Note: Total memory usage is the summation of OptiStruct usage listed in memory estimation section in the .out file and the MUMPS usage listed in .stat file. The MUMPS estimation may also be listed in the memory estimation section of the .out file for later releases.

Cluster with separate machines:

• A generally cautious method is to specify the number of threads -nt equal to the number of cores per socket, and to specify the number of MPI processes -np equal to the number of sockets in each machine. For example, if a node is equipped with two typical Intel Xeon Ivy bridge CPUs,

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 57

you can set two MPI processes per node (-np=2), and 12 cores per MPI process (-nt=12) since a typical Ivy bridge CPU consists of 12 cores.

Will DDM use less memory on each node than in the serial run? Yes, memory per MPI process for a DDM solution is significantly reduced compared to serial runs. DDM is designed for extremely large models on machine clusters. The scaling of out-of-core mode on multiple nodes is very good because the total I/O amount is distributed and the smaller I/O is better cached by system memory.

Will DDM use less disk space on each node than in the serial run? Yes. Disk space usage is also distributed.

Can DDM be used in Normal Mode Analysis and Dynamic Analysis? No, DDM is only supported for Static Analysis.

Can DDM be used in Optimization runs? Yes, DDM can be used in Static Analysis and Optimization.

Can DDM be used if I have only one subcase? Yes, the solver utilizes multiple processors to perform matrix factorizations and analysis.

If I have multiple static subcases, should I use LDM or DDM? This depends on the available memory and disk-space resources. LDM will not reduce the memory and disk-space requirement for each MPI process. If memory is a bottleneck, DDM is recommended. Additionally, if multiple subcases share the same boundary conditions, LDM will not parallelize the model effectively and again, and DDM is recommended in this case.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 58

3.3 Multi-Model Optimization

In addition to Load Decomposition Method and Domain Decomposition Method , OptiStruct SPMD includes another approach for MPI-based parallelization called Multi-Model Optimization (MMO) for optimization of multiple structures with common design variables in a single optimization run.

ASSIGN, MMO can be used to include multiple solver decks in a single run. Common design variables are identified by common user identification numbers in multiple models. Design variables with identical user identification numbers are linked across the models. Responses in multiple models can be referenced via the DRESPM continuation lines on DRESP2/DRESP3 entries. Common responses in different models can be qualified by using the name of the model on the DRESPM continuation line. The model names can be specified via ASSIGN, MMO for each model.

Figure 10: Example use case for Multi-Model Optimization

Multi-Model Optimization is a MPI based parallelization method, requiring OptiStruct MPI executables for it to run. Existing solver decks do not need any additional input, can be easily included, and are fully compatible with the MMO mode. MMO allows greater flexibility to optimize components across structures. The –mmo run option can be used to activate MMO in OptiStruct.

3.3.1 Supported Solution Sequences

a. All optimization types are currently supported. b. Multi-body Dynamics (OS-MBD) and Geometric Nonlinear Analysis (RADIOSS Integration) are currently not supported. c. MMO currently cannot be used in conjunction with the Domain Decomposition Method . d. The DTPG and DSHAPE entries are supported, however linking of design variables is not. For example, it makes no difference to the solution if multiple DSHAPE entries in different slave files contain the same IDs or not.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 59

Note: a. The number of processes should be equal to one more than the number of models. b. Refer to Launching OptiStruct SPMD for information on launching Multi-Model Optimization in OptiStruct. c. If multiple objective functions are defined across different models in the master/ slaves, then OptiStruct always uses minmax [Objective(i)] (where, i is the number of objective functions) to define the overall objective for the solution. d. The following entries are allowed in the master deck: Control Cards SCREEN, DIAG/OSDIAG, DEBUG/OSDEBUG, TITLE, ASSIGN, RESPRINT, DESOBJ, DESGLB, REPGLB, MINMAX, MAXMIN, ANALYSIS, LOADLIB

Bulk Data Cards DSCREEN, DOPTPRM (see section below), DRESP3, DRESP3, DOBJREF, DCONSTR, DCONADD, DREPORT, DREPADD, DEQATN, DTABLE, PARAM

DOPTPRM parameters (these work from within the master deck – all other DOPTPRM’s should be specified in the slave) CHECKER, DDVOPT, DELSHP, DELSIZ, DELTOP, DESMAX, DISCRETE, OBJTOL, OPTMETH, SHAPEOPT

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 60

3.4 Failsafe Topology Optimization

OptiStruct SPMD includes another approach for MPI-based parallelization called Failsafe Topology Optimization (FSO) for topology optimization of structures. Regular Topology Optimization runs may not account for the feasibility of a design in situations where a section of the structure fails. FSO divides the structure into damage zones and generates multiple models (equal to the number of failure zones), wherein each model is the same as the original model minus one failure zone. In this process, the FSO method is applied by running Topology Optimization simultaneously for all such generated models and a final design is output which is optimized to account for all generated models.

Typically, the number of damage zones is large, which means the number of SPMD domain is large. Such a job needs to be run on multiple nodes with cluster setup.

Activation a. The FAILSAFE continuation line on the DTPL Bulk Data entry, the failsafe topology script run option (-fso), and the number of processors (-np) can be used to activate Failsafe Topology Optimization. For example

/Altair/hwsolvers/script/optistruct filename.fem –np 20 –fso.

Note: The option for executable, which is equivalent to script option –fso, is - fsomode.

b. The number of processors should be set equal to the number of damage zones in the original model + 1. To determine the number of damage zones, you can run the Topology Optimization model (with FAILSAFE continuation line) in serial mode (or a check run) and look at the .out file. The number of MPI processes for Failsafe Optimization is displayed in the .out file. This determines the number of processors (–np) for the subsequent MPI (–fso) run. Refer to Launching FSO for information on launching Failsafe Topology Optimization in OptiStruct.

Output

a. Separate _FSOi folders are created for damage zone. Each folder contains the full Topology Optimization results for the corresponding model. For example, the folder _FSO1 would contain topology results (.out, .stat, .h3d, _des.h3d files, and so on) for the first damaged model (original topology model minus the first damage zone), and so on.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 61

b. In the main working directory, the Damage Zones are output for both the first layer and the overlap layer (if it is not deactivated) to the _fso.h3d file. The Damage Zones can then be visualized in HyperView. c. Additionally, in the main working directory, the final Failsafe Topology Optimization results are output to the _des.h3d file. It is recommended to compare these results with the initial non-Failsafe Topology results to get a sense of the modified design.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 62

3.5 Supported Platforms

Supported platforms and MPI versions for OptiStruct SPMD are listed in Table 5:

Table 5: Supported Platforms for OptiStruct SPMD

Application Version Supported Platforms MPI

OptiStruct SPMD 2017 Linux 64 bit IBM Platform MPI 9.1.2 (formerly HP-MPI) or

Intel MPI - Version 5.1.3 (223)

Windows 64 bit IBM Platform MPI 9.1.2 (formerly HP-MPI) or

Intel MPI - Version 5.1.3

MS MPI – Version 5.0.12435.6

Note: OptiStruct does not support MPICH for MPI-based runs.

SPMD in OptiStruct is implemented by the following MPI-based functions: • Load Decomposition Method • Domain Decomposition Method • Multi-Model Optimization • Failsafe Topology Optimization

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 63

3.6 Frequently Asked Questions

Can I run OptiStruct SPMD over LAN? It is possible to run OptiStruct SPMD over LAN. Follow the corresponding MPI manual to setup different working directories of each node the OptiStruct SPMD is launched.

Is it better to run on cluster of separate machines or on shared memory machine(s) with multiple CPU’s? There is no single answer to this question. If the computer has sufficient memory to run all tasks in- core, expect faster solution times as MPI communication is not slowed down by the network speed. But if the tasks have to run out-of-core, then computations are slowed down by disk read/write delay. Multiple tasks on the same machine may compete for disk access, and (in extreme situations) even result in wall clock time slower than that for serial (non-MPI) runs.

How do I determine the number of nodes that I can use for each of the three MPI-based parallelization processes?

The flowchart below provides a quick reference to determine the number of nodes for each parallelization process.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 64

Figure 11:

How many MPI processes (-np) and threads (-nt) per process should I use for OptiStruct SPMD? To run parallel MPI processes, distributed memory (with parallel access) is essential. If a single node contains multiple sockets (each with a single Processor), then theoretically, an equivalent number of MPI processes (equal to the number of sockets) can be run on the node, provided sufficient RAM is available to handle all MPI processes simultaneously in parallel. However, if sufficient distributed memory is not available in the RAM, then it is typically more efficient to use Shared Memory Parallelization (SMP) instead of SPMD and use multiple logical processors/cores within the node in parallel via the –nt run option. When each node has sufficient RAM to execute only a single serial OptiStruct run, use multiple logical processors/cores to activate SMP on each node (using more than four cores, -nt=4 is usually not effective).

For example:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 65

On a 4 node cluster (with 2 sockets each) and if overall each node contains 8 cores, you can run:

Insufficient RAM:

optistruct -ldm -np 4 –nt 4

Sufficient RAM:

optistruct -ldm -np 8 –nt 4

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 66

3.7 OptiStruct SPMD Setup

Follow the steps detailed in this section to install and launch OptiStruct SPMD.

3.7.1 Installing OptiStruct SPMD on Linux Machines

3.7.1.1 System Requirements

a. Operating system: Linux 64 b. The MPI library: IBM Platform-MPI (Formerly HP-MPI) or Intel MPI are included with the HyperWorks 2017 installation. If the HyperWorks installation is available on all machines in the cluster, then both IBM Platform MPI 9.1.2 and Intel MPI are preinstalled (at $ALTAIR_HOME/mpi) and available for SPMD-cluster runs.

3.7.1.2 Launching OptiStruct SPMD

There are several ways to launch parallel programs with OptiStruct SPMD. Remember to propagate environment variables when launching OptiStruct SPMD, if needed. Please refer to the respective MPI vendor’s manual for more details. In OptiStruct 14.0, commonly used MPI runtime software are automatically included as a part of the HyperWorks installation. The various MPI installations are located at $ALTAIR_HOME/mpi.

Note: • A minimum of 3 processes are needed to launch OptiStruct SPMD. • OptiStruct SPMD must match the MPI implementation you use.

Using Solver Scripts On a single host (for IBM Platform MPI 9.1.2 (formerly HP-MPI)) using solver script: • Load Decomposition Method

[optistruct@host1~]$ $ALTAIR_HOME/scripts/optistruct –mpi [MPI_TYPE] -ldm –np [n] [INPUTDECK] [OS_ARGS]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 67

• Domain Decomposition Method

[optistruct@host1~]$ $ALTAIR_HOME/scripts/optistruct –mpi [MPI_TYPE] –ddm –np [n] [INPUTDECK] [OS_ARGS]

• Multi-Model Optimization

[optistruct@host1~]$ $ALTAIR_HOME/scripts/optistruct –mpi [MPI_TYPE] –mmo –np [n] [INPUTDECK] [OS_ARGS]

• Failsafe Topology Optimization

[optistruct@host1~]$ $ALTAIR_HOME/scripts/optistruct [INPUTDECK] [OS_ARGS] –mpi [MPI_TYPE] -fso –np [n]

Where

[MPI_TYPE]: is the MPI implementation used: pl for IBM Platform MPI 9.1.2 i for Intel MPI

-- ( [MPI_TYPE]is optional, default MPI implementation on Linux machines is i

[n]: is the number of processors

[INPUTDECK]: is the input deck file name

[OS_ARGS]: lists the arguments to OptiStruct

-- ([OS_ARGS] is optional)

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 68

Note:

• Adding the command line option -testmpi runs a small program which verifies whether your MPI installation, setup, library paths and so on are accurate. • OptiStruct SPMD can also be launched using the Run Manager GUI. • It is also possible to launch OptiStruct SPMD without the GUI/solver scripts.

• Adding the optional command line option –mpipath PATH helps the user to find the MPI installation if it is not included in the current search path or when multiple MPI’s are installed.

• If no SPMD run option (-ldm/-mmo/-ddm/-fso) is specified, then LDM is run by default

3.7.1.3 Launching OptiStruct SPMD Using Direct Calls to the Executable

On a single host (for IBM Platform MPI 9.1.2 and Intel MPI: • Load Decomposition Method

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/ linux64/optistruct_spmd [INPUTDECK] [OS_ARGS] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/ linux64/optistruct_spmd [INPUTDECK] [OS_ARGS] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/ linux64/optistruct_spmd [INPUTDECK] [OS_ARGS] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/ linux64/optistruct_spmd [INPUTDECK] [OS_ARGS] -fsomode

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 69

Where optistruct_spmd is the OptiStruct SPMD executable and can be one of:

Linux Intel MPI Executable: = optistruct_14.0_linux64_impi Linux IBM Platform MPI Executable: = optistruct_14.0_linux64_plmpi

[n]: is the number of processors

[INPUTDECK]: is the input deck file name

[OS_ARGS]: lists the arguments to OptiStruct other than –mpimode

-- ([OS_ARGS] is optional. Please refer to Run Options for further information)

Note: Running OptiStruct SPMD using direct calls to the executable requires an additional command line option –mpimode/-ddmmode/-mmomode/-fsomode (as shown above). If one of these run options is not used, there will be no parallelization and the entire program will be run on each node.

On a Linux Cluster (for IBM Platform MPI 9.1.2): • Load Decomposition Method

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_2017_linux64_plmpi [INPUTDECK] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 70

$ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/ optistruct_2017_linux64_plmpi [INPUTDECK] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/ optistruct_2017_linux64_plmpi [INPUTDECK] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/ optistruct_2017_linux64_plmpi [INPUTDECK] -fsomode

Where

: is a text file which contains process counts and a list of programs.

Example: 4 CPU job on 2 dual-CPU hosts (the two machines are named: c1 and c2) using IBM Platform MPI 9.1.2. The following example shows the contents of the :

[optistruct@host1~]$ cat hostfile

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 71

-h c1 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/ optistruct_2017_linux64_plmpi [INPUTDECK] -mpimode -h c2 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/ optistruct_2017_linux64_plmpi [INPUTDECK] –mpimode

On a Linux Cluster (for IBM Intel MPI): • Load Decomposition Method

[[optistruct@host1~]$ mpirun –f -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_2017_linux64_impi [INPUTDECK] [OS_ARGS] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpirun –f -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_2017_linux64_impi [INPUTDECK] [OS_ARGS] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpirun –f -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_2017_linux64_impi [INPUTDECK] [OS_ARGS] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpirun –f -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_2017_linux64_impi [INPUTDECK] [OS_ARGS] -fsomode

Where

: is a text file which contains process counts and a list of programs. Line format is as follows:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 72

Note: • One host requires only one line. • Running OptiStruct SPMD using direct calls to the executable requires an additional command line option –mpimode/-ddmmode/-mmomode/-fsomode (as shown above). If one of these run options is not used, there will be no parallelization and the entire program will be run on each node.

Example: 4 CPU job on 2 dual-CPU hosts (the two machines are named: c1 and c2). The following example shows the contents of the :

[optistruct@host1~]$ cat hostfile c1 c2

3.7.2 Installing OptiStruct SPMD on Windows Machines

3.7.2.1 System Requirements

a. Operating system: Windows Vista/7/8.1; 64-bit only b. The MPI library: IBM Platform-MPI (Formerly HP-MPI) or Intel MPI are included with the HyperWorks 2017 installation. If the HyperWorks installation is available on all machines in the cluster, then both IBM Platform MPI 9.1.2 and Intel MPI are preinstalled (at $ALTAIR_HOME/mpi) and available for SPMD-cluster runs. The Microsoft MPI (MS-MPI) is available as an installation package ($ALTAIR_HOME/mpi/win64/ms-mpi) and can be installed by the user.

3.7.2.2 Launching OptiStruct SPMD

There are several ways to launch parallel programs with each MPI. Below are some typical ways to launch OptiStruct SPMD. Remember to propagate environment variables when launching OptiStruct SPMD, if needed. Please refer to corresponding MPI’s manual for more details.

Note: • A minimum of 3 processes are needed to launch OptiStruct SPMD. • OptiStruct SPMD must match the MPI implementation you use.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 73

Using Solver Scripts On a single host using solver script (for HP-MPI, Platform MPI 9.1.2, Intel MPI, and MS MPI: • Load Decomposition Method

[optistruct@host1~]$ $ALTAIR_HOME/hwsolvers/scripts/optistruct.bat –mpi [MPI_TYPE] -ldm –np [n] [INPUTDECK] [OS_ARGS]

• Domain Decomposition Method

[optistruct@host1~]$ $ALTAIR_HOME/hwsolvers/scripts/optistruct.bat bat –mpi [MPI_TYPE] –ddm –np [n] [INPUTDECK] [OS_ARGS]

• Multi-Model Optimization

[optistruct@host1~]$ $ALTAIR_HOME/hwsolvers/scripts/optistruct.bat –mpi [MPI_TYPE] –mmo –np [n] [INPUTDECK] [OS_ARGS]

• Failsafe Topology Optimization

[optistruct@host1~]$ $ALTAIR_HOME/hwsolvers/scripts/optistruct.bat [INPUTDECK] [OS_ARGS] –mpi [MPI_TYPE] -fso –np [n]

Where

[MPI_TYPE]: is the MPI implementation used: pl for IBM Platform MPI 9.1.2 i for Intel MPI ms for MS MPI

-- ( [MPI_TYPE]is optional, default MPI implementation on Windows machines is i

[n]: is the number of processors

[INPUTDECK]: is the input deck file name

[OS_ARGS]: lists the arguments to OptiStruct

-- ([OS_ARGS] is optional)

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 74

Note:

• Adding the optional command line option –mpipath PATH helps the user to find the MPI installation if it is not included in the current search path or when multiple MPI’s are installed.

• If no SPMD run option (-ldm/-mmo/-ddm/-fso) is specified, then LDM is run by default

3.7.2.3 Launching OptiStruct SPMD Using Direct Calls to the Executable

On a single host (for IBM Platform MPI 9.1.2): • Load Decomposition Method

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] [OS_ARGS] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] [OS_ARGS] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpirun -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] [OS_ARGS] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpirun -np [n]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 75

$ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] [OS_ARGS] -fsomode

Where

[n]: is the number of processors

[INPUTDECK]: is the input deck file name

[OS_ARGS]: lists the arguments to OptiStruct other than –mpimode

-- ([OS_ARGS] is optional. Please refer to Run Options for further information)

On a Single Host (for Intel MPI and MS MPI): • Load Decomposition Method

[optistruct@host1~]$ mpiexec -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] [OS_ARGS] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpiexec -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] [OS_ARGS] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpiexec -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] [OS_ARGS] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpiexec -np [n]

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 76

$ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] [OS_ARGS] -fsomode

Where optistruct_spmd is the OptiStruct SPMD executable and can be one of: Windows Intel MPI Executable: = optistruct_2017_win64_impi.exe Windows Microsoft MPI Executable: = optistruct_2017_win64_msmpi.exe

[n]: is the number of processors

[INPUTDECK]: is the input deck file name

[OS_ARGS]: lists the arguments to OptiStruct other than –mpimode

-- ([OS_ARGS] is optional. Please refer to Run Options for further information)

On a Multiple Window Hosts (for IBM Platform MPI 9.1.2): • Load Decomposition Method

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpirun –f

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 77

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] -ddmmode

• Multi-Model Optimization

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] -mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpirun –f

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-h [host i] -np [n] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_2017_win64_plmpi.exe [INPUTDECK] -fsomode

Where

: is a text file which contains process counts and a list of programs.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 78

Example: 4 CPU job on 2 dual-CPU hosts (the two machines are named: c1 and c2). The following example shows the contents of the :

[optistruct@host1~]$ cat hostfile -h c1 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ optistruct_2017_win64_plmpi.exe [INPUTDECK] -mpimode -h c2 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ optistruct_2017_win64_plmpi.exe [INPUTDECK] –mpimode [optistruct@host1~]$ cat appfile -h c1 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_spmd [INPUTDECK] -mpimode -h c2 –np 2 $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/optistruct_spmd [INPUTDECK] –mpimode

On a Multiple Window Hosts (for Intel MPI and MS MPI): • Load Decomposition Method

[optistruct@host1~]$ mpiexec –configfile [config_file]

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-host [host i] –n [np] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] - ddmmode -mpimode

• Domain Decomposition Method

[optistruct@host1~]$ mpiexec –configfile [config_file]

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-host [host i] –n [np] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] - ddmmode

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 79

• Multi-Model Optimization

[optistruct@host1~]$ mpiexec –configfile [config_file]

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-host [host i] –n [np] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] - mmomode

• Failsafe Topology Optimization

[optistruct@host1~]$ mpiexec –configfile [config_file]

Inside the , the following sentence can be repeated based on the number of hosts within the cluster:

-host [host i] –n [np] $ALTAIR_HOME/hwsolvers/optistruct/bin/win64/ [INPUTDECK] - fsomode

Where

: which contains the command for each host optistruct_spmd is the OptiStruct SPMD executable and can be one of: Windows Intel MPI Executable: = optistruct_2017_win64_impi.exe Windows Microsoft MPI Executable: = optistruct_2017_win64_msmpi.exe

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 80

Example: 4 CPU job on 2 dual-CPU hosts (the two machines are named: c1 and c2). The following example shows the contents of the :

Proprietary Information of Altair Engineering AcuSolve Installation Overview 4

AcuSolve Installation Overview

AcuSolve supports a broad range of computing architectures. For most cases, the default settings used by the AcuSolve installation are sufficient to run the software on the system that it has been installed on. However, there are occasions where users wish to customize the installation, or where the default settings are not sufficient. This chapter discusses some of the more common manners in which the AcuSolve installation is customized. Additionally, instructions for testing the installation are provided.

The AcuSolve CFD solutions package contains all tools and utilities necessary to construct and solve CFD simulations. The primary executables used in this process are AcuConsole and AcuSolve. This chapter describes common aspects of the installation and set-up procedure for the AcuSolve CFD solutions package.

AcuConsole is a powerful, yet easy to use GUI-Based Computational Fluid Dynamics (CFD) preprocessor for AcuSolve. It can work with very complex geometries, build simulations from CAD or existing mesh and import 3D CAD models from ProE, Parasolid, ACIS, and discrete formats. AcuConsole can also import an existing mesh in various formats. Through an intuitive menu structure, AcuConsole helps the user fully set up a problem, generate an input file, and launch the solver. CAE automation is available through the Python scripting language, allowing users to customize menus and functions for specific applications.

AcuSolve is a general-purpose finite element based Computational Fluid Dynamics (CFD) flow solver with superior robustness, speed, and accuracy. AcuSolve may be used by designers and research engineers with all levels of expertise, either as a standalone product or seamlessly integrated into a powerful design and analysis application. Quality solutions can be obtained quickly without iterating on solution procedures or worrying about mesh quality or topology. The fluid structure interaction (FSI) capabilities available in AcuSolve enable the user to perform multi-physics analysis of complex scenarios in an efficient manner. The interfaces in HyperMesh and HyperView ensure a smooth integration of AcuSolve into the HyperWorks framework.

Although AcuSolve and AcuConsole are referred to as separate products in the discussion above, they are installed through a single installation package.

General Installation Steps AcuSolve is available as a stand-alone installation package that may be installed separately or in conjunction with a full HyperWorks installation. When installing as part of HyperWorks, users may download the installation package that contains AcuSolve and have the master installation install it as part of the full installation. Alternatively, users may install AcuSolve outside of the HyperWorks 2017 Advanced Installation Guide 82

HyperWorks environment by installing it as a stand-alone package. If users intend to use functionality of HyperWorks tools that rely on an AcuSolve installation, AcuSolve should be installed in the same directory as HyperWorks. HyperWorks tools check in this installation location for the existence of the acusolve directory in order to use many of these functions. The full details of the HyperWorks installation process are provided in the previous chapters.

This chapter covers the following:

• 4.1 Configuring an AcuSolve Installation (p. 83) • 4.2 Installing AcuSolve 2017 Alongside Older Distributions on Linux (p. 95) • 4.3 Testing the AcuSolve Installation (p. 96)

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 83

4.1 Configuring an AcuSolve Installation

Upon successful completion of the installer, the AcuSolve installation is configured to run the solver using a number of defaults. These default settings are appropriate for the majority of computer systems. However, there are occasions when users need to modify the default behavior of the AcuSolve installation. Some examples of this include users who wish to change the default message passing type used by AcuSolve, the default licensing type used by the solver, etc. These types of system level modifications are handled through a single configuration file that is located within the AcuSolve installation directory. The full path to the configuration file is found within the installation at $INSTALL_DIR/altair/acusolve/platform/script/Acusim.cnf.

Where platform is one of win64 or linux64 and $INSTALL_DIR is the installation directory chosen by the user. The Acusim.cnf file is an ASCII text file. The file contains name=value pairs that act to configure various aspects of the installation. A number of name=value options are included in the default file that is provided with the distribution. The following example shows a default configuration file:

#======# # "Acusim.cnf": System and site dependent configuration # #======#======# Parallel processing #======message_passing_type = impi view_message_passing_type= impi

automount_path_remove= _none automount_path_name = _none automount_path_replace= _none collect_subdomain_output= true

host_lists= _auto num_processors= 1 #======# License manager #======lm_license_file= _auto lm_port= 41994 lm_server_host= _undefined lm_daemon= _auto lm_checkout= _auto

lm_service_type= hwu

A number of important default options are specified in the configuration file shown above. Among these are the message_passing_type and view_message_passing_type. These options instruct

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 84

AcuSolve’s parallel executables to run using the Intel Message Passing Interface when running on more than a single processor. Another important configuration option that is set in the default file is lm_service_type=hwu. This instructs AcuSolve to seek a license from a HyperWorks Units based license server.

Note that additional system level options may be set in the Acusim.cnf file. For a full discussion of configuration files and command line options, please refer to Chapter 1 of the AcuSolve Programs Reference Manual. This manual is included in the /doc directory of the AcuSolve installation in PDF format, and is also available in HTML format through the HyperWorks help system.

4.1.1 Configuring AcuSolve to Use Legacy ACUSIM Licensing

Upon successful completion of the installation process, the AcuSolve installation is configured to use HyperWorks units licensing. Users who wish to use a legacy ACUSIM license need to modify the AcuSolve installation accordingly. All modifications necessary to accomplish this are made by editing the system level configuration file.

The full path to the configuration file is found within the installation at $INSTALL_DIR/altair/ acusolve/platform/script/Acusim.cnf.

Where platform is one of win64 or linux64 and $INSTALL_DIR is the installation directory chosen by the user.

To enable ACUSIM licensing, users need to specify (at a minimum) the following options:

lm_service_type= classical

lm_server_host= _auto

lm_license_file= _auto

lm_port= 41994

lm_daemon= _auto

lm_checkout= _auto

Once this change is made, all licensed applications will rely on a connection to a legacy ACUSIM license server. The settings in the above example make use of the _auto option for lm_license_file, lm_server_host, lm_daemon, lm_checkout, and lm_server_host. The following table describes how this setting is interpreted for each option:

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 85

Setting Interpretation

lm_license_file=_auto Resolves to $ACUSIM_HOME/$ACUSIM_MACHINE/ license.dat

lm_server_host=_auto Resolves to the host name of the local machine

lm_daemon=_auto Resolves to $ACUSIM_HOME/$ACUSIM_MACHINE/ bin/acuLmd

lm_checkout=_auto Resolves to $ACUSIM_HOME/$ACUSIM_MACHINE/ bin/acuLmco

The use of the _auto setting is intended to simplify the setting of parameters for classical ACUSIM licensing. For more complex installations, the _auto option will not be appropriate for all settings and users will need to configure the options listed above appropriately. The following example illustrates the necessary settings to check out a license from a network license server named “pharaoh”:

lm_service_type= classical

lm_server_host= pharaoh

lm_port= 41994

lm_daemon= /scratch/HW_2017/altair/acusolve/linux64/ bin/acuLmd

lm_checkout= /scratch/HW_2017/altair/acusolve/linux64/ bin/acuLmco

Note that the _auto option could have been used for the lm_daemon and lm_checkout options. Additional information on the proper settings for the license server can be found in the AcuSolve Programs Reference Manual.

4.1.2 Configuring AcuSolve for Distributed Memory Parallel Computing

AcuSolve’s parallel computing model relies on a shared file system to function properly. Each MPI process reads and writes its own files. Using this model, all data I/O happens to a commonly accessible file system. For Linux platforms, this is typically accomplished by running simulations from a Network File System (NFS). For Windows platforms, this is accomplished by creating network shares. All compute nodes involved in the computation need to access the file system using the same path. Note

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 86 that AcuSolve uses no temporary files on disk during execution and does not write any files to /tmp on the local hard drive.

4.1.3 Configuring AcuSolve for Parallel Processing on Windows Platforms

AcuSolve supports a variety of parallel executables and message passing types on Windows platforms. For distributed memory message passing, users may select between Intel MPI (mp=impi), Platform MPI (mp=pmpi), MPICH (mp=mpi), and Microsoft MPI (mp=msmpi). These options are supported by all parallel executables. For shared memory message passing, AcuSolve supports the use of openmp. This option is supported for all parallel executables with the exception of AcuView. The following sections discuss the steps necessary to run AcuSolve in parallel on Windows platforms.

4.1.4 Configuring AcuSolve for Parallel Processing on a Single Windows Host

When running in parallel on a single Windows host, there are no special set-up steps that need to be taken if you intend to use Intel MPI, Platform-MPI, or openMP. The HyperWorks installer package handles all set-up necessary to accommodate these scenarios and it is not necessary to perform any additional set-up work. Microsoft MPI, however, does require an additional step to install. The following sections discuss the set up process necessary to run each type of MPI on a single host Windows system.

Intel MPI When running Intel MPI on a single Windows host, there is no need to perform any additional configuration to AcuSolve. The following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job on a single Windows host using Intel MPI:

acuRun -pb Test -mp impi -np 4 -nt 1 -hosts localhost

Platform MPI 9.1.2 When running Platform MPI 9.1.2 on a single Windows host, there is no need to perform any additional configuration to AcuSolve. The following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job on a single Windows host using Platform MPI 9.1.2:

acuRun -pb Test -mp pmpi -np 4 -nt 1 -hosts localhost

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 87

Open MP When running Open MP on a single Windows host, there is no need to perform any additional configuration to AcuSolve. The following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job on a single Windows host using Open MP:

acuRun -pb Test -mp openmp -np 4 -nt 4 -hosts localhost

MPICH When running on a Windows host, MPICH relies on a Simple Multi-Purpose Daemon (SMPD) service to manage the parallel processes. This service must be installed on any system that is taking part in the parallel computation. For the case of a single host, this only requires installation of the service on the local machine. The SMPD service must be installed by a user with administrative privileges. Once the service has been installed on a given system, it will be available for use in parallel processes run by any user of the system. The service will also be started automatically after restarts of the machine. The service may be installed in one of two ways:

Using AcuRun The MPICH SMPD service may be installed using the following command, executed from an AcuSolve command prompt by a user with administrative privilege:

acuRun -pb Test -mp mpi -ismpi

Using the smpd.exe executable directly The MPICH SMPD service may be installed using the following command, executed from any command prompt by a user with administrative privileges:

/mpi/win64/mpich/bin/smpd.exe -install

Once the service has been installed, the following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job on a single Windows host using MPICH:

acuRun -pb Test -mp mpi -np 4 -nt 1 -hosts localhost

Microsoft MPI The Microsoft MPI (MS MPI) redistribution agreement does not allow shipping of run time libraries outside of the MS MPI installation package. Therefore, the MS MPI run time environment is not installed by the HyperWorks installer. Instead, the MS MPI installer is shipped with the AcuSolve installation and

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 88 made available for installation by users when desired. To configure your system to run AcuSolve using MS MPI, it is necessary to install the MS MPI run time environment. This is accomplished by executing the installer that is located in the following path:

/mpi/win64/ms-mpi/MSMpiSetup.exe

At the conclusion of this installation, the following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job on a single Windows host using MS MPI:

acuRun -pb Test -mp msmpi -np 4 -nt 1 -hosts localhost

4.1.4.1 Intel MPI

When running AcuSolve in parallel across multiple Windows hosts using Intel MPI, it is necessary to complete some additional configuration steps that are not performed by the HyperWorks installation package. These additional steps consist of installing and starting the hydra service to manage job launching on remote hosts, and registering user credentials to enable the execution of the processes on the remote hosts. The following sections discuss how to accomplish this.

Installing the Hydra Service The hydra service is used by Intel MPI to manage the parallel processes when running across multiple hosts. As such, the hydra service must be installed by a user with administrative privileges on each host that is to participate in the parallel run. Once the service has been installed on a given system, it will be available for use in parallel processes run by any user of the system. The service will also be started automatically after restarts of the machine. The service may be installed in one of two ways: • Using AcuRun The Intel MPI hydra service may be installed using the following command, executed from an AcuSolve command prompt by a user with administrative privileges:

acuRun -pb Test -mp impi -ismpi

• Using Commands The Intel MPI hydra service may also be installed and started using the following commands, executed from any command prompt by a user with administrative privileges:

/mpi/win64/intel-mpi/bin/hydra_service.exe -install

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 89

/mpi/win64/intel-mpi/bin/hydra_service.exe -start

Registering Credentials Before Intel MPI can launch parallel processes on remote machines, it must be able to access the remote system using a user ID and password. To avoid repeatedly typing a user name and password for each remote machine when a parallel process is launched, Intel MPI allows you to register your credentials for the remote machines. This can be accomplished using the command line utility, or by using a graphical user interface. To accomplish this from the command line, the following sequence of commands can be executed from any command prompt:

/mpi/win64/intel-mpi/bin/mpiexec.exe -register

When prompted, enter the user name and password that will be used to authenticate the user account on the remote machines.

To register your credentials using a graphical user interface, double-click on the following executable:

/mpi/win64/intel-mpi/bin/wmpiregister.exe

Launching the Parallel Job Once the previously described set-up steps are completed, the following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job across two Windows hosts using Intel MPI:

acuRun -pb Test -mp impi -np 4 -nt 1 -hosts pharaoh,seneca

4.1.4.2 Platform MPI 9.1.2

When running AcuSolve in parallel across multiple Windows hosts using Platform MPI 9.1.2, it is necessary to complete some additional configuration steps that are not performed by the HyperWorks installation package. These additional steps consist of installing and starting the SMPD service to manage job launching on remote hosts, and registering user credentials to enable the execution of the processes on the remote hosts. The following sections discuss how to accomplish this.

Installing the SMPD Service The SMPD service is used by Platform MPI 9.1.2 to manage the parallel processes when running across multiple hosts. As such, the SMPD service must be installed by a user with administrative privileges

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 90 on each host that is to participate in the parallel run. Once the service has been installed on a given system, it will be available for use in parallel processes run by any user of the system. The service will also be started automatically after restarts of the machine. The service may be installed in one of two ways: • Using AcuRun The Platform MPI 9.1.2 SMPD service may be installed using the following command, executed from an AcuSolve command prompt by a user with administrative privileges:

acuRun -pb Test -mp pmpi -ismpi

• Using Commands The Platform MPI 9.1.2 SMPD service may also be installed and started using the following commands, executed from any command prompt by a user with administrative privileges:

/mpi/win64/platform-mpi/sbin/pcmpiwin32service.exe -install /mpi/win64/platform-mpi/sbin/pcmpiwin32service.exe -start

Registering Credentials Platform MPI 9.1.2 can be used to run AcuSolve in parallel across multiple hosts without the need to enter any credentials on the remote systems. This is accomplished by specifying the -nopass option to the mpirun command (using the -mpirun_options command line argument of acuRun). Although this approach provides the easiest setup, it does prevent you from automatically mounting remote shares onto the compute nodes that are participating in the computation. Therefore, it will be necessary to use the full network path to the problem directory when launching the job. For example, to run a simulation from the \temp\channel directory of a drive that is mounted on the machine named pharaoh, the following entries would be added into the local Acusim.cnf file:

mpirun_options = -nopass pdir = \\pharaoh\temp\channel

If you do wish to automatically mount network shares, then it is necessary to register your credentials with Platform MPI 9.1.2. To accomplish this, the following commands can be executed from any command prompt:

/mpi/win64/platform-mpi/bin/mpidiag.exe -s remoteHostName -cache

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 91

When prompted, enter the password that will be used to authenticate the current user account on the remote machines for the mpi runs. To confirm that the password is cached, the following command can be used:

/mpi/win64/platform-mpi/bin/mpidiag.exe -s remoteHostName -iscached

The mpidiag utility also has some other helpful options for diagnosing problems with distributed memory parallel runs. The following command can be used to check if the remote share "T:" is available on a remote machine named pharaoh:

/mpi/win64/platform-mpi/bin/mpidiag.exe -s pharaoh -dir T:

Launching the Parallel Job Once the previously described setup steps are completed, the following command can be executed from an AcuSolve command prompt to launch a 4 processor parallel job across two Windows host using Platform MPI 9.1.2:

acuRun -pb Test -mp pmpi -np 4 -nt 1 -hosts pharaoh,seneca

4.1.4.3 MPICH

When running AcuSolve in parallel across multiple Windows hosts using MPICH, it is necessary to complete some additional configuration steps that are not performed by the HyperWorks installation package. These additional steps consist of installing and starting the SMPD service to manage job launching on remote hosts, and registering user credentials to enable the execution of the processes on the remote hosts. The following sections discuss how to accomplish this.

Installing the SMPD Service The SMPD service is used by MPICH to manage the parallel processes when running across multiple hosts. As such, the SMPD service must be installed by a user with administrative privileges on each host that is to participate in the parallel run. Once the service has been installed on a given system, it will be available for use in parallel processes run by any user of the system. The service will also be started automatically after restarts of the machine. The service may be installed in one of two ways: • Using AcuRun

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 92

The MPICH SMPD service may be installed using the following command, executed from an AcuSolve command prompt by a user with administrative privileges:

acuRun -pb Test -mp mpi -ismpi

• Using Commands The MPICH SMPD service may also be installed and started using the following commands, executed from any command prompt by a user with administrative privileges:

/mpi/win64/mpich/bin/smpd.exe -install /mpi/win64/mpich/bin/smpd.exe -start

Registering Credentials Before MPICH can launch parallel processes on remote machines, it must be able to access the remote system using a user ID and password. To avoid repeatedly typing a user name and password for each remote machine when a parallel process is launched, MPICH allows you to register your credentials for the remote machines. This can be accomplished using the command line utility, or by using a graphical user interface. To accomplish this from the command line, the following command can be executed from any command prompt:

/mpi/win64/mpic/bin/mpiexec.exe -register

When prompted, enter the password that will be used to authenticate the current user account on the remote machines.

To register your credentials using a graphical user interface, double-click on the following executable:

/mpi/win64/mpich/bin/wmpiregister.exe

acuRun -pb Test -mp mpi -np 4 -nt 1 -hosts pharaoh,seneca

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 93

4.1.4.4 Microsoft MPI (MS MPI)

The Microsoft MPI (MS MPI) redistribution agreement does not allow shipping of run time libraries outside of the MS MPI installation package. Therefore, the MS MPI run time environment is not installed by the HyperWorks installer. Instead, the MS MPI installer is shipped with the AcuSolve installation and made available for installation by you when desired. To configure your system to run AcuSolve using MS MPI, it is necessary to install the MS MPI run time environment. This is accomplished by executing the installer that is located in /mpi/win64/ms-mpi/MSMpiSetup.exe For distributed memory parallel operation, MS MPI relies on a SMPD process to manage the parallel processes. This SMPD process is different from those of MPICH and Intel MPI in the sense that it does not get installed as a service onto the system. Therefore, it is necessary for you to confirm that the service is running on each host that is participating in your distributed memory parallel run. The process can be started by issuing the following command on each host:

/bin/smpd.exe -d

acuRun -pb Test -mp msmpi -np 4 -nt 1 -hosts pharaoh,seneca

4.1.5 Configuring the AcuSolve Environment on Linux Platforms

To configure your environment to run the AcuSolve programs on Linux platforms, a number of environment variables must be set. These variables are set by sourcing the appropriate set up file for your shell.

For CSH or CSH variants (i.e. csh, tcsh, etc.), the following file must be sourced:

$INSTALL_DIR/altair/acusolve/linux64/script/acusim.csh

Where $INSTALL_DIR is the installation directory chosen by the user.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 94

To automatically source this file every time a new shell is opened, place the following line in your ~/.cshrc file (or equivalent):

source $INSTALL_DIR/altair/acusolve/linux64/script/acusim.csh

For BASH or BOURNE shell variants (i.e. sh, bash, etc.), the following file must be sourced:

$INSTALL_DIR/altair/acusolve/linux64/script/acusim.sh

To automatically source this file every time a new shell is opened, place the following line in your ~/.profile file (or equivalent):

. $INSTALL_DIR/altair/acusolve/linux64/script/acusim.sh

Alternatively, users can execute $INSTALL_DIR/altair/scripts/acusim to launch a subshell that has the appropriate environment configured.

4.1.5.1 Configuring Platform MPI 9.1.2 on Linux Platforms

Platform MPI 9.1.2 (mp=pmpi) is shipped with the AcuSolve installation and is available for use on 64 bit Linux and Windows platforms. By default, PMPI uses the ssh remote shell protocol on Linux to launch the parallel processes on the various compute nodes used in the computation. ssh needs to be configured to permit the connection on all hosts without typing a password. This is accomplished through the use of rsa keys. Please refer to the ssh manual pages to generate and install RSA keys (ssh-keygen tool).

It is also possible to use rsh instead of ssh to launch AcuSolve when using Platform MPI 9.1.2. Refer to the rsh manual pages for installation instructions of the rsh utility.

Note that the default remote shell can be changed by setting the following line in the Acusim.cnf file:

remote_shell = /usr/bin/rsh remote_shell = /usr/bin/ssh

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 95

4.2 Installing AcuSolve 2017 Alongside Older Distributions on Linux

Starting in version 12.0, significant changes have been made to the directory structure in which AcuSolve is installed. AcuSolve now installs using the following structure: $INSTALL_DIR/altair/ acusolve/linux64/.

Where $INSTALL_DIR is the installation directory chosen by the user. The $ACUSIM_VERSION variable is no longer used. However, for backward compatibility, it is set to an empty string on Linux 64 platforms.

For users that wish to link the new directory structure within a legacy installation on Linux platforms, this can be accomplished by soft linking the resulting directory structure to the “latest” link, or by soft linking it to a link with the version as the name. Below are some examples:

• HyperWorks and AcuSolve installed in: /scratch1/Applications/HyperWorks/2017 • Legacy AcuSolve installation in: /scratch1/Applications/Acusim/LINUX64 From within the legacy AcuSolve installation, the following command should be executed:

ln -sf /scratch1/Applications/HyperWorks/2017/altair/acusolve/linux64 ./V14.0

ln –sf /scratch1/Applications/HyperWorks/2017/altair/acusolve/linux64 ./latest

Changes have also been made to the manner in which the AcuSolve environment is configured on Linux platforms. The necessary shell configuration file names have been modified to script/acusim.sh (bourne shell and variants) and script/acusim.csh (c-shell and variants), and the structure of them has changed. Installation paths are automatically detected based on the location of the files and the entire distribution directory structure may be moved without the need to modify any files. For backward compatibility, the acusim.sh and acusim.csh files are soft linked to the file names of the previous convention. However, users should transition to the new convention as soon as older releases are phased out of use. Note that the automatic detection of the installation path requires the use of the lsof command in the csh environment. If this command is not installed on your system, then the / script/acuCrtcsh program can be run to hard code the paths in the acusim.csh file.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 96

4.3 Testing the AcuSolve Installation

Two simple test cases are provided with the AcuSolve distribution that can be used to test the functionality of an AcuSolve installation. The two examples are located in the /model_files/examples directory of the AcuSolve installation. Instructions for executing the examples are provided below.

4.3.1 Channel Example

The channel example contains all input files necessary to run the solver from the command line. It does not include a database file for use with AcuConsole. The intent of the channel example is to provide a quick check that the solver installation is functional and returning suitable results. This example problem solves a laminar flow in a channel. Run the problem using the following procedure: • Windows: Double click on AcuSolve desktop shortcut to open a command prompt window • Linux: Open a terminal where the AcuSolve environment is set

1. Navigate to the model_files/examples/channel directory of the AcuSolve installation. 2. To run the example on 2 processors on a machine named “eagle”, execute the following command:

acuRun –np 2 –hosts eagle

3. To translate the solution to a raw table format:

acuTrans -out -to table

4. To get statistics on the nodal output:

acuTrans -out -to stats

a) Compare the solution with the exact solution limits:

x-velocity 0 1

y-velocity 0 0

z-velocity 0 0

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 97

pressure 0 0.2

The AcuSolve solution should be within (+-) 1.e-3 of the above. 5. To get surface traction:

acuTrans -osi -osiv step,trac -to table

a) Compare the last row of file channel_srf1.osi with the exact solution:

x-traction 0.2

y-traction 0

z-traction 0

6. To translate nodal values to be viewed by AcuFieldView:

acuTrans -out -to fieldview

4.3.2 Pipe Example

The pipe example contains an AcuConsole database as well as files necessary to run the particle tracer (AcuTrace). The purpose of this example is to test the GUI tools that are included in the AcuSolve distribution. To test the main installation, use the following procedure:

1. Depending on your operating system, complete one of the following:

Option Description

Windows Double click on the AcuConsole icon on the desktop

Linux In a terminal window with the AcuSolve environment set, issue the acuConsole command

2. Click File > Open and select the pipe.acs file from the model_files/examples/pipe directory of the AcuSolve installation. 3. Click Tools > Generate Mesh.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 98

4. In the dialog, make sure Export ams file and Launch AcuMeshSim are both set to On. 5. Click OK. When the Mesh Generator finishes, a message box is displayed. 6. Click Open. 7. Click Tools > AcuSolve. 8. In the dialog, make sure Generate AcuSolve input files and Launch AcuSolve are both set to On. 9. Click OK. The acuRun script is executed, which in turn executes three codes: • acuPrep: AcuSolve data pre-processor • acuView: View factor module • acuSolve: Main solver 10. Click Tools > AcuFieldView.

11. Select pipe.1.Log from the file browser, and click Ok. The AcuFieldView visualizer package opens with a default view of the temperature field on the outer wall of the pipe.

Proprietary Information of Altair Engineering FEKO Installation Overview 5

FEKO Installation Overview FEKO supports a broad range of computing architectures ranging from personal computers to large compute clusters. Please consult the FEKO Installation Guide for a complete description of the installation procedures and configuration options. RADIOSS HMPP to MADYMO 7.1 MPP Coupling 6

RADIOSS HMPP to MADYMO 7.1 MPP Coupling MADYMO® has been developed at TASS-SAFE.

The terms and conditions governing the license of MADYMO software consist solely of those set forth in the written contracts between TASS-SAFE or TASS-SAFE authorized third parties and its customers. The software may only be used or copied in accordance with the terms of these contracts.

Supported Platforms

Table 6: RADIOSS 11/MADYMO MPP

Platform ID Architecture OS Executable Interconnect Remarks Name Type

Linux64_plmpi INTEL EM64T RHEL 5.9 e_2017_linux64_plmpiGigabit, Needs Platform and AMD RHEL 6.2 Infiniband MPI PLMPI Operton 2.2.7 or higher SLED 11 SP2

SLES 11 SP2

This chapter covers the following:

• 6.1 System Requirements (p. 101) • 6.2 Installing RADIOSS and Activating the License (p. 102) • 6.3 Installing MADYMO and Activating the License (p. 103) • 6.4 Configuring the Machine (p. 104) • 6.5 Launching a RADIOSS to MADYMO Computation (p. 105) HyperWorks 2017 Advanced Installation Guide 101

6.1 System Requirements

a. Operating system: Intel Xeon EM64T or AMD64 systems under Linux with glibc 2.3 or higher: RHEL 5.8 RHEL 6.2 SLED 11 SP2 SLES 11 SP2 b. The interconnect to use is Gigabit Ethernet or Infiniband. c. It is highly recommended to have the same hardware and OS level on all nodes. d. The MPI library: HPMPI 2.2.7 must be installed on all the cluster nodes. Currently the RADIOSS to MADYMO 7.1 MPP coupling does not work with HPMPI 2.3 or higher.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 102

6.2 Installing RADIOSS and Activating the License

1. Install HyperWorks 2017 package for Linux64. 2. Follow the steps documented in the HyperWorks 2017 Installation Guide to complete the installation and connect to the Altair License 2017 server.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 103

6.3 Installing MADYMO and Activating the License

1. Install MADYMO according to the installation instructions, using the linux24-em64t platform. 2. Install the MADYMOlicense.

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 104

6.4 Configuring the Machine

The configuration of the machine must conform to the specifications in Installing RADIOSS HMPP on x86-64 Linux Cluster using Platform MPI 9.1.2.

MADYMO environment needs to be reachable from all the computation nodes, with the same account to run RADIOSS.

Environment Variables Setting MADHOME, MADETCPATH, MADINCPATH, MADBINPATH must be set. For MADYMO 71, MADHOME will be set to the complete path to repository madymo6x.

In a csh shell: setenv MADETCPATH $MADHOME/share/etc:. setenv MADINCPATH $MADHOME/share/etc:. setenv MADBINPATH $MADHOME/linux24_em64t/bin

In sh/ksh/bash shell: export MADETCPATH=$MADHOME/share/etc:. export MADINCPATH=$MADHOME/share/etc:. export MADBINPATH=$MADHOME/linux24_em64t/bin

Install HyperWorks 2017 package for Linux 64. The radflex_2017_linux64 tool needs the liblmx- altair.13.0.0.so library from the HyperWorks 2017 installation. Set the LD_LIBRARY_PATH environment to $ALTAIR_HOME/hw/lib/linux64:

[radioss@host1] setenv LD_LIBRARY_PATH $ALTAIR_HOME/hw/lib/linux64

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 105

6.5 Launching a RADIOSS to MADYMO Computation

[radioss@host1~]$ cp $ALTAIR_HOME/hwsolvers/bin/linux64/radflex_2017_linux64 . [radioss@host1~]$ $ALTAIR_HOME/hwsolvers/bin/linux64/s_2017_linux64 - input[ROOTNAME]_0000.rad –np [n] [radioss@host1~]$ $MADHOME/linux24-em64t/bin/madymo_cpl -config cpl.conf .xml ROOTNAME_0001.rad

Where cpl.conf : is a config file containing all parameters needed to execute the coupling: COUPLING=mpi MPILIB=hpmpi NPMAD=1 NPFE=1 MPIRUN=/opt/ibm/platform_mpi/mpirun FEARGS="" FEBIN=./radioss.sh

COUPLING=mpi, MPILIB=hpmpi: gives the RADIOSS HMPP MADYMO 7.1 coupling method. These parameters should remain unchanged.

NPMAD: Number of processors for MADYMO NPFE: Number of processors for RADIOSS MPIRUN: Path and command to HP MPI mpiprun executable FEARGS: Specific arguments to RADIOSS FEBIN: Script and PATH to execute RADIOSS

The Radioss.sh script is as follow:

#! /bin/sh np="-np 1"

# Parse the command line arguments. while [ $# -gt 0 ]; do case "$1" in (-nrproc) shift np="-np $1" ;; (*) break ;; esac shift done

job_name="$1"

Proprietary Information of Altair Engineering HyperWorks 2017 Advanced Installation Guide 106

echo STARTING ENGINE COMPUTATION echo np: $np echo file: "${job_name}" echo working dir: `pwd`

echo $FEMPIRUN -e MPI_WORKDIR=`pwd` \ $np $ALTAIR_HOME/hwsolvers/bin/e_14.0_linux64_plmpi -input ${job_name} –nt [threads]

eval $FEMPIRUN -e MPI_WORKDIR=`pwd` \ $np $ALTAIR_HOME/hwsolvers/bin/e_14.0_linux64_plmpi -input ${job_name} –nt [threads] }

Where

[n] is the number of SPMD domains.

[Threads] is the number of threads per domain.

Proprietary Information of Altair Engineering