FEM Simulations: Effects of Improvements in Information Technologies on the Computational Time with Large Full Vehicle FEM Models
Total Page:16
File Type:pdf, Size:1020Kb
FEM simulations: effects of improvements in information technologies on the computational time with large full vehicle FEM models A. Ghelardini, G. Mancini, C. Goracci, A. Cera, A. Corbizi, D. Russo Trenitalia S.p.A., Florence, Italy Abstract New well designed computing platforms for FEM simulators and new hardware and software computing technologies have allowed to address FEM related scalability constraints. The latest 64 bit computer technologies are tremendously improving Technical and Research Department capability to deal with more complex and realistic simulation models, simultaneously reducing the department FEM related response times. Introduction Trenitalia is the main Train Operator Company in Italy, it is responsible for managing the development, construction and maintenance of the rail transportation system (Image 1) in the country. In this capacity, the Trenitalia Technical and Research Department uses ANSYS Mechanical for the following activities: • Design optimization for implementation of new equipment on existing locomotives, coaches and wagons of Trenitalia fleet. • Stress strength structural checks to comply with safety transportation rules for new vehicles. • Maintenance engineering planning for bogies and/or body frames deteriorated by fatigue and corrosion phenomena. Looking for FEM optimized computing platforms The need for larger analysis models and shorter computer response times led Trenitalia to evaluate new calculation solutions. In looking for more computational power to improve mechanical stress simulation capability, Trenitalia Technical and Research Department, in cooperation with Information Technology Department, investigated how finite element programs interact with modern operating systems. This research activity demonstrated the following main constraints in dealing with production-size models: • Model size is limited by the amount of real memory used by 32-bit FEM programs (especially in their internal databases) in a 32-bit computing solution. • Solution times are reduced using multiprocessors platforms. • Hardware architecture bottlenecks in memory and storage sub-systems increase elapsed times. To address the first issue, Trenitalia started investigated 64-bit technology from the older Alpha-based solutions to the current Itanium-based ones, and even to the latest AMD64/EM64T platforms. Both shared memory and massive parallel architecture technologies have been evaluated, with efficiency requirements and economical constraints suggesting a scalable SMP architecture. Economical, technical and maintenance needs led Trenitalia to carefully evaluate the new x86-64 technology of the latest PC-based AMD64/EM64T systems for us to understand if the new systems were ready for reliable high performance computing solutions. Image 1: The ETR 500 italian high speed train The architectural core improvements in x86-64 capable processors allowed us to get very good integer and floating point mathematical results, and the full-duplex star topology of PCI-Express modern workstations gave sufficient bandwidth to move Gigabytes of data to/from memory from/to storage sub-system. These systems were less expensive than traditional solutions and were going to be mainstream in the market, but would they be able to accomplish their tasks? Benchmarking New Systems In December 2004, two 64-bit operating systems were tested on the same hardware (an ordinary monoprocessor 2GB AMD64 personal computer) with the same 0.35 million degree of freedom test model: • The most recent version of Linux (kernel 2.6.9) with a native 64-bit ANSYS 9.0 got an elapsed of 270 seconds. • The Release Candidate of Microsoft Windows x64 Edition with the 32-bit version of ANSYS 9.0 got an elapsed of 180 seconds. The same phenomena was observed on a dual Xeon 32-bit SMP test platform. Better SMP scalability, greater efficiency in thread and memory management and maintenance constraints led us to select the new MS Windows XP X64 Ed. operating system. Several field tests were planned in 2005 to evaluate reliability, even while the x64 Operating System was still in a beta phase, with the current versions available of Win 32 ANSYS on a 4GB two way AMD64 platform. Trenitalia found that the x64 operating system was indeed the best operating system for “/3GB compliant” (i.e. large addresses) Win 32 professional programs (i.e.: ANSYS 9.0A1 Win 32). ANSYS 9.0A1 (this is the code version identity in output logs of ANSYS 9.0 SP1) was finally able to manage up to 3.7 GB of memory. Nevertheless, what Trenitalia still needed was a true, native 64-bit version of ANSYS for Win x64 able to address more than 4 GB of memory. At last, Trenitalia could define their ideal reference platform: a two- way Win x64 SMP workstation able to support at least 16 GB of memory with a low latency memory and storage sub-system. Due to platform and drivers constraints with more than 4 GB of memory installed, it took a long time (until Summer 2005) and considerable tuning efforts to acquire the first full working prototype of the reference platform. Real production models (Image 2 and Image 3) were used to benchmark (thanks to side- by-side trials) with the latest Intel and AMD based platforms. Trenitalia selected the AMD64 (table 1) solution due to its actual greater efficiency in executing 64-bit programs, improved scalability on SMP systems and better performances while solving very complex models. The latest dual-core CPUs were also tested in a real production environment with the same production test models. Image 2: Static strength results from stress analysis of a bogie frame for a high-speed car The widespread availability of the new high performance 64-bit operating system, the quick improvements in quality in 64 bit developing tools, the investments in ANSYS parallel compliant high performance sparse solver (implicit SMP solver with large memory capability), the huge amount of memory addressable by 64-bit technologies and an ANSYS management investment in emerging 64-bit technologies, allowed ANSYS to quietly publish a 64-bit native ANSYS 10 Win x64 beta product at the end of Summer 2005. Trenitalia jumped on the new program files and began to test the beta product. Image 3: First natural frequency from modal analysis of railway car body frame ANSYS 10 Win x64 beta was able to use considerably more than 4 GB of memory, and Trenitalia immediately started to develop some million degree of freedom production models. While solving them with the beta build (the ANSYS Win 32 version was not even able to open them), Trenitalia demonstrated some important issues in the same beta product, especially those dealing with the efficiency of the sparse solver (in 10.0 x64 beta, while solving large models, the sparse solver was slower than the older 9.0 sp1 32 bit one). After having notified this limitation to our local ANSYS office, we were allowed to directly report feedbacks to the appropriate manager at ANSYS and, even if ANSYS had only a few weeks to improve their code before the scheduled official release, they quickly identified the problems and fixed them in time for the official ANSYS 10.0 sp1 Win x64 product. This effort enabled the sparse solver to gain a 30% in efficiency from the beta version. Deploying 64 bit workstations In Febrary 2006, as soon as Ansys 11.0 sp1 for Windows XP Professional x64 Edition was officially released, Trenitalia deployed several 64 bit workstations and the dealing with some MDOF FEM models feature became an usual capability of the Technical and Research Department. In December 2006, while testing the new Ansys 11.0 version with a brand new production model (modal analysis of a full body frame vehicle of the latest Alstom ETR600) (Image 4), Trenitalia was able to identify a critical Sparse Solver related bug. Trenitalia demonstrated the problem with Ansys 10.0 sp1 and Ansys 11.0 while solving on multiprocessor systems on 32 and 64 bit Windows SMP platforms; monoprocessor systems worked fine while solving the same model. In January 2007 Trenitalia contacted again Ansys Support and was quickly forwarded to report directly to Ansys Sparse Solver Developer Department. Trenitalia sent Ansys a model (only for internal debug efforts) and told Ansys how to reproduce the bug. Ansys confirmed the bug on SMP platforms even with the Itanium-64, Linux-32 and Linux- EM64T binaries; the Linux-AMD64 binaries worked fine. This was the proof of a cross- platform compiler family inducted error and not a source code bug, so Trenitalia suggested to investigate very carefully about the high performance mathematical libraries used to implement Ansys Sparse Solver. While waiting for a fix, Trenitalia production platforms were constrained on monoprocessor solvers only. Image 3: The new ETR 600 high speed tilting train In Febrary 2007 Ansys reported that they had identified the problem, had developed a pair of workarounds and actually they started working with the compiler manufacturer to totally fix it. It took more than six month of hard work and product quality testing to upgrade the developing tools (the Fortran compiler and its high performance mathematical libraries) to their latest version. In October 2007 Ansys finally published the long waited Ansys 11.0 sp1 executables with their updated mathematical libraries. Trenitalia successfully tested the binaries on her SMP platforms and deployed them on production workstations, while removing the monoprocessor constraint for the Sparse Solver. Last but not least, the compiler and libraries upgrade enabled Ansys