nanoHUB.org online simulations and more Network for Computational Nanotechnology (NCN) Parallel Computing for Realistic Nanoelectronic Simulations Gerhard Klimeck, NCN Technical Director Purdue University September 12th, 2005 Univ. of Florida, Univ.of Illinois, MIT, Morgan State, Northwestern, Purdue, Stanford, UTEP Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Simulation is essential for online simulations and more Nanotechnology Development Hint from the Semiconductor Industry: • No new devices / circuits designed without software! Simulation Problem: • Accepted nano simulation tool suite does NOT exist. Approach: • Conduct research in Modeling and Simulation of: • Nanoelectronics • Nanoelectromechanics • Nano-bio sensors • Computational science • DEVELOP and DEPLOY to nanoscience and nanotechnology community Characterization Fabrication Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Online simulation and MORE online simulations and more collaboration animations Real users! Real Usage! online simulation >6,600 nanoHUB.oUsersrg >71,000 simulations learning modules seminars courses, tutorials Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Middleware for Deployment online simulations and more Remote access to simulators and compute power nanoHUB.org Unique Middleware internet tool Physical Machine Mostly Batch Operation Old -system: PUNCH Jobs are run on a specific hardware Most jobs are serial Most jobs are batch oriented and not interactive New System: In-VIGO Jobs can be distributed to various machines Virtual machines can generate specific OS on various machines Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Middleware for Deployment online simulations and more Remote access to simulators and compute power nanoHUB.org Unique Middleware internet Virtual Machine tool Physical Machine RMeomsotltye Bdaetsckht oOpp (eVrNatCio)n Old -system: PUNCH Jobs are run on a specific hardware Most jobs are serial Job Most jobs are batch oriented and Manager not interactive New System: In-VIGO Jobs can be distributed to various machines Virtual machines can generate TeraGrid Cluster specific OS on various machines (future) Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Parallel Computing: online simulations and more Why and How? Motivation: Approaches: • Serial computation on single CPU • Maximize the work on each CPU requires • Minimize communication between CPUs .Too much time . Communication is your “enemy” .Too much memory ⇒Identify “identical code segments” that can Available Resources: be treated independently of each other • Traditional parallel computers .with shared memory Technologies: .expensive in FLOPS/$ • Batch processing with shells • Cluster computers / Beowulfs • Independent executables - trivial .Distributed memory - independent • Automated compiler-based parallelism computers on a dedicated network • Typically limited to shared memory machines .“cheap” in FLOPS/$ and few CPUs / threads Objective: • OpenMP - shared memory model • Distribute workload on N CPUs • MPI - message passing interface • Only communicate where programmer decides • Best achievable: Tparallel = Tserial / N Minimize communication => Keep available N CPUs busy • • Runs on almost all parallel computers • Runs on clusters / beowulfs Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Really Embarrassing Parallel online simulations and more Time 1 2 3 4 5 ... N-1 •Spawn independent jobs one single time •Possibly trivial data aggregation ⇒shell-based parallelism or ⇒MPI-based parallelism Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Embarrassingly Parallel online simulations and more 0 Time 1 2 3 4 5 ... N-1 0 1 2 3 4 5 ... N-1 •Data needs to aggregated by one master CPU ... •Master distributes work to slaves repeatedly => MPI-based approaches Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Coherently Parallel Processes online simulations and more Lockstep Parallelism 1 2 3 4 5 6 •Compute Time 1 2 3 4 5 6 •Communicate odd <=> odd+1 •Communicate 1 2 3 4 5 6 odd <=> odd-1 •Communicate 1 2 3 4 5 6 all-2-all, aggregate 1 2 3 4 5 6 •Compute ... •Repeat Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Parallelization on Multiple Levels online simulations and more •Sometimes the work cannot be easily distributed amongst many CPUs .Maybe there are not enough members in the high level loop .Maybe parallelism is only efficient on a few CPUs on the low level loop •Sometimes the levels of available parallelism change .User runs an experimentwith just a single bias point .User runs a model that has no “k” level parallelism Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Presentation Outline online simulations and more •NanoHUB.org - Online Simulation and More •Parallel Applications: .Motivation .Approaches •Embarrassingly Parallel .Genetic Algorithm Optimization .GENES - Genetically engineered nanoelectronic structures •Lock-Step Parallelism .NEMO 3-D - multimillion atom quantum dot simulation .Parallel performance .Simulation results •Multi-Level Parallelism .NEMO 1-D - a real CAD tool with dynamic parallelism requirements Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Basic Genetic Algorithm online simulations and more S1’ S1’ S1 S1’ S1’ S1 E E v v o o l u l u S2’ S2’ S2 F F S2’ S2’ S2 t C t i i o i i o t t o n n n n m e … e … … … … … s s b s s i n -R -R … Si’ Si’ e … Si’ Si’ a a S n n e k k t i i n n ... Si’’ Si’’ s ... Si’’ Si’’ g g … ... ... … ... ... SN SN’ SN SN’ SN’ RIP SN’ RIP Generation M Generation M+1 • Genetic algorithm parameter optimization is based on: .Survival of good parameter sets .Evolution of new parameter sets .Survival of a diverse population • Optimization can be performed globally, rather than locally. Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Basic Evolution Operations online simulations and more • Each set (Si) consists of several parameters (Pj) • The parameters Pj can be of different kinds: real, integers, symbols, …. Gross Exploration Fine Tuning Set PI PI Set 1 1’ P2 Q2 … … Pk Pk Set PI PI Set 3 3’ P2 P2’ Set QI QI Set 2 2’ … … Q2 P2 Pk Pk … … Qk Qk Mutation operation Crossover operation •Crossover explores different combinations of existing genes. •Creation of new gene values. Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Global Optimization via Genetic Algorithms online simulations and more sin(x) sin(y) sin(x ! 4) sin(y ! 4) F(x, y) = + 0.7 x y (x ! 4) (y ! 4) Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Global Optimization: online simulations and more Genetic Algorithm Development Genetic Algorithm Convergence pop = 100, 300 generations, steady-state (10%), 2-point crossover p = 0.85, mutatation p = 1/2 Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Embarrassingly Parallel online simulations and more 0 Time 1 2 3 4 5 ... N-1 0 1 2 3 4 5 ... N-1 •Data needs to aggregated by one master CPU ... •Master distributes work to slaves repeatedly => MPI-based approaches Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Presentation Outline online simulations and more •NanoHUB.org - Online Simulation and More •Parallel Applications: .Motivation .Approaches •Embarrassingly Parallel .Genetic Algorithm Optimization .GENES - Genetically engineered nanoelectronic structures •Lock-Step Parallelism .NEMO 3-D - multimillion atom quantum dot simulation .Parallel performance .Simulation results •Multi-Level Parallelism .NEMO 1-D - a real CAD tool with dynamic parallelism requirements Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org GENES - RTD Structural Analysis online simulations and more •Allow genetic algorithm to vary 5 • Start from “random” population of different structural parameters: 5 parameters. .3 Thicknesses: well, barrier, spacer • Well width is larger than nominal. .2 Dopings: low doped spacer, • No intentional doping is larger unintentional doping in center than nominal. Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org online simulations and more Mapping of Orbitals to Bulk Bandstructure Bulk Semiconductors are described by: • Conduction and valence bands, bandgaps (direct, indirect), effective masses • 10-30 physically measurable quantities Tight Binding Models are described by: Atomic Orbitals Structure size: 0.2nm • Orbital interaction energies. • 15-30 theoretical parameters z High Dimensional y Fitting Problem (1/4)Vsa, pc anion s orbital x coupled to cation p orbital 15-30 theoretical interaction energies 10-30 data points of bands and masses Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Semiconductor Compounds: online simulations and more cation: In, Ga, Al anion: Sb, As, P In Ga Al •Match experimental data in various electron transport Sb areas of the Brillouin zone: .Effective masses of electrons at Γ, X and L .Effective masses of holes at Γ .Bandedges at Γ, X and L •Each individual material As poses a 15 dimensional fitting problem. z P y (1/4)Vsa, pc anion s orbital x coupled to cation p orbital Gerhard Klimeck Network for Computational Nanotechnology nanoHUB.org Simulation in Si and Ge: 2nd Nearest Neighbor Model - online simulations and more Optimization in a 37-dimensional Space Value Target Rel. Error Description 0.0006 0.0000 5.72E-04 Vhh_E Si Target: 0.6760 0.7500 9.87E-02 kz_X_Delta •Energy at X 1.1330