Full-System Simulation

The leader in full-system simulation virtutech

Jakob Engblom Business Development Manager [email protected]

The leader in full-system simulation virtutech • Simulation: a way to study the world •Build a model of a system • Try scenarios on this model – Experimental, not analytical approach • Understand the real system by working with the model – More available –More inspectable – Less dangerous

The leader in full-system simulation virtutech • Simulation gets closer to real world – More details – Fewer assumptions – High computational workload

• Analytical models – Efficient predictors – Low computational workload – ... but more removed from world=less accurate

The leader in full-system simulation virtutech

P Prog roce ram ssor

WhatPe rdo we need to simulate? iphe rals

Stim uli

The leader in full-system simulation virtutech

source code analysis printf method analytical methods source code semantic statistics correctness proof macros

code transformation object code code augmentation OS emulation

OS profilers execution environment virtual machines

simulation instruction set full system simulation

simulation register transfer level

simulation gate level

Instruction set level is essentially the level where software engineers interface with hardware engineers. Therefore, it is the lowest level accessible to software, and at the same time it is the best defined, least complicated, and most stable level.

The leader in full-system simulation virtutech

Virtutech Simics

One physical Virtual computer systems of computer many different types

The leader in full-system simulation virtutech

AllAll software:software: arbitraryarbitrary && User program unmodified.unmodified. SameSame asas onon aa realreal system.system.

Target operating system ArbitraryArbitrary;; currentlycurrently Simulated target hardware supportssupports Alpha,Alpha, AMD64,AMD64, ARMARM IA64,IA64, MIPS,MIPS, PowerPC, Sparc, x86 Simics PowerPC, Sparc, x86

Linux,Linux, Solaris,Solaris, Host operating system WindowsWindows

Host hardware PCPC oror SparcSparc

The leader in full-system simulation virtutech

User program

Middleware DB Server Complete Operating system real Drivers Firmware software

CPU Simulated PCI Network CPU Disk hardware IDE LCD RAM I2C FLASH ROM KBD Hardware

User program Real user Middleware DB Servers program

Operating system Simulated OS, services, CPU some HW

RAM

Hardware

The leader in full-system simulation virtutech • Complete and correct function – From a software perspective – All accessible values represented • ... including supervisor-level registers in processors • ... including model-specific registers – Memory-management units – All computations bit-correct • No difference from real target •Runs any software: – Firmware, OS, drivers, networks, applications, middleware • Hardware modeled as a set of devices – I/O-space or memory mapped – Complete configuration and control register set – Transaction-level of modeling

• Next slide: example of detail required

5 January 2004 Copyright © 1998-2004 Virtutech, All rights reserved. CONFIDENTIAL 11 ActualActual screenshot screenshot fromfrom within within Simics, Simics, The leader in full-system simulation virtutech simulatingsimulating a a P4 P4 PC PC runningrunning Windows Windows XP XP

The leader in full-system simulation virtutech

SimulatedSimulated networknetwork ofof simulatedsimulated machinesmachines

InterfaceInterface toto realreal networknetwork

Real network of physical machines

The leader in full-system simulation virtutech

• Globally synchronized timing – Across processors – Across machines – One processor stops Æ all stop • Correct relative speed – 500 Mhz processor will execute 10 times more instructions than a 50 Mhz processor, in the same time – For greater accuracy: select CPI values – Across the network • Virtualized time – Insulated from external time – Time does not advance when simulation is stopped

The leader in full-system simulation virtutech To simulate the Galaxies universe, the units of simulation have to be galaxies

AtomLevel of abstraction The Scope of model Universe Simulating a single atom, we can use the incredible detail Reasonable to of quantom simulate: scope mechanics and proportional string theory to abstraction

String theory

The leader in full-system simulation virtutech

• Grand-Prix Legends • ”Life-like” action: –Momentum –Friction – Steering –Engine torque • Not nuts & bolts of cars

Grand-Prix Legends

ESSES, 4 Sept 2003 (c) Virtutech, Confidential 16 Increasing the Scope

The leader in full-system simulation virtutech • Interesting systems are larger than single CPU • Multiprocessors – Homogeneous like servers – Heterogeneous like mobile phones • Distributed systems – Local-Area Networks – Embedded CAN buses – Networks-on-chips • Need to simulate – Shared memory machines –Many machines – Interconnection networks

The leader in full-system simulation virtutech • Gate-level simulation: very very slow – Every gate simulated, subcycle simulation • Cycle-accurate simulation: very slow – Hardware timing at cycle level – Usually unnecessary • Instruction-set simulation (ISS): slow to fast – Depends on implementation technique – Abstract away most of timing • Emulation: fastest • Simics is a very fast ISS: – 100s of MIPS as the base model... – ...but can do cycle-accurate simulation when needed

The leader in full-system simulation virtutech

Detailed hardware sim

cycle-accurate simulator (>10,000x)

accuracy fast full-system simulation (20-400x) emulator (5x)

Virtualization speed 10 KIPS 1000 MIPS

The leader in full-system simulation virtutech

Programs Programs

Target OS Target OS

Network Target HW Target HW simulation Simics

Host operating system TwoTwo oror moremore machinesmachines cancan bebe simulatedsimulated withinwithin aa singlesingle simulatorsimulator Host hardware

The leader in full-system simulation virtutech

Simulated network Programs Simulated network Programs behavesbehaves thethe samesame asas onon aa singlesingle simulatingsimulating Target OS machinemachine Target OS

Network Target HW Target HW simulation Simics Simics

Host operating system Host operating system Real-lifeReal-life networknetwork connection Host hardware connection Host hardware

The leader in full-system simulation virtutech • Checkpointing – Store current state; pick up and continue later • Determinism – Same initial state gives same execution; replay input/output • Controllability – Any part of machine or state can be changed • Visibility (insight without intrusion) – All state can be observed, traced, logged; unintrusively • Sandboxing – Simulated machine complete isolated

The leader in full-system simulation virtutech • Software monitors are intrusive... Simics is not

WindowsWindows XPXP TaskTask Manager:Manager: 2%2% CPU,CPU, 33 MBMB RAMRAM

The leader in full-system simulation virtutech

• Simulation Tool in 1977: DEC VAX-11/780 – 200,000 USD (1977 dollars) – 1 VAX MIPS – simulation technology ~200x – cost for simulated server hour: 4,000 USD

Photo courtesy of The Computer Museum History Center • Simics in 1999: Dell PC (PIII 700) – 10000 USD – approx 1400 VAX MIPS – simulation technology ~40x – cost for simulated server hour: 4 USD

• Simics in 2004: Dual Xeon 3G 1U Server – 5000 USD – approx 15000 VAX MIPS – simulation technology ~15x Photo courtesy of Intel Photo courtesy – cost for simulated server hour: 10 cents

… a relative cost reduction of over 10,000x!

Windows NT VxWorks on on x86 Linux on PowerPC Itanium

Solaris on Sun SunFire

Windows XP/64 on AMD Hammer

Linux All running on a on x86 Linux host

The leader in full-system simulation virtutech

The leader in full-system simulation virtutech • Embedded hardware systems – Often 10000+ USD development platforms – Simulation: •Cheaper • More convenient, easily replicated • Early availability • Virtual testlabs – Replace custom hardware with PCs – Cheaper than custom hardware – Easier setup – Checkpoint & replicate setups – Sufficient performance •“Virtual hardware”: – Better than the real thing!

The leader in full-system simulation virtutech Handoff to the software team, Board design Board prototype when ”working” Classic hardware exists model Software development

Board prototype With Board design & Time saved Simics build simulator Software development Handoff to the software team, using a simulation of the hardware platform

The leader in full-system simulation virtutech • Low-level software development – Supervisor-level (OS) & interrupt code debug • Single-step interrupt handlers! – Inspection of system state – Device access tracing & breakpoints – Debugging unfinished operating systems – Developing drivers • Trace & break device accesses

• High-level software development – Powerful debugger, with checkpointing – Deterministic multitasking

The leader in full-system simulation virtutech • Trace performance events: – Cache misses – TLB misses –Page faults – Disk accesses – Hardware accesses – Memory access patterns • Without intrusion • Validity of performance measurements: – Relative measures are OK – Absolute measures more questionable

The leader in full-system simulation virtutech • Model hardware – Architecture – Devices – Processors • Debug hardware designs – Including the software interface & drivers • Stimulate HW with real workloads – On real operating systems • HW/SW cosimulation – At various levels of detail • Shortens time to market dramatically

The leader in full-system simulation virtutech • System-Level Verification – Transaction level testing –Run real code – Generate real transaction streams – Interactive testing, not fixed transactions/bit patterns • System-Level Testbench – Design phase • Early validation of functionality • Against behavioral model – Implementation phase • Validation of RTL implementation – Hardware prototyping phase • Validation of prototype chips & systems

The leader in full-system simulation virtutech

Memory Devices

Application

OS Behavioral Model of Drivers Custom Hardware CPU Core

Simics SimicsSimics SimicsSimics API: API: interconnectioninterconnection transaction-basedtransaction-based modelmodel oror cycle-based cycle-based

The leader in full-system simulation virtutech

Interface: Memory Devices Interface: clock-cyclesclock-cycles and and signalssignals Application

OS Transactor Drivers

CPU Core Detailed model ConversionConversion from from and to Simics and to Simics RTL-level simulation Simics transactionstransactions

The leader in full-system simulation virtutech

SystemCSystemC interconnectinterconnect modelmodel Application Device OS Drivers Memory Behavioral Model of CPU Core Custom Device Hardware Simics

Device Interface:Interface: cycle-cycle- or or event-based event-basedSystem C framework

The leader in full-system simulation virtutech • Develop network stacks & protocols – Instrument the network –Trace traffic – Inject packets – No interference from other traffic – Synchronous breaks at important events • Try network configurations –Large networks – Pathological topologies – Routing delays, faulty packets, ...

The leader in full-system simulation virtutech • Fault injection – Fault tolerant systems – Safety critical systems

–Repeatable – No physical damage – Full insight – Regression testing

The leader in full-system simulation virtutech

C orru m pt easu C reme orrupt nts regis ter value Sensor s CPU

Transient errors in xmit Device RAM t anen Perm rors bit er Device Bridge Network rk netwo rrupt g a Co nplug plu ire ets; u Un nt m pack ice ll e te ev Ki s d bsy su 5 January 2004 Copyright © 1998-2004 Virtutech, All rights reserved. CONFIDENTIAL 38 Fault Injection & Checkpointing

The leader in full-system simulation virtutech

Restore checkpoint

Boot system SW Run 1

Position workload Check results Take checkpoint CKP Did the Restorefault checkpoint Runaffect 2 the Check results injected result? fault Check results

The leader in full-system simulation virtutech • Boundary case testing – Extremely small or large configurations •”264 bytes of RAM” • Not limited by physical availability – Interrupt arrival rates – Network packet arrival rates – Network packet size – Network configuration size • 247 nodes in a fibre-channel network

The leader in full-system simulation virtutech • Enable hands-on experience • Computer architecture • Embedded systems programming • Operating systems – Debug half-finished systems – Same setup for all students, easy handins • System management – Easy to restore system state – No risk to real machines and networks

The leader in full-system simulation virtutech

The leader in full-system simulation virtutech • Without proper stimuli, model is useless

•Feed mechanism – How to get information into the simulation

•Data generation – What to supply to the simulation – Can get tricky

The leader in full-system simulation virtutech •Fixed inputs – Spec benchmarks: loaded from disk •Network – Load generation on simulated machines – Interface to a real network • Interactive use • Load generators on real machines • Keyboard & mouse – Map directly to real device – Easy for PC-on-PC-style – Interactive user

The leader in full-system simulation virtutech

• Phones, navigation computers, PDAs, etc. • Application development – Use GUI to provide interactive sessions with user – Keyboard, joystick, touch screen – Not radio data etc.

The leader in full-system simulation virtutech • Special simulated devices – Sensors & actuators • Data sources – Statistical models of real system behavior – Simulation models of physical reality – Hardware-in-the-loop simulation

The leader in full-system simulation virtutech • Stimuli = hardware configuration • Booting an operating system – Test of OS software vs hardware – Reconfigure hardware, alter devices • Self-configuring systems – Networks & other distributed systems – Master election, device discovery, etc. – Adding/removing simulated nodes

The leader in full-system simulation virtutech • Problem: simulation is slow – Especially for detailed architectural simulation – Slowdown 10000: • 1 minute real time = 7 days simulation time • Scale (down) workloads to fit – Smaller data sets – How to make representative of full runs? – Tricky problem in its own right

The leader in full-system simulation virtutech

The leader in full-system simulation virtutech • Computer architecture research – 90% or more done in simulation – Measure of success: effect of modification to a reference machine – What is a significant result? -5%?+10%? •How real is the machine modified? – SimpleScalar is not a real processor • = need for quite extensive modeling

The leader in full-system simulation virtutech • Mark Hill et al, IEEE Computer Feb 2003 • Investigating potential pitfalls of simulation • Detailed microarchitectural modeling – Pipeline, caches, reordering, the works – Randomized L2 miss time (80-89 cycles) • Several runs with same workload • Variable results!

The leader in full-system simulation virtutech 4.0

3.5 max avg 3.0 min Cycles Per Trans. (millions) 2.5 16 32 64 • WCR (16,32) = 18%ROB (“Wrong Size Conclusion Ratio”) • WCR (16,64) = 7.5% • WCR (32,64) = 26%

The leader in full-system simulation virtutech

3.4

3.2

3.0 32 64 2.8 Cycles Per Trans. (millions) 2.6 5 101520 Sample Size (number of runs)

The leader in full-system simulation virtutech • Conclusions: – Simulation no different from runs on real HW – Use standard statistics – Non-overlapping confidence intervals

• Danger of determinism in simulation – Testing a single path of a program – Induce variability by randomization

The leader in full-system simulation virtutech

• Virtutech develops customized virtual development platforms • Based on full-system simulation technology • Flagship product: Simics • We help our customers – Save money – Develop better products faster

The leader in full-system simulation virtutech

• Founded in 1998 in Stockholm, Sweden – Based on research begun in 1991 – Spin-off from Swedish Institute of Computer Science • Launched U.S. Operations in 2002 – Sales/marketing/support in San Jose, Boston and Stockholm •Early Customers – Ericsson and Sun Microsystems were first customers • Currently – The market is opening up & understanding our message – US and European operations expanding rapidly

The leader in full-system simulation virtutech • Significantly Reduce Development Hardware Costs – Immediately reduce capital and operational costs by replacing development hardware with virtual development platforms. – “Build a $2M system on a $2K PC.”

• Accelerate Software and Hardware Development – Provide software teams with virtual hardware many months prior to the availability of physical hardware. – “Gain 6-18 months in time-to-market.”

• Increase Quality, Performance, and verify High Availability – Complete, non-intrusive control and visibility of the software running on Simics virtual hardware. – “The ultimate debug and test environment.”

The leader in full-system simulation virtutech

This is the power of full-system simulation!

The leader in full-system simulation virtutech

The leader in full-system simulation virtutech • Simics is offered to academia almost for free • Academia gets the full Simics tool, all targets

• Personal licenses – Free, renewable yearly – Node-locked to a single computer

• Site licenses – Free first year – 2500 USD for 40 licenses after that – Much cheaper than buying specialized hardware labs