Last time… • Multithreading executes instructions from different threads • Coarse-grained multithreading threads on CSE 490/590 Architecture misses • Most of the OoO superscalar units are idle.

Multithreading II

Steve Ko Computer Sciences and Engineering University at Buffalo

CSE 490/590, Spring 2011 CSE 490/590, Spring 2011 2

Vertical Multithreading Superscalar Machine Efficiency

Issue width Issue width Instruction Instruction issue issue Second interleaved Completely idle cycle cycle-by-cycle (vertical waste)

Time Time Partially filled cycle, Partially filled cycle, i.e., IPC < 4 i.e., IPC < 4 (horizontal waste) (horizontal waste)

• What is the effect of cycle-by-cycle interleaving?

3 4 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

Chip (CMP) Ideal Superscalar Multithreading Issue width [Tullsen, Eggers, Levy, UW, 1995] Issue width

Time Time

• What is the effect of splitting into multiple processors?

• Interleave multiple threads to multiple issue slots with no restrictions 5 6 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

C 1 O-o-O Simultaneous Multithreading [Tullsen, Eggers, Emer, Levy, Stamm, Lo, DEC/UW, 1996] IBM Power 4 Single-threaded predecessor to • Add multiple contexts and fetch engines and allow Power 5. 8 execution units in! instructions fetched from different threads to issue simultaneously out-of-order engine, each may! • Utilize wide out-of-order superscalar issue issue an instruction each cycle.! queue to find instructions to issue from multiple threads • OOO already has most of the circuitry required to schedule from multiple threads • Any single thread can utilize whole machine

7 8 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

Power 4 Power 5 data flow ...

2 commits Power 5 (architected register sets)

Why only 2 threads? With 4, one of the shared 2 fetch (PC), resources (physical registers, cache, memory 2 initial decodes bandwidth) would be prone to bottleneck 9 10 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

Changes in Power 5 to support SMT CSE 490/590 Administrivia • Increased associativity of L1 instruction cache and the • Quiz 2 (next Friday 4/8): After midterm until next instruction address translation buffers Monday • Added per thread load and store queues • Increased size of the L2 (1.92 vs. 1.44 MB) and L3 caches • Added separate instruction prefetch and buffering per thread • Increased the number of virtual registers from 152 to 240 • Increased the size of several issue queues • The Power5 core is about 24% larger than the Power4 core because of the addition of SMT support

11 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011 12

C 2 -4 Hyperthreading (2002) Initial Performance of SMT • First commercial SMT design (2-way SMT) • Pentium 4 Extreme SMT yields 1.01 for – Hyperthreading == SMT SPECint_rate and 1.07 for SPECfp_rate • Logical processors share nearly all resources of the physical processor – Pentium 4 is dual threaded SMT – Caches, execution units, branch predictors – SPECRate requires that each SPEC benchmark be run against a vendor-selected number of copies of the same benchmark • Die area overhead of hyperthreading ~ 5% • When one logical processor is stalled, the other can make • Running on Pentium 4 each of 26 SPEC benchmarks progress paired with every other (262 runs) speed-ups from 0.90 – No logical processor can use all entries in queues when two threads are active to 1.58; average was 1.20 • Processor running only one active software thread runs at approximately same speed with or without hyperthreading • Power 5, 8-processor server 1.23 faster for • Hyperthreading dropped on OoO based followons to SPECint_rate with SMT, 1.16 faster for SPECfp_rate Pentium-4 (Pentium-M, Core Duo, Core 2 Duo), until revived with Nehalem generation machines in 2008. • Power 5 running 2 copies of each app speedup • Intel Atom (in-order core) has two-way vertical multithreading between 0.89 and 1.41 – Most gained some – Fl.Pt. apps had most cache conflicts and least gains

13 14 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

SMT adaptation to parallelism type Icount Choosing Policy

For regions with high thread level For regions with low thread level parallelism (TLP) entire machine parallelism (TLP) entire machine Fetch from thread with the least instructions in flight. width is shared by all threads width is available for instruction level parallelism (ILP) Issue width Issue width

Time Time

Why does this enhance throughput?

15 16 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

Summary: Multithreaded Categories Uniprocessor Performance (SPECint)

Simultaneous 10000 3X Superscalar Fine-Grained Coarse-Grained Multiprocessing Multithreading From Hennessy and Patterson, : A Quantitative ??%/year Approach, 4th edition, 2006 1000

52%/year VAX-11/780) 100 (vs.

Performance 10 25%/year Time (processor cycle) Time

1 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 Thread 1 Thread 3 Thread 5 • VAX : 25%/year 1978 to 1986 Thread 2 Thread 4 Idle slot • RISC + x86: 52%/year 1986 to 2002 17 18 CSE 490/590, Spring 2011 CSE 490/590,• RISC Spring + x86: 2011 ??%/year 2002 to present

C 3 Parallel Processing: Déjà vu all over again? Symmetric Multiprocessors

“… today’s processors … are nearing an impasse as technologies approach the speed of light..” David Mitchell, The : The Time Is Now (1989) Processor Processor • Transputer had bad timing (Uniprocessor performance↑) ⇒ Procrastination rewarded: 2X seq. perf. / 1.5 years • “We are dedicating all of our future product development to multicore designs. CPU-Memory … This is a sea change in computing” Paul Otellini, President, Intel (2005) bridge • All companies to MP (2X CPUs / 2 yrs) I/O bus ⇒ Procrastination penalized: 2X sequential perf. / 5 yrs Memory I/O controller I/O controller I/O controller Manufacturer/Year AMD/’09 Intel/’09 IBM/’09 Sun/’09 symmetric • All memory is equally far Graphics Processors/chip 6 8 8 16 away from all processors output Threads/Processor 1 2 4 8 • Any processor can do any I/O Networks (set up a DMA transfer) Threads/chip 6 16 32 128 19 20 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

A Producer-Consumer Example Synchronization tail head The need for synchronization arises Producer Consumer whenever there are concurrent processes in a system R R R producer Rtail tail head (even in a uniprocessor system)

Consumer: Producer-Consumer: A consumer consumer Producer posting Item x: Load Rhead, (head) must wait until the producer process has Load Rtail, (tail) spin: Load Rtail, (tail) produced data Store (Rtail), x if Rhead==Rtail goto spin Rtail=Rtail+1 Store (tail), R Load R, (Rhead) Mutual Exclusion: Ensure that only one P1 P2 tail R =R +1 process uses a resource at a given time head head Store (head), Rhead process(R) Shared The program is written assuming Resource instructions are executed in order. Problems?

21 22 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

A Producer-Consumer Example Sequential Consistency continued A Memory Model

Producer posting Item x: Consumer: P P P P P P

Load Rtail, (tail) Load Rhead, (head) 1 Store (R ), x spin: Load R , (tail) 3 tail tail M Rtail=Rtail+1 if Rhead==Rtail goto spin 2 Store (tail), Rtail Load R, (Rhead) 4 Rhead=Rhead+1 “ A system is sequentially consistent if the result of Can the tail pointer get updated Store (head), Rhead any execution is the same as if the operations of all before the item x is stored? process(R) the processors were executed in some sequential order, and the operations of each individual processor Programmer assumes that if 3 happens after 2, then 4 appear in the order specified by the program” happens after 1. Leslie Lamport

Problem sequences are: Sequential Consistency = 2, 3, 4, 1 arbitrary order-preserving interleaving 4, 1, 2, 3 of memory references of sequential programs

23 24 CSE 490/590, Spring 2011 CSE 490/590, Spring 2011

C 4 Acknowledgements

• These slides heavily contain material developed and copyright by – Krste Asanovic (MIT/UCB) – David Patterson (UCB) • And also by: – Arvind (MIT) – Joel Emer (Intel/MIT) – James Hoe (CMU) – John Kubiatowicz (UCB)

• MIT material derived from course 6.823 • UCB material derived from course CS252

CSE 490/590, Spring 2011 25

C 5