Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 18-600 Foundations of Computer Systems Lecture 9: “Modern Superscalar Out-of-Order Processors” John P. Shen & Gregory Kesden September 27, 2017 Lecture #7 – Processor Architecture & Design Lecture #8 – Pipelined Processor Design Lecture #9 – Superscalar O3 Processor Design ➢ Required Reading Assignment: • Chapter 4 of CS:APP (3rd edition) by Randy Bryant & Dave O’Hallaron. ➢ Recommended Reading Assignment: ❖ Chapter 5 of Shen and Lipasti (SnL). 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 1 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 18-600 Foundations of Computer Systems Lecture 9: “Modern Superscalar Out-of-Order Processors” A. Instruction Flow Techniques a. Control Flow Prediction b. Dynamic Branch Prediction B. Register Data Flow Techniques a. Resolving Anti and Output Dependencies b. Resolving True Dependencies c. Dynamic Out-of-Order Execution C. Memory Data Flow Techniques a. Memory Data Dependencies b. Load Bypassing & Load Forwarding 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 2 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Modern Superscalar Processor Organization Fetch Instruction/Decode Buffer ▪ Buffers provide r Decode e d r decoupling O Dispatch Buffer n I Dispatch ▪ In OOO designs Reservation they also facilitate Issue Stations f r (re-)ordering o e t d Execute r u O O Finish ▪ More details on Reorder/ Completion Buffer specific buffers to r Complete e d r follow O Store Buffer n I Retire 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 3 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Designs of Inter-stage Buffers Stage i Stage i • • • 1 n ( in order ) Buffer (1) Buffer (n) • • • 1 n ( in order ) Stage i + 1 Stage i +1 • • • Scalar Pipeline Buffer In-order Parallel Buffers (simple register) (wide-register or FIFO) Stage i • • • ( any order ) Buffer (>_ n) ( any order ) Stage i + 1 • • • (multiported SRAM and CAM) Out-of-order Pipeline Stages 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 4 From Lec #8 … Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 3 Major Penalty Loops of Pipelined Processors W F ALU BRANCH D PENALTY M PENALTY LOAD (0 cycle) (2 cycles) PENALTY (1 cycle) E LOAD E PENALTY ALU BRANCH (1 cycle) PENALTY M D PENALTY (0 cycle) (2 cycles) F W Performance Objective: Reduce CPI as close to 1 as possible. Best Possible for Real Programs is as Low as CPI = 1.15. CAN WE DO BETTER? … CAN WE ACHIEVE IPC > 1.0? 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 5 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Three Impediments to Superscalar Performance I-cache F Branch Instruction FETCH Flow Predictor Instruction D BRANCH Buffer PENALTY (2 cycles) DECODE DISPATCH E Integer Floating-point Media Memory Reservation ALU PENALTY Stations (0 cycle) M Memory Data LOAD EXECUTE Flow PENALTY W (1 cycle) Reorder Buffer Register (ROB) Data COMMIT Flow Store D-cache Queue 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 6 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Three Flow Paths of Superscalar Processors ➢ Wide Instruction Fetching ➢ Dynamic Branch Prediction I-cache Branch Instruction FETCH Flow Predictor Instruction Buffer DECODE DISPATCH Integer Floating-point Media Memory Reservation Stations Memory Data EXECUTE Flow Reorder Buffer Register (ROB) Data COMMIT Flow Store D-cache Queue 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 7 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Control Flow Graph (CFG) Your program is actually BB 1 main: a control flow graph addi r2, r0, A addi r3, r0, B Shows possible paths of addi r4, r0, C BB 1 addi r5, r0, N control flow through BB 2 add r10,r0, r0 bge r10,r5, end basic blocks loop: lw r20, 0(r2) lw r21, 0(r3) BB 2 bge r20,r21,T1 BB 3 BB 4 sw r21, 0(r4) BB 3 Control Dependence b T2 T1: Node X is control sw r20, 0(r4) BB 4 T2: dependent on Node Y if addi r10,r10,1 BB 5 addi r2, r2, 4 the computation in Y addi r3, r3, 4 BB 5 addi r4, r4, 4 determines whether X blt r10,r5, loop end: executes 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 8 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Mapping CFG to Linear Instruction Sequence A A A C B B C D D B D C 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 9 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Branch Types and Implementation Types of Branches Conditional or Unconditional? Subroutine Call (aka Link), needs to save PC? How is the branch target computed? Static Target e.g. immediate, PC-relative Dynamic targets e.g. register indirect Conditional Branch Architectures Condition Code “N-Z-C-V” e.g. PowerPC General Purpose Register e.g. Alpha, MIPS Special Purposes register e.g. Power’s Loop Count 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 10 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition What’s So Bad About Branches? Robs instruction fetch bandwidth and ILP Use up execution resources Fragmentation of I-cache lines Disruption of sequential control flow Need to determine branch direction (conditional branches) Need to determine branch target Example: We have a N-way superscalar processor (N is large) A branch every 5 instructions that takes 3 cycles to resolve What is the effective fetch bandwidth? 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 11 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Disruption of Sequential Control Flow Fetch Instruction/Decode Buffer Decode Dispatch Buffer Dispatch Reservation Issue Stations Branch Execute Finish Reorder/ Completion Buffer Complete Store Buffer Retire 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 12 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Riseman and Foster’s Study ➢ 7 benchmark programs on CDC-3600 ➢ Assume infinite machine: • Infinite memory and instruction stack, register file, fxn units Consider only true dependency at data-flow limit ➢ If bounded to single basic block, i.e. no bypassing of branches maximum speedup is 1.72 ➢ Suppose one can bypass conditional branches and jumps (i.e. assume the actual branch path is always known such that branches do not impede instruction execution) Br. Bypassed: 0 1 2 8 32 128 Max Speedup: 1.72 2.72 3.62 7.21 24.4 51.2 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 13 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Dynamic Branch Prediction Tasks ➢Target Address Generation • Access register • PC, GP register, Link register • Perform calculation • +/- offset, auto incrementing/decrementing Target Speculation ➢Condition Resolution • Access register • Condition code register, data register, count register • Perform calculation • Comparison of data register(s) Condition Speculation 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 14 Dynamic Branch Target Prediction nPC to Icache prediction FA-mux specu. target nPC(seq.) = PC+4 Branch PC Fetch specu. cond. Predictor (using a BTB) Decode Buffer BTB Decode update (target addr. Dispatch Buffer nPC=BP(PC) and history) Dispatch Reservation Stations Issue Branch Execute Finish Completion Buffer 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 15 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Target Prediction: Branch Target Buffer (BTB) key tag data Associative Memory (CAM) A small “cache-like” memory in the instruction fetch stage Remembers previously executed branches, their addresses (PC), information to aid target prediction, and most recent target addresses I-fetch stage compares current PC against those in BTB to “guess” nPC If matched then prediction is made else nPC=PC+N If predict taken then nPC=target address in BTB else nPC=PC+N When branch is actually resolved, BTB is updated 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 16 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition More on BTB (aka BTAC) Typically a large associative structure idx key Pentium3: 512 entries, 4-way; Opteron: 2K entries, 4-way tag data Entry format Valid bit, address tag (PC), target address, fall-through BB address (length of BB), branch type info, branch direction prediction decoder BTB provides both target and direction prediction Multi-cycle BTB access? The case in many modern processors (2 cycle BTB) N-Way Start BTB access along with I-cache in cycle 0 Set-Associative Memory In cycle 1, fetch from PC+N (predict not-taken) k-bit index 2k • N blocks In cycle 2, use BTB output to verify 1 cycle fetch bubble if branch was taken 9/27/2017 (©J.P. Shen) 18-600 Lecture #9 17 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition Branch Target Prediction for Function Returns In most languages, function calls are fully nested If you call A() B() C() D() Your return targets are PCc PCb PCa PCmain Return address stack (RAS) A FILO structure for capturing function return addresses Operation On a function call retirement, push call PC into the stack On a function return, use the top value in the stack & pop A 16-entry RAS can predict returns almost perfectly Most programs do not have such a deep call tree Sources of RAS inaccuracies Deep call statements (circular buffer overflow – will lose older calls) Setjmp and longjmp C functions (irregular call semantics) 9/27/2017 (©J.P.