Lecture 7: Pipelining

18-447 Computer Architecture Lecture 7: Pipelining

Prof. Onur Mutlu Carnegie Mellon University Spring 2015, 1/30/2015

Agenda for Today & Next Few Lectures

 Single-cycle Microarchitectures

 Multi-cycle and Microprogrammed Microarchitectures

 Pipelining

 Issues in Pipelining: Control & Data Dependence Handling, State Maintenance and Recovery, …

 Out-of-Order Execution

 Issues in OoO Execution: Load-Store Handling, …

Recap of Last Lecture

 Multi-cycle and Microprogrammed Microarchitectures

 Benefits vs. Design Principles

 When to Generate Control Signals

 Microprogrammed Control: uInstruction, uSequencer, Control Store

 LC-3b State Machine, Datapath, Control Structure

 An Exercise in Microprogramming

 Variable Latency Memory, Alignment, Memory Mapped I/O, …

 Microprogramming

 Power of abstraction (for the HW designer)

 Advantages of uProgrammed Control

 Update of Machine Behavior

Review: A Simple LC-3b Control and Datapath

4 C.4. THE CONTROL STRUCTURE 9

R IR[15:11]

BEN

Microsequencer

6 Simple Design of the Control Structure

Control Store

2 6 x 35

Microinstruction

9 26

(J, COND, IRD)

Figure C.4: The control structure of a microprogrammed implementation, overall block diagram on the LC-3b instruction being executed during the current instruction cycle. This state carries out the DECODE phase of the instruction cycle. If the IRD control signal in the microinstruction corresponding to state 32 is 1, the output MUX of the microsequencer (Figure C.5) will take its source from the six bits formed by 00 concatenated with the four opcode bits IR[15:12]. Since IR[15:12] speciﬁes the opcode of the current LC- 3b instruction being processed, the next address of the control store will be one of 16 addresses, corresponding to the 14 opcodes plus the two unused opcodes, IR[15:12] = 1010 and 1011. That is, each of the 16 next states is the ﬁrst state to be carried out after the instruction has been decoded in state 32. For example, if the instruction being processed is ADD, the address of the next state is state 1, whose microinstruction is stored at location 000001. Recall that IR[15:12] for ADD is 0001. If, somehow, the instruction inadvertently contained IR[15:12] = 1010 or 1011, the A Simple Datapath Can Become Very Powerful Review: The Power of Abstraction

 The concept of a control store of microinstructions enables the hardware designer with a new abstraction: microprogramming

 The designer can translate any desired operation to a sequence of microinstructions

 All the designer needs to provide is

 The sequence of microinstructions needed to implement the desired operation

 The ability for the control logic to correctly sequence through the microinstructions

 Any additional datapath elements and control signals needed (no need if the operation can be “translated” into existing control signals) 7

Review: Advantages of Microprogrammed Control

 Allows a very simple design to do powerful computation by controlling the datapath (using a sequencer)

 High-level ISA translated into microcode (sequence of u-instructions)

 Microcode (u-code) enables a minimal datapath to emulate an ISA

 Microinstructions can be thought of as a user-invisible ISA (u-ISA)

 Enables easy extensibility of the ISA

 Can support a new instruction by changing the microcode

 Can support complex instructions as a sequence of simple microinstructions

 Enables update of machine behavior

 A buggy implementation of an instruction can be fixed by changing the microcode in the field

8 Wrap Up Microprogrammed Control

 Horizontal vs. Vertical Microcode

 Nanocode vs. Microcode vs. Millicode

 Microprogrammed MIPS: An Example

9 Horizontal Microcode

 A single control store provides the control signals

Microcode ALUSrcA storage IorD Datapath IRWrite Outputs control ouPCWritetputs MIPS design PCWriteCond …. From n-bit mPC input output signal control P&H, Appendix D Input

1 bit

- k Sequencing Microprogram counter control Adder

Address select logic

Inputs from instruction register opcode field

 Two-level control store: the first specifies abstract operations 1-bit signal means do this RT (or combination of RTs) Microcode “PC  PC+4” storage “PC  ALUOut” D“PCata pa tPC[h 31:28 ],IR[ 25:0 ],2’b00” Outputs c“oIRn trol MEM[ PC ]” outputs “A  RF[ IR[ 25:21 ] ]” “B  RF[ IR[ 20:16 ] ]” …………. ……. n-bit mInPCput input 1 m-bit input

Sequencing Microprogram counter control ROM Adder

Address select logic k-bit control signal output

[Based on original figure from P&H CO&D, COPYRIGHT

If done right (i.e., m<

 Nanocode: a level below traditional microcode

 microprogrammed control for sub-systems (e.g., a complicated floating-point module) that acts as a slave in a microcontrolled datapath

 Millicode: a level above traditional microcode

 ISA-level subroutines that can be called by the microcontroller to handle complicated operations and system functions

 E.g., Heller and Farrell, “Millicode in an IBM zSeries processor,” IBM JR&D, May/Jul 2004.

 In both cases, we avoid complicating the main u-controller  You can think of these as “microcode” at different levels of abstraction 12 Nanocode Concept Illustrated

a “mcoded” processor implementation

ROM processor datapath mPC

We refer to this a “mcoded” FPU implementation as “nanocode” when a mcoded ROM arithmetic subsystem is embedded datapath in a mcoded system mPC 13 Microcoded Multi-Cycle MIPS Design

 Any ISA can be implemented with a microprogrammed microarchitecture

 P&H, Appendix D: Microprogrammed MIPS design

 We will not cover this in class

 However, you can do an extra credit assignment for Lab 2

14 Microcoded Multi-Cycle MIPS Design

 Advantages

 Disadvantages

 You should be very familiar with this right now

18 Microprogrammed vs. Hardwired Control

 Advantages

 Disadvantages

 You should be very familiar with this right now

19 Can We Do Better?

 What limitations do you see with the multi-cycle design?

 Limited concurrency

 Some hardware resources are idle during different phases of instruction processing cycle

 “Fetch” logic is idle when an instruction is being “decoded” or “executed”

 Most of the datapath is idle when a memory access is happening

20 Can We Use the Idle Hardware to Improve Concurrency?

 Goal: More concurrency  Higher instruction throughput (i.e., more “work” completed in one cycle)

 Idea: When an instruction is using some resources in its processing phase, process other instructions on idle resources not needed by that instruction

 E.g., when an instruction is being decoded, fetch the next instruction

 E.g., when an instruction is being executed, decode another instruction

 E.g., when an instruction is accessing data memory (ld/st), execute the next instruction

 E.g., when an instruction is writing its result into the register file, access data memory for the next instruction 21 Pipelining

22 Pipelining: Basic Idea

 More systematically:

 Pipeline the execution of multiple instructions

 Analogy: “Assembly line processing” of instructions

 Idea:

 Divide the instruction processing cycle into distinct “stages” of processing

 Ensure there are enough hardware resources to process one instruction in each stage

 Process a different instruction in each stage

 Instructions consecutive in program order are processed in consecutive stages

 Benefit: Increases instruction processing throughput (1/CPI)

 Downside: Start thinking about this… 23 Example: Execution of Four Independent ADDs

 Multi-cycle: 4 cycles per instruction

F D E W F D E W F D E W F D E W Time

 Pipelined: 4 cycles per 4 instructions (steady state)

F D E W F D E W Is life always this beautiful? F D E W F D E W Time

The Laundry Analogy

6 PM 7 8 9 10 11 12 1 2 AM Time Task order A B

 “place one dirty load of clothes in the washer”

 “when the washer is finished, place the wet load in the dryer”

 “when the dryer6 PM is7 finished,8 take9 out10 the11 dry 1load2 and1 fold2 AM ” Time  “when folding is finished, ask your roommate (??) to put the clothes Task away” order A - steps to do a load are sequentially dependent B - no dependence between different loads

C - different steps do not share resources

D 25 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] Pipelining Multiple Loads of Laundry 6 PM 7 8 9 10 11 12 1 2 AM Time Task order 6 PM 7 8 9 10 11 12 1 2 AM TimeA Task B order AC

6 PM 7 8 9 10 11 12 1 2 AM Time

Task order 6 PM 7 8 9 10 11 12 1 2 AM Time A - 4 loads of laundry in parallel Task order B - no additional resources A C - throughput increased by 4 B D - latency per load is the same C

D 26 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 6 PM 7 8 9 10 11 12 1 2 AM PipeliningTime Multiple Loads of Laundry: In Practice Task order A 6 PM 7 8 9 10 11 12 1 2 AM Time B Task order C A D B

6 PM 7 8 9 10 11 12 1 2 AM Time

Task order 6 PM 7 8 9 10 11 12 1 2 AM TimeA

Task B order C A

D B

C the slowest step decides throughput D 27 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 6 PM 7 8 9 10 11 12 1 2 AM PipeliningTime Multiple Loads of Laundry: In Practice Task order A6 PM 7 8 9 10 11 12 1 2 AM Time Task B order C A

D B

6 PM 7 8 9 10 11 12 1 2 AM Time

Task order 6 PM 7 8 9 10 11 12 1 2 AM Time A A

 Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)

 Repetition of identical operations

 The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)

 Repetition of independent operations

 No dependencies between repeated operations

 Uniformly partitionable suboperations

 Processing can be evenly divided into uniform-latency suboperations (that do not share resources)

 Fitting examples: automobile assembly line, doing laundry

 What about the instruction processing “cycle”? 29 Ideal Pipelining

combinational logic (F,D,E,M,W) BW=~(1/T) T psec

T/2 ps (F,D,E) T/2 ps (M,W) BW=~(2/T)

T/3 T/3 T/3 BW=~(3/T) ps (F,D) ps (E,M) ps (M,W)

30 More Realistic Pipeline: Throughput

 Nonpipelined version with delay T BW = 1/(T+S) where S = latch delay

T ps

 k-stage pipelined version

BWk-stage = 1 / (T/k +S ) Latch delay reduces throughput (switching overhead b/w stages) BWmax = 1 / (1 gate delay + S )

T/k T/k ps ps

31 More Realistic Pipeline: Cost

 Nonpipelined version with combinational cost G Cost = G+L where L = latch cost

G gates

 k-stage pipelined version Latches increase hardware cost Costk-stage = G + Lk

G/k G/k

32 Pipelining Instruction Processing

33 Remember: The Instruction Processing Cycle

1. Instruction Fetch fetch (IF) 2. Instruction Decode decode and

register Evaluate operand Address fetch (ID/RF) 3. Execute/Evaluate memory address (EX/AG)  Fetch Operands 4. Memory operand fetch (MEM) 5. Store/writeback Execute result (WB)  Store Result

34 Remember the Single-Cycle Uarch

Instruction [25– 0] Shift Jump address [31– 0] left 2 26 28 0 PCSrc1 1=Jump PC+4 [31– 28] M M u u x x ALU Add result 1 0 Add RegDst Shift Jump left 2 4 Branch MemRead Instruction [31– 26] Control MemtoReg PCSrc2=Br Taken ALUOp MemWrite ALUSrc RegWrite

Instruction [25– 21] Read Read register 1 PC address Read data 1 Instruction [20– 16] Read register 2 Zero Instruction 0 Registers Read ALU [31– 0] 0 ALU M Write data 2 result Address Read 1 Instruction register M data u M memory x u Instruction [15– 11] Write x u 1 Data x data 1 memory 0 Write bcond data 16 32 Instruction [15– 0] Sign extend ALU control

Instruction [5– 0]

ALU operation

T BW=~(1/T) Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 35 Dividing Into Stages 200ps 100ps 200ps 200ps 100ps IF: Instruction fetch ID: Instruction decode/ EX: Execute/ MEM: Memory access WB: Write back register file read address calculation 0 M u x 1 ignore for now

Add

4 Add Add result Shift left 2

Read PC Address register 1 Read data 1 Read register 2 Zero Instruction Registers ALU Read 0 ALU Write data 2 result Address Read 1 data Instruction register M u Data M RF memory Write x u memory x data 1 0 write Write data 16 32 Sign extend

Is this the correct partitioning? Why not 4 or 6 stages? Why not different boundaries? 36 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] Instruction Pipeline Throughput

Program execution 2 2004 400 6 600 8 800 1 0 1000 1 12002 140014 160016 180018 o rder Time (in instructions) Instruction Data lw $1, 100($0) Reg ALU Reg fetch access

Instruction Data lw $2, 200($0) Reg ALU Reg 800ps8 ns fetch access

Instruction lw $3, 300($0) 8800ps ns fetch ... 800ps8 ns

Program 200 400 600 800 1000 1200 1400 execution 2 4 6 8 10 12 14 Time order (in instructions) Instruction Data lw $1, 100($0) Reg ALU Reg fetch acc ess Instruction Data lw $2, 200($0) 2 ns Reg ALU Reg 200ps fetch access

Instruction Data lw $3, 300($0) Reg ALU Reg 200ps2 ns fetch access

200ps2 ns 200ps2 ns 2200ps ns 200ps2 ns 200ps2 ns

5-stage speedup is 4, not 5 as predicted by the ideal model. Why?

37 Enabling Pipelined Processing: Pipeline Registers

IF: Instruction fetch ID: Instruction decode/ EX: Execute/ MEM: Memory access WB: Write back register file read address calculation 0 0 No resource is used by more than 1 stage! M M u u x x 1 1

IF/ID ID/EX EX/MEM MEM/WB

M +4

AdAddd +4

D E

AdAdd d

PC nPC 4 4 PC AdAddd rerseusltult ShSifht ift

leflte f2t 2 n

Read

PCPC AdAddredsresss t rergeisgtiesrte 1r 1

c ReRaeda d

u M

r data 1

t data 1

PC Read

s Read

n ZeZreoro I register 2 register 2 W

Instruction Registers ALU Instruction D Registers ReRaeda d ALU ALU memory Write 0 0 ALU RReaeda d

Write data 2 Aout Address 1 data 2 rerseusltult Address data 1 IR register M data Instruction register M M u M u DaDtat a MDR memory E u u WWriteri te x x mmememoroyry B x x dadtata 1 1 0 0

WWritreit e

dadtata

W E 1616 3232 M SiSgnig n B

exetextnednd

Imm Aout

IF/ID ID/EX EX/MEM MEM/WB IF/ID ID/EX EX/MEM MEM/WB

Add Add Add 4 Add 4 Add Add result result Shift Shift left 2 left 2

n Read o

n Read i

t register 1

PC Address o i

c Read

PC Address t register 1 u c Read

r data 1

u t

r Read data 1 s t Read

n Zero s

I register 2

Instruction n Zero

I register 2 Instruction Registers Read ALU ALU memory Registers Read 0ALU Read memory Write data 2 0 ALU result Address Read 1 Write register data 2 M result Address data 1 register M Ddaattaa M u M Write u x Data memory u Write x u x data 1 memory x data 1 0 Write 0 Write data data 16 32 16 32Sign Sign extend Pipelined Operationextend Example All instruction classes must follow the same path lw Instruction fetch and timing through the pipeline stages. lw lw lw 0 000 Instruction decode lw MM Execution Memory uuu Any performance impact? xxx Write back 111

IIIIFIFF//I/IDIDD IIDID//E/EXX EEEXXX//M/MMEEEMM MMMEEEMMM///W/WWBBB

AAAdddddd

AAdddddd 444 AAAdddddd Add rrrreeesssuuulltltt SShhhiififtftt t lllelleefftft t 222

n Read n

n RReeaadd n n Read

n Read

o Read

i t

o register 1

i t

t register 1

PC Address i register 1 t Addddrreessss t rreeggisisteterr 11 PC c PC AAddddrreessss t register 1 Read

PPCC c

PC Address c RReeaadd c c RReeaadd

c Read

r u

u data 1

u r

t data 1 r

r data 1

t t r ddaatata 1 1

t Read data 1

t t

s RReeaadd s

s Read s

n Zero n n ZZeerroo

I register 2 n

n ZZeerroo I

I register 2

n register 2 Zero I Instruction I rreeggisisteterr 22 I register 2 IInInnsssttrtrruuucccttitiioiioonnn Registers ALU memory RReeegggiiisiisstteteerrrsss RReeeaaaddd AALLUU AALLUU mmeeemmooorrrryyy Write Read 000 AALLUU Address RRReeeaaaddd WWrrriiitittetee dddaaattataa 2 22 0 rrreeesssuuulltltt AAdddddrrreeessss Read 111 rreeggiisstteerr data 2 MM result dddaaattttataa 1 rreeggiisistteterrr MM DDaatata data MM uuu DDaattaa MM u memory uuu WWrrriiitittetee xxx memory u data mmeemmoorryy xxx dddaaatttataa 111 1 000 WWWrrriiitiittetee 0 Wdartiate dddaaattataa 1166 3322 1166 Sign 3322 SSiiigiggnnn extend eeexxxtteteennnddd

IF/ID ID/EX EX/MEM MEM/WB IF/ID ID/EX EX/MEM MEM/WB

Add Add 4 Add Add result 4 Add Add result Shift 97108/Patterson Shift left 2 Figure 06.15 left 2

n Read

o i

t register 1 PC Address n Read

c Read

i u Address t register 1 PC r data 1

c Read

t Read

u s

r data 1 n

t Zero

I Read register 2 Instruction s

n Registers ZeroALU I register 2 Instruction memory Read 0 ALU ReWgisritter s data 2 ALU result Address Read 1 memory Read 0 ALU data Write registerdata 2 M result Address Read 1 u data M register M Data Data u Write u x M memory memory u x Write data x 1 x 0 data 1 Write 0 Write data data 16 32 16 32Sign Sign extend extend

97108/Patterson Figure 06.15 lw $10, 20($1) Instruction fetch sub $11, $2, $3 lw $10, 20($1)

0 Instruction decode Execution M u sub $11, $2, $3 lw $10, 20($1) 00x 1MM uu Memory Write back xx 11 IF/ID ID/EX EX/MEM MEM/WB

IF/ID ID/EX EX/MEM MEM/WB Add IF/ID ID/EX EX/MEM MEM/WB

Add Add 4 Add Add result Shift Add 44 AAdddd Add left 2 rreessuulltt SShhiifftt

n Read lleefftt 22

o i

PC Address t register 1

c Read u

r data 1 t

n Read

o i

n Zero i

t register 1

Address I register 2 PC t register 1

PC AddresInsstruction c Read

c Read

u Registers ALU u

r dRaetaa 1d ALU r memory t data 1 t Read 0 Read s WReriated Address s data 2 result 1

n Zero

n Zero data I rreeggisistteerr 2 M IInnssttrruuccttiioonn I register 2 M RReeggiisstteerrss u AALLUU memory RReeaadd 0 AALLUU Data u memory WWrriittee data 2 0x result Address RReeaadd 1 data 2 result Addressmemory data 1x drreeaggtaiisstteerr 1MM data u 0MM u Write Datta u WWrriittee xx u data memorry xx ddaattaa 11 00 16 32 WWrriittee Sign ddaattaa extend 1166 3322 Pipelined OperationSSiiggnn Example eexxtteenndd Clock 1 ClocCk lo5ck 3 sluwb $$1101,, 2$02(,$ $13) slwub $ $1101, ,2 $02($, 1$)3 lw $10, 20($1) IInsttrructtiion ffettch IInsttrructtiion deecode Execution sub $11, $2, $3 sluwb $ $1101, ,2 $02(,$ $13) slwub $ $1101, ,2 $02($, 1$)3 00 MM uu Execution Memory WWrriittee bbaacckk xx 11

IIFF////IIIIDD IIDD//EEXX EEXX////MMEEMM MMEEMM//WWBB

AAdddd

AAdddd 44 AAdddd Add rreessuullttt SShhiiififfftt lleeffttt 22

n Read

n n n Reeaadd

n RReeaadd

i i

t register 1

t t PC Address t rreeggiiisstteerr 11 PC AAdddrreessss t register 1

PPCC AAddddrreessss c Read

c c c Read c Reaadd

c RReeaadd

u r

u data 1

r t

r ddaattaa 11

t t

t data 1 t

t Read

s RReeaadd s

s Read

s s

n Zero

n n n Zero n ZZerroo I register 2 Zero

n Zero

I I I register 2 I rreeggiiisstteerr 22 Instruction I register 2 IIInnssttrruucctttiioonn Registers ALU RReeggiisstteerrss RReeaadd AALLUU AALLUU mmeemmoorryy Write Read 00 ALU RReeaadd Wrriittee ddaatttaa 22 rreessuullltlttt AAddddrrreessss Read 11 rrreeggiiisistteerrr MM ddaattaa register M MM uu DDaattaa M DDaattaa uu WWrriitttee xx mmeemmoorryy data mmeemmoorryy xx ddaattaa 11 00 Is life always this beautiful? WWrrriiiittee ddaattttaa 1166 3322 SSiiiiggnn eexxtttteenndd

Clock 3 CCClloollccoCkkc lk5o6 c21k 4

sub $11, $2, $3 lw $10, 20($1) 40 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] sub $11, $2, $3 lw $10, 20($1) sub $11, $2, $3 Instruction00 fetch Instruction decode M uu Execution Memory Write back 0xx 11M u x 1 IIFF//IID IID//EEXX EEXX//MEM MEEM//WBB

IF/ID ID/EX EX/MEM MEM/WB AAdddd

Add 44 Add AAdddd Add rreesuulltt Shhiifftt 4 Add Add lleefftt 22 result Shift

n Read

n Read left 2

i i

t register 1 PPC AAddddrreessss t register 1 c Read

c Read

u r

r data 1

t data 1 t Read

n Read

s o

n Zero i n Zero

I register 2 t PC AddreIsnsstruction I register 21 Instruction c Registers Read ALU u Registers Reaadd ALU ALLU memory r data 1 memory t WRerriiatteed 00 Address Reeaadd s ddaattaa 22 rreesuulltt Address 11

n register M Zero ddaattaa I register 2 M Instruction u M Registers Read u ALU ALU Daattaa u memory Wrriittee 0xx Read u data 2 result Addressmeemoorry 1xx dreaagttaister 11M data 00M u Wrriittee Data Write x u ddaattaa memory x data 1 0 116 332 Write SSiiggnn data exxttennd 16 32 Sign extend ClockC l6ock 4 Clock 2 Illustrating Pipeline Operation: Operation View

t0 t1 t2 t3 t4 t5

Inst0 IF ID EX MEM WB Inst1 IF ID EX MEM WB Inst2 IF ID EX MEM WB Inst3 IF ID EX MEM WB Inst4 IF ID EX MEM IF ID EX IF ID steady state (full pipeline) IF

41 Illustrating Pipeline Operation: Resource View

t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

IF I0 I1 I2 I3 I4 I5 I6 I7 I8 I9 I10

ID I0 I1 I2 I3 I4 I5 I6 I7 I8 I9

EX I0 I1 I2 I3 I4 I5 I6 I7 I8

MEM I0 I1 I2 I3 I4 I5 I6 I7

WB I0 I1 I2 I3 I4 I5 I6

42 Control Points in a Pipeline

PCSrc

0 M u x 1

IF/ID ID/EX EX/MEM MEM/WB

Add

Add Add 4 result Branch Shift RegWrite left 2

n Read MemWrite

o i PC Address t register 1

c Read u

r data 1 t Read ALUSrc s MemtoReg

n ZeZreoro Instruction I register 2 Registers Read ALU ALU memory 0 Read Write data 2 result Address 1 register M data M u Data Write x u memory x data 1 0 Write data Instruction [15– 0] 16 32 6 Sign ALU extend control MemRead

Identical set of control points as the single-cycle datapath!! 43 Control Signals in a Pipeline

 For a given instruction

 same control signals as single-cycle, but

 control signals required at different cycles, depending on stage

 Option 1: decode once using the same logic as single-cycle and buffer signals until consumed WB

Instruction M WB Control EX M WB

IF/ID ID/EX EX/MEM MEM/WB

 Option 2: carry relevant “instruction word/field” down the pipeline and decode locally within each or in a previous stage Which one is better?

44 Pipelined Control Signals

PCSrc

ID/EX 0 M u WB x EX/MEM 1 Control M WB MEM/WB

EX M WB IF/ID

Add

4 Add result

t i r Branch

W Shift

g e

e left 2

i r

R ALUSrc

m g

n Read

t M PC Address register 1 R

c Read

t u

r data 1 t

Read m

s e

n Zero

I register 2 Instruction M Registers Read ALU memory 0 ALU Write data 2 result Address Read 1 register M data u Data M Write x memory u x data 1 0 Write data

Instruction 16 32 6 [15– 0] Sign ALU MemRead extend control

Instruction [20– 16] 0 ALUOp M Instruction u [15– 11] x 1 RegDst

 Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)

 Repetition of identical operations

 The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)

 Repetition of independent operations

 No dependencies between repeated operations

 Uniformly partitionable suboperations

 Processing an be evenly divided into uniform-latency suboperations (that do not share resources)

 Fitting examples: automobile assembly line, doing laundry

 What about the instruction processing “cycle”? 46 Instruction Pipeline: Not An Ideal Pipeline

 Identical operations ... NOT!  different instructions  not all need the same stages Forcing different instructions to go through the same pipe stages  external fragmentation (some pipe stages idle for some instructions)

 Uniform suboperations ... NOT!  different pipeline stages  not the same latency Need to force each stage to be controlled by the same clock  internal fragmentation (some pipe stages are too fast but all take the same clock cycle time)

 Independent operations ... NOT!  instructions are not independent of each other Need to detect and resolve inter-instruction dependencies to ensure the pipeline provides correct results  pipeline stalls (pipeline is not always moving) 47 Issues in Pipeline Design

 Balancing work in pipeline stages

 How many stages and what is done in each stage

 Keeping the pipeline correct, moving, and full in the presence of events that disrupt pipeline flow

 Handling dependences

 Data

 Control

 Handling resource contention

 Handling long-latency (multi-cycle) operations

 Handling exceptions, interrupts

 Advanced: Improving pipeline throughput

 Minimizing stalls 48

Causes of Pipeline Stalls

 Stall: A condition when the pipeline stops moving

 Resource contention

 Dependences (between instructions)

 Data

 Control

 Long-latency (multi-cycle) operations

49 Dependences and Their Types

 Also called “dependency” or less desirably “hazard”

 Dependences dictate ordering requirements between instructions

 Two types

 Data dependence

 Control dependence

 Resource contention is sometimes called resource dependence

 However, this is not fundamental to (dictated by) program semantics, so we will treat it separately

50 Handling Resource Contention

 Happens when instructions in two pipeline stages need the same resource

 Solution 1: Eliminate the cause of contention

 Duplicate the resource or increase its throughput

 E.g., use separate instruction and data memories (caches)

 E.g., use multiple ports for memory structures

 Solution 2: Detect the resource contention and stall one of the contending stages

 Which stage do you stall?

 Example: What if you had a single read and write port for the register file?

51 Data Dependences

 Types of data dependences

 Flow dependence (true data dependence – read after write)

 Output dependence (write after write)

 Anti dependence (write after read)

 Which ones cause stalls in a pipelined machine?

 For all of them, we need to ensure semantics of the program is correct

 Flow dependences always need to be obeyed because they constitute true dependence on a value

 Anti and output dependences exist due to limited number of architectural registers

 They are dependence on a name, not a value

 We will later see what we can do about them 52

Data Dependence Types

Flow dependence

r3  r1 op r2 Read-after-Write r5  r3 op r4 (RAW)

Anti dependence

r3  r1 op r2 Write-after-Read r1  r4 op r5 (WAR)

Output-dependence

r3  r1 op r2 Write-after-Write r5  r3 op r4 (WAW) r3  r6 op r7

53 lw $10, 20($1) Instruction fetch sub $11, $2, $3 lw $10, 20($1)

0 Instruction decode Execution M u sub $11, $2, $3 lw $10, 20($1) 00x 1MM uu Memory Write back xx 11 IF/ID ID/EX EX/MEM MEM/WB

IF/ID ID/EX EX/MEM MEM/WB Add IF/ID ID/EX EX/MEM MEM/WB

Add Add 4 Add Add result Shift Add 44 AAdddd Add left 2 rreessuulltt SShhiifftt

n Read lleefftt 22

o i

PC Address t register 1

c Read u

r data 1 t

n Read

o i

n Zero i

t register 1

Address I register 2 PC t register 1

PC AddresInsstruction c Read

c Read

u Registers ALU u

r dRaetaa 1d ALU r memory t data 1 t Read 0 Read s WReriated Address s data 2 result 1

n Zero

IIFF////IIIIDD IIDD//EEXX EEXX////MMEEMM MMEEMM//WWBB

AAdddd

AAdddd 44 AAdddd Add rreessuullttt SShhiiififfftt lleeffttt 22

n Read

n n n Reeaadd

n RReeaadd

i i

t register 1

t t PC Address t rreeggiiisstteerr 11 PC AAdddrreessss t register 1

PPCC AAddddrreessss c Read

c c c Read c Reaadd

c RReeaadd

u r

u data 1

r t

r ddaattaa 11

t t

t data 1 t

t Read

s RReeaadd s

s Read

s s

n Zero

n n n Zero n ZZerroo I register 2 Zero

n Zero

I I I register 2 I rreeggiiisstteerr 22 Instruction I register 2 IIInnssttrruucctttiioonn Registers ALU RReeggiisstteerrss RReeaadd AALLUU AALLUU mmeemmoorryy Write Read 00 ALU RReeaadd Wrriittee ddaatttaa 22 rreessuullltlttt AAddddrrreessss Read 11 rrreeggiiisistteerrr MM ddaattaa register M MM uu DDaattaa M DDaattaa uu WWrriitttee xx mmeemmoorryy data mmeemmoorryy xx ddaattaa 11 00 What if the SUB were dependent on WWLW?rrriiiittee ddaattttaa 1166 3322 SSiiiiggnn eexxtttteenndd

Clock 3 CCClloollccoCkkc lk5o6 c21k 4

sub $11, $2, $3 lw $10, 20($1) 54 Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] sub $11, $2, $3 lw $10, 20($1) sub $11, $2, $3 Instruction00 fetch Instruction decode M uu Execution Memory Write back 0xx 11M u x 1 IIFF//IID IID//EEXX EEXX//MEM MEEM//WBB

IF/ID ID/EX EX/MEM MEM/WB AAdddd

Add 44 Add AAdddd Add rreesuulltt Shhiifftt 4 Add Add lleefftt 22 result Shift

n Read

n Read left 2

i i

t register 1 PPC AAddddrreessss t register 1 c Read

c Read

u r

r data 1

t data 1 t Read

n Read

s o

n Zero i n Zero

55 Readings for Next Few Lectures

 P&H Chapter 4.9-4.11

 Smith and Sohi, “The Microarchitecture of Superscalar Processors,” Proceedings of the IEEE, 1995

 More advanced pipelining

 Interrupt and exception handling

 Out-of-order and superscalar execution concepts

56 How to Handle Data Dependences

 Anti and output dependences are easier to handle

 write to the destination in one stage and in program order

 Flow dependences are more interesting

 Five fundamental ways of handling flow dependences

 Detect and wait until value is available in register file

 Detect and forward/bypass data to dependent instruction

 Detect and eliminate the dependence at the software level

 No need for the hardware to detect dependence

 Predict the needed value(s), execute “speculatively”, and verify

 Do something else (fine-grained multithreading)

 No need to detect

57 Interlocking

 Detection of dependence between instructions in a pipelined processor to guarantee correct execution

 Software based interlocking vs.

 Hardware based interlocking

 MIPS acronym?

58 Approaches to Dependence Detection (I)

 Scoreboarding

 Each register in register file has a Valid bit associated with it

 An instruction that is writing to the register resets the Valid bit

 An instruction in Decode stage checks if all its source and destination registers are Valid

 Yes: No need to stall… No dependence

 No: Stall the instruction

 Advantage:

 Simple. 1 bit per register

 Disadvantage:

 Need to stall for all types of dependences, not only flow dep.

Not Stalling on Anti and Output Dependences

 What changes would you make to the scoreboard to enable this?

60 Approaches to Dependence Detection (II)

 Combinational dependence check logic

 Special logic that checks if any instruction in later stages is supposed to write to any source register of the instruction that is being decoded

 Yes: stall the instruction/pipeline

 No: no need to stall… no flow dependence

 Advantage:

 No need to stall on anti and output dependences

 Disadvantage:

 Logic is more complex than a scoreboard

 Logic becomes more complex as we make the pipeline deeper and wider (flash-forward: think superscalar execution) 61 Once You Detect the Dependence in Hardware

 What do you do afterwards?

 Observation: Dependence between two instructions is detected before the communicated data value becomes available

 Option 1: Stall the dependent instruction right away

 Option 2: Stall the dependent instruction only when necessary  data forwarding/bypassing

 Option 3: …

62 We did not cover the following slides in lecture. These are for your preparation for the next lecture. Data Forwarding/Bypassing

 Problem: A consumer (dependent) instruction has to wait in decode stage until the producer instruction writes its value in the register file

 Goal: We do not want to stall the pipeline unnecessarily

 Observation: The data value needed by the consumer instruction can be supplied directly from a later stage in the pipeline (instead of only from the register file)

 Idea: Add additional dependence check logic and data forwarding paths (buses) to supply the producer’s value to the consumer right after the value is available

 Benefit: Consumer can move in the pipeline until the point the value can be supplied  less stalling 64 A Special Case of Data Dependence

 Control dependence

 Data dependence on the Instruction Pointer / Program Counter

65 Control Dependence

 Question: What should the fetch PC be in the next cycle?

 Answer: The address of the next instruction

 All instructions are control dependent on previous ones. Why?

 If the fetched instruction is a non-control-flow instruction:

 Next Fetch PC is the address of the next-sequential instruction

 Easy to determine if we know the size of the fetched instruction

 If the instruction that is fetched is a control-flow instruction:

 How do we determine the next Fetch PC?

 In fact, how do we know whether or not the fetched instruction is a control-flow instruction? 66