SPARC Joint Programming Specification 1 Implementation Supplement: Sun Ultrasparc III

SPARC Joint Programming Specification 1 Implementation Supplement: Sun UltraSPARC III

Sun Microsystems Proprietary/Need-to-Know JRC Contributed Material

Working Draft 1.0.5, 10 Sep 2002

Part No.: 806-6754-1 Working Draft 1.0.5, 10 Sep 2002 Copyright 2001 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303 U.S.A. All rights reserved. Portions of this document are protected by copyright 1994 SPARC International, Inc. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, SunSoft, SunDocs, SunExpress, and Solaris are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements. RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015(b)(6/95) and DFAR 227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON- INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a. Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Des parties de ce produit pourront être dérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd. La notice suivante est applicable à Netscape Communicator™: Copyright 1995 Netscape Communications Corporation. Tous droits réservés.

Sun, Sun Microsystems, the Sun logo, AnswerBook2, docs.sun.com, et Solaris sont des marques de fabrique ou des marques déposées, ou marques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays. Toutes les marques SPARC sont utilisées sous licence et sont des marques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc.

L’interface d’utilisation graphique OPEN LOOK et Sun™ a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés. Sun reconnaît les efforts de pionniers de Xerox pour la recherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industrie de l’informatique. Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre se conforment aux licences écrites de Sun. CETTE PUBLICATION EST FOURNIE "EN L’ETAT" ET AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, N’EST ACCORDEE, Y COMPRIS DES GARANTIES CONCERNANT LA VALEUR MARCHANDE, L’APTITUDE DE LA PUBLICATION A REPONDRE A UNE UTILISATION PARTICULIERE, OU LE FAIT QU’ELLE NE SOIT PAS CONTREFAISANTE DE PRODUIT DE TIERS. CE DENI DE GARANTIE NE S’APPLIQUERAIT PAS, DANS LA MESURE OU IL SERAIT TENU JURIDIQUEMENT NUL ET NON AVENU.

Please Recycle Contents

1. Overview 1 1.1 Navigating the UltraSPARC III Implementation Supplement 1 1.2 Fonts and Notational Conventions 2 1.3 The UltraSPARC III Processor 2 1.3.1 Component Overview 2 1.3.2 Instruction Issue Unit (IIU) 4 1.3.3 Integer Execution Unit (IEU) 5 1.3.4 Data Cache Unit (DCU) 5 1.3.5 Floating Point and Graphics Unit (FGU) 6 1.3.6 External Memory Unit (EMU) 6 1.3.7 System Interface Unit (SIU) 7 1.4 Chip Differences from UltraSPARC I, II 7 1.4.1 Bootbus Limitations 8 1.4.2 Instruction Set Extensions 8 VIS Extensions 8 Interval Arithmetic Support 9 1.4.3 Instruction Differences 9 1.4.4 Memory Subsystem 10 Caches 10 Cache Flushing 11 Translation Lookaside Buffers (TLBs) 12 1.4.5 Interrupts 13 1.4.6 Address Space Size 13 1.4.7 Error Correction 13

Contents iii Sun Microsystems Proprietary/Confidential – JRC Contributed Material 1.4.8 Registers 14 Address Space Identifier (ASI) Registers 14 Ancillary State Registers (ASRs) 15 1.4.9 Noncacheable Store Compression 15 1.4.10 Summary of Differences 16

2. Definitions and Acronyms 19

3. Architectural Overview 23

4. Data Formats 25 4.2.3 Floating-Point, Quad-Precision 25 4.3 Graphics Data Formats 25

5. Registers 27 5.1.7 Floating-Point State Register (FSR) 27 FSR_nonstandard_fp (NS) 27 FSR_floating-point_trap_type (ftt) 28 FSR_current_exception (cexc) 28 5.2.1 PSTATE Register 29 5.2.9 Version (VER) Register 29 5.2.11 Ancillary State Registers (ASRs) 30 Performance Control Register (PCR) (ASR 16) 30 Performance Instrumentation Counter (PIC) Register (ASR 17) 30 Dispatch Control Register (DCR) (ASR 18) 30 5.2.12 Registers Referenced Through ASIs 33 Data Cache Unit Control Register (DCUCR) 33 Data Watchpoint Registers 35 Instruction Trap Register 36

6. Instructions 37 6.1 Processor Pipeline 37 6.1.1 Instruction-Fetch Stages 39 A-stage (Address Generation) 39 P-stage (Predictor Address Generation) 39 F-stage (Fetch) 39

iv SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material B-stage (Branch Target Computation) 39 6.1.2 Instruction Issue Stages 40 I-stage (Instruction Issue) 40 R-stage (Register) 40 6.1.3 Integer Instruction Execution: E-stage (Execute) 41 6.1.4 Floating-Point and VIS Instruction Execution 42 C-stage (Cache) 42 M-stage (Miss) 42 W-stage (Write) 42 X-stage (Extend) 43 6.1.5 Trap (T) and Done (D) Stages 43 T-stage (Trap) 43 D-stage (Done) 43 6.2 Grouping Rules 43 6.2.1 Execution Order 44 6.2.2 Integer Register Dependencies to Instructions in the MS Pipeline 44 6.2.3 Integer Instructions Within a Group 45 6.2.4 Same-Group Bypass 45 6.2.5 Floating Point Unit Operand Dependencies 45 Latency and Destination Register Addresses 45 PDIST Special Cases 46 Helpers 46 Floating-Point Grouping Rules 47 6.3 Conditional Moves 51 6.4 Instruction Latencies and Dispatching Properties 52

7. Traps 61 7.1.2 Error_state 61 7.4.2 Trap Type (TT) 61

8. Memory Models 63 8.1 Programmer-Visible Properties of Models 64 8.1.1 Differences Between Memory Models 64 8.1.2 Current Memory Model 65 8.2 Memory Location Identification 65

Working Draft 1.0.5, 10 Sep 2002 Contents v Sun Microsystems Proprietary/Confidential – JRC Contributed Material 8.3 Memory Accesses and Cacheability 65 8.3.1 Coherence Domains 66 Cacheable Accesses 66 Noncacheable and Side-Effect Accesses 66 8.3.2 Global Visibility and Memory Ordering 67 8.4 Memory Synchronization 68 8.4.1 MEMBAR #Sync 68 8.4.2 MEMBAR Rules 68 8.5 Atomic Operations 69 8.6 Nonfaulting Load 71 8.7 Prefetch Instructions 71 8.8 Block Loads and Stores 72 8.9 I/O and Accesses with Side Effects 72 8.9.1 Instruction Prefetch to Side-Effect Locations 73 8.9.2 Instruction Prefetch Exiting Red State 73 8.9.3 UltraSPARC III Internal ASIs 74 8.10 Store Compression 74

A. Instruction Definitions: UltraSPARC III Extensions 77 A.2 Alignment Instructions (VIS I) 78 A.3 Three-Dimensional Array Addressing Instructions (VIS I) 78 A.4 Block Load and Store Instructions (VIS I) 79 A.5 Byte Mask and Shuffle Instructions (VIS II) 83 A.13 Floating-Point Add and Subtract 83 A.26 Load Floating-Point 84 A.27 Load Floating-Point Alternate 84 A.30 Load Quadword, Atomic 84 A.33 Logical Operate Instructions (VIS I) 85 A.35 Memory Barrier 86 A.42 Partial Store (VIS I) 91 A.43 Partitioned Add/Subtract (VIS I) 91 A.44 Partitioned Multiply (VIS I) 92 A.47 Pixel Formatting (VIS I) 92 A.47.5 FPMERGE Instruction 92 A.49 Prefetch Data 93 A.49.1 SPARC V9 Prefetch Variants 93

vi SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material A.55 Set Interval Arithmetic Mode (VIS II) 94 A.59 SHUTDOWN Instruction (VIS I) 94 A.61 Store Floating Point 94 A.62 Store Floating Point into Alternate Space 95

B. IEEE Std 754-1985 Requirements for SPARC V9 97 B.3 Overflow, Underflow, and Inexact Traps 97 B.6 Floating-Point Nonstandard Mode 98 B.6.1 Subnormal Operands 98 B.6.2 Subnormal Results 99 B.6.3 NaN Operands 100

C. Implementation Dependencies 101 C.1 List of Implementation Dependencies 102 C.2 SPARC V9 General Information 112 C.2.1 Level 2 Compliance (Impl. Dep. #1) 113 C.2.2 Unimplemented Opcodes, ASIs, and the ILLTRAP Instruction 113 C.2.3 Trap Levels (Impl. Dep. #37, 38, 39, 40, 114, 115) 113 C.2.4 Trap Handling (Impl. Dep. #16, 32, 33, 35, 36, 216, 217) 114 C.2.5 TPC/TnPC and Resets 114 C.2.6 SIR Support (Impl. Dep. #116) 114 C.2.7 TICK Register 115 C.2.8 Population Count Instruction (POPC) 115 C.2.9 Secure Software 115 C.3 SPARC V9 Integer Operations 115 C.3.1 Integer Register File and Window Control Registers (Impl. Dep. #2) 116 C.3.2 Clean Window Handling (Impl. Dep. #102) 116 C.3.3 Integer Multiply and Divide 116 C.3.4 Version Register (Impl. Dep. #2, 13, 101, 104) 116 C.4 SPARC V9 Floating-Point Operations 117 C.4.1 Subnormal Operands/Results; NaN Operands 117 C.4.2 Overflow, Underflow, and Inexact Traps (Impl. Dep. #3, 55) 117 C.4.3 Quad-Precision Floating-Point Operations (Impl. Dep. #3) 118 C.4.4 Floating-Point Upper and Lower Dirty Bits in FPRS Register 119 C.4.5 Floating-Point Status Register (Impl. Dep. #13, 19, 22, 23, 24) 119

Working Draft 1.0.5, 10 Sep 2002 Contents vii Sun Microsystems Proprietary/Confidential – JRC Contributed Material C.5 SPARC V9 Memory-Related Operations 121 C.6 Non-SPARC V9 Extensions 123 C.6.3 DCR Register Extensions 123 C.6.4 Other Extensions 123

D. Formal Specification of the Memory Models 125

E. Opcode Maps 127

F. Memory Management Unit 129 F.1 Virtual Address Translation 129 F.2 Translation Table Entry (TTE) 130 F.4 Hardware Support for TSB Access 131 F.4.1 Typical TLB Miss/Refill Sequence 132 F.4.2 Faults and Traps 133 F.8 Reset, Disable, and RED_state Behavior 133 F.10 Internal Registers and ASI Operations 134 F.10.3 Accessing MMU Registers 134 F.10.3 Instruction/Data MMU TLB Tag Access Registers 134 F.10.4 I/D TLB Data In, Data Access, and Tag Read Registers 134 Data In and Data Access Registers 134 F.10.6 I/D Translation Storage Buffer Base Registers 135 F.10.7 I/D TSB Extension Registers 136 F.10.12 I/D TLB CAM Diagnostic Register 137 F.12 Translation Lookaside Buffer Hardware 138 F.12.2 TLB Replacement Policy 138 F.12.3 TSB Pointer Logic Hardware Description 140

G. Assembly Language Syntax 143

H. Software Considerations 145

I. Extending the SPARC V9 Architecture 147

J. Programming with the Memory Models 149

K. Changes from SPARC V8 to SPARC V9 151

L. Address Space Identifiers 153

viii SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material L.1 Address Space Identifiers and Address Spaces 153 L.2 ASI Values 153 L.3 ASI Assignments 154 L.3.1 Supported SPARC V9 ASIs 154 L.3.2 UltraSPARC III ASI Assignments 154 L.3.3 Special Memory Access ASIs 157 Block Load and Store ASIs 157 Partial Store ASIs 157

M. Caches and Cache Coherency 159 M.1 Cache Organization 159 M.1.1 Virtual Indexed, Physical Tagged Caches (VIPT) 159 M.1.2 Physical Indexed, Physical Tagged Caches (PIPT) 160 Instruction Cache (I-Cache) 160 External and Write Caches (E-Cache, W-Cache) 161 M.2 Cache Flushing 162 M.2.1 Address Aliasing Flushing 162 M.2.2 Committing Block Store Flushing 163 M.2.3 Displacement Flushing 163 M.3 Coherence Tables 164 M.3.1 Processor State Transition and the Generated Transaction 164 M.3.2 Snoop Output and Input 166 M.3.3 Transaction Handling 170

N. Interrupt Handling 175 N.4 Interrupt ASR Registers 175 N.4.2 Interrupt Vector Dispatch Register 175 N.4.3 Interrupt Vector Dispatch Status Register 175 N.4.5 Interrupt Vector Receive Register 175

O. Reset and RED_state 177 O.1 RED_state Characteristics 177 O.2 Resets 178 O.2.1 Hard Power-on Reset (Hard POR, Power-on Reset, POK Reset) 178 O.2.2 System Reset (Soft POR, Fireplane Reset, POR) 179 O.2.3 Externally Initiated Reset (XIR) 179

Working Draft 1.0.5, 10 Sep 2002 Contents ix Sun Microsystems Proprietary/Confidential – JRC Contributed Material O.2.4 Software-Initiated Reset (SIR) 179 O.3 RED_state Trap Vector 179 O.4 Machine States 180

P. Error Handling 185 P.1 Error Classes 186 P.2 Corrective Actions 186 P.2.1 Fatal Error (FERR) 186 P.2.2 Precise Traps 187 P.2.3 Deferred Traps 188 Error Barriers 188 TPC, TNPC, and Deferred Traps 188 Enabling Deferred Traps 189 Errors Leading to Deferred Traps 189 Special Access Sequence for Recovering Deferred Traps 189 Deferred Trap Handler Functionality 189 P.2.4 Disrupting Traps 190 P.2.5 Multiple Traps 191 P.2.6 Entering RED_state 191 P.3 Memory Errors 191 P.3.1 E-cache Data ECC Error 191 Hw_corrected E-cache Data ECC Errors 191 Sw_correctable E-cache Data ECC Errors 192 Uncorrectable E-cache Data ECC Errors 194 P.3.2 Errors on the System Bus 196 Hw_Corrected System Bus Data and MTag ECC Errors 196 Uncorrectable System Bus Data ECC Errors 197 Uncorrectable System Bus MTag Errors 198 System Bus BERR Errors 198 System Bus Timeout Errors 199 System Bus Hardware Timeouts 200 P.3.3 Memory Errors and Prefetch 200 Memory Errors and Prefetch by the Instruction Fetcher and I-cache 200 P.3.4 Memory Errors and Interrupt Transmission 201 P.3.5 Cache Flushing in the Event of Multiple Errors 202

x SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material P.4 Error Registers 202 P.4.1 E-cache Error Enable Register 202 P.4.2 Asynchronous Fault Status Register 204 AFSR Fields 204 Clearing the AFSR 206 P.4.3 ECC Syndromes 208 E_SYND 208 M_SYND 212 P.4.4 Asynchronous Fault Address Register 213 P.5 Error Reporting Summary 214 P.6 Overwrite Policy 217 P.6.1 AFAR Overwrite Policy 217 P.6.2 E_SYND Data ECC Syndrome Overwrite Policy 218 P.6.3 M_SYND MTag ECC Syndrome Overwrite Policy 218 P.7 Multiple Errors and Nested Traps 219 P.8 Further Details on Detected Errors 219 P.8.1 E-cache Data ECC Error 220 The UCC Error 220 The UCU Error 221 The EDC Error 221 The EDU Error 222 The WDC Error 223 The WDU Error 223 The CPC Error 224 The CPU Error 224 P.8.2 System Bus ECC Errors 224 The CE Error 224 The UE Error 225 The EMC Error 225 The EMU Error 225 The IVC Error 226 The IVU Error 226 P.9 Further Details of ECC Error Processing 226 P.9.1 System Bus ECC Error Detection 226 P.9.2 System Bus ECC Error Injection 227 P.9.3 E-cache ECC Errors 228

Working Draft 1.0.5, 10 Sep 2002 Contents xi Sun Microsystems Proprietary/Confidential – JRC Contributed Material P.9.4 When Are Traps Taken? 228 P.9.5 When Are Interrupts Taken? 232 P.9.6 Error Barriers 233 P.10 UltraSPARC III Behavior Under Asynchronous Error Conditions 235 P.10.1 External Cache Access Errors 235 P.10.2 UltraSPARC III Behavior with Error from System Bus 242 P.11 External Memory Unit Error Handling 251 P.11.1 Asynchronous Fault Status Register 252 P.11.2 EMU Error Status Register 253 P.11.3 EMU Error Mask Register 257 P.11.4 EMU Shadow Register 258 P.11.5 EMU Shadow Scan Chain Order 259 Mask Chain Order 259 Shadow Scan Chain Order 259

Q. Performance Instrumentation 265 Q.1 Performance Control and Counters 265 Q.2 Performance Instrumentation Counter Events 268 Q.2.1 Instruction Execution Rates 268 Q.2.2 IIU Statistics 269 Q.2.3 IIU Stall Counts 269 Q.2.4 R-stage Stall Counts 270 Q.2.5 Recirculate Counts 270 Q.2.6 Memory Access Statistics 271 Q.2.7 System Interface Statistics 272 Q.2.8 Software Statistics 272 Q.2.9 Floating-Point Operation Statistics 272 Q.2.10 Memory Controller Statistics 273 Q.2.11 PCR.SL and PCR.SU Encoding 273

R. Specific Information About Fireplane Interconnect 277 R.1 Power Management 277 R.1.1 Low Power Mode 277 R.2 Fireplane Interconnect ASI Extensions 278 Fireplane Port ID Register 278 Fireplane Configuration Register 279

xii SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material Fireplane Interconnect Address Register 283 R.3 RED_state and Reset Values 283

S. Summary of Differences Between UltraSPARC III and SPARC64 V 285

T. UltraSPARC III Chip Identification 287 T.1 Allocation of Identification Codes for UltraSPARC III 287 T.2 Identification Registers on UltraSPARC III 288 T.2.1 JTAGID (JTAG Identification Register) 288 T.2.2 Version Register (V9) 289 T.2.3 FIREPLANE_PORT_ID MID field 289 T.2.4 Serial ID Register 290 T.2.5 Revision Register 290 T.2.6 Identification Code on UltraSPARC III Package 291

U. Memory Controller 293 U.1 Memory Subsystem 293 U.2 Programmable Registers 297 U.2.1 Memory Timing Control I (Mem_Timing1_CTL) 297 Timing Diagrams for Mem_Timing1_CTL Fields 301 Settings of Mem_Timing1_CTL 302 U.2.2 Memory Timing Control II (Mem_Timing2_CTL) 303 Timing Diagram for Mem_Timing2_CTL Fields 305 Settings of Mem_Timing2_CTL 305 U.2.3 Memory Timing Control III (Mem_Timing3_CTL) 306 rfr_int Field of Mem_Timing3_CTL Register 306 Settings of Mem_Timing3_CTL in Energy Star 1/32 Mode 307 U.2.4 Memory Timing Control IV (Mem_Timing4_CTL) 308 Settings of Mem_Timing4_CTL in Energy Star 1/32 Mode 308 U.2.5 Memory Address Decoding Registers 309 Address Decoding Logic 310 Bank-Size Settings 311 U.2.6 Memory Address Control Register 313 banksel_n_rowaddr_size Encoding 316 Settings of Mem_Address_Control 319 Bank Select/Row and Column Address Generation Logic 320 U.3 Memory Initialization 321

Working Draft 1.0.5, 10 Sep 2002 Contents xiii Sun Microsystems Proprietary/Confidential – JRC Contributed Material U.4 Energy Star 1/32 Mode 323

V. Debug and Diagnostics Support 325 V.1 Diagnostics Control and Accesses 325 V.2 Floating-Point Control 326 V.3 Data Cache Unit Control Register (DCUCR) 326 V.4 Instruction Cache Diagnostic Accesses 326 V.4.1 Instruction Cache Instruction Fields 327 V.4.2 Instruction Cache Tag/Valid Fields 327 IC_tag: I-cache tag numbers 328 V.4.3 Instruction Cache Snoop Tag Fields 329 V.5 Branch Predictor Array Accesses 331 V.6 Data Cache Diagnostic Accesses 331 V.6.1 Data Cache Data Field 332 V.6.2 Data Cache Tag/Valid Fields 333 V.6.3 Data Cache Microtag Fields 334 V.6.4 Data Cache Snoop Tag Access 334 V.6.5 Data Cache Invalidate 335 V.7 External Cache Diagnostics Accesses 336 V.7.1 External Cache Control Register 336 V.7.2 External Cache Data/ECC Fields 337 V.7.3 External Cache Data Staging Registers 337 V.7.4 External Cache Tag/State Field Diagnostics Accesses 339 V.8 Write Cache Diagnostic Accesses 340 V.8.1 Write Cache Diagnostic Valid Bits Register 340 V.8.2 Write Cache Diagnostic Bank Valid Bits Register 341 V.8.3 Write Cache Diagnostic Data Register 342 V.8.4 Write Cache Tag/Valid Fields 343 V.8.5 Write Cache Snoop Tag Register 344 V.9 Integer Unit Design for Test (DFT) 345 V.9.1 IU Shadow Scan Registers 345 V.9.2 IU Observability Bus Signals 346

Bibliography 351

xiv SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Confidential – JRC Contributed Material S. CHAPTER 1

Overview

In this chapter, we discuss these topics: ■ Navigating the UltraSPARC III Implementation Supplement on page 1 ■ The UltraSPARC III Processor on page 2 ■ Chip Differences from UltraSPARC I, II on page 7

For a general overview of SPARC V9 architecture, please refer to Section 1.3 of Commonality.

1.1 Navigating the UltraSPARC III Implementation Supplement

We suggest that you approach this Implementation Supplement SPARC V9 Joint Programming Specification as follows. First, familiarize yourself with the UltraSPARC III processor and its components by reading these sections: ■ The UltraSPARC III Processor on page 2 ■ Component Overview on page 2

If you are familiar with UltraSPARC I or II, then review their differences from UltraSPARC III in this section: ■ Chip Differences from UltraSPARC I, II on page 7

Study the terminology in Chapter 2, Definitions and Acronyms. Then, for details of architectural changes, see the remaining chapters in this portion of the book as your interests dictate. The Sun Microsystems implementation portion of the book (UltraSPARC-III) closely follows the organization of The SPARC Architecture Manual- Version 9. We suggest you keep a copy close at hand.

We added two new chapters: Chapter , Memory Controller, and Chapter , Debug and Diagnostics Support.

Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.2 Fonts and Notational Conventions

Fonts and notational conventions are as described in Section 1.2 of Commonality.

1.3 The UltraSPARC III Processor

The UltraSPARC III processor is a high-performance, highly integrated superscalar processor implementing the 64-bit SPARC V9 RISC architecture. It can sustain the execution of up to four instructions per cycle, even in the presence of conditional branches and cache misses, mainly because the units asynchronously feed instructions and data to the rest of the pipeline. Instructions that are predicted to be executed are issued in program order to multiple functional units, executed in parallel, and for added parallelism can be completed out-of-order. To further increase the number of instructions executed per cycle, instructions from two basic blocks can be issued in the same group.

The chip fully implements the 64-bit SPARC V9 architecture. It supports a 64-bit virtual address space and a 43-bit physical address space. The core instruction set has been extended to include graphics instructions that provide the most common operations related to two-dimensional image processing, two- and three- dimensional graphics and image compression algorithms, and parallel operations on pixel data with 8- and 16-bit components.

The execution time of an application is the product of three factors: ■ The number of instructions generated by the compiler ■ The average number of cycles required per instruction ■ The cycle time of the processor

The architecture and implementation coupled with new compiler techniques make it possible to reduce each component while not degrading the other two.

1.3.1 Component Overview

The UltraSPARC III processor contains these components: ■ Instruction Issue Unit (IIU) ■ Floating Point and Graphics Unit (FGU) ■ Integer Execution Unit (IEU) ■ Data Cache Unit (DCU) ■ External Memory Unit (EMU) ■ System Interface Unit (SIU)

2 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material FIGURE 1-1 illustrates the major units, which are described in the next subsections.

Instruction Issue Unit (IIU) Instruction Cache Instruction Queue Steering Logic 4 instructions

Floating Point Integer Execution Unit (FGU) Unit (IEU) Fp Multiply Dependency / Trap Logic FpRF Fp Add / Subtract Fp Divide WARF ALU pipes (0 & 1) Graphics Unit Load/Store/Special pipe

Data Cache Unit (DCU)

Data Write Store Cache Cache Queue

E$ SRAM External Memory System Interface System Interconnect 288 Unit (EMU) Unit (SIU) Local Memory 144 DRAM E$ Tags SRAM Snoop pipe Data Switch DRAM Ctrlr Ctrlr Ctrlr Ctrlr

FIGURE 1-1 UltraSPARC III Major Units

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 3 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.3.2 Instruction Issue Unit (IIU)

The IIU fetches instructions and dispatches them into the execution pipelines. Major blocks are defined in Table 1-1.

TABLE 1-1 Instruction Issue Unit Major Blocks

Name Description

Instruction cache 32-Kbyte, 4-way associative, 32-byte line, 128 bits/cycle Branch Predict Array 16K entries, 2-bit predictor/entry PC tracking logic Instruction Translation 8-Kbyte pages: 128-entry, 2-way associative Buffer 64-Kbyte, 512-Kbyte, and 4-Mbyte locked pages: 16-entry fully associative Instruction queue 16-entry Predecoding logic Predecode bits are generated on an instruction cache fill and saved in the instruction cache. Steering Logic Steers up to four instructions per cycle into the six execution pipes defined as follows: Pipe Action A0 Integer Execute and Floating-Point Load A1 Integer Execute and Floating-Point Load MS Load/Store/Special FGM Floating Point and Graphics Multiply FGA Floating Point and Graphics Add BR Branch Next Fetch Address A stage logic to determine the address of the next four calculation instruction fetch groups.

4 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.3.3 Integer Execution Unit (IEU)

The IEU carries out execution of all integer arithmetic, logical, and shift instructions. Table 1-2 describes the IEU major blocks.

TABLE 1-2 Integer Execution Unit Major Blocks

Block Description

Working register file 7 read ports, 3 write ports Architectural register file 3 write ports, 1 transfer port Two integer execution pipelines 64-bit ALU and shifters; one 64-bit virtual address for Floating-Point Load Virtual address adder for the MS pipe 64 bit Arithmetic and special instruction unit Latency-tolerant integer instructions e.g., mulx, mulscc, tadd Trap generation and recirculate logic Register and pipeline dependency maxtl = 5 checking and bypassing logic 8 register file windows

1.3.4 Data Cache Unit (DCU)

The DCU handles all sourcing and sinking of data for load and store instructions. TABLE 1-3 describes the data cache major blocks.

TABLE 1-3 Data Cache Unit Major Blocks

Block Description

Data cache 64-Kbyte, 4-way associative, 32-byte line, write-through. Provides low latency data source for loads. Write cache 2-Kbyte, 4-way associative, 64-byte line. Reduces store bandwidth to the external cache by merging stores. Data Translation Buffer 8-Kbyte pages: 512-entry, 2-way associative, 64-Kbyte, 512-Kbyte, and 4-Mbyte locked pages: 16-entry, fully associative. Store queue Decouples the pipeline from the latency of store operations. Allows the pipeline to continue flowing while the store waits for data and eventually writes into the write cache. Stores may be grouped with the instruction upon which they depend.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 5 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.3.5 Floating Point and Graphics Unit (FGU)

The Floating Point and Graphics Unit provides five pipelines to execute all floating- point instructions and most VIS graphics instructions. A combination of two floating-point or VIS operations can be started each cycle. Most floating-point instructions complete in four cycles (exceptions are div and sqrt).

Table 1-4 defines the FGU major blocks.

TABLE 1-4 Floating Point and Graphics Unit (FGU) Major Blocks

Name Description

Floating-point register file 32-, 64-bit registers. 5 read and 4 write ports Floating-point multiply pipe Floating-point add/sub pipe Floating-point graphics Shares FRF ports with the floating-point add/sub pipe ALU pipe Floating-point graphics Shares FRF ports with the floating-point multiply pipe multiply pipe Floating-point div/sqrt pipe Shares FRF ports with the floating-point multiply pipe

1.3.6 External Memory Unit (EMU)

The External Memory Unit controls the operation of the external cache and external DRAM memory. Table 1-5 describes the major blocks of the EMU.

TABLE 1-5 Major Units of the External Memory Unit

Name Description

External cache tags 90 Kbytes, 1-cycle throughput, 2-cycle latency. Writeback buffer 512 bytes. Holds 2 or 4 dirty external cache lines, depending on the external cache configuration. External cache control Schedules and services the primary-cache and system requests for external cache data. The unit also controls the read and update of external cache tags to maintain cache coherency. DRAM controller Supports up to 4 banks, 8 Gbytes of EDO DRAM.

The External Cache Unite (ECU) and Memory Control Unit (MCU) are contained in the EMU and included the above description.

6 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.3.7 System Interface Unit (SIU)

Table 1-6 describes the SIU major blocks.

TABLE 1-6 System Interface Unit Major Blocks

Name Description

External data switch Control logic schedules all movement of data between the data control switch and local memory. UltraSPARC III data Control logic schedules all movement of data between the data switch control switch and UltraSPARC III/local memory/Sun™ Fireplane Interconnect. Data path control Control logic and data path for incoming data. Control registers control Programmable control registers, control logic for error reporting, interrupt, ASI read/write access. Pending queues Coherent pending queue (CPQ); noncoherent pending queue (NCPQ) to store all pending transactions; outgoing request queue (ORQ); and pending tag array (PTA). Pending queue control Control logic to enqueue and dequeue the transactions in CPQ and NCPQ; control logic for snoop pipeline (PTA), bootbus interface, HBM control, and E* control. Transaction outgoing Buffers for requests and data from E$ and W$ and control logic to buffer generate the outgoing Fireplane Interconnect transaction and data.

1.4 Chip Differences from UltraSPARC I, II

The UltraSPARC III processor differs from previous UltraSPARC processors in several key areas, including: ■ Bootbus limitations ■ Instruction set extensions ■ Instruction differences ■ Memory subsystem ■ Interrupts ■ Address space size ■ Error correction ■ Registers ■ Noncacheable store compression

This section describes the differences and concludes with a summary table of differences.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 7 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.4.1 Bootbus Limitations

All bootbus addresses must be mapped as side-effect pages with the TTE E bit set. In addition, programmers must not issue the following memory operations to any bootbus address: ■ Prefetch instructions ■ Block load and store instructions ■ Any memory operations with ASI_PHYS_USE_EC or ASI_PHYS_USE_EC_LITTLE ■ Partial store instructions

1.4.2 Instruction Set Extensions

UltraSPARC III has added Sun proprietary extensions to the SPARC V9 Instruction Set Architecture (ISA), in addition to those implemented in UltraSPARC I. The extensions were defined with close cooperation of the software groups (WABI, Graphics, OS, Compiler) that will use them. The extensions are in the areas of VIS extensions, prefetch enhancement, and interval arithmetic support.

VIS Extensions

Three new VIS instructions were added: ■ Byte Mask — Sets the Graphics Status Register (GSR) for a following byte shuffle operation. One byte mask can be issued per instruction group as the last instruction of the group. Byte Mask is a break-after instruction. ■ Byte Shuffle — Allows any set of 8 bytes to be extracted from a pair of double- precision, floating-point registers and written to a destination double-precision, floating-point register. The 32-bit byte mask field of the GSR specifies the pattern of source bytes for the byte shuffle instruction. ■ Edge(ncc) — Two variants: the original instruction sets the integer condition codes, and the new instruction does not set condition codes. Differences between the variants are as follows:

Edge Edgencc

Sets integer condition codes Does not set integer condition codes Single-instruction group Groupable with other instructions

Because of implementation restrictions in the pipeline, all instructions that set condition codes and execute in the MS pipeline stage must be in a single instruction group.

8 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Interval Arithmetic Support

One new instruction was added to improve the efficiency of interval arithmetic computations. The Set Interval Arithmetic Mode (SIAM) instruction enables the rounding mode bits in the Floating-Point Status Register (FSR) to be overridden without the overhead of modifying the RD field of the FSR. Updates directly to FSR are expensive because they flush the pipeline.

Please refer to A.55 in Commonality for details.

1.4.3 Instruction Differences

Several instructions changed. ■ SHUTDOWN — Energy Star compliance is achieved through a different mechanism than that used by the UltraSPARC I processor. For compatibility, the SHUTDOWN instruction in UltraSPARC III executes as a NOP. See SHUTDOWN Instruction (VIS I) on page 94 for more details. ■ FLUSH — Since the processor maintains consistency between the instruction cache and all store and atomic instructions, the FLUSH instruction is used only to clear the pipeline. Unlike the case with the UltraSPARC I processor, the FLUSH address is ignored. It is not used for instruction cache flushing and is not propagated to the system. A single FLUSH at the end of sequence of stores is sufficient to synchronize the pipeline. ■ Floating-point conversion instructions — Because of implementation restrictions, the following integer-to-floating-point conversion instructions generate an unfinished_FPop exception for certain ranges of integer operands, as shown in TABLE 1-7.

TABLE 1-7 Integer/Floating-Point to Floating-Point/Integer unfinished_FPop Exception Conditions

Instruction Unfinished Trap Ranges

FsTOi result < − 231, result ≥ 231, Inf, NaN FsTOx |result| ≥ 252, Inf, NaN FdTOi result < − 231, result ≥ 231, Inf, NaN FdTOx |result| ≥ 252, Inf, NaN FiTOs operand < − 222, operand ≥ 222 FxTOs operand < − 222, operand ≥ 222 FxTOd operand < − 251, operand ≥ 251

When the above instructions take an unfinished_FPop trap, system software must properly emulate the instruction and resume execution.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 9 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ■ Floating-point subnormal/NaN handling — Because of implementation restrictions, the processor generates an unfinished_FPop exception over a different range of subnormal and NaN operands and results than that of the UltraSPARC I processor. When an unfinished_FPop trap is generated, it is expected that system software will properly emulate the instruction and resume execution. See NaN Operands on page 100 for more details on FPop operand ranges that generate an unfinished_trap. ■ Ticc reserved ﬁeld checking — The processor checks the reserved field of the Ticc instruction for zero and generates an illegal_instruction trap if the field is nonzero. Neither UltraSPARC I nor UltraSPARC II processors checked the Ticc reserved field for zero.

1.4.4 Memory Subsystem

The memory subsystem design is new. Differences include changes in the caches, cache flushing, and translation lookaside buffers (TLBs).

Caches

The UltraSPARC III memory system comprises five caches; four on-chip and one external to the chip. ■ Data cache (D-cache) — A 64-Kbyte, 4-way associative, virtually indexed, physically tagged (VIPT) cache. The D-cache is write-through, no write-allocate, not included in the external cache. The line size is 32 bytes, no subblocking. The D-cache needs to be flushed only if an alias is created with virtual address bit 13. VA<13> is the only virtual bit used to index the D-cache. ■ Instruction cache (I-cache) — A 32-Kbyte, 4-way associative, physically indexed, physically tagged (PIPT) cache. The I-cache is not included in the external cache. The line size is 32 bytes, no subblocking. The I-cache is kept consistent with the store stream of the processor as well as with external stores from other processors. The I-cache never needs to be flushed, not even for address aliases. ■ Write cache (W-cache) — A 2-Kbyte, 4-way associative, physically indexed, physically tagged (PIPT) cache. The line size is 64 bytes with 32-byte subblocks. The W-cache reduces bandwidth to the external cache by coalescing and bursting stores to the external cache. The W-cache is included in the external cache; all lines in the W-cache have a corresponding line allocated in the external cache. The data state of the W-cache line always supersedes the state of the data in the corresponding external cache line. It is necessary to flush the W-cache for stable storage.

10 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ■ External cache (E-cache) — A 1- to 8-Mbyte, direct mapped, physically indexed, physically tagged cache. The E-cache is write-allocate, write-back. It is necessary to flush the E-cache for stable storage.

Cache Flushing

The UltraSPARC III D-cache differs in size and organization from the UltraSPARC I D-cache and so requires changes to the algorithms used to flush the cache.

The virtually indexed caches need to be flushed when a virtual address alias is created. Caches that contain modified data need to be flushed for stable storage.

Following are flushing requirements for specific caches: ■ Data cache — The UltraSPARC III D-cache is the only cache that needs to be flushed when a virtual address alias is created. Like the UltraSPARC I D-cache, the UltraSPARC III D-cache uses one virtual address bit for indexing the cache and thus creates an alias boundary of 16 Kbytes for the D-cache. ■ Instruction cache — The processor maintains consistency of the on-chip I-cache with the stores from all processors so that a FLUSH instruction is needed only to ensure the pipeline is consistent. This means a single flush is sufficient at the end of a sequence of stores that updates the instruction stream to ensure correct operation. Unlike the case with the UltraSPARC I processor, the FLUSH instruction does not propagate externally since all I-caches in an UltraSPARC III multiprocessor system are maintained consistent. Since the I-cache is a PIPT cache, it does not have to be flushed for virtual address aliases. The I-cache never contains modified data, so it does not have to be flushed for stable storage. ■ External and write caches — Since the E-cache and W-cache can contain modified data, they must be flushed for stable storage. The W-cache is included in the E- cache, so it is sufficient to flush a block from the E-cache; if there is a corresponding block in the W-cache, it will also be flushed. The recommended procedure to flush modified data from the E-cache back to memory is as follows:

■ Load the block (64 bytes) into the floating-point registers by using FP loads or Block Load. ■ Write the floating-point registers to memory with a Block Store Commit. ■ Issue MEMBAR #Sync to ensure completion. The Block Store with Commit instruction will invalidate the block from both the E-cache and the W-cache. Both of these caches are physically indexed, so they do not need to be flushed for address aliases.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 11 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Translation Lookaside Buffers (TLBs)

The instruction and data translation lookaside buffers share the same organization in the UltraSPARC III processor. The instruction and data (I/D) TLBs use a 16-entry, fully associative TLB to hold entries for 4-Mbyte and 512-Kbyte page sizes and all locked pages of any size.

In addition to the small, fully associative TLB, UltraSPARC III has two large TLBs used exclusively for 8-Kbyte page entries. For data accesses, the TLB is a 512-entry, 2-way associative D-TLB. For instruction accesses, the TLB is a 128-entry, 2-way associative I-TLB.

Note – The lock bit is not used in the 8-Kbyte TLBs.

Other TLB differences are described below. ■ TLB flushing — Both the instruction and data TLBs now have a demap-all operation that removes all unlocked Translation Table Entries (TTEs). See I/D TLB CAM Diagnostic Register on page 137 for details. ■ TTE format — The UltraSPARC III processor now has the additional elements in the TTE format:

■ Physical Address field: Expanded from 28 bits (PA<40:13>,TTE<40:13>) to 30 bits (PA<42:13>,TTE<42:13>). See Translation Table Entry (TTE) on page 130 for more information. ■ Synchronous Fault Status Registers (SFSR) extensions — A new fault type was added to the FT field of the SFSR to indicate an I/D TLB miss, and one status bit was added to the I/D TLB SFSRs:

■ NF: Set to signify that the faulting operation was a speculative load instruction. See I/D Synchronous Fault Status Registers (SFSR) in Commonality for more details on the SFSR. ■ I/D Translation Storage Buffer (TSB) Register — Three new register extensions of the I/D TSB register were added to the UltraSPARC III processor. These registers allow a different TSB virtual address base to be used for each of the three virtual address spaces (primary, secondary, nucleus) in the DTLB and two virtual address spaces (primary, nucleus) in the ITLB. On an I/D TLB miss, the processor selects which TSB Extension Register to use to form the TSB base address, based on the virtual space accessed by the faulting instruction. See I/D TSB Extension Registers on page 136 for more details. ■ TLB Data Access Register — The access address for the TLB Data Access Register has been expanded to enable access to three TLBs, each with up to 512 entries. See I/D TLB CAM Diagnostic Register on page 137 for details.

12 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ■ TLB Diagnostic Register — A new register replaces the function of the diagnostic bits in the TTE. See I/D TLB CAM Diagnostic Register on page 137.

1.4.5 Interrupts

The UltraSPARC III processor extends the sun-5 interrupt architecture previously implemented in the UltraSPARC I processor in these areas: ■ Module ID fields — Extended from 5 bits to 10 bits. ■ Interrupt Transmit BUSY/NACK bits — Extended from 1 pair to 32 pairs, enabling pipelining of outgoing interrupts. ■ Data Dispatch and Receive Registers — Expanded from 3 to 8, enabling up to 64 bytes to be transmitted in an interrupt. ■ System Tick Interrupt bit — Added to the soft interrupt register. ■ Interrupts — Now occur only before AX pipe instructions.

For more details, please refer to Appendix N, Interrupt Handling.

1.4.6 Address Space Size

The UltraSPARC III processor extends both the virtual and physical address space previously implemented. It implements the full 64-bit virtual address range defined in the SPARC V9 architecture. There are no VA holes, as in UltraSPARC I. The physical address range has also been extended from 41 bits to 43 bits.

Address space with PA<42> = 1 is considered as the noncacheable address space. Physical address 4000000000016 to 7FFFFFFFFFF16 is in the noncacheable area.

1.4.7 Error Correction

Error correction differs from the UltraSPARC I and II handling, as follows: ■ External cache — The processor uses ECC protection on the E-cache instead of parity protection. It requires software correction and recovery for single bit E- cache ECC read errors, which are signaled as a precise error. See E-cache Data ECC Error on page 191 for details. ■ System interface — A new ECC code has been defined for ECC protection across 132 data bits (9 ECC bits) and 3 MTag bits (4 ECC bits) on the system bus and on the data switch. The syndromes for these codes differ from the syndromes used previously. The processor requires software correction and recovery for single-bit system ECC errors, which are signaled as disrupting errors. See ECC Syndromes on page 208 for details.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 13 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.4.8 Registers

Differences in registers include enhancements to ASI registers and ASR registers.

Address Space Identifier (ASI) Registers

Changes to the ASI registers include those to the following registers: ■ SRAM diagnostic registers — Several new diagnostic ASI registers were added for the following on-chip SRAMs: ■ Write cache ■ Branch predict array ■ I/D TLB CAM Changes were made to fields of existing UltraSPARC II diagnostic ASI registers for the following on-chip SRAMs:

■ Data cache ■ External cache ■ Instruction cache These ASI registers were removed:

■ UDB Error Register ■ UDB Control Register See Chapter , Debug and Diagnostics Support, for more details. ■ Instruction Trap Register — An instruction breakpoint register was added to enable an illegal_instruction trap to be generated on an arbitrary opcode. See Section N.5, Software Interrupt Register,inCommonality for details. ■ Asynchronous Fault Status Register (AFSR) — Several changes were made to add new fault types (E-cache ECC errors) and remove old fault types (SDB errors). See Asynchronous Fault Status Register on page 204 for details. ■ Asynchronous Fault Address Register (AFAR) — The AFAR was extended to handle a 43-bit physical address. It is now updated on several errors that previously did not capture the address. See Asynchronous Fault Status Register on page 204 for details. ■ Software Interrupt Register (SOFTINT) — The SOFTINT register has an additional bit added to signal SYSTEM TICK COMPARE interrupts. See Section N.5, Software Interrupt Register,inCommonality for details. ■ System Interface Registers — The UPA interface ASI has been reused for two new Fireplane Interconnect registers: a configuration register and an address register. See Fireplane Interconnect ASI Extensions on page 278 for details.

14 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Ancillary State Registers (ASRs)

Changes to the ASRs include changes to the following registers: ■ System Tick and Compare — Two new ASRs were added to support a system clock: ASR 1816, a System Tick Register (analogous to the per-processor tick register ASR 416), and ASR 1916, a System Tick Compare Register (analogous to the per-processor tick compare register 1716). For details, see TICK Register on page 115. ■ Graphics Status Register (GSR) — New fields were added to the GSR:

■ 32-bit MASK field used by the BSHUFFLE instruction ■ 1-bit IM field to enable interval arithmetic round mode ■ 2-bit IRND field to specify round mode for interval arithmetic For details see Section 5.2.11 of Commonality. ■ Performance Control Register (PCR) — The Performance Control Register has been extended to enable a larger number of performance events to be measured. For details, see Performance Instrumentation Counter Events on page 268. ■ Dispatch Control Register (DCR) — Many control fields were added to the Dispatch Control Register to aid in debugging first silicon. For details, see Dispatch Control Register (DCR) (ASR 18) on page 30.

1.4.9 Noncacheable Store Compression

Like previous implementations, the UltraSPARC III processor uses a 16-byte buffer to merge adjacent noncacheable stores into a single external data transaction. This merging greatly increases store bandwidth to the graphics frame buffer. A change in the algorithm for determining when to break merging improves store bandwidth to graphics devices.

For more details see Store Compression on page 74.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 15 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 1.4.10 Summary of Differences

TABLE 1-8 summarizes the differences between the UltraSPARC II and UltraSPARC III processors.

TABLE 1-8 UltraSPARC III and UltraSPARC II Differences (1 of 3)

Element UltraSPARC III Processor UltraSPARC II Processor trap Waits for all outstanding instructions and Waits for all outstanding instructions. all deferred errors. (Note: MEMBAR #Sync has error isolation, too.) See Deferred Traps on page 188. PSTATE.tle (trap little- Unchanged. 0. endian after WDR/XIR/ See TABLE O-1 on page 180. See Table 10-1 in UltraSPARC I & II User’s SIR/RED Manual. Internal ASI load/store MEMBAR #Sync is required Makes side effects visible. and MEMBAR #Sync/ 1. Makes side effects visible. DONE RETRY / 2. Between internal ASI store and internal ASI load. See TABLE 8-4 on page 72. MEMBAR Semantics See TABLE 8-1 on page 68 MEMBAR #LoadLoad NOP All loads wait for completion of all loads. MEMBAR #StoreLoad #Sync All loads wait for completion of all loads. MEMBAR #LoadStore NOP All stores wait for completion of all loads. MEMBAR NOP All stores wait for completion of all stores. #StoreStore, STBAR MEMBAR #Lookaside #Sync Waits until all outstanding memory accesses are complete. MEMBAR #Sync #Sync #Sync (waits for all outstanding instructions and all deferred errors). FLUSH Flushes all fetched instructions (address Creates an I$ invalidate request. (MEMBAR don’t care). #StoreStore is needed in PSO and See Instruction Differences on page 9. RMO.) Store and block store Always creates I$ invalidate request. Never creates I$ invalidate request. Memory model support TSO only TSO (Total Store Order) [arch outstanding bug #7012] Process loads in program order. Process stores in program order. PSO (Partial Store Order) Stores are not ordered with respect to each other. RMO (Relaxed Memory Order No priority rules for any two memory references. prefetch instruction Supported Supported. See Instruction Differences on page 9.

16 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material TABLE 1-8 UltraSPARC III and UltraSPARC II Differences (2 of 3)

Element UltraSPARC III Processor UltraSPARC II Processor Load miss Recirculates (blocking) Goes to load buffer (nonblocking). Store miss Goes to W$ Goes to store buffer. Virtual address 64 bit 44 bit (with VA hole) Physical address 43 bit 41 bit SHUTDOWN instruction NOP (privileged). Goes to power-down mode. Demap for IMMU Doesn’t support demap with secondary Supported. context. See TABLE F-14 in Commonality. TPC/TNPC after XIR Inaccurate. Accurate. (TPC=PC and ~1F16, TNPC=TPC+4) TPC/TnPC and Resets on page 114. Note: Neither UltraSPARC III nor UltraSPARC II can return to previous condition after XIR. TICK_CMPR.tick_cmpr 0. Unknown. after POR See TABLE O-1 on page 180 GSR.IM after POR 0. Unknown. See TABLE O-1 on page 180. OBS control By obs[0] pin and %dcr.obs. %dcr.obs ASR register See OBS document.

PIC overflow Causes trap type 4F16. Does not check. See Performance Control and Counters on page 265. Reserved field of TCC Checked. Does not check. instruction See Instruction Differences on page 9. Subnormal handling for More cases cause unfinished_FPop trap Some cases cause unfinished FP_op trap. FPU than with UltraSPARC II. sequence_error by FPU Never causes sequence_error May cause sequence_error. FLUSH instruction causes Never. May cause data_access_MMU_miss; trap? may cause data_access_exception prefetcha instruction to MEMBAR #Sync is required. MEMBAR #Sync is not required because internal ASI prefetcha is NOP. TTE<49:43> <49:48>is reserved <49:41>is Diag <47> snoop bit Async fault status register *** Bit assignment is completely different *** I/D TSB Extension ASI Available. Not available. Registers ASI store/load to/from I$ Forbidden. No problem. valid bit during I$ on PRM open bug #7138 I$ Physically indexed, physically tagged Physically indexed, physically tagged. (with microtag).

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 1 Overview 17 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material TABLE 1-8 UltraSPARC III and UltraSPARC II Differences (3 of 3)

Element UltraSPARC III Processor UltraSPARC II Processor D$ Virtually indexed, physically tagged. Virtually indexed, physically tagged. DMMU-SFSR/DMMU-SFAR dtlb miss trap update. dtlb miss trap does not update. FSR.NS=1 & NaN taken fp_exception_ieee_754. Trap not taken. fadd/fsub/fdivs with Sometimes takes fp_exception_ieee_754 or Takes no trap. NaN/Subnormal operand unfinished_fpop trap.

18 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material S. CHAPTER 2

Definitions and Acronyms

Terminology and acronyms specific to UltraSPARC III, the Sun Microsystems implementation of SPARC V9, are presented below. For definition of terms that are common to all implementations, please refer to Chapter 2 of Commonality.

ARF Architectural Register File.

Ax Either the A0 or A1 pipeline.

BBC Bootbus controller.

CDS Crossbar Data Switch.

CSR Control Status Register.

CPQ Coherent pending queue.

D-cache processor Data memory cache.

DCU Data Cache Unit.

DFT Designed for test.

DQM Data input/output Mask. Q stands for either input or output.

E-cache External memory cache (external to the processor chip).

ECU External Cache Unit.

EMU External Memory Unit.

FFA Synonym for FGA.

FGA Floating-point/Graphics ALU pipeline.

FGM Floating-point/Graphics Multiply pipeline.

FGU Floating Point and Graphics Unit.

19 Sun Microsystems Proprietary/Need -To-Know – JRC Contributed Material FP0 Synonym for FGM.

FP1 Synonym for FGA.

FRF Floating-point Register File.

HBM Hierarchical Bus Mode.

HPE Hardware Prefetch Enable.

I-cache processor Instruction memory.

IEU Instruction Execution Unit.

IIU Instruction Issue Unit.

LPA Local Physical Address.

MCU Memory Control Unit.

module A master or slave device that attaches to the shared-memory bus.

MOESI A cache-coherence protocol. Each of the letters stands for one of the states that a cache line can be in, as follows: M, modified, dirty data with no outstanding shared copy; O, owned, dirty data with outstanding shared copy(s); E, exclusive, clean data with no outstanding shared copy; S, shared, clean data with outstanding shared copy(s); I, invalid, invalid data.

NCPQ Noncoherent pending queue.

ORQ Outgoing request queue.

PIPT Physically indexed, physically tagged.

PIVT Physically indexed, virtually tagged.

PTA Pending tag array.

RTO Read to own.

RTOR Read to own remote. A reissued RTO transaction.

RTS Read to share.

RTSM Read to share Mtag. An RTS to modify MTag transaction.

SAM SPARC Architecture Manual, Version 9.

scrub To write data from the W-cache to the E-cache.

SDRAM Synchronous Dynamic Random Access Memory

SIG Single-Instruction Group; sometimes shortened to “single-group.”

SIU System Interface Unit.

20 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material SPE Software prefetch enable.

SSM Scalable shared memory.

UE User process error.

victimize [Error handling]

VIPT Virtually indexed, physically tagged.

VIS Visual Instruction Set.

VIVT Virtually indexed, virtually tagged.

WAS Write after write.

WRF Working Register File.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 2 Definitions and Acronyms 21 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 22 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material S. CHAPTER 3

Architectural Overview

Please refer to Chapter 3 in Commonality.

23 Sun Microsystems Proprietary/Need -To-Know – JRC Contributed Material 24 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material S. CHAPTER 4

Data Formats

Please refer to Chapter 4 in Commonality. That chapter contains information from Chapter 4, Data Formats,fromThe SPARC Architecture Manual, version 9.

Following is information specific to the UltraSPARC III implementation of SPARC V9: ■ Floating-Point, Quad-Precision on page 25 ■ Graphics Data Formats on page 25

Note that section headings below correspond to those of Chapter 4 in Commonality.

4.2.3 Floating-Point, Quad-Precision

TABLE 4-6 in Chapter 4 of Commonality describes the memory and register alignment for doubleword and quadword. Please note the following:

Implementation Note – Floating-point quad is not implemented in the processor. Quad-precision operations, except ﬂoating-point multiply-add and multiply-subtract, are emulated in the OS kernel.

4.3 Graphics Data Formats

Graphics instructions are optimized for short integer arithmetic, where the overhead of converting to and from floating point is significant. Image components can be 8 or 16 bits; intermediate results are 16 or 32 bits.

Sun frame buffer pixel component ordering is α,G,B,R.

25 Sun Microsystems Proprietary/Need -To-Know – JRC Contributed Material Please refer to Section 4.3 in Commonality for details of pixel graphics format (4.3.1), Fixed16 Graphics format (4.3.2), and Fixed32 Graphics format (4.3.3).

26 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material S. CHAPTER 5

Registers

For general information, please see Chapter 5 in Commonality. Implementation- specific information about the following registers is presented below (section numbers correspond to those in Commonality): ■ FSR_nonstandard_fp (NS) on page 27 ■ PSTATE Register on page 29 ■ Version (VER) Register on page 29 ■ Performance Control Register (PCR) (ASR 16) on page 30 ■ Performance Instrumentation Counter (PIC) Register (ASR 17) on page 30 ■ Dispatch Control Register (DCR) (ASR 18) on page 30 ■ Data Cache Unit Control Register (DCUCR) on page 33 ■ Data Watchpoint Registers on page 35 ■ Instruction Trap Register on page 36

For information on MMU registers, please refer to Section F.10, Internal Registers and ASI Operations, on page 134.

In addition, several registers are described in Chapter , Debug and Diagnostics Support, including the following: ■ Miscellaneous registers supporting diagnostic accesses ■ Integer Unit Design for Test (DFT) on page 345 (shadow scan copies of IU registers)

The SPARC V9 architecture also defines two implementation-dependent registers: the IU Deferred-Trap Queue and the Floating-point Deferred-Trap Queue (FQ); the UltraSPARC III processor does not implement either of these queues. See Chapter 5 of Commonality for more information about deferred-trap queues.

5.1.7 Floating-Point State Register (FSR)

FSR_nonstandard_fp (NS)

If a floating-point operation generates a subnormal value on UltraSPARC III and FSR.NS = 1, the value is replaced by a floating-point zero value of the same sign

27 Sun Microsystems Proprietary/Need -To-Know – JRC Contributed Material (impl. dep. #18). This replacement is usually performed in hardware. However, for the following cases when a subnormal value is generated in the course of the instruction and FSR.NS =1,anfp_exception_other exception with FSR.ftt =2 (unfinished_FPop) is taken and trap handler software is expected to replace the subnormal value with a zero value of the appropriate sign: ■ fadd of numbers with opposite signs ■ fsub of numbers with the same signs ■ fdtos

FSR_floating-point_trap_type (ftt)

UltraSPARC III triggers fp_exception_other with trap type unfinished_FPop under the conditions described in Section B.6. These conditions differ from the JPS1 “standard” set and are described in Section 5.1.7 of Commonality (impl. dep. #248).

FSR_current_exception (cexc)

UltraSPARC III follows the specification in Table 5-10 of Commonality, for treatment of FSR.cexc when a floating-point operation causes an overflow or underflow, with differences in two cases. Those cases are listed in the following table, with the differences marked in bold italic font: TABLE 5-10 Setting of FSR.cexc bits

Exception(s) Current Detected Trap Enable Exception in f.p. Mask bits bits (in fp_exception_ operation (in FSR.TEM) FSR.cexc) ieee_754 of uf nx OFM UFM NXMTrap Occurs? ofc ufc nxc Notes - ✔ ✔ x 0 1 yes 0 10 ✔ - ✔ 0 x 1 yes 1 0 0 (2) Notes: (2) Overflow is always accompanied by inexact.

Although UltraSPARC III hardware returns the FSR.cexc settings shown above, it is assumed that the trap handler for the fp_exception_IEEE_754 trap will change FSR.cexc to correspond to the table in Commonality before FSR.cexc is observed by nonprivileged software. Therefore, as observed by nonprivileged software, the system will appear to adhere to Table 5-10 in Commonality.

28 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 5.2.1 PSTATE Register

The UltraSPARC III processor supports two additional sets (privileged only) of eight 64-bit global registers: interrupt globals and MMU globals. These additional registers are called the trap globals. Two 1-bit fields, PSTATE.IG and PSTATE.MG, were added to the PSTATE register to select which set of global registers to use.

When PSTATE.AM = 1, UltraSPARC III writes the full 64-bit program counter value to the destination register of a CALL, JMPL,orRDPC instruction. When PSTATE.AM = 1 and a trap occurs, UltraSPARC III writes the full 64-bit program counter value to TPC[TL] (impl. dep. # 125).

When PSTATE.AM = 1 and an exception occurs, UltraSPARC III writes the full 64-bit address to the D-SFAR (impl. dep. #241).

Note – Exiting RED_state by writing 0 to PSTATE.RED in the delay slot of a JMPL instruction is not recommended. A noncacheable instruction prefetch can be made to the jmpl target, which can be in a cacheable memory area. That case may result in a bus error on some systems and cause an instruction_access_error trap. Programmers can mask the trap by setting the NCEEN bit in the E-cache Error Enable Register to 0, but this solution masks all noncorrectable error checking. Exiting RED_state with DONE or RETRY avoids the problem.

5.2.9 Version (VER) Register

The version register, VER, is a read-only register that specifies fixed parameters of an implementation. It is read with the RDPR instruction.

On UltraSPARC III, the VER register reads as listed in TABLE 5-11.

TABLE 5-11 VER Register Encoding in UltraSPARC III

Bit VER Field Value

63:48 manuf 003E16 (Sun’s JEDEC code) (impl. dep. #104) 47:32 impl 001416 (impl. dep. #13) 31:24 mask Tape-out # Ver.mask value 3.2 5216 3.4 5416 3.9 5916 3.B 5B16 3.C 5C16 If any additional revision of UltraSPARC III is fabricated, it will receive a mask number greater than 5C16. 23:16 Reserved 0

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 5 Registers 29 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material TABLE 5-11 VER Register Encoding in UltraSPARC III

Bit VER Field Value 15:8 maxtl 5 7:5 Reserved 0 4:0 maxwin 7

5.2.11 Ancillary State Registers (ASRs)

Please refer to Section 5.2.11 of Commonality for details of the ASRs.

Performance Control Register (PCR) (ASR 16)

UltraSPARC III implements the PCR register as described in Section 5.2.11 of Commonality.

Bits 47:32, 26:17, and bit 3 of PCR are unused in UltraSPARC III (impl. dep. #207). They read as zeroes and writes to them are ignored. PCR.NC =0.

In UltraSPARC III, access to PCR is strictly privileged; when PSTATE.PRIV =0,an attempt to execute either an RDPCR or a WRPCR instruction causes a privileged_opcode exception (impl. dep. #250).

See Appendix Q, Performance Instrumentation, for a detailed discussion of the PCR and PIC register usage and event count definitions.

Performance Instrumentation Counter (PIC) Register (ASR 17)

The PIC register is implemented as described in SPARC JPS1 Commonality.

For PICL/PICU encodings of specific event counters, see Appendix Q, Performance Instrumentation.

Note – PICU and PICL were previously referred to as PIC1 and PIC0, respectively, in early UltraSPARC III documentation.

Dispatch Control Register (DCR) (ASR 18)

The Dispatch Control Register is accessed through ASR 18 and should only be accessed in privileged mode. Nonprivileged accesses to this register cause a privileged_opcode trap.

See also TABLE O-1 on page 180 for the state of this register after reset.

30 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material The Dispatch Control Register is illustrated in FIGURE 5-1. The OBS and IFPOE bits are described in the subsections below.

— — OBS BPE RPE SI — IFPOE MS 63 14 13 12 11 6 5 4 3 2 1 0

FIGURE 5-1 Dispatch Control Register (ASR 18)

DCR bits 13:12 are unused on UltraSPARC III; writes to them are ignored and they read as 0 (impl. dep. #203).

UltraSPARC III implements the standard semantics for DCR.BPE, DCR.RPE, DCR.SI, and DCR.MS, as described in Section 5.2.11 of SPARC JPS1 Commonality (impl. dep. #204).

DCR Control for Observability Bus (OBS) (Impl. Dep. #203). To accommodate a system request to enable control of the observability bus through software, a major change was made in the control configuration of the bus. In addition to using pulses at obsdata<0>, bits 11:6 of the Dispatch Control Register (DCR) can be programmed to select the set of signals to be observed at obsdata<9:0>. Note that this feature is an addition to the existing control setup and is available only on versions of UltraSPARC III beyond TO_2.0 (tapeout 2.0).

TABLE 5-12 shows the mapping between the settings on bits 11:6 of the DCR and the values seen on the observability bus.

TABLE 5-12 Signals Observed at obsdata<9:0> for Settings on Bits 11:6 of the DCR

DCR bits Signal obsdata obsdata obsdata obsdata obsdata obsdata obsdata obsdata obsdata 11:6 source <9> <8> <7> <6> <5> <4> <3> <2> <1>

101xxx ECC* 0 0 0 ec_cor ec_uncor sys_uncor sys_cor 0 0 100xxx Clk grid l2clk/4 l2clk/2 l2clk l1clk/4 l1clk/2 l1clk 1 1 1 110xxx IU* a0_valid a1_valid ms_valid br_valid fa_valid fm_valid ins_comp mispred recirc 111000 IOT delta cbu<8> cbu<7> cbu<6> cbu<5> cbu<4> cbu<3> cbu<2> cbu<1> impctl<2> (up) 111001 IOT delta cbd<8> cbd<7> cbd<6> cbd<5> cbd<4> cbd<3> cbd<2> cbd<1> impctl<3> (down) 111011 IOL delta cbu<8> cbu<7> cbu<6> cbu<5> cbu<4> cbu<3> cbu<2> cbu<1> impctl<0> (up) 111010 IOL delta cbd<8> cbd<7> cbd<6> cbd<5> cbd<4> cbd<3> cbd<2> cbd<1> impctl<1> (down) 111110 IOR delta cbu<8> cbu<7> cbu<6> cbu<5> cbu<4> cbu<3> cbu<2> cbu<1> impctl<4> (up)

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 5 Registers 31 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material TABLE 5-12 Signals Observed at obsdata<9:0> for Settings on Bits 11:6 of the DCR (Continued)

DCR bits Signal obsdata obsdata obsdata obsdata obsdata obsdata obsdata obsdata obsdata 11:6 source <9> <8> <7> <6> <5> <4> <3> <2> <1>

111111 IOR delta cbd<8> cbd<7> cbd<6> cbd<5> cbd<4> cbd<3> cbd<2> cbd<1> impctl<5> (down) 111101 IOB delta cbu<8> cbu<7> cbu<6> cbu<5> cbu<4> cbu<3> cbu<2> cbu<1> impctl<6> (up) 111100 IOB delta cbd<8> cbd<7> cbd<6> cbd<5> cbd<4> cbd<3> cbd<2> cbd<1> impctl<7> (down)

* ECC = Error Correcting Code; IU = Integer Unit

Note that in every valid setting of the DCR obs control bits, bit 11 must be set to 1.

Below are some requirements and recommendations for using the DCR to control the observability bus. ■ Use only one or the other mode of control for the observability bus; that is, control either by pulses at obsdata[0] or by programming of the DCR bits. ■ As long as the por_n pin is asserted, the state of obsdata<9:0> will always be 0. Once the device has been reset, the default state becomes visible on the bus. Note that the DCR OBS control bits are reset to all 0’s on a software POR trap. Until the DCR bit 11 is programmed to 1, the obsdata<0> mode of control will have precedence. ■ There is a latency of approximately 5 CPU cycles between writing the DCR and the signals corresponding to that setting appearing at obsdata<9:0>.

For more information on the observability bus, please refer to IU Observability Bus Signals on page 346.

Single Issue Disable (bit 3). When DCR.SI = 0, the processor is in “single- issue” mode, that is, only allows one instruction to be outstanding at a time. Under certain conditions when DCR.SI = 0 and the data cache is enabled (DCUCR.DC =1)1, the processor can stop dispatching instructions and deadlock. The data cache should normally be disabled (DCUCR.DC = 0) when operating in single-issue mode (DCR.SI = 0). If the D-cache must be enabled during single-issue operation, processor lock-up can be avoided by placing a benign MS pipeline instruction (such as “rd %y, %g0”) after each store or after each block of stores (because stores do not block other stores from retiring).

1. The precise conditions for deadlock: processor in single-issue mode (DCR.SI = 0), data cache enabled (DCUCR.DC = 1), a store hits the Dcache, followed by any load steered to the MS pipeline with no non-load MS instructions between the store and the load. Reference: UltraSPARC III Erratum #16, Bug ID#6888.

32 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Interrupt Floating-Point Operation Enable (bit 1). The IFPOE bit enables system software to take interrupts on FP instructions. This enable bit is cleared by hardware at power-on. System software must set the bit as needed. When this bit is enabled, UltraSPARC III forces an fp_disabled trap when an interrupt occurs on FP- only code. The trap handler is then responsible for checking whether the FP is indeed disabled. If it is not, the trap handler then enables interrupts to take the pending interrupt (impl. dep. #203).

Note – This behavior deviates from SPARC V9 trap priorities in that interrupts are of lower priorities than the other two types of FP exceptions (fp_exception_ieee_754, fp_exception_other).

■ This mechanism is triggered for an FP instruction only if none of the approximately 12 preceding instructions across the two integer, load/store, and branch pipelines are valid, under the assumption that they are better suited to take the interrupt (only one trap entry/exit). ■ Upon entry, the handler must check both TSTATE.PEF and FPRS.FEF bits. If TSTATE.PEF = 1 and FPRF.FEF = 1, the handler has been entered because of an interrupt, either interrupt_vector or interrupt_level. In such a case:

■ The fp_disabled handler should enable interrupts (that is, set PSTATE.IE = 1), then issue an integer instruction (for example, add %g0,%g0,%g0). An interrupt is triggered on this instruction.

■ UltraSPARC III then enters the appropriate interrupt handler (PSTATE.IE is turned off here) for the type of interrupt.

■ At the end of the handler, the interrupted instruction is RETRY’d after returning from the interrupt. The add %g0,%g0,%g0 is RETRY’d.

■ The fp_disabled handler then returns to the original process with a RETRY.

■ The “interrupted” FPop is then retried (taking an fp_exception_ieee_754 or fp_exception_other at this time if needed).

Multiscalar Dispatch Enable (MS) (bit 0). In UltraSPARC III, DCR.MS operates as described in Section 5.2 of Commonality and has no additional side effects (impl. dep. #204).

5.2.12 Registers Referenced Through ASIs

Data Cache Unit Control Register (DCUCR)

ASI 4516 (ASI_DCU_CONTROL_REGISTER),VA=016

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 5 Registers 33 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material The Data Cache Unit Control Register contains fields that control several memory- related hardware functions. The functions include Instruction, Write and data caches, MMUs, and watchpoint setting. Most of DCUCR’s functions are described in Section 5.2.12 of Commonality; details specific to UltraSPARC III are described in this section.

After a power-on reset (POR), all fields of DCUCR are set to 0. After a WDR, XIR, or SIR reset, all fields of DCUCR except WE are set to 0 and WE is left unchanged (impl. dep. #240).

See TABLE O-1 on page 180 for the state of this register after reset or RED_state trap.

The Data Cache Unit Control Register, as implemented in UltraSPARC III, is illustrated in FIGURE 5-2 and described in TABLE 5-13. In the table, bits are grouped by function rather than by strict bit sequence.

— CP CV ME RE— — — SL WE PM VM PR PW VRVW — DMIM DC IC

63 50 49 48 4746 45 44 43 42 4140 33 32 25 24 23 22 21 20 4 3 2 1 0

FIGURE 5-2 DCU Control Register Access Data Format (ASI 4516)

TABLE 5-13 DCU Control Register Description

Bits Field Type Use — Description MMU Control 49 CP RW UltraSPARC III implements this cacheability bit as described in Section 5.2.12 of Commonality. (impl. dep. #232) 48 CV RW UltraSPARC III implements this cacheability bit as described in Section 5.2.12 of Commonality. (impl. dep. #232) 3 DM DMMU Enable. Implemented as described in Section 5.2.12 of Commonality. 2 IM IMMU Enable. Implemented as described in Section 5.2.12 of Commonality. Store Queue Control 47 ME RW Noncacheable Store Merging Enable. If cleared, no merging of noncacheable, non- side-effect store data will occur. Each noncacheable store will generate a system bus (Fireplane) transaction. (impl. dep. #240) 46 RE R-A-W Bypass Enable. If cleared, no bypassing of data from the store queue to a dependent load instruction will occur. All load instructions will have their R-A-W predict field cleared. (impl. dep. #240) Second Load Control 42 SL Second Load Steering Enable. If cleared, all load type instructions will be steered to the MS pipeline and no floating-point load type instructions will be issued to the Ax pipelines. (impl. dep. #240)

34 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material TABLE 5-13 DCU Control Register Description (Continued)

Bits Field Type Use — Description Cache Control 41 WE Write Cache Enable. If 0, all W-cache references will be handled as W-cache misses. Each store queue entry will perform an RMW transaction to the E-cache, and the W- cache will be maintained in a clean state. Software is required to flush the W-cache (force it to a clean state) before setting this bit to 0. (impl. dep. #240) 1 DC UltraSPARC III implements DCUCR.DC. When DCUCR.DC = 0 (D-cache is disabled), the data cache is not updated. When the D-cache is reenabled, software must flush any inconsistent lines from the D-cache (impl. dep. #252). 0 IC UltraSPARC III implements DCUCR.IC. When DCUCR.IC = 0 (I-cache is disabled), the instruction cache is not updated. When the I-cache is reenabled, software must invalidate any inconsistent lines from the instruction cache (impl. dep. #253). Watchpoint Control 40:33 PM<7:0> DCU Physical Address Data Watchpoint Mask. Implemented as described in Section 5.2.12 of Commonality. 32:25 VM<7:0> DCU Virtual Address Data Watchpoint Mask. Implemented as described in Section 5.2.12 of Commonality. 24, 23 PR, PW DCU Physical Address Data Watchpoint Enable. Implemented as described in Section 5.2.12 of Commonality. 22, 21 VR, VW DCU Virtual Address Data Watchpoint Enable. Implemented as described in Section 5.2.12 of Commonality.

Under certain conditions when DCR.SI = 0 and the data cache is enabled (DCUCR.DC = 1), the processor can stop dispatching instructions (deadlock). The data cache should normally be disabled (DCUCR.DC = 0) when operating in single- issue mode (DCR.SI = 0). See Single Issue Disable (bit 3) on page 32 for a description of this condition and methods to avoid it when DCUCR.DC =1.

Data Watchpoint Registers

On UltraSPARC III, watchpoint comparison is only done on the MS (memory) pipeline of the processor; any second-issued Ax pipe FP loads will not trigger a watchpoint. For reliable use of the watchpoint mechanism, the second FP load feature (DCUCR.SL) must be disabled (impl. dep. #244).

Note – The first implementation of UltraSPARC III supports a 43-bit physical address space. Software is responsible to write a zero-extended 64-bit address into the PA Data Watchpoint Register.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 5 Registers 35 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Instruction Trap Register

UltraSPARC III implements the Instruction Trap Register (impl. dep. #205).

The Instruction Trap Register enables an illegal_instruction trap to be generated when an instruction opcode is dispatched from the R stage of the UltraSPARC III pipeline. This facility is used to implement instruction breakpoints.

In UltraSPARC III, the least significant 11 bits (bits 10:0) of a CALL or branch (BPcc, FBPcc, Bicc, BPr) instruction in the instruction cache contain the sum of the least significant 11 bits of the architectural instruction encoding (as appears in main memory) and the least significant 11 bits of the virtual address of the CALL/branch instruction (impl. dep. #245).

Therefore, software on UltraSPARC III that writes the Instruction Trap Register to cause a trap on CALL or branch instruction must either ■ set bits 10:0 of the Mask field to 0 to mask out the implementation-dependent bits from the comparison or ■ place in bits 10:0 of the Match field the sum of bits 10:0 of the instruction and bits 12:2 of the instruction’s virtual address (that is, Match<10:0> = instruction<10:0> + VA<12:2>)

On UltraSPARC III, the instruction breakpoint facility is not fully implemented. Although instruction breakpoints do work in the majority of cases, the possibility exists that: ■ An instruction breakpoint that should have been taken will be missed. One example1 is an instruction in the delay slot of a DCTI in which the static prediction bit predicts “not taken”. If the DCTI is actually taken (and therefore the delay slot executed), the delay slot instruction may be requeued from the processor’s mispredict queue (MQ); in that case, the breakpoint (instruction trap) will not be triggered. ■ Arbitrary instructions will be lost from the instruction stream (a potentially catastrophic error)

1. Reference: UltraSPARC III Erratum #46, Bug ID#7046

36 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material S. CHAPTER 6

Instructions

This chapter departs from the organization of Chapter 6 in Commonality. Instead, it focuses on the needs of compiler writers and others who are interested in scheduling instructions to optimize program performance on the UltraSPARC III pipeline. The chapter discusses these subjects: ■ Processor Pipeline on page 37 ■ Grouping Rules on page 43 ■ Conditional Moves on page 51 ■ Instruction Latencies and Dispatching Properties on page 52, with a table that lists these characteristics for the complete UltraSPARC III instruction set.

Please refer to Chapter 6 of Commonality for general information on instructions.

6.1 Processor Pipeline

The UltraSPARC III processor pipeline consists of 15 stages. Instructions pass through the pipeline in order of groups of up to six instructions. The pipeline stages are referred to by the following mnemonic single-letter names, with the reference shown below:

A PFBI J SRECMWXTD

Address generation Predictor address generation Fetch Branch target computation Instruction issue J: extra S: extra Register Execute Cache Miss Write eXtend Trap Done

37 Sun Microsystems Proprietary/Need -To-Know – JRC Contributed Material Rather than instruction execution in a single pipeline, several separate pipelines are each dedicated to execution of a particular class of instructions. The execution pipelines start after the R-stage of the pipe. Not all the execution pipelines use all of the pipeline stages, and in some cases (long-latency, floating-point pipes), the stage names must be extended with a cycle number, such as, D1, D2, etc. The following sections provide a stage-by-stage description of the pipeline. While reading the following sections, you may find it useful to refer to FIGURE 6-1, which illustrates at which pipeline stage each of the large architectural structures resides. A

Instruction Cache P

BP 32 KB, 4-way, 32-Byte lines ITLB F

Instruction Instruction Queue 4 X 4 Steering B, I

Dependency Working Register File 7R 3W Check R

Floating Point VA + D W Register File A0 A1 D$ E (MS) /2 / 2 64-KB DTLB Special Unit

4-way D$ Tag C

Miss Sxt/Align = M

FP Mul / Divide W FP Add / Sub ALU (FGA) Graphics Mul (FGM) Graphics

Store Queue T

WWW Architectural Register File 3W 1T D

Write Cache 2 KB

FIGURE 6-1 Pipeline Diagram

38 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 6.1.1 Instruction-Fetch Stages

The instruction-fetch pipeline stages, A, P, F, and B, are described below.

A-stage (Address Generation)

The address stage generates and selects the fetch address to be used by the instruction cache in the next cycle. The address that can be selected in this stage for instruction fetching comes from several sources including: ■ Sequential PC ■ Branch target (from I-stage) ■ Trap target ■ Interrupt ■ Predicted return target ■ Jmpl target

This stage also contains the instruction prefetch buffer, which holds up to eight instructions prefetched from the sequential stream.

P-stage (Predictor Address Generation)

The predictor address stage starts fetching four instructions from the instruction cache. Since the I-cache has a two-cycle latency, the P-stage and the F-stage are both used to complete an I-cache access. Although the I-cache has a two-cycle latency, it is pipelined and can access a new set of four instructions every cycle. The address used to start an I-cache access was generated in the previous cycle.

F-stage (Fetch)

The F-stage is used for the second half of the I-cache access. At the end of this stage, up to four instructions from an I-cache line (32 bytes) are latched for use in the I- stage. An I-cache fetch group is not permitted to cross an I-cache line.

The P-stage also accesses the Branch Predictor (BP). The BP is a small, single-cycle access SRAM whose output is latched at the end of the P-stage. The BP predicts the direction of all conditional branches, based on the PC of the branch and the direction history of the most recent conditional branches.

B-stage (Branch Target Computation)

The B-stage is the final stage of the I-Fetch pipeline, A P F B. In this stage, the four fetched instructions are first available in a register. The processor analyzes the instructions, looking for DCTIs that can alter the path of execution. It finds the first DCTI, if any, among the four instructions and computes (if PC relative) or predicts

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 39 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material (if register based) its target address. If this DCTI is predicted taken, the target address is passed to the A-stage to begin fetching from that stream; if predicted not taken, the target is passed on to the CTI queue for use in case of mispredict. Also in the B-stage, the computation of the hit or miss status of the instruction fetch is begun, so that the validity of the four instructions can be reported to the instruction queue.

6.1.2 Instruction Issue Stages

The I-stage and R-stage make up the instruction issue stages. The J-stage and the S- stage are additional stages for use when needed.

I-stage (Instruction Issue)

In the I-stage, the four instructions fetched from the I-cache are entered into the instruction queue. The instruction queue is a 4-wide-by-4-deep structure. Each instruction position from the I-cache is directly wired to one of the 4-deep queue positions. No rotation is done to put the instructions into age order.

Each instruction in the fetch group is steered to one of six possible execution pipelines: A0, A1, BR, MS, FGA, or FGM, described in TABLE 6-1.

TABLE 6-1 Execution Pipelines

Pipeline Description

A0 Integer ALU and Floating-point Load pipe 0 A1 Integer ALU and Floating-point Load pipe 1 BR Branch pipe MS Memory/Special pipe FGM Floating Point/Graphics multiply pipe FGA Floating Point/Graphics ALU pipe

R-stage (Register)

The integer working register file is accessed during the R-stage for the operands of the instructions (up to three) that have been steered to the A0, A1, and MS pipelines. At the end of the R-stage, results from previous instructions are bypassed in place of the register file operands if required.

Up to two floating-point or graphics instructions are sent to the Floating Point/ Graphics Unit in this stage.

40 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material The register and pipeline dependencies between the instructions in the group and the instructions in the execution pipelines are calculated concurrently with the register file access. If a dependency is found, the dependent instruction is held in the R-stage until the dependency is released.

The S- and R-stages of each pipeline form a 2-entry queue. The head end of this queue is the R-stage. If the queue contains one instruction, then it is in R; if the queue contains two instructions, then the older instruction is in R and the younger instruction is in S. In every cycle, zero or one instruction may be removed from this queue (that is, going from R to E), and zero or one instruction may be inserted into this queue (that is, going from J to either R or, if R is occupied, to S).

When the queue contains two instructions (that is, if S is occupied), then no instructions may be added to the queue in that cycle, even if one is being removed. If the queue is “out of order” (that is, it contains an instruction that is younger than another instruction still in J), then no instructions may be added to the queue in that cycle, even if one is being removed. Otherwise, the queue may receive a new instruction in any cycle.

6.1.3 Integer Instruction Execution: E-stage (Execute)

Integer instructions in the A0 and A1 pipelines compute their results in the E-stage. The instructions include most arithmetic, all shift, and all logical instructions. Their results are available for bypassing to dependent instructions that are in the R-stage, resulting in single-cycle execution for most integer instructions. The A0 and A1 pipelines are the only two sources of bypass results in the E-stage.

Other integer instructions are steered to the MS pipe and if necessary are sent with their operands to the special execution unit in this stage. Most MS instructions do not proceed [?] to the System Interface Unit (SIU). They can start their execution during the E-stage but will not produce any results to be bypassed until the C-stage.

Load instructions steered to the MS pipe start accessing the data cache during the E- stage. Since the D-cache is SAM [?] accessed, loads do not have to wait for their virtual address to be computed before starting the D-cache access.

Concurrently with the start of the D-cache access, the virtual address is calculated in an adder in the MS pipeline. The virtual address adder drives the data TLB, which starts its access at the end of the E-stage.

Floating-point and graphics instructions access the floating-point register file in the E-stage to obtain their operands. At the end of the E-stage, the results from previous completing floating-point/graphic instructions can be bypassed to the E-stage instructions.

Conditional branch instructions in the BR pipe resolve their directions in the E-stage. Based on their original predicted direction, a mispredict signal is computed and sent to the A-stage for possible refetching of the correct instruction stream.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 41 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material JMPL and RETURN instructions compute their target addresses in the E-stage of the MS pipe. The results are sent to the A-stage to start fetching instructions from the target stream.

6.1.4 Floating-Point and VIS Instruction Execution

Execution of floating-point and VIS instructions is done in the C-stage, M-stage, W- stage, and X-stage.

C-stage (Cache)

The data cache delivers results for doubleword (64-bit) and unsigned word (32-bit) integer loads in the C-stage. The D-TLB is accessed in parallel with the D-cache access and provides a translated physical address at the end of the C-stage.

Special instruction unit results are produced at the end of this stage and can be bypassed to waiting dependent instructions in the R-stage—minimum 2-cycle latency for SIU instructions. The integer pipelines, A0 and A1, write their results back to the working register file in the C-stage.

The C-stage is the first stage of execution for floating-point and VIS instructions in the FGA and FGM pipelines.

M-stage (Miss)

Data cache misses are detected in the M-stage by a comparison of the physical address from the DTLB to the physical address in the D-cache tags. If the load required additional alignment or sign extension (such as signed word, all halfword, and all byte loads), that alignment is carried out in this stage, resulting in a 3-cycle latency for those load operations. D-cache miss requests are sent to the external cache at this time. This stage is used for the second execution cycle of floating-point and VIS instructions.

Load data are available to the floating-point pipelines in the M-stage. Although this is one cycle later than the integer pipelines, a perceived 2-cycle floating-point load latency results from the floating-point execution pipes being pushed back one cycle relative to the integer execution pipelines. The floating-point register file access occurs in E-stage, one cycle after integer register file access.

W-stage (Write)

In the W-stage, the MS integer pipe results are written into the working register file. The W-stage is also used as the third execution cycle of floating-point and VIS instructions.

42 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material X-stage (Extend)

The X pipeline stage is the last execution stage for most floating-point operations (except divide and sqrt) and for all VIS instructions. Floating-point results from this stage are available for bypass to dependent instructions that will be entering the C- stage next cycle.

6.1.5 Trap (T) and Done (D) Stages

This section describes the stages that interrupt or complete instruction execution.

T-stage (Trap)

Traps, including floating-point and integer traps, are signalled in this stage. Any instructions that are younger than the trapping instruction must invalidate their results before reaching the D-stage to prevent their results from being erroneously written into the architectural or floating-point register files.

D-stage (Done)

Integer results are written into the architectural register file in this stage. At this point, they are fully committed and are visible to any traps generated from younger instructions in the pipeline.

Floating-point results are written into the floating-point register file in this stage. These results are visible to any traps generated from younger instructions.

6.2 Grouping Rules

A group is a collection of instructions that launch into the execution pipelines in the same cycle. For example, if four instructions are being evaluated for dispatch in the R-stage and only three of those are allowed to execute in the E-stage, those three instructions are considered to be a group. The fourth instruction, which was held back, is not considered part of the group of instructions that was dispatched.

Instruction grouping rules are necessary for the following reasons. ■ Each pipeline can run only a subset of instructions. ■ Resource dependencies may require special spacing of instructions. ■ Data dependencies may require separation of instructions. ■ Some instructions are multicycle in nature and require special sequencing flows. ■ Special rules can help ease the logic/timing burden.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 43 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Before continuing, we define a few terms that apply to instructions.

break-before: The instruction will always be the first instruction of a group.

break-after: The instruction will always be the last instruction of a group.

single-instruction group (SIG): The instruction will not issue with any other instructions; it will be the only instruction in the group.(SIG is sometimes shortened herein to “single-group.”)

instruction latency: The number of processor cycles after issuing an instruction that a following data-dependent instruction can issue.

blocking, multicycle: The instruction reserves one or more of the execution pipelines for more than one cycle. The reserved pipelines are not available for other instructions to issue into until the blocking, multicycle instruction completes.

6.2.1 Execution Order

● Rule: Within the R-stage, some of the instructions can be dispatched and others cannot. If an instruction is younger than an instruction that is not able to dispatch, then the younger instruction will not be dispatched. “Younger” and “older” refer to instruction order within the program. If two instructions are executed sequentially, then the instruction with the smaller address is considered the older of the two. That is, if the two instructions are executed individually, the instruction that would be executed first is the older instruction.

6.2.2 Integer Register Dependencies to Instructions in the MS Pipeline

● Rule: If an operand of an instruction in the R-stage matches the destination register of an instruction in the MS pipeline’s E-stage, then instruction in the R- stage may not proceed. The MS pipeline has no E-stage bypass. If an operand of an instruction in the R-stage matches the destination register of an instruction in the MS pipeline’s C-stage, then the instruction in the R-stage may not proceed if the instruction in the MS pipeline’s C-stage does not generate its data until the M-stage. For example, LDSB does not have the load data until the M-stage, but LDX has its data in the C-stage. Thus, LDX would not cause an interlock, but LDSB would. Most instructions in the MS pipeline have their data by the M-stage, so there is no dependency check on the MS pipeline’s M-stage destination register. In the case of multicycle MS instructions, the data are always available by the M-stage of the last helper flow.

44 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 6.2.3 Integer Instructions Within a Group

● Rule: Integer instructions within a group are not allowed to write the same destination register. By not writing the same destination register at the same time, we simplify bypass logic and register file write-enable determination and we simplify potential WAW errors as well. The instructions are break-before second destination is written. This rule applies only to integer instructions writing integer registers. Floating-point instructions and floating-point loads (done in the integer A0, A1, and MS pipelines) can be grouped so that two or more instructions in the same group can write the same floating-point destination register.

6.2.4 Same-Group Bypass

● Rule: Same-group bypass is disallowed. The group bypass rule states that no instruction can bypass its result to another instruction in the same group. The one exception to this rule is store. A store instruction can get its store data (rd), but not its address operands (rs1, rs2), from an instruction in the same group.

6.2.5 Floating Point Unit Operand Dependencies

The FPU grouping rules are better understood after a review of FPU operand dependencies, as described below.

Latency and Destination Register Addresses

Floating-point operations have longer latencies than do integer instructions. Moreover, the various floating-point instructions have varying latencies, and we must check the precision of the floating-point operands. Floating-point operands can be generated by the FGA, FGM, Ax, and MS pipelines.

In the current implementation, there are no floating-point latencies that are less than two or more than four cycles (except for floating-point divide and square root and PDIST → PDIST). Thus, in the Integer Execution Unit, we create a pipeline of destination register addresses that runs from the E-stage to the W-stage for the FGA and FGM pipelines.

The floating-point results from Ax and MS pipelines are limited to floating-point loads, which are available to the floating-point pipelines in the M-stage. Thus, we need to create a pipeline of destination register addresses that runs only from the E- stage to the C-stage.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 45 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Associated with these destination register address pipelines is latency information that is calculated in the E-stage and based on the instruction opcode. The FGA and FGM pipelines can run instructions with variable latency. This additional logic enables compares only on the cycles where floating-point data are not available.

Floating-point divide and square root instructions are variable latency, and that latency is fairly large. Divides and square root operations are not pipelined, so the div/sqrt unit can be “busy.” There are two stages for the divide and square root, which the Integer Execution Unit calls the first part and the finishing part. Each of these stages has a destination register address. We propagate the address from the first part to the finishing part when the Floating Point and Graphics Unit tells us that the first part of the operation is complete. We dequeue the address from the finishing part when the Floating Point and Graphics Unit tells us that the operation is complete.

With all of the various pipelines combined, 14 stages of the pipeline can have a destination register that we must check before we can release a floating-point instruction in the R-stage.

Each floating-point operand and result can be either double or single precision. Single-precision registers can be contained within a double-precision address without there being an exact match. To make the comparison, we use a special 5-bit comparator that takes into account the precision of the two register addresses. For example, floating-point double-precision register %f0 encompasses floating-point single precision registers %f0 and %f1. So, a comparison between single precision %f1 and double-precision %f0 should result in the match logic indicating a dependency.

PDIST Special Cases

PDIST-to-dependent-PDIST is handled as a special case with 1-cycle latency. PDIST latency to any other dependent operation is 4-cycle latency. In addition, a PDIST cannot be issued if there is an ST, block store, or partial store instruction in the M- stage of the pipeline. PDIST issue is delayed if there is a store type instruction two groups ahead of it.

Helpers

Sometimes, an instruction as part of its operation requires multiple flows in the pipeline. We call those extra flows after the initial instruction flow helper cycles. The only pipeline that executes such instructions is the MS pipeline. If an instruction requires a helper, that helper is generated in the R-stage. The help generation logic will generate only as many helpers as the instruction requires.

46 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Most of the time, we can determine the number of helpers by examining only the opcode. However, some recirculate cases run the recirculated instruction differently than the original flow down the pipe, and some instructions, like integer multiply and divide, require variable numbers of helpers. Other helper lengths are determined by outside units, for example, the DCU for atomic memory instructions.

Floating-Point Grouping Rules

● Rule: Instructions requiring helpers are always break-after. There can be no instruction in a group that is younger than an instruction that requires helpers. Another way of saying this is “an instruction that requires helpers will be the youngest in its group.” This rule preserves the in-order execution of the integer instructions and simplifies the logic.

● Rule: Helpers are always single-group. Helpers block the pipe from executing other instructions; thus, instructions with helpers are blocking. A helper cycle is always alone in a group. No other instruction will ever be dispatched from the R-stage if there is a helper cycle in the R-stage.

● There are no special rules concerning integer set-cc and integer branch instructions. Integer set-cc instructions can be grouped in any way with integer branches. In fact, any number of set-cc instructions in any order relation to the branch are allowed, provided that they do not violate any other rules. No special rules apply to this specific case. Integer set-cc instructions in the A1 and A0 pipelines can compute a taken/untaken result in the E-stage, which is the same stage in which the branch is evaluating the correctness of its prediction. The control logic guarantees that the correct condition codes are used in the evaluation. The set-cc instructions in the MS pipeline are all Ax instructions. Since all Ax instructions require helpers, a set-cc instruction and dependent branch instruction will never be in the same group. The Ax set-cc makes its condition codes available in the C-stage of its last helper cycle. This case lines up perfectly with the following branch’s E-stage prediction evaluation, so there is no special rule with set-cc instructions in the MS pipeline.

● Rule: Window changing instructions are single-group. The window changing instructions SAVE, RESTORE, and RETURN are all single- group instructions. These instructions are never grouped with any other instruction. This rule greatly simplifies the tracking of register file addresses.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 47 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ● Rule: Window changing instructions force bubbles after. The window changing instructions SAVE, RESTORE, and RETURN also force a subsequent pipeline bubble. A bubble is distinct from a helper cycle in that there is nothing valid in the pipeline within a bubble. During the bubble, control logic transfers the new window from the Architectural Register File (ARF) to the Working Register File (WRF).

● Rule: Disallow fast window-changing flip-flopping. In this case, fast flip-flopping means changing from one window to the next window and back again before the first change has left the pipeline. Consider this sequence of code.

ADD → %l0 R E C M W X T D SAVE R E C M W X v D RESTORE R E C M W X T D

Both SAVE and RESTORE instructions transfer the contents of the new windows in their E-stages from the architectural register file to the working register file. The ARF is written in the D-stage of all instructions. In this example, if the RESTORE is allowed, the value of %l0 in the WRF will not reflect the result of the ADD because the transfer in the RESTORE’s E-stage occurs before the ADD’s result is written to the ARF. The restore is held in the R-stage until all previous window ARF results are written.

● Rule: Write ASR and Write PR instructions are single-group. WRASR and WRPR are always the youngest instructions in a group. That case prevents problems with an instruction being dependent on the result of the write, which occurs late in the pipeline.

We may decide at a later date to make these instructions single-group.

● Rule: Write ASR and Write PR force seven bubbles after. To guarantee that any instruction that starts in the R-stage is started with the most up-to-date status registers, WRASR and WRPR force bubbles after they are dispatched. Thus, if a WRASR or a WRPR instruction is in the pipeline anywhere from the E-stage to the T-stage, no instructions are dispatched from the R-stage (bubbles are forced in).

● Rule: Read ASR and Read PR force up to six bubbles before (break-before multicycle). An instruction can update the ASRs and PRs. Therefore, if an RDASR or RDPR instruction is in the R-stage and any valid instruction is in the integer pipelines from the E-stage to the X-stage, UltraSPARC III does not allow the RDASR and RDPR instructions to be dispatched. Instead, we wait for all pipeline state to write the ASRs and privileged registers and then read them.

48 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ● Rule: Block Load and Block Store are single-group and multicycle. For simplicity in the Integer Execution Unit and for the memory system, BLD and BST are single-group instructions with helpers.

● Rule: FLUSH is single-group and seven bubbles after. To simplify the Instruction Issue Unit and Integer Execution Unit, the FLUSH instruction is single-group. This make instruction cancellation and issue easier. FLUSH is held in the R-stage until the store queue and the pipeline from E-stage through D-stage is empty.

● Rule: FLUSHW is single-group. To simplify the Integer Execution Unit handling of the register file window flush, the FLUSHW instruction is single-group.

● Rule: MEMBAR (#Sync, #Lookaside, #StoreLoad, #Memissue) is single-group. To simplify the Integer Execution Unit and memory system, MEMBAR is a single- group instruction. MEMBAR will not dispatch until the DCU is prepared for these instructions.

● Rule: SAVED and RESTORED are single-group. To simplify the Integer Execution Unit’s window tracking, SAVED and RESTORED are single-group instructions.

● Rule: Software-initiated reset (SIR) is single-group. For simplicity, SIR is a single-group instruction.

● Rule: Load FSR (LDFSR) is single-group and forces seven bubbles after. For simplicity, LDFSR is a single-group instruction.

● Rule: DONE and RETRY are single-group. For simplicity, we handle the DONE and RETRY (trap exit) instructions as single- group. During this time, we are attempting to update the working register file, restore the state prior to the trap, etc., so it is easier to make these instructions single-group.

● Rule: DONE and RETRY force seven bubbles after. It takes a few cycles to properly restore the pre-trap state and the working register file from the architectural register file, so we force bubbles after the trap exit instructions to give us the cycles to do it all. We will not accept a new instruction until the trap exit instruction leaves the pipeline (also known as D+1).

● Rule: Floating-point divide/square-root is busy. The floating-point divide/square-root unit is a nonpipelined unit. The Integer Execution Unit sets a busy bit for each of the two stages of the div/sqrt and depends on the Floating Point and Graphics Unit to clear them. Only the first part of the div/ sqrt is considered to have a busy unit, so once the first part is complete, a new floating-point div/sqrt operation can be started.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 49 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material ● Rule: Floating-point divide/square-root needs a write slot in FGM pipeline. In the stage in which a div/sqrt is moved from the first part to the last part, we cannot issue any instructions to the FGM pipeline. This constraint provides the write slot in the FGM pipeline so the div/sqrt can write the floating-point register file.

● Rule: Floating-point store is dependent on floating-point divide/square-root. The floating-point divide/square-root unit has a latency longer than the normal pipeline. As a result, should a floating-point store depend on the result of a floating- point divide/square-root, the floating-point store instruction may not be dispatched until the floating-point divide/square-root instruction has completed. This restriction is somewhat pessimistic since the store instruction bypasses its store data late in the pipeline. Thus, it should be possible to release the floating-point store somewhat before the floating-point divide/square-root is completed. At this time, we are implementing the pessimistic solution. This approach may change should the floating-point performance require it.

● Rule: Unused floating-point source operands have %f0 dependency. The following floating-point instructions will exhibit dependencies on %f0 when the prior floating-point instruction in an older instruction group use %f0 as the destination register. The false dependency is not recognized for same-group instructions. The intergroup dependency checking logic does not treat these instructions specially even though they do not use an frs1 and/or frs2 operand.

■ %f0 dependency of frs2: fzero{s} fone{s} fsrc1{s}, fnot1{s},

■ %f0 dependency of frs1: fpack16 , fpackfix , fsqrt{s,d,q},

● Rule: Graphics Status Register (GSR) Write instructions are break-after. The SIAM, BMASK, and FALIGNADDR instructions write the GSR. The BSHUFFLE and FALIGNDATA instructions read the GSR in their operation. Because of GSR write latency, a GSR reader cannot be in the same group as a GSR writer unless the GSR reader is older than the GSR writer. The simplest solution to this dependency is to make all GSR write instructions break-after.

Note – The WRGSR instruction is not included in this rule as a special case. The WRGSR instruction is already break-after by virtue of being a WRASR instruction.

50 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material 6.3 Conditional Moves

The compiler needs to have a detailed model of the implementation of the various conditional moves so it can optimally schedule code. TABLE 6-2 describes the implementation of the five classes of SPARC V9 conditional moves in the UltraSPARC III pipeline. FADD and ADD instructions (shaded rows) are also described as a reference for comparison with the conditional move instructions.

TABLE 6-2 SPARC V9 Conditional Moves

RD Busy Instruction Latency Pipes Used Cycles Groupable Dependency

FMOVicc 3 cycles FGA and BR 1 Yes iCC -0 FMOVfcc 3 cycles FGA and BR 1 Yes fCC -0 FMOVr 3 cycles FGA and MS 1 Yes N/A FADD 4 cycles FGA 1 Yes N/A ADD 1 cycle One of A0 or A1 1 Yes N/A MOVcc 2 cycles MS and BR 1 Yes iCC;0 MOVR 2 cycles MS and BR 1 Yes N/A

Where:

Rd latency — The number of processor cycles until the destination register is available for bypassing to a dependent instruction.

Pipes used — The UltraSPARC III pipelines that the instruction busies when it is issued. The pipelines are:

A0 Integer execution (arithmetic,logical,shift, 2nd FP load) A1 Integer execution (arithmetic,logical,shift, 2nd FP load) MS Load/Store BR Branch FGA Floating/Graphics Add

Busy cycles — The number of cycles that the pipelines are not available for other instructions to be issued. A value of 1 signifies a fully pipelined instruction.

Groupable — Whether instructions using pipelines other than those used by the conditional move can be issued in the same cycle as the conditional move.

Working Draft 1.0.5, 10 Sep 2002 S. Chapter 6 Instructions 51 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material {i,f}CC dependency — The number of cycles that a CC setting instruction must be scheduled ahead of the conditional move in order to avoid incurring pipeline stall cycles.

6.4 Instruction Latencies and Dispatching Properties

In this section, a machine description is given in the form of a table (TABLE 6-3) dealing with dispatching properties and latencies of operations. The static or nominal properties are modelled in the following terms (columns in TABLE 6-3), which are discussed below. ■ Latencies ■ Blocking properties in dispatching ■ Pipe resources (Ax, FGA, FGM, MS, BR) ■ Break rules in grouping (before, after, single-group)

The dynamic properties such as effect of cache miss are dealt with elsewhere.

Latency. In the Latency column, latencies are minimum cycles at which a dependent operation (consumer) can be dispatched relative to the producer operation without causing a dependency stall.

Operations like ADDcc produce two results, one in the destination register and another in condition code. For such operations, latencies are stated as a pair x,y where x is for the destination register dependence and y is for the condition code.

A zero latency implies that the producer and consumer operations can be grouped together in a single group, as in {SUBcc, BE %icc}.

Operations like UMUL have different latencies, depending on operand values. These are given as a range, min–max, for example, 6–8 in UMUL. Operations like LDFSR involve waiting for a specified condition. Such cases are described by footnotes and a notation like 32+ for CASA (meaning at least 32 cycles).

Cycles for branch operations (like BPcc) give the dispatching cycle of the retiring target operation relative to the branch. A pair of numbers, like 0,8, is given, depending on the outcome of a branch prediction, where 0 means a correct branch prediction and 8 means a mispredicted case.

Special cases such as FCMP(s,d), in which latencies depend on the type of consuming operations are described in footnotes (bracketed, e.g., [1]).

52 SPARC V9 JPS1 Implementation Supplement: Sun UltraSPARC-III • Working Draft 1.0.5, 10 Sep 2002 Sun Microsystems Proprietary/Need-To-Know – JRC Contributed Material Blocking. The Blocking column gives the number of cycles during which further dispatching is blocked. Operations like FDIVd have limited blocking property, that is, the blocking is limited to another FD operation in succession. Such cases are noted with footnotes.

Pipe. The Pipe column specifies the resource usage. Operations like MOVcc require more than one resource, as designated by the notation MS & BR. The operation LDF can dispatch to either MS or Ax as indicated.

Break and SIG. Grouping properties are given in columns Break and SIG (single-instruction group). In the Break column, an entry can be “Before,” meaning that this operation causes a break in a group so that the operation starts a new group. Operations like RDCCR require dispatching to be stalled until all operations in flight are completed (reach D-stage); in such cases, details are provided in a footnote reference in the Break column.

Operations like ALIGNADDR must be the last in an instruction group, causing a break in the group of type “After.”

Certain operations must be a single-instruction group. They are designated by “Yes” in the SIG column. Break “Before” and “After” are implied by SIG “Yes.”

TABLE 6-3 UltraSPARC III Instruction Latencies and Dispatching Properties (1 of 7)