ISBN 978-0-557-01974-8 90000 > 9 780557 019748 OpenSPARC™ Internals OpenSPARC T1/T2 CMT Throughput Computing David L. Weaver, Editor Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054 U.S.A. 650-960-1300 Copyright 2002-2008 Sun Microsystems, Inc., 4150 Network Circle • Santa Clara, CA 950540 USA. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. For Netscape Communicator, the following notice applies: Copyright 1995 Netscape Communications Corporation. All rights reserved. Sun, Sun Microsystems, the Sun logo, Solaris, OpenSolaris, OpenSPARC, Java, MAJC, Sun Fire, UltraSPARC, and VIS are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. or its subsidiaries in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements. RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227- 7015(b)(6/95) and DFAR 227.7202-3(a). DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. ISBN 978-0-557-01974-8 First printing, October 2008 Contents Preface . xiii 1 Introducing Chip Multithreaded (CMT) Processors . 1 2 OpenSPARC Designs . 7 2.1 Academic Uses for OpenSPARC . 8 2.2 Commercial Uses for OpenSPARC . 8 2.2.1 FPGA Implementation. 9 2.2.2 Design Minimization. 9 2.2.3 Coprocessors . 9 2.2.4 OpenSPARC as Test Input to CAD/EDA Tools. 10 3 Architecture Overview. 11 3.1 The UltraSPARC Architecture . 12 3.1.1 Features . 12 3.1.2 Attributes. 13 3.1.2.1 Design Goals . 14 3.1.2.2 Register Windows . 14 3.1.3 System Components . 14 3.1.3.1 Binary Compatibility . 14 3.1.3.2 UltraSPARC Architecture MMU . 15 3.1.3.3 Privileged Software . 15 3.2 Processor Architecture . 15 3.2.1 Integer Unit (IU) . 16 3.2.2 Floating-Point Unit (FPU). 16 3.3 Instructions . 17 3.3.1 Memory Access. 17 3.3.1.1 Memory Alignment Restrictions . 18 3.3.1.2 Addressing Conventions . 18 3.3.1.3 Addressing Range . 18 3.3.1.4 Load/Store Alternate . 19 3.3.1.5 Separate Instruction and Data Memories . 19 3.3.1.6 Input/Output (I/O) . 20 3.3.1.7 Memory Synchronization. 20 v vi Contents 3.3.2 Integer Arithmetic / Logical / Shift Instructions . 20 3.3.3 Control Transfer . 20 3.3.4 State Register Access . 21 3.3.4.1 Ancillary State Registers . 21 3.3.4.2 PR State Registers . 21 3.3.4.3 HPR State Registers. 22 3.3.5 Floating-Point Operate . 22 3.3.6 Conditional Move . 22 3.3.7 Register Window Management . 22 3.3.8 SIMD. 22 3.4 Traps . 23 3.5 Chip-Level Multithreading (CMT) . 23 4 OpenSPARC T1 and T2 Processor Implementations . 25 4.1 General Background . 25 4.2 OpenSPARC T1 Overview. 27 4.3 OpenSPARC T1 Components . 29 4.3.1 OpenSPARC T1 Physical Core . 29 4.3.2 Floating-Point Unit (FPU). 30 4.3.3 L2 Cache . 31 4.3.4 DRAM Controller . 31 4.3.5 I/O Bridge (IOB) Unit . 31 4.3.6 J-Bus Interface (JBI) . 32 4.3.7 SSI ROM Interface . 32 4.3.8 Clock and Test Unit (CTU) . 32 4.3.9 EFuse. 33 4.4 OpenSPARC T2 Overview. 33 4.5 OpenSPARC T2 Components . 34 4.5.1 OpenSPARC T2 Physical Core . 35 4.5.2 L2 Cache . 35 4.5.3 Memory Controller Unit (MCU) . 35 4.5.4 Noncacheable Unit (NCU) . 36 4.5.5 System Interface Unit (SIU) . 36 4.5.6 SSI ROM Interface (SSI). 36 4.6 Summary of Differences Between OpenSPARC T1 and OpenSPARC T2 . 36 4.6.1 Microarchitectural Differences . 37 4.6.2 Instruction Set Architecture (ISA) Differences . 37 4.6.3 MMU Differences . 39 4.6.4 Performance Instrumentation Differences . 40 4.6.5 Error Handling Differences . 40 4.6.6 Power Management Differences . 41 4.6.7 Configuration, Diagnostic, and Debug Differences. 42 vii 5 OpenSPARC T2 Memory Subsystem — A Deeper Look . 43 5.1 Caches . 44 5.1.1 L1 I-Cache. 44 5.1.2 L1 D-Cache . 44 5.1.3 L2 Cache . 45 5.2 Memory Controller Unit (MCU) . 47 5.3 Memory Management Unit (MMU). 50 5.3.1 Address Translation Overview . 50 5.3.2 TLB Miss Handling. 51 5.3.3 Instruction Fetching. 52 5.3.4 Hypervisor Support . 53 5.3.5 MMU Operations . 54 5.3.5.1 TLB Operation Summary. 54 5.3.5.2 Demap Operations . 54 5.4 Noncacheable Unit (NCU). 55 5.5 System Interface Unit (SIU). 55 5.6 Data Management Unit (DMU) . 56 5.7 Memory Models. 56 5.8 Memory Transactions. 57 5.8.1 Cache Flushing . 58 5.8.2 Displacement Flushing . 58 5.8.3 Memory Accesses and Cacheability . 59 5.8.4 Cacheable Accesses. 59 5.8.5 Noncacheable and Side-Effect Accesses . 60 5.8.6 Global Visibility and Memory Ordering . 60 5.8.7 Memory Synchronization: MEMBAR and FLUSH. 61 5.8.8 Atomic Operations . 62 5.8.9 Nonfaulting Load . 63 6 OpenSPARC Processor Configuration . 65 6.1 Selecting Compilation Options in the T1 Core . 66 6.1.1 FPGA_SYN . ..