Opensparc T1 on Xilinx Fpgas – Updates Thomas Thatcher Paul Hartke Thomas.Thatcher@Sun.Com Paul.Hartke@Xilinx.Com Opensparc Engineering Xilinx University Program

Home , MicroBlaze, SPARC

OpenSPARC T1 on Xilinx FPGAs – Updates Thomas Thatcher Paul Hartke [email protected] [email protected] OpenSPARC Engineering Xilinx University Program

RAMP Retreat – August 2008, Stanford Agenda • Quick OpenSPARC Overview • Progress timeline • Current Status > OpenSPARC T1 1.6 Release > OpenSPARC Book > OpenSPARC FPGA Board > Multi-core T1 design > T1 core on BEE3 • Roadmap • Q & A

2 www.opensparc.net RAMP Retreat-Aug 2008 What is OpenSPARC? • Open-Sourced versions of Sun's Microprocessor Products > RTL, Verification Env,documentation, system software > Available for download at www.opensparc.net • Two Processors Available > OpenSPARC T1 > 8 cores, 4 hardware threads per core > 1 floating-point unit external to core, shared by all cores > 4 banks of L2 cache > OpenSPARC T2 > 8 cores, 8 hardware threads per core > Floating-point internal to core, one per core > 8 banks of L2 cache

3 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC T1

• SPARC V9 implementation • Eight cores, four threads each – 32 simultaneous threads • All cores connect through a 134.4 GB/s crossbar switch • High BW 12-way associative 3 MB on-chip L2 cache • 4 DDR2 channels (23 GB/s) • 70W power • ~300M transistors

4 www.opensparc.net RAMP Retreat-Aug 2008 Sun/Xilinx Partnership: Big Goals • Proliferation of OpenSPARC technology • Proliferation of Xilinx FPGA technology • Make OpenSPARC FPGA friendly > Create reference design with complete system functionality > Boot Solaris/Linux on the reference design > Open it up > Seed ideas in the community

Enable multi-core research 5 www.opensparc.net RAMP Retreat-Aug 2008 Timeline

July 06 Jan 07 June 07 Jan 08 Aug 08 OpenSPARC T1

Sun/Xilinx OpenSpARC Stand-alone OpenSolaris on Collaboration T1 on ML411 program under ML411 board Begins board hypervisor First ML505 Support Today

6 www.opensparc.net RAMP Retreat-Aug 2008 New Developments • OpenSPARC T1 1.6 Release • OpenSPARC Book • New OpenSPARC Development Kit > ML505 board with XC5VLX110T FPGA • Multi-core Design • OpenSPARC T1 core running on BEE3 Board

7 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC T1 1.6 Release • Released May, 2008 • Implementation of 4-thread T1 core on Virtex 5 FPGAs > ML505-V5LX110T board > EDK Project files (for EDK 9.2) > Scripts to run complete RTL regression on hardware • Complete setup to boot Solaris > Networking support, including telnet and ftp • Quick start ace files included > Creates an out-of-the-box experience > T1 core boots OpenSolaris in 30 minutes

8 www.opensparc.net RAMP Retreat-Aug 2008 Hardware Block Diagram

MultiPort Xilinx Embedded Developer’s FPGA Boundary Memory Controller External DDR2 Dimm (EDK) Design Cache-processor interface (CPX) MCH-OPB MemCon

CCX-FSL Microblaze Proc Microblaze Debug UART SPARC T1 Core Interface SPARC T1 UART

processor- Fast Simplex 10/100 Ethernet cache interface Links interface (PCX) (FSL) IBM Coreconnect Developed and OPB Bus Working

9 www.opensparc.net RAMP Retreat-Aug 2008 Software Setup • OpenSolaris is booted from a RAM disk Image • Memory Allocation: > 1 MB used by Microblaze firmware > 1 MB used for OpenSPARC Boot PROM image > 80 MB for RAM disk image > Leaving 174 MB for OpenSPARC RAM • Microblaze firmware does address translation to map SPARC addresses to board addresses.

10 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC Development Kit • A kit for OpenSPARC development now available > Board based on the ML505, but with an XC5VLX110T FPGA > Includes USB interface for FPGA programming > Tested with OpenSPARC T1 release 1.6 release design > Eliminates the need to buy a board and then upgrade the FPGA • Shipping now! • Kit Includes: > Board, with power supply and 256 MB DRAM > Platform USB download cable > Host to host SATA crossover cable > Compact flash card with OpenSPARC T1 1.6 ace files

11 www.opensparc.net RAMP Retreat-Aug 2008 Kit Contents

12 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC Kit Donation Program • Sun will donate OpenSPARC Development Kits to qualified universities > Web address below: • See the web page for more details • Also available from directly from Digilent > http://www.digilentinc.com

http://www.opensparc.net/edu/university-program.html

13 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC Internals Book • Covers both OpenSPARC T1 and T2 • Includes > Architectural Overview > Development environments for OpenSPARC > Source (RTL) code overview > Configuring, extending, and verifying OpenSPARC > Porting operating systems to OpenSPARC • 350 Pages • Available in both hardcopy (Amazon.com) and PDF format • Sign-up sheet for early release PDF copy (by poster)

14 www.opensparc.net RAMP Retreat-Aug 2008 Implementing a Multi-core design • We have created a multi-core system by interconnecting two boards. > Opens the door to multi-core designs on BEE3 board • Uses Xilinx Aurora link-layer protocol running over RocketIO™ GTP serial tranceivers > Connected through the SATA connectors on the board • Each GTP channel is 16 bits at 75 Mhz > Connected to Microblaze through an FSL FIFO

15 www.opensparc.net RAMP Retreat-Aug 2008 Multi-core System Block Diagram

Xilinx Embedded Xilinx CacheLink (XCL) FPGA Boundary Developer’s External DDR2 Dimm (EDK) Design

Fast Simplex Links (FSL) MemCon

CCX-FSL Microblaze Proc Microblaze Debug UART SPARC T1 Core Interface SPARC T1 UART Aurora over GTP Ethernet

Developed and FSL connected Aurora-over-GTP Working module to connect to other board New

16 www.opensparc.net RAMP Retreat-Aug 2008 Dual-core system implementation

SATA Cable

Master Node 17 www.opensparc.net RAMP Retreat-Aug 2008 Four-core system implementation

SATA Cable SMA cables

SATA Cable

Master Node

18 www.opensparc.net RAMP Retreat-Aug 2008 Initial Configuration • Master FPGA hosts entire OpenSPARC Address space. > However, Each client MicroBlaze will run firmware code out of its own memory • Both boards have the same bit file > Avoids need to develop and implement separate bit files > CPU ID set by DIP switches on the board • However, software will be different for each board > Master software: services all memory requests > Slave software: only routes memory requests to the other board

19 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC on the BEE3 • BEE3 board uses Virtex 5 FPGAs (same family as ML505) • Re-implemented Release 1.6 design on BEE3 > Very easy to re-target design. > Updated design to EDK 10.1 > Re-implemented on both XC5VLX110T and XC5VLX155T > Generated ace files for BEE3 board > Verified OpenSolaris Boot • Seamless and trouble-free porting experience • Validated BEE3 board infrastructure • Stop by to see our demo!

20 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC on the Bee3 Details

• Single 4-thread OpenSPARC core – 62.5MHz OpenSPARC; 125MHz Microblaze MPMC/MIG – 6-LUTs: 59,350 / 97,280 61% – 36kbit BRAM: 147 / 212 69% – Ethernet in design but not tested Microblaze – 1.5 hour implementation time • EDK project based on Bee3 EDK reference design 4-thread – Uses EDK MPMC4 and MIG OpenSPARC core

21 www.opensparc.net RAMP Retreat-Aug 2008 Bee3 OpenSPARC Four-Core System Diagram (Master-node Configuration)

305.00

105.00 105.00 QSH “cross-over” cable 25.00 25.00

Fujitsu 2x2 15.00 Fujitsu 2x2 30.00 RJ45 CX4 CX4 RJ45 35.00 20.00 40.00 P P P P J T DDR2 DIMM0 DDR2 DIMM0 C C C C A I I I I - - - - G E E E E 65.00 60.00 x x x x p p p p

DDR2 DIMM1 DDR2 DIMM1 r r r r e e e e

Ring Wiring s s s s s s s s

8 8 8 8 5

DDR2 DIMM2 DDR2 DIMM2 x x x x 0

p 4 4 4 4 i n

G G G G M M M M 2

DDR2 DIMM3 DDR2 DIMM3 CCX link m A A A A B B B B m R R R R

H D D D D

D D D D

D D D D a 1 1 3 6 F 7 7 F 7 7 d

R R R R

6 6 6 6 r V L X T 2 2 2 2 5 5VLXT 100.00 6 6 6 6 - - - - 133 133 133 133 - - - - 6 6 6 6 2 2 2 2

6 6 FF1136 6 6 R R R R 7 7 7 7 D D D D

D D D D D D D D

R R R R 4 1 - 2 p B B B B V A A A A i CX4 CX4 n M M M M G G G G

21.00 QSH-DP- User1 User2 QSH-DP- 4 4 4 4 8 1 -

40x2 2 72* 40x2 p V i 040 5VLXT 5VLXT 040 n 180.00 29.00 CX4 PCI-E PCI-E CX4 2 4 2

Ring Wiring 8X 8X 380.00 p . i 5 n V V V V V V

V A 0 8 8 0 8 8 T ...... X 72* 72* 1 1 1 1 1 1 CCX link P W PCI-E PCI-E R Q

CX4 8X 8X CX4 0 S 4 H

0 Ring Wiring - - 4 4 4 4 D P

QSH-DP- User3 User4 QSH-DP- D G G G G M M M M - - 0 A A A A H 40x2 72* B B B B 40x2 4 78.00 S 0 R R R R

D D D D

040 5VLXT 5VLXT 040 Q CCX link D D D D

D D D D 1 1 3 6 F

7 7 F 7 7

R R R R

6 6 6 6 V L X T 2 2 2 2 5

CX4 CX4 6 6 6 6 ------6 6 6 6 5VLXT 2 2 2 2 180.00 6 6 6 6 R R R R 7 7 FF1136 7 7 150.00 D D D D

D D D D D D D D

R R R R B B B B

133 133 133 133 A A A A M M M M G G G G

Q 4 4 4 4 0 S 4 H 0 - - D P P D

DDR2 DIMM0 DDR2 DIMM0 - - 0 H 4 S 0 Master Node DDR2 DIMM1 DDR2 DIMM1 Q DDR2 DIMM2 DDR2 DIMM2

DDR2 DIMM3 DDR2 DIMM3 40.00 10.00

18.00 23.00 102.00 70.00 107.00 Master Node Ring Wiring QSH “cross-over” cable CCX link 22 www.opensparc.net RAMP Retreat-Aug 2008 Bee3 Multi-core Node Block Diagram

Xilinx Embedded Xilinx CacheLink (XCL) FPGA Boundary Developer’s External DDR2 Dimm (EDK) Design

Fast Simplex Links (FSL) MemCon

CCX-FSL Microblaze Proc Microblaze Debug UART SPARC T1 Core Interface SPARC T1 UART RinRingg Ring WiWriniringg Wiring Ethernet

Developed and FSL interconnect to other Working FPGA(s) New Maintain compatability between ml505_v5lx110t and Bee3 designs 23 www.opensparc.net RAMP Retreat-Aug 2008 Bee3 OpenSPARC Four-Core System Diagram (Full Mesh interconnect)

305.00

105.00 105.00 QSH “cross-over” board 25.00 25.00

Fujitsu 2x2 15.00 Fujitsu 2x2 30.00 RJ45 CX4 CX4 RJ45 35.00 20.00 40.00 P P P P J T DDR2 DIMM0 DDR2 DIMM0 C C C C A I I I I - - - - G E E E E 65.00 60.00 x x x x p p p p

DDR2 DIMM1 DDR2 DIMM1 r r r r e e e e

Ring Wiring s s s s s s s s

8 8 8 8 5

DDR2 DIMM2 DDR2 DIMM2 x x x x 0

p 4 4 4 4 i n

G G G G M M M M 2

DDR2 DIMM3 DDR2 DIMM3 CCX link m A A A A B B B B m R R R R

H D D D D

D D D D

D D D D a 1 1 3 6 F 7 7 F 7 7 d

R R R R

6 6 6 6 r V L X T 2 2 2 2 5 5VLXT 100.00 6 6 6 6 - - - - 133 133 133 133 - - - - 6 6 6 6 2 2 2 2

6 6 FF1136 6 6 R R R R 7 7 7 7 D D D D

D D D D D D D D

R R R R 4 1 - 2 p B B B B V A A A A i CX4 CX4 n M M M M G G G G

21.00 QSH-DP- User1 User2 QSH-DP- 4 4 4 4 8 1 -

40x2 2 72* 40x2 p V i 040 5VLXT 5VLXT 040 n 180.00 29.00 CX4 PCI-E PCI-E CX4 2 4 2

Ring Wiring 8X 8X 380.00 p . i 5 n V V V V V V

V A 0 8 8 0 8 8 T ...... X 72* 72* 1 1 1 1 1 1 CCX link P W PCI-E PCI-E R Q

CX4 8X 8X CX4 0 S 4 H 0 - - 4 4 4 4 D P

QSH-DP- User3 User4 QSH-DP- D G G G G M M M M - - 0 A A A A H 40x2 72* B B B B 40x2 4 78.00 S 0 R R R R

D D D D

040 5VLXT 5VLXT 040 Q D D D D

D D D D 1 1 3 6 F

7 7 F 7 7

R R R R

6 6 6 6 V L X T 2 2 2 2 5

CX4 CX4 6 6 6 6 ------6 6 6 6 5VLXT 2 2 2 2 180.00 6 6 6 6 R R R R 7 7 FF1136 7 7 150.00 D D D D

D D D D D D D D

R R R R B B B B

133 133 133 133 A A A A M M M M G G G G

Q 4 4 Ring4 Wi4 ring 0 S 4 H 0 - - D P P D

DDR2 DIMM0 DDR2 DIMM0 - - 0 H

4 CCX link S 0

DDR2 DIMM1 DDR2 DIMM1 Q DDR2 DIMM2 DDR2 DIMM2

DDR2 DIMM3 DDR2 DIMM3 40.00 10.00

18.00 23.00 102.00 70.00 107.00

Ring Wiring QSH “cross-over” board CCX link 24 www.opensparc.net RAMP Retreat-Aug 2008 Bee3 OpenSPARC Eight-Core, Two-Board System Diagram • Use Aurora over CX4 cables to connect “extended” four- core Bee3 board to “base” four-core Bee3 board. > Maintain native OpenSPARC T1 4-way L2 architecture > [Almost] full OpenSPARC T1 32-thread system!

305.00 305.00 105.00 105.00 25.00 25.00 105.00 105.00 25.00 25.00

Fujitsu 2x2 15.00 Fujitsu 2x2 CX4 30.00 CX4 Fujitsu 2x2 15.00 Fujitsu 2x2 RJ45 RJ45 35.00 30.00 RJ45 CX4 CX4 RJ45 35.00 20.00 40.00 20.00 40.00 P P P P J T C C C C A P P P P I I I I J - - - - G T C C C C E E E E

A 65.00 x x x x I I I I

- - - - G 60.00 p p p p E E E E 65.00 r r r r x x x x

60.00 e e e e p p p p s s s s r r r r s s s s e e e e

s s s s 8 8 8 8 5 s s s s x x x x 0 8 8 8 8

5 p x x x x 4 4 4 4 0 i n

4 4 4 4 G G G G M M M M 2 i n m

A A A A G G G G M M M M 2 B B B B m m A A A A R R R R

B B B B

m H D D D D

R R R R D D D D

H D D D D

D D D D

1 1 3 6 F

D D D D F e

7 7 7 7 d

D D D D

R R R R a

1 1 3 6 F F e

7 7 7 7 d 6 6 6 6

r R R R R

V L X T e 5 2 2 2 2

6 6 6 6 6 6 6 6 r V L X T 5 100.00

5VLXT ------2 2 2 2 6 6 6 6 6 6 6 6

5VLXT 100.00 - - - - 2 2 2 2 - - - - 6 6 6 6 6 6 6 6 2 2 2 2 FF1136 R R R R 6 6 6 6 FF1136 7 7 7 7 R R R R 7 7 7 7 D D D D

D D D D D D D D

R R R R D D D D 4 1

- R R R R 2 4 p 1 B B B B V - A A A A i 2 n p B B B B V A A A A i n M M M M G G G G

M M M M G G G G 21.00

21.00 4 4 4 4 4 4 4 4 8 1 8 1 - 2 - p 2 p V i V i n n 180.00 180.00 29.00 CX4 cables 29.00 2 2 4 4 2 2

p 380.00 380.00 p . . i i 5 n 5 n V V V V V V

V V V V V V V V A A 0 8 8 0 8 8 0 8 8 0 8 8 T ...... T ...... X X 1 1 1 1 1 1 1 1 1 1 1 1

P P W W R R Q 0 Q S 4 0 S H 0 4 - H - 0 4 4 4 4 D - P - 4 4 4 4 D P D P G G G G M M M M - -

P D 0 G G G G M M M M A A A A H - B B B B - 4 78.00 0 S A A A A H 0 R R R R B B B B

D D D D 78.00 Q S

0 R R R R

D D D D

D D D D D D D D

1 1 3 6 F

F D D D D

7 7 7 7

D D D D R R R R

1 1 3 6 F

6 6 6 6 F 7 7 7 7

V L X T

2 2 2 2 5

R R R R 6 6 6 6

6 6 6 6 ------V L X T 2 2 2 2 5 6 6 6 6 6 6 6 6 5VLXT 2 2 2 2 180.00 ------6 6 6 6 6 6 6 6 R R R R 2 2 2 2 7 7 7 7 150.00 5VLXT 180.00

FF1136 6 6 6 6 D D D D

R R R R D D D D 7 7 7 7 150.00

D D D D FF1136 D D D D

R R R R D D D D B B B B D D D D A A A A

R R R R M M M M G G G G B B B B

A A A A Q 4 4 4 4 M M M M G G G G 0 S 4 Q H 4 4 4 4 0 0 - S - D 4 P H 0 P D - - - - D P 0 H P 4 D S - 0 - 0 Q H 4 S 0 Q

40.00 10.00

18.00 23.00 40.00 10.00 102.00 70.00 107.00 18.00 23.00 102.00 70.00 107.00 25 www.opensparc.net RAMP Retreat-Aug 2008 Roadmap • OpenSPARC T1 release 1.7 > Setup to boot Ubuntu Linux > Update of EDK project to EDK 10.1 > Improvements to memory controller > Improvements to place and route > Multi-core design • Future Work (possible projects) > Connect T1 core directly to system > Increase size of L1 caches > Current size doesn't efficiently utilize the Block RAMs > Should be able to quadruple size without increasing logic

26 www.opensparc.net RAMP Retreat-Aug 2008 Summary • OpenSPARC: The tools you need to do multi-core research! > Complete EDK project to implement a system > Implemented on both BEE3 board and OpenSPARC Kit > Complete verification environment > Complete software stack > OpenSolaris Boot demonstrated > Ubuntu Linux boot underway

27 www.opensparc.net RAMP Retreat-Aug 2008 OpenSPARC momentum Innovation will happen everywhere

Innovation Happens Everywhere > 8400

downloads 28 www.opensparc.net FCRC-RAMP-2007-San Diego Team

Ismet Bayraktaroglu Thomas thatcher

Durgam Vahia Paul Hartke (Xilinx) Not Pictured: Gopal Reddy

29 www.opensparc.net RAMP Retreat-Aug 2008