The IBM Power Micro-architecture Report for COMP9244: Software View of Processor Architectures
Godfrey van der Linden
2006-08-01
Abstract The IBM PowerPC instruction set architecture and the implementations of it have pow- ered many different computer systems. It is a second generation RISC design that incorpo- rates many instruction extensions designed to ease the generation of quality code by modern compilers. The RISC design lends itself to scaling from very small implementations designed for embedded applications, through super–computers, to standalone desktop and server ma- chines. Although the design is fundamentally RISC, the IBM designers have explored much of the landscape of modern computer architectures, from large super–scalar processors down to tiny single issue pipelined processor cells, suitable for use in custom designed ASICs. This paper explores the history of the architecture and some of the unusual instruction set choices that the original PowerPC designers made, before investigating some of the PowerPC im- plementations. Modern PowerPC processors, such as the POWER4, have very powerful and general supervisor and hypervisor mode functionality. The later sections of this paper discusses operating system design and some of the some of the issues posed by the POWER4 architecture.
1 Introduction as mobile phones. In this introduction we shall briefly discuss the development history of the PowerPC(PPC) Continuing with IBM’s long history of com- architecture and a quick survey of the PPC ex- puter design innovation, the PowerPC archi- tensions to traditional RISC architectures. In tecture and the various implementations of it, two middle sections will discuss the large main- such as the POWER5, are worthy representa- frame desktop implementations and followed tives of their world class design. IBM lead the by the smaller embedded processor variations. way with many performance innovations in the Finally, in Section 4, issues and design features 1950s and ’60s; it is interesting that so many of will be discussed from the viewpoint of an op- the technologies that they invented have been erating system designer. of direct use in their RISC architectures some In this document it is assumed that the 40 years later. reader is familiar with both modern RISC pro- Apple Computer shocked the personal com- cessor architectures and also with operating puter world when they announced their inten- system design issues. tion of switching to the Intel Core Duo line of processors. IBM and Freescale seem to be un- 1.1 History of the PowerPC concerned, certainly IBM will be shipping as many of these processors as their fabricators In 1985 IBM started working on a second can make attempting to satisfy the insatiable generation RISC architecture, known as the gaming console market. In a remarkable coup “AMERICA architecture”, [CM90]. In 1986 both MicroSoft’s XBox 360 and Sony PlaySta- the development on the RS/6000 was started, tion 3 will be powered by PowerPC chips. which was the first implementation of the Freescale is concentrating their design efforts AMERICA architecture, in February 1990, the with low power signal processing chips aim- first RS/6000 machines shipped, [IBM01] with ing for the handheld embedded market, such a renamed POWER architecture. At the end
1 of development of the RS/6000 series IBM had 1.2.1 Branch registers: Link and Count reduced the chip count of the CPU from eleven The PowerPC ISA uses a dedicated link regis- chips down to a single chip in the low end ma- ter, , to save the return address for a function chines. lr call. It is the callees responsibility to save the In the early ’90s Apple and IBM were col- lr if it isn’t a leaf function. Using a special reg- laborating on a number of projects. During ister for the link register can make the return this time IBM approached Apple to collabo- jump faster since the hardware may not need rate on the development of a family of single– to go through a register read pipeline stage for chip processors based on the POWER archi- function returns. tecture. Soon after, Apple asked Motorola to The ISA also has a dedicated register join the collaboration to leverage Motorola’s for controlling loops, the count register ctr. experience in manufacturing high–volume pro- This register is automatically decremented and cessors. This collaboration was known as the tested against 0 by the branch conditional in- 1 AIM alliance; the result was a modification of structions. By using this special register the POWER, known as the PowerPC instruction branch processor can determine the result of set architecture(ISA). PowerPC added single– branch early in the pipeline. precision floating point, general register–to– In the PowerPC architecture the branch register multiply and divide and removed some and instruction fetch units are closely associ- POWER instructions, such as MQ register ated, in fact he branch unit is split into two based multiplies. PowerPC also added a par- with the branch prediction unit very close — allel 64–bit instruction set mode. in terms of clock cycles — to the fetch unit. The 1993, the POWER 2 was released The lr and ctr registers can be implemented and the POWER3 was released in 1997. The in the part of the branch unit that is near the POWER3 was the first implementation of fetch unit and the instruction set can the use the full 32 and 64–bit PowerPC ISA. The these registers to implement indirect function POWER4 was released in 2001 and has be- calls. See [HP03, Appendix C] for details. come the basis for later development of rela- tively large systems. The PowerPC 970(2002) 1.2.2 Load and Store extensions and POWER5(2005) extending the POWER4 micro-architecture. Atomic update The Load and Re- serve(lwarx) and Store Condi- tional(stwcx) instructions provide the PowerPC shared–storage atomic–update 1.2 PowerPC instruction set archi- mechanism. Lwarx instruction loads a tecture value from a memory location and marks that location as reserved. The subse- The PowerPC instruction set architecture is quent stwcx stores a new value, if the part of a group of second generation RISC ISAs reservation is still valid. A reservation that were developed in the late ’80s and early is invalidated when another processor ’90s. This document will concentrate on those successfully commits a stwcx instruc- parts of the PowerPC ISA that are unconven- tion. Using this mechanism a general tional. In most ways the PowerPC ISA is con- atomic operation suite can be built such ventionally RISC in that only load and store as atomic increment, compare and swap operations are used to access memory; that and test and set. See [WSM+5b]. is, there are no memory operand addressing modes. However unlike many RISC instruction With update Indexed load and store instruc- sets, the PowerPC has some extended instruc- tions have a ’with update’ variant, which tions that are capable of modifying multiple updates the source index register with target registers or are inherently multi-cycle in the computed effective address after the nature. load or store is initiated. The load in- 1(A)pple, (I)BM and (M)otorola
2 struction will modify two registers; some- use of memory barriers. However the de- thing that is unusual for RISC instruc- tails are so complex and difficult to explain tion sets. This variation is essentially a that the average programmer is left behind. convenience to reduce the number of in- Which means that some system programmers structions executed inside a loop itera- may utilise sophisticated synchronisation tech- tion. See [WSM+5a]. niques internally in the operating system, but probably needs to publish quite conservative, Multiple Another pair of convenience in- i.e. slow, services to external non-kernel pro- structions are the load and store multi- grammers. ple register instructions lmw/stmw. These In PowerPC ISA there are four data instructions are intended for the pream- synchronisation instructions: isync, eieio, ble/postamble of functions to save and lwsync and sync. restore non-volatile registers, as de- fined by the application binary interface isync Instruction synchronise This instruc- (ABI ). These instruction do not really tion insures that all preceding instruc- increase program performance directly tions are completed before the isync as they still take one cycle per regis- completes and stops all subsequent in- ter to implement but rather are used to structions from scheduling until comple- increase the instruction stream density. tion. It also invalidates any instructions See [WSM+5a]. in the I-cache with respect to any out- String/misaligned A variation of the standing instruction cache block invali- Load/Store Multiple, the Load/Store dations, icbi, from the bus. In lock string instruction. These instructions acquisition, this instruction is used im- load a series of words from an arbitrary mediately after the stwcx complete test byte address in to or out of multiple reg- to guarantee that no later instructions isters. See [WSM+5a]. are executed speculatively until the re- sults are known. In general isync can Byte reversed The load/store word/halfword be thought of as a load barrier. instructions all support a byte swapped variant, where the (half–)word at the eieio Enforce in–order execution of I/O The specified location can be byte–swapped behavior of this instruction depends on on the way to or from memory. As many the type of caching enabled in preced- of the POWER implementations have ing loads and stores. Memory mapped switchable endianess these instructions device registers are mapped cache– are used to efficiently work with cross– inhibited/guarded (see section ??). For endian integers. See [WSM+5a]. this type of mapped register operation, this instruction guarantees load and store 1.2.3 Shared Storage ordering such that all preceding loads and stores complete to main memory be- The PowerPC specifies a weakly consistent fore the eieio completes. This only ef- storage model. This means that each proces- fects loads and stores, if ordering of all in- sor’s hardware on the memory bus are not re- structions is required then the sync must sponsible for maintaining coherence. The de- be used. For the default cache–enabled sign takes the approach that a programmer case the instruction guarantees store or- must maintain a consistent view of memory dering but its use is deprecated in favour with the appropriate use of synchronisation op- of the lwsync instructions, when avail- erations in code, see section 1.7 Shared Storage able, see below. of [WSM+5b]. The strength, and weakness, with the lwsync Light–weight synchronise This in- weakly consistent memory model is that the struction guarantees order of store in- programmer is responsible for maintaining co- structions, hence a memory barrier, to herence across the processors by appropriate system memory. However it doesn’t work
3 predictably on memory mapped device sraw/addze This pair of instructions — shift memory, in which case the, much slower, right algebraic/add zero bit extended — sync should be used. This instruction is used by a compiler to optimise an im- precedes the atomic update for mutex mediate power of two division of signed unlocking and guarantees that all of the or unsigned integers as an algebraic shift code in a critical section is complete be- operation. fore the lock is dropped. Only imple- mented in the POWER4 derived micro- cntlz Count leading zeros Very useful instruc- architectures. tion for computing the log2 of a value. Very useful when implementing power of sync Heavy–weight synchronise When order- two allocators or logarithmic instrumen- ing against a device register is required tation. it is necessary to issue a sync instruction to guarantee ordering. Sync extends the Move program counter (PC ) In the Ap- guarantee of lwsync to include a round ple dynamic library system, the program trip to the system memory controller so counter is required to be passed as an that outstanding bus snoops can com- argument to the dynamic code patcher. plete. This is an unfortunate design as RISC architectures rarely make the PC directly available in the ISA. Apple has been us- 1.2.4 Miscellaneous instructions ing a branch and link to the next instruc- The designers of the PowerPC ISA added a tion,BCL 20, 31, $+4, which stores the number of unusual instructions. These instruc- PC in the link register. The PPC970 tions seem to be provided for the use of C com- branch predictor has been specially mod- pilers to make some of the more difficult C se- ified to not update the link register stack mantics easier to implement. for this particular instruction.
32–bit immediate values An immediate 2 POWER4 based micro- 32–bit value is generated using an ori rt, r0,
4 Figure 1: Block diagram of the POWER4 processor. [BBF+01]
Figure 2: Block diagram of the Cell processor. Where: SPU means synergistic processor ele- ment, LS is local store, EIB is element interconnect bus, PPU is PowerPC processing unit, MIC is memory interrupt controller and BIC is I/O bus interface controller. [Kre05]
5 system. The fabric is used by the L2 cache to branch stalls, so the designers introduced a communicate to an on–chip local L3 controller very large branch prediction unit with sophisti- and directory, with a memory controller built cated subroutine and other indirection logging. in. The P4 uses three branch history tables: the The internal micro-architecture of the local predictor table has 16k 1–bit predictors, core is a speculative, superscalar, out–of– the global predictor table also has 16k 1–bit order(OOO) execution design, see figure 1; it predictors — indexed by hashing the result of can fetch, decode and crack up to eight instruc- the last 11 branches with the current branch tions per cycle, issue up to five operations per address and finally a selector table of 16k 1–bit cycle and maintain a sustained completion rate predictors — indexed the same as the global of five operations per cycle — most instruc- branch table. According to [TDF+01], “This tions crack to one operation, a few crack to combination of branch prediction tables has more than one operation. With large register been shown to produce very accurate predic- rename pools, other OOO resources and long tions across a wide range of workload types.” pipelines, the P4 can have over two hundred in- Certainly a large number of transistors have structions in flight. To exploit instruction level been dedicated to branch prediction. In addi- parallelism there are eight execution units each tion to the branch prediction unit the P4 ar- of which can have an instruction issued each cy- chitecture maintains a stack of the last link cle, though this rate can not be sustained due register values for target prediction of return to the formation of five instruction groups. addresses. Also the POWER4 has a 32 entry To minimise the resources used to keep direct mapped cache for count register target track of such a large number of instructions address prediction. in flight, the P4 decodes and schedules instruc- To keep the memory flowing through the tions into instruction groups of five. The group caches of the large SMP systems, the PowerPC is encoded such that each slot in the group can ISA only specifies a weakly consistent storage only be scheduled on specific execution unit model. However a rich set of synchronisation pipelines, and the fifth slot may only contain primitives are provided for a programmer to a branch instruction or a no–op. This group optimise shared storage accesses. In subsec- then has rename and other OOO resources as- tion 4.3 the operating system issues of weakly sociated with it and is tracked through the en- consistent model will be explored more com- tire execution pipeline. Finally when every in- pletely. struction in the group has completed the entire As a part of IBM’s commitment to reliabil- group is committed, hence the sustained com- ity, availability and serviceability, the designers pletion rate of five instructions per cycle. have introduced into the P4 the ability to log- With this micro-architecture’s power mem- ically partition an SMP system into a number ory bandwidth could become a bottleneck. The of logical subsystems. Each processor has a P4 has very large caches to minimise mem- logical partition ID, a real mode offset register ory access latencies with: a 64KiB directly (RMOR), a real mode limit register (RMLR) mapped I-cache, 32KiB two–way set associa- and a hypervisor RMOR. These registers and a tive D-cache, 3 x 480KiB (1.41MiB total) inde- few others can only be modified in a new priv- pendent 4–8 way set associative L2 caches and ileged mode called the hypervisor mode. Once 8–way set associative L3 cache directory for up the RMOR/RMLR registers are set by the hy- to 32MiB of off–chip L3 cache. The L1 and L2 pervisor the processor will be incapable of ac- caches are 128b per line and the L3 cache has cessing memory out of that range of physical 512b per line. Finally the memory hardware is memory. With these tools the operating sys- capable of recognising up to eight software ini- tem can divide a large SMP into a series of logi- tiated contiguous data streams and will start cal partitions that can be individually failed — scheduling pre-fetch of contiguous data. without bringing the entire system down – un- To achieve very high potential clock rates til a service technician can be scheduled. This the P4 has a sixteen cycle instruction pipeline, technique has other uses too, such as processor which can cause very long miss–predicted virtualisation.
6 2.1 PowerPC 970 ity with the POWER4, [SKT+05]. The POWER5(P5 ) is a two–way simultaneous mul- The PowerPC970 family of chips are all 64–bit tithreaded(SMT ) dual–core chip that has a PowerPC ISA microprocessors, based upon the very similar architecture to the P4. P5 systems POWER4 micro-architecture with vector — are capable of running up to 64 processors or single–instruction, multiple–data (SIMD)— 32 chips, twice the number of processors for instruction extensions known as VMX,[ppc05]. P4s. The processor is designed for desktop and low– end servers applications for uni-processors up The P5 designers realised that the addition to four way SMP system configurations. of SMT and 64 processors SMP would present tremendous problems to the P4 memory de- Apple Computer was a big customer for the sign where the L3 cache was between the inter– 970 design, which they marketed as the G5. processor fabric and the memory bus. Thus They requested a few changes in the basic P4 when a processor core had an L3 hit it would design, specifically the requested support for issue a memory cycle on the processor fabric, Motorola’s SIMD instruction set, support for with twice as many processors on this bus and a binary compatibility 32–bit execution mode more fetches issuing from each chip this caused is supported for 32–bit process support and a an unacceptable degradation in memory per- couple of smaller changes to support the Mac formance. The P5 has moved the L3 to di- OS X application binary interface, see Move rectly connect to the L2 controller, so that L2 program counter in section 1.2.4 above. Ap- fills no longer cause transactions on the fab- ple had wanted to design a 970/G5 laptop sys- ric. In addition to moving the L3 cache, IBM tem but the power consumed and heat dissi- designers added an on–chip memory controller pated presented insurmountable problems to to improve the speed of main memory access. Apple’s system designers. The 2.5GHz 970MP These two changes greatly decrease the latency dual core chip has a peak power consumption to main memory of a cache miss and also en- of 100W. hances system reliability by reducing system The primary limitation of the 970 over the chip count. P4 architecture are in the caching hierarchy. The 970 does not have a L3 cache and only The caches were also somewhat reorganised has a single 512KiB L2 cache, though the low– from the P4 architecture. The P5’s L1 caches power 970FX and dual–core 970MP expanded have been doubled in size by increasing the I- it to 1MiB. Otherwise the caches are the same cache and D-caches to 2–way and 4–way re- structure as the P4’s. spectively. The L2 cache is now a 10–way 128b Probably the biggest architectural differ- line with 640KiB for each of the 3 cache slices ence between the P4 and the 970 family is for a total of 1.875MiB of L2 cache. The L3 di- the addition of the VMX, Vector/SIMD Mul- rectory is very different from the P4. The new timedia eXtension, instructions. Two new 36MiB cache is divided into three slices where execution units were added to the architec- each slice is organised as 12MiB, 12–way set ture: the vector permutation (VPERM ) and associative with 256b per line. the vector ALU (VALU ). The VPERM has a Too maintain structural compatibility the 19 stage pipeline. The VALU is further di- P5 has an identical instruction pipeline to the vided into 3 units for fixed point, complex– P4 processor. So code that is optimised to run fixed and floating point instructions with 19, on the P4 will still run optimally on the P5. 22 and 25 pipeline stages respectively. The Al- However with the addition of SMT it should tiVec/VMX ISA seems to be very highly re- be possible to maintain higher utility of the spected in the industry in comparison to the chips execution units, the P4 usually achieved SIMD extensions available on other processors. on 25% utilisation of each of its eight execution units. Other resources where also increased to 2.2 POWER5 micro-architecture better support SMT for instance the various register files where increased in size to support The key goal of the POWER5, was to main- much more register renaming. In IBM bench- tain both binary and structural compatibil- marks these design changes have led to a 40%
7 improvement in processor throughput over a pect that many of the technologies that have large test base. been explored for the very high speed Cell To deal with potential resource starvation broadband engine will make their way into the for a single thread the processor supports eight POWER6, see subsection 3.3. priority levels per thread. These are used to control the relative number of decode cycles each thread gets and hence the number of in- 3 PowerPC based micro- structions executed. Also the processor mon- architectures itors various recourse such as the global com- pletion table and the load miss queue, to de- In this section the smaller embedded and set– tect significant resource starvation. In the case top implementations of the PowerPC ISA will that resource starvation is found the offend- be described. In general most of these proces- ing resource hunger thread is throttled back sors continue to use the 32–bit instruction set. by lowering its priority. The PowerPC 440 will be discussed first, fol- The P5 is capable of running in a single lowed by the Blue Gene Compute Chip and the threaded(ST ) mode as some applications that Cell Broadband Engine. are execution unit or bus bandwidth limited. In this mode all of the additional resources 3.1 PowerPC 440 needed to support SMT are devoted to the execution of a single thread, for instance the The PowerPC 440 (PPC440 ) core is a rela- full 120 rename registers become available to a tively high–performance, superscalar processor single thread. The processor can dynamically core intended for embedded applications. The switch between ST and SMT execution modes. PPC 440 core is also available as part of the Chip power is becoming an important met- IBM ASIC library of processor cores. The tar- ric and such a complex chip as the P5, with its get market is the usual list of embedded appli- 237 million transistors, would be very power cations such as digital cameras, laser printers, hunger. To offset this power consumption switches and network cards, [ppc99]. In an un- problem, the P5 chip has implemented a dy- expected twist the PPC440 core has become namic power management system that dynam- the basis of the of the Blue Gene/L Compute ically shuts–down inactive parts of the chip. Chip, see section 3.2 below. Each unit can be clock gated into a low power The core features a two–way superscalar state, on a cycle by cycle bases, with no per- design, with 3 execution pipelines and OOO formance loss. issue, execution and completion. The pipeline itself is 7 stages long. In addition to the stan- dard 32–bit RISC PowerPC ISA some 24 DSP 2.3 POWER6 micro-architecture — operations are also available including a single– what is known cycle throughput 16x16+32 to 32 multiply ac- According to reports from the 2006 Interna- cumulate instruction. tional Solid–State Circuits Conference in San The basic design is quite modular and the Francisco, IBM presented many papers on the library core con be configured with a num- up–coming POWER6. It is expected to be ber of different subsystems including 0 - 64KiB shipping in 2H2007 and will clock in the 4–4.5 caches. The processor is very low power with GHz range, with speeds of 5.6GHz achieved in 2.5mW/MHz or about 1.4W at 555MHz. At the lab. Simultaneously IBM claims to have the peak of 555MHz, the processor delivers kept a lid on power consumption; “Despite the 1000 MIPS using the Dhrystone 2.1 bench- speeds, it will have a lower power density than mark. in some chips found in today’s desktops” said The 3 execution units are: a load/store Bernard Meyerson, chief technologist of IBM’s pipeline, simple integer pipeline and com- Systems and Technology Group, [Ass06, Feb. plex integer pipeline. The pipeline itself has 07]. seven stages: instruction fetch, pre-decode, Although we do not yet know many details decode/issue, read registers, execution stage of the POWER6 micro-architecture, we sus- 1/EA address compute, execution stage 2/data
8 cache access and register writeback. The lack Debug There are two debugging interfaces on of renaming registers is a serious limitation for the PPC440 core, the JTAG and instruc- exploiting any ILP that may be available. tion trace ports. The PPC440 has separate instruction and data caches, factory configurable up to 64KiB 3.2 Blue Gene/L Supercomputer and are highly associative, the 64KiB cache is 128–way set associative, the associatively The fully populated IBM Blue Gene/L system varies with cache size. With the high associa- installed at Lawrence Livermore National Lab- tively advanced cache partitioning is possible. oratories in Livermore, CA topped the Novem- The caches can be separated into normal, tran- ber 2005 TOP500 Supercomputer list with sient and locked regions, where: normal regions a sustained 280.6 TFlops. At the time no have traditional cache replacement, transient other system exceeded 100 TFlop/s and the regions are used temporarily then not used Blue Gene/L is expected to top the next few again — typical of data streams and locked re- TOP500 lists. The Blue Gene/L system is gions is used for code that is not to be cast out an unorthodox entry into the supercomputer of the cache. field, in that it uses slow unsophisticated pro- As with most embedded system designs, cessors as its workhorse compute engine. Each the PPC440 is designed with a number of ex- Blue Gene/L compute chip is a 4MiB DRAM ternal interfaces: with a pair of 700MHz PowerPC440 proces- sor cores sharing the DRAM as a L3 cache. Processor local bus (PLB) Three separate Each core has 2 FPUs, 32KiB I and D-caches, PLB interfaces are used to access system a very small 2KiB L2 cache and 16KiB of lo- resources: one for instruction fetches, cal scratch SRAM. They also share some I/O one for data reads and the last fro data related subsystems for inter-processor commu- writes. Each PLB controller is a 128–bit nication. Each chip can be thought of as a bus master. 4MiB DRAM cache with a small compute core in one corner of it, [IJEBP+05]. Device control register bus (DCR) The In Blue Gene/L two chips are packaged DCR bus is a configuration bus for com- with 2x512MiB of RAM onto a single proces- ponents external to the core. The DCR sor board. These boards are collected together bus is used to manage status and configu- into a processor ’card’. These cards are col- ration registers and reduces PLB traffic. lected into a cabinet with 1024 nodes, or 2048 System resources on the DCR bus are processors, drawing a total of 29kW and has a protected to some extent as they are not performance of 2.9/5.8 TFlops. A fully config- part of the system memory map. ured Blue Gene/L system has 64k processing Auxiliary processor unit (APU ) Instruc- nodes, draws a tiny 1.8MW and achieves an tions within the PowerPC ISA have been astounding 76 MFlops/W, [cF05]. reserved for APUs to execute instructions In comparison the ASC Purple using in the stream. The APU interface pro- 10240 POWER5 processors draws 7.5MW and vides a dual–issue pipeline design and achieves 63.3TFlops or 8.4 MFlops/W. The can use a full 128–bit load/store path to ASC Purple machine is typical of the high– the D-cache. Some uses of the APU is to power compute node based high performance implement a full PowerPC Floating Point systems and also typical has required quite Unit, multimedia macros, DSP or other carefully designed cooling systems. Of course custom functions. the choice of slow processing nodes means that only problems that are massively par- External interrupt control (EIC ) The allel will benefit, in fact it is easy to con- EIC can extend the PPC440 interrupt ceive of workloads that would run slower on system to external interrupt systems. the Blue Gene/L than more traditional high– These external interrupts are level sen- performance computers even though they do sitive and can be wired to critical or not compare to the Blue Gene/L in processing non-critical interrupts internally. power.
9 3.3 Cell broadband engine cycles to unwind and data loads have a four cycle latency. Like the 970 the Cell’s PPU does The Cell broadband engine is the product of a implement the complete VMX/AltiVec SIMD collaboration between IBM, Sony and Toshiba. instructions set. The PPU is intended to run The design goal of the chip is to support the operating system and control the work flow the next generation of Sony PlayStation gam- through the SPUs and the new memory flow ing console. Sony required substantial perfor- controller. mance of the design and it seems that IBM has achieved this goal, the chip has a design fre- Each SPE has a vector processing unit, quency of 4.0GHz and can achieve a startling, known in IBM inimitable three letter acronym though theoretical, peak single–precision per- way as an SPU, and 256KiB of local static formance of 192GFlops at 3.0GHz2. The mem- RAM. The design of the SPU The SPUs, ory interface has also been redesigned and is ca- are derived and ’inspired’ from and by pable of an I/O bandwidth of some 25.6GB/s, the VMX/AltiVec implementation and Sony provided by two Rambus XDR memory con- PlayStation 2’s ’Emotion Engine’. The vec- trollers, [Kre05]. tor instruction set mostly traditional with per- A Cell chip is IBM’s first instantiation of a haps an unusual number of registers. The SPU architecture known as the ’Broadband Proces- supports 128 128–bit wide registers. With this sor Architecture’(BPA). The new architecture large register set and the fast local store the was designed for specific workloads such as: designers have not implemented caches in the cryptography, graphics and lighting, physics, processor. The designer hope that a carefully fast–Fourier, matrix computations and other designed algorithm should not need a cache. scientific tasks. This instantiation is a multi– The local store is populated using dedicated processor on a chip, with a completely re- SPE local DMA controllers. The DMA engines designed 64–bit PowerPC core and eight ’Syn- can access both main memory and other pro- ergistic Processor Elements(SPEs)’, which are cessors local stores. Although the DMA trans- SIMD signal processors. It would seem that actions are coherent, once the data is cached in IBM had an additional design goal of maximis- an SPE’s local store its coherence is not main- ing floating point performance per watt even tained, leaving coherence and synchronisation though the architecture runs at very high fre- to the programmer, see section 4.3. quency. The block diagram, figure 2, from Intra-chip communications use the element [Kre05], shows the structure of the Cell chip; interconnect bus (EIB). The EIB consists of as you can see the Cell is a multi–SIMD pro- four busses, two clockwise and two anticlock- cessor, or a MIMD. wise. Each bus is sixteen bytes wide and data With the Cell IBM has reworked the 64– clocks every two processor clocks, or a theoret- bit PowerPC core, which is the controlling ical maximum bandwidth of 128BiB/second. processor or (PPU ). It implements the 64– The EIB is organised as a token ring where bit PPC instruction set, though internally the each processor on the ring is no more than five PPU has a much simpler design than the steps away from any other processor. How- super–scalar POWER4 designs with their com- ever, the bus will automatically partition itself plex multi–functional–unit out–of–order issue if necessary when two non-overlapping proces- architecture. Instead the Cell implements two sors are communicating. For example if proces- thread SMT with dual, in–order, instruction sor A is moving data to B and C is communi- issue to a three computation functional units. cating with C they can share the same proces- Like the POWER4, the Cell was designed for sor bus as B’s DMA engine does not transmit a very fast clock, some 4.0GHz for the Cell packets targeted for itself on the bus. and has a long, 21 stage, pipeline. However as In conclusion, the Cell’s potential is enor- the Cell is not super–scalar to any extent IBM mous but its unconventional architecture will has limited instruction hazard and cache miss make severe demands on tool and software de- penalties: a branch misprediction takes eight signers to try to make use of this potential. 2IBM has not stated the power consumption of the Cell processor yet, but Microprocessor Report reckons that it will be 80W or peak 2400MFlop/W
10 4 Software considerations for ify a virtual memory address. The segment ta- operating systems on the ble is backed by a small fully–associative SLB cache and the virtual memory has the tradi- POWER4 tional TLB. Finally there are small cache’s for performing direct EA to real address conver- While researching the POWER architecture sion. several features of the design suggested them- selves as being of interest or concern to oper- This structure allows the operating system ating system designers. In this section some to maintain a single much larger 65–bit virtual of those areas will be explored. The 32– address space which combines segment table bit PowerPC440 based architectures are tra- provided virtual specifier IDs (VSIDs) with ef- ditional pipelined RISC processors and do not fective address to specify a real address. The implement many of the features here discussed. designers intended the segmentation to allow Hence this section will concern itself with the for large portions of the virtual address space 64–bit POWER family, such as the POWER4, to be shared between different user tasks. For PowerPC970 and Cell processors. instance to mapping large shared libraries, an In subsection 4.1 we will discuss the en- OS kernel, a micro-kernel and rootserver com- hanced memory management options available bination or even large shared memory pools using the new two level effective–address to created and manipulated directly by client pro- real–address translation using segment tables. cesses through such APIs as mmap. A single seg- Subsection 4.2, will explore a new API for con- ment can map single use effective address space text dependant pre-fetching of cache data or into a VSID, but an entire segment table will alternatively structuring data in such a way map into many different VSID specified parts that automatic pre–fetching is triggered. Sub- of the VM. It should be possible to limit the section 4.3, explores the implications of the number of TLB interactions using this design. POWER families weakly consistent memory models. Subsection 4.4, covers issues result- The VM page table itself has been en- ing from large branch misprediction penalties hanced to support multiple page sizes, on and interactions with typical operating sys- POWER4 they can be 4KiB and 16MiB, but tem designs. The hypervisor facilities of the the Cell can support 64KiB, 1MiB and 64MiB POWER architecture are the subject of sub- pages. Use of these new page sizes should be section 4.5. Subsection 4.6, discusses the inter- explored for the operating system as a whole, nal implementation of the POWER architec- though it is obvious that the kernel/rootserver ture and how structuring code carefully may and large user shared libraries can make imme- greatly enhance performance. Finally in sub- diate use of them. section 4.7, we will briefly introduce the reader to some of the performance tools available to The POWER family has had a long stand- the reader for analysing PowerPC970 perfor- ing problem with slow TLB interactions. Any mance. operating system engineer designing an imple- mentation on the POWER architecture should 4.1 Segmented Memory spend considerable time characterising the per- formance of the many levels of look–aside A 64–bit virtual address spaces is too huge buffers implemented in this segmented mem- to maintain a traditional 4KiB page table for ory model. each process. The 64–bit PowerPC architec- ture specifies a virtual storage model where the We believe that it will be worthwhile to ex- user address space (effective address or EA) is periment with the new feature to determine if a subset of an OS maintained virtual memory TLB sharing between processes is practical and address space. An effective address maps into performant. With judicious uses of the VSID the virtual address space using 256MiB seg- identifier an operating system may be designed ments, where each segment maps an effective that will only infrequently require a TLB inval- address ID ESID and effective address to spec- idation.
11 4.2 Cache pre-fetching would need to add this functionality. If every client needs the functionality then good design An argument could be made 3 that an demands that the functionality must be pushed operating system is a massive multi- up to the service provider. This mechanism of plexer/demultiplexer engine. Good system associating a function pointer with the com- call design usually limits the number of en- plete context data could be made a general try points into an OS and each entry point is service provided by the operating system and as general as possible. Hardware interrupts are coded as efficiently as possible. also demultiplexed and routed to appropriate Many processors that conform to the later drivers. Traditionally all of this demultiplexing 64–bit POWER design also have hardware de- is implemented using C’s very efficient pointer tected data and instruction streaming. When to function. Once a demultiplex has been the cache controller notices that a series of decided the layer of code probably has some cache–misses have been contiguous, controller context that needs to be re-established. If the automatically issues pre-fetch requests. With decision point could also arrange for the con- this facility in mind we should investigate text to be pre-loaded, later cache misses may mechanisms to identify associated data and at- be avoided. tempt to group all of the data required sequen- We propose a new API allows a system en- tially. This is easy within one module but if we gineer to register not only a pointer to function can identify opportunities across several API and data context pointer, but also the context layers the rewards may be significant. To iden- size with a parent data provider. The parent tify such cases we may be able to run an in- could decide whether it implemented the de- strumented version of software to identify sep- cision or not, but it is responsible for passing arate layer data structures, by profiling per- the request onto the ultimate decision maker. haps, then try to arrange this memory to live The decision maker would associate the set of in a single allocated data structure. Interest- addresses with a particular result and pre-load ingly the hardware can identify cache–missing context data. These pre-load requests would streams in either direction, which means that be made before the data is requested by the both up calls and down calls through a stack normal stream of instructions, and thus should would benefit. shorten the data cache delays. Extensive profiling would be required to Consider as a concrete example of the use prove that the pre-fetching technology dis- of this API a PCI device driver. At the time cussed in this section do not have a detrimen- when the driver registers its interrupt handler tal impact on the system due to an increase in and data context, it would also associate the worthless cache invalidation. Further research context length with the interrupt handler. The could define probability bounds where a cache– interrupt demultiplexer would store a handler line that is pre-fetched is sufficiently likely to function, a context data pointer and the con- be used. text’s length. When an interrupt for the card occurs, the interrupt controller would quickly 4.3 Shared Storage spin over the data context once per cache line touching the entire context into data cache, be- The POWER architecture specifies a weakly fore branching to the handler function. By the consistent storage model. Later processors time the indirect function call is resolved at do extensive instruction and write buffer re- the context data should be closer to the caches ordering. Data–hazards are dealt with by the than would otherwise occur. internal instruction dispatcher and the cache It would appear at first glance that the pro- snooping will deal with cache–line inconsisten- posed API constitutes an unnecessary layer vi- cies. However store re-ordering is independent olation, after all each interrupt handler could of the caching specified on the memory, the easily pre-load its own data directly. But that CPU never guarantee’s the order of stores. means that all well designed interrupt handlers Although it is possible to describe this is- 3But not here!
12 sue in only a few sentences, the implications of rules that the general programming commu- of it are wide ranging and performance criti- nity could follow and that are optimal. To cer- cal. A system designer, considering POWER, tain of a safe, near optimal consistency solution should devote some time experimenting with would require a team of application, operating the weakly consistent storage model, research- system and platform hardware engineers. The ing interactions and performance impact on a solution developed with would only be nar- particular platform/memory controller combi- rowly optimal, another revision of the proces- nation. This research should be carefully doc- sor or memory system would probably invali- umented and distilled, ideally in such a way date the synchronisation and may even lead to that an average programmer can grasp enough inconsistent data. of the subtleties to seek expert advice when At Apple Computer, we had the Mac OS required. X kernel running for some six months, before Given weakly consistent memory, the pro- we discovered the cause of a rare bug where grammer is expected to implement sufficient two threads could both modify lock protected synchronisation between different entities on a data. We knew that each thread took the lock, modern computer system; for instance between yet the corruption still occurred. It was dis- processes that can run on different cores but on covered that write re-ordering could unlock a the same caches, on different processors on the lock before some memory stores are completed, same memory, or on a NUMA machine. Com- thus defeating the lock. At that time we de- municating with devices on the system bus also cided to use the sync operation before mutex presents different but related synchronisation unlock. sync before unlocking a mutex. Years issues. later, after gaining more experience, we deter- Optimising synchronising performance is mined that a sync was unnecessary and the the real issue, IBM engineers having ducked higher performance lwsync instruction would consistency, have left consistency to the pro- be sufficient. grammer and system designers. If a naive, but safe, consistency model is used, such as using 4.4 Indirect function performance the sync instruction exclusively it would ut- terly destroy the performance of the system; The design goal of achieving a very high pro- the POWER core would be reduced to little cessor frequency, required the POWER4 to im- better than early non-pipelined CPU perfor- plement very long instruction pipelines. This mance without even a cache as all memory op- long pipelines implies that branch mispredic- erations must complete. Hence it is natural for tion will be expensive. They are, take twelve system designers to choose the weakest, that cycles to resolve, [BBF+01]. As the POWER4 is highest performance, of the many synchro- architecture is super–scalar, with up to ten nisation options provided be the architecture functional units, it is possible that a hun- as possible. However the experience at Apple dred instructions must be unrolled4. To limit Computer (conversations with Apple [Eng06]) these penalties the branch prediction unit, on with these barrier operations lwsync, sync, POWER4 derived architectures, is one of the eieio and the instruction order barrier isync most sophisticated such units implemented on has been painful. Each revision of the CPU any RISC processor, see section 2. and memory controller combination has had PowerPC implements pointers to functions subtly different semantics. Apple spent sig- by using the count register(CTR) with the nificant amounts of time and effort to achieve bcctr instruction. The CTR and, related, link a high–performance, consistent, set of barrier register(LR) can be located in the instruction operations. The task is so complex that it is fetch unit. The fetch unit can use these regis- unlikely even now that Apple achieved either ters to determine its path through the instruc- cross processor generation portability or opti- tion stream. On the POWER4, these archi- mal performance. tectural registers are backed by sixteen physi- It may be impossible to design a simple set cal rename registers and so some care must be 4This does not occur due to the instruction grouping, see subsection 4.6.
13 taken by the fetch unit to use the appropriate words it does not deal with user tasks directly physical register. but rather controls and services various oper- In subsection 4.2, we noted that much of ating systems on the platform, [AAB+05]. the demultiplexing performed by an operating With the POWER4 IBM introduced the system is implemented using C function point- first phase of their virtualisation solution, ers. The performance of these instructions is which culminated in the POWER5. The critical. Software designers on the POWER4 POWER4 only had limited hypervisor func- platforms should be aware of the very high cost tionality, in the form of logical partitioning of of branch misprediction, however they may a system. A system’s memory could be par- not know that all bcctr indirect branches use titioned into smaller contiguous chunks. The the branch predictor’s target address predictor. processor’s memory management unit (MMU) The nature of demultiplexing will often result enforces a mapping of real addresses onto this in high miss rates on branch target predictions. partition. While a process is executing in a This is a shame; if the designers required al- partition, the MMU is incapable of generat- lowed the actual value to be used rather than ing real addresses that are not in the partition. the predicted value it is often possible to order The hypervisor kept track execution state and instructions so as to make the target available switched the MMU mapping when switching early. Consider that many indirect functions to a new state in a different partition. This have two or more arguments. Just marshalling required a total TLB invalidation, as the pre- the arguments would give sufficient time for the vious virtual memory mapping would be use- CTR value to become available. less. Hence the performance of switching par- As the CTR register cache is direct mapped titions would probably be very slow. A sched- and indexed by the instruction address of the uler which has a concept of processor/partition branch the indirect function call will often have affinity is required. Or, alternatively, a multi- different target addresses and will be mispre- processor system could be mapped such that dicted. This is especially problematic for C++ processors are assigned to logical partitions virtual function calls which all indirect through and only occasionally would an administrator a class’s virtual table. switch the configuration around. Given the deep pipeline in the POWER4 With the POWER5 the limitations of micro-architecture, mispredicted branches can the POWER4’s virtualisation have been ad- take twelve cycles to resolve; it would be bet- dressed. The design criteria was to enable ef- ter to use an extension to the bcctr instruction ficient and flexible virtualisation to hypervisor that disables the branch target prediction. Al- aware operating systems, this design technique though we will pay the twelve cycle penalty and is sometimes known as paravirtualisation. The stall the pipeline, we are no worse off and the client operating systems be assigned exclusive processor has not spent a lot of energy com- access to particular PCI slots or can share I/O puting values that it will then discard. resources with other clients. IBM has an implementation of a hyper- 4.5 POWER family hypervisor visor on their iServer and pServer systems. IBM were probably the first company to ex- They needed to design a solution that could plore virtualisation, their 370 series of com- efficiently share I/O systems across multiple puters which introduced the concept of totally logical partitions. A traditional design would independent virtual machines in the VM/370 locate drivers in hypervisor space, but that operating system in 1972. In recent years Intel would lead to a less robust hypervisor and processors have also become so powerful that it could potentially compromise security. It was has become practical to run multiple instances decided to dedicate a logical partition to hard- of an operating system on one computer. The ware drivers and to cede sufficient privilege to POWER4 derived hardware implements virtu- this partition to allow it to issue remote logical alisation enhancements, known collectively as partition DMAs. The approach was to allow hypervisor functions. A hypervisor is a sort a partition to instantiate virtual I/O adapters of a small ’operating system’ system. In other in its local partition. The virtual adapter has
14 such definitions as DMA windows, interrupts A group can not be formed arbitrarily, only and other adapter information such as real certain operation types may appear in each of physical hardware would have. In addition hy- the five slots, this simplifies the routing of op- pervisor operations allows partitions to pass erations among the different issue queues. As their virtual adapter details to the I/O service only a single group can be issued per cycle, it in a secure way, which passes a capability to is very important for instructions to be sched- the I/O partition to perform the physical I/O. uled in such a way as to maximise the util- The design that IBM developed has many ity of each group, however these instructions similarities with a modern micro-kernel such must also conform to the group formation lim- as L4. L4 could be a hypervisor that con- itations. For a compiler to do a good job of trols the various logical partitions. Certain instruction scheduling it will need to model the enhancements would have to be made to L4 complex instruction cracking and group forma- to support the POWER5 completely, such as tion architecture. address translation for remote partition DMA This is very difficult as the processor is so and scheduling with processor/partition affin- complex and is probably out of reach for hand ity. A root-server task would be used to control coding of large amounts of assembler. In sec- the assignment of processor, memory and I/O tion 4.7 we shall discuss some of the tools avail- resources to each logical partition all of which able to the programmer for optimising their would be mediated by the micro-kernel. code. With the aid of the performance tools it should be possible for an assembler engineer 4.6 Instruction grouping to iteratively approach optimal code, in much the same way that [GCC+05] did for the Ita- Large super–scalar processors such as the nium, we suspect that very similar improve- POWER4 derived designs have some unex- ments can be achieved over the code generated pected similarities with VLIW and EPIC archi- by the GCC group of compilers. tectures, in that they both benefit greatly from IBM are reported to have made extensive appropriate instruction scheduling. Otherwise changes to the GCC compilers to better sup- it is unlikely that the processor could discover port POWER instruction scheduling but they sufficient instruction level parallelism to keep discovered that the GCC compiler’s internal the functional units as saturated. Although model was not powerful enough to accurately the typical operating system engineers will not model the POWER family. The IBM XL fam- be impacted by such details, they should be ily of compilers do not suffer from this limi- aware that a compiler’s code generation qual- tation. If high performance is critical to your ity will impact significantly the performance of application then the use of the IBM provided their OS. Most operating systems have a small compilers will generate much better code than amount of assembler for particularly time criti- GCC and is highly recommended. cal functions, and hand-coding these optimally is difficult.. 4.7 PowerPC performance tools POWER4 derived processors have up to ten functional units, see figure 1, which are As just mentioned the easiest way of gaining fed by several issues queues. If every opera- high performance is to use one of the IBM XL tion is treated seperately then processor con- family of compilers, these compilers accurately trol would become too complex and expensive. model the instruction break up and operation The IBM designers decided to combine oper- grouping on POWER processors. IBM have ations into groups of five, with a single group also published an excellent book, [BBF+01], issued on each cycle and then it is tracked un- that describes the architecture of the proces- til entirely completed. We have been careful to sors and its caches, and has many code ex- refer to operations rather than instructions, as amples optimising performance for particular some PowerPC instructions need to be broken tasks. One of the nicest features of this book up into multiple operations, for example the is a small table of optimal implementations for store multiple registers instructions is issued certain numerical kernels. This table can al- as a separate operation per register. low a developer to determine where they can
15 achieve real performance improvements in their sations. For instance highly performant Pow- algorithm. Apple has provided a less detailed erPC7400 code — the previous generation of guide to optimising for the PowerPC970 in chips that Apple called the G4’ — will not per- [App]. form optimally on the 970. This means that it In addition to the standard Unix tools such is impossible for a single app. to be optimal as prof and gprof to determine hotspots in over many generations of a CPU, despite the applications, an operating system may choose fact that the CPUs are binary compatible with to publish the PowerPC performance regis- each other. ters. AIX makes them available through the pmcount infrastructure. This allows a program to record details of its own performance for a 5 Conclusion programmer to analyse. Unfortunately it takes Like other RISC platforms, the PowerPC ar- a lot of skill to interpret these raw numbers in chitecture has been found to be flexible and a meaningful way. adaptable; from the small PowerPC440 ASIC Apple has developed a highly regarded core up to the massive super–scalar POWER5 suite of software, Computer Hardware Under- and everything in between. The POWER4 standing Developer (CHUD), which they pro- derived architectures are particularly well de- vide free as part of the Apple development signed, it is interesting that IBM managed to suite. It is used to optimise applications run- compress what is essentially a mainframe pro- ning on the PowerPC970 or ’G5’ processors cessor and ship it as the PowerPC970 micro- that power their machines. CHUD has an ex- processor for use in desktop computers. tension to the MacOSX kernel that collects Given the 20 year history of the architec- statistics in the kernel’s privileged space. A ture, it is likely that IBM will continue to performance monitoring application interfaces evolve the PowerPC for the foreseeable future. with the CHUD extension to start and stop It is ironic that Apple computer has dropped data collection. These data are then com- the PowerPC at the same time as MicroSoft bined with fully symbolled object code by the and Sony have committed to developing their Shark application to identify problematic lines respective third generation of gaming consoles of source code. Shark can even be configured based with it. to collect statistics within the kernel itself and The instruction set architecture has stood provided with kernel symbols it can be used up to the test of time remarkably well, IBM’s to identify expensive kernel operations — al- designers obviously realised that C would be though it finds that the kernel seems to spend the most important programming language to a lot of time in the enable interupt function support and tailored the instruction set to suit much to the confusion to non-kernel program- this language well. The bit masking instruc- mers. Other performance application in the tions demonstrate their commitment to opti- suite can collect complicated call graphs. mising C, these instructions are obviously tai- Apple has also made a cycle accurate G5 lor made for the C bit-field manipulation. simulator, [App], known as Amber, available If the PowerPC architecture has fault it for developers. Although not useful to identify would be the weakly consistent memory model. problem code in a large application, once an Even talented programmers get lost in the area of concern has been identified it is invalu- complexity and design trade-offs required to able in analysing and optimising especially crit- write highly performant and correct code. This ical sections of code. A combination of Amber is a failure of design, the decision to leave syn- and hand–coded assembler will allow a talented chronisation up to software requires an exten- engineer to iteratively approach an optimal so- sive and difficult training period for the average lution for the POWER4 derived architectures. engineer. If, as Hennesey claims, it is possible It should be noted that optimal code is pro- to develop a high performance processor with- cessor architecture dependant, changing such out store re-ordering and strongly consistent things as the size of the caches or the internal memory this would be ideal. In IBM’s defence architecture will invalidate most of the optimi- however it should be possible to achieve greater
16 code efficiency with their flexible synchronisa- A final observation on all modern proces- tion model, if a software engineer knows pre- sors, is the trend towards very complicated cisely the context under which the code will super–scalar processors with deep pipelines run. and other VLIW style limitations. The impor- It is interesting that IBM is continuing with tance of a good compiler can not be minimised, the development of very high clock frequen- scheduling instructions to avoid pipeline stalls cies processors after their main desktop com- has never been so difficult, nor so essential. puter competitors Intel and AMD have both Code that is extensively scheduled, however, retreated from it and are trying to change the will not be optimal on different instantiations marketing ground to performance/Watt. IBM of the hardware even with 100% instruction will be shipping the Cell broadband engine set compatibility. Logically, a developer should chips with an astounding 4.0GHz clock fre- optimise for the slowest processor, as the slow- quency, and should be available early in the est processor needs the most help compared to third quarter of this year, 2006. The POWER6 later, faster processors. However this usually is also due to be released soon and, given the leaves some of the new generation hardware ca- performance of the Cell, it seem likely that pability unexploited, not something that prod- IBM will achieve their very aggressive goal of uct marketers ever like selling. The only alter- 4.5 to 5.0GHz frequencies. Even at this incred- native is unpalatable, to ship multiple object ibly high frequency the Cell processor should code with each image scheduled for a partic- only draw some 80 Watts, which is reasonable ular processor implementation. There are ob- for a desktop system. Cooling the PlayStation vious limits to this approach, should a devel- 3 will be quite a dilemma for the Sony design- oper release different images depending on each ers. cache’s size too?
References
[AAB+05] W. J. Armstrong, R. L. Arndt, D. C. Boutcher, R. G. Kovacs, D. Larson, K. A. Lucke, N. Nayar, and R. C. Swanberg. Advanced virtualization capabilities of POWER5 systems. IBM Journal of Research and Development, 49(4/5):523–532, July 2005. [App] Apple Developer Connection. G5 performance programming. [Ass06] Associated Press. IBM unveils super-fast microprocessor, 2006. [BBF+01] Steve Behling, Ron Bell, Peter Farrell, Holger Holthoff, Frank O’Connell, and Will Weir. The POWER4 Processor Introduction and Tuning Guide. IBM Redbooks, 2001. [cF05] Wu chun Feng. The importance of being low power in high performance computing. CTWatch Quarterly, 1(3), August 2005. [CM90] John Cocke and V. Markstein. The evolution of RlSC technology at IBM. IBM Journal of Research and Development, 34(1):4–11, 1990. [Eng06] Apple CoreOS Engineers. Conversations with Apple CoreOS engineers over an eight year period. Private communications, 1997-2006. [GCC+05] Charles Gray, Matthew Chapman, Peter Chubb, David Mosberger-Tang, and Gernot Heiser. Itanium—a system implementor’s tale. In Proceedings of the USENIX Annual Technical Conference, 2005. [HP03] John L. Hennessy and David A. Patterson. Computer Architecture — A Quantitative Approach, chapter Appendix-C, pages C1–C44. Morgan Kaufmann, third edition, 2003.
17 [IBM01] IBM. A brief hstory of RISC, the IBM RS/6000 and the IBM eServer pSeries. IBM Archives 2416RS01, IBM, 2001.
[IJEBP+05] S. S. Iyer, Jr. J. E. Barth, P. C. Parries, J. P. Norum, J. P. Rice, L. R. Logan, and D. Hoyniak. Embedded dram: Technology platform for the blue gene/l chip, 2005.
[Kre05] Kevin Krewell. Cell moves into the limelight. Microprocessor Reports, February 2005.
[ppc99] The PowerPC 440 Core. IBM Microelectronics Division, Research Triangle Park, NC, 1999.
[ppc05] IBM PowerPC 970FX RISC Microprocessor User’s Manual. IBM, version 1.6 edition, December 2005.
[SKT+05] B. Sinharoy, R. N. Kalla, J. M. Tendler, R. J. Eickemeyer, and J. B. Joyner. POWER5 system microarchitecture, 2005.
[TDF+01] Joel M. Tendler, Steve Dodson, Steve Fields, Hung Le, and Balaram Sinharoy. POWER4 system microarchitecture. White paper, IBM Server Group, 2001.
[WSM+5a] Joe Wetzel, Ed Silha, Cathy May, Brad Frey, Junichi Furukawa, and Giles Frazier. PowerPC User Instruction Set Architecture Book 1. IBM, version 2.02 edition, January 2005a.
[WSM+5b] Joe Wetzel, Ed Silha, Cathy May, Brad Frey, Junichi Furukawa, and Giles Frazier. PowerPC Virtual Environment Architecture Book II. IBM, version 2.02 edition, January 2005b.
18