DRAM Memory System Organization
Total Page:16
File Type:pdf, Size:1020Kb
CHAPTER 10 DRAM Memory System Organization Previous chapters examine the basic building In the past few decades, the growth rate of DRAM blocks of DRAM devices and signaling issues that device storage capacity has roughly paralleled the constrain the transmission and subsequent storage growth rate of the size of memory systems for desk- of data into the DRAM devices. In this chapter, basic top computers, workstations, and servers. The paral- terminologies and building blocks of DRAM memory lel growth rates have dictated system designs in that systems are described. Using the building blocks multiple DRAM devices must be connected together described in the previous chapters, the text in this to form memory systems in most computing plat- chapter examines the construction, organization, and forms. In this chapter, the organization of different operation of multiple DRAM devices in a larger mem- multi-chip DRAM memory systems and different ory system. This chapter covers the terminologies interconnection strategies deployed for cost and per- and topology, as well as the organization of various formance concerns are explored. types of memory modules. In Figure 10.1, multiple DRAM devices are intercon- nected together to form a single memory system that is managed by a single memory controller. In modern computer systems, one or more DRAM memory con- 10.1 Conventional Memory System trollers (DMCs) may be contained in the processor The number of storage bits contained in a given package or integrated into a system controller that DRAM device is constrained by the manufacturing resides outside of the processor package. Regardless process technology, the cell size, the array effi ciency, of the location of the DRAM memory controller, its and the effectiveness of the defect-cell remapping functionality is to accept read and write requests to mechanism for yield enhancement. As the manu- a given address in memory, translate the request to facturing process technology advances in line with one or more commands to the memory system, issue Moore’s Law, the number of storage bits contained in those commands to the DRAM devices in the proper a given DRAM device doubles every few years. How- sequence and proper timing, and retrieve or store data ever, the unspoken corollary to Moore’s Law states on behalf of the processor or I/O devices in the sys- that software written by software companies in the tem. The internal structures of a system controller are Pacifi c Northwest and elsewhere will automatically examined in a separate chapter. This chapter focuses expand to fi ll available memory in a given system. on the organization of DRAM devices in the context of Consequently, the number of storage bits contained multi-device memory systems. in a single DRAM device at any given instance in time has been and will continue to be inadequate to serve as the main memory for most computing 10.2 Basic Nomenclature platforms with the exception of specialty embedded The organization of multiple DRAM devices into a systems. memory system can impact the performance of the 409 410 Memory Systems: Cache, DRAM, Disk Channel? Rank? Bank?Row? Column? Data Channel Address =? Rank Address = ? Memory Bank Address = ? Controller Row address =? Column Address =? Command Sequence FIGURE 10.1: Multiple DRAM devices connected to a processor through a DRAM memory controller. memory system in terms of system storage capac- of the word bank can be inferred from the context ity, operating data rates, access latency, and sustain- in each case, the overloading and repeated use of able bandwidth characteristics. It is therefore of great the word introduces unnecessary confusion into importance that the organization of multiple DRAM discussions about DRAM memory systems. In this devices into larger memory systems be examined in section, the usage of channel, rank, bank, row, and detail. However, the absence of commonly accepted column is defi ned, and discussions in this and sub- nomenclature has hindered the examination of DRAM sequent chapters will conform to the usage in this memory-system organizations. Without a common chapter. basis of well-defi ned nomenclature, technical articles and data sheets sometimes succeed in introducing confusion rather than clarity into discussions on 10.2.1 Channel DRAM memory systems. In one example, a technical Figure 10.2 shows three different system control- data sheet for a system controller used the word bank lers with slightly different confi gurations of the DRAM in two bulleted items on the same page to mean two memory system. In Figure 10.2, each system con- different things. In this data sheet, one bulleted item troller has a single DRAM memory controller (DMC), proclaimed that the system controller could support and each DRAM memory controller controls a single 6 banks (of DRAM devices). Then, several bulleted channel of memory. In the example labelled as the items later, the same data sheet stated that the same typical system controller, the system controller con- system controller could support SDRAM devices with trols a single 64-bit-wide channel. In modern DRAM 4 banks. In a second example, an article in a well- memory systems, commodity DRAM memory mod- respected technical journal examined the then-new ules are standardized with 64-bit-wide data busses, i875P system controller from Intel and proceeded and the 64-bit data bus width of the memory mod- to discuss the performance advantage of the system ule matches the data bus width of the typical per- controller due to the fact that the i875P system con- sonal computer system controller.1 In the example troller could control 2 banks of DRAM devices (it can labelled as Intel i875P system controller, the system control two entire channels). controller connects to a single channel of DRAM with In these two examples, the word bank was used a 128-bit-wide data bus. However, since commod- to mean three different things. While the meaning ity DRAM modules have 64-bit-wide data busses, 1Commodity memory modules designed for error correcting memory systems are standardized with a 72-bit-wide data bus. Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 411 the i875P system controller requires matching pairs Direct RDRAM memory modules are designed with of 64-bit wide memory modules to operate with the 16-bit-wide data busses, and high-performance sys- 128-bit-wide data bus. The paired-memory module tem controllers that use Direct RDRAM, such as the confi guration of the i875P is often referred to as a Intel i850 system controller, use matched pairs of dual channel confi guration. However, since there is Direct RDRAM memory modules to form a 32-bit- only one memory controller, and since both memory wide channel that operates in lockstep across the two modules operate in lockstep to store and retrieve physical channels of memory. data through the 128-bit-wide data bus, the paired- In contrast to system controllers that use a single memory module confi guration is, logically, a 128-bit- DRAM memory controller to control the entire mem- wide single channel memory system. Also, similar to ory system, Figure 10.3 shows that the Alpha EV7 pro- SDRAM and DDR SDRAM memory systems, standard cessor and the Intel i925x system controller each have “Typical” 64 One “physical channel” of 64 bit width system controller DMC DDR One DMC: One logical 64 bit wide channel 64 DDR Intel i875P Two “physical channels” of 64 bit wide busses DMC 128 system controller One DMC: One logical 128 bit wide channel 64 DDR 16 D-RDRAM Two “physical channels” of 16 bit width Intel i850 32 DMC system controller One DMC: One logical 32 bit wide channel 16 D-RDRAM FIGURE 10.2: Systems with a single memory controller and different data bus widths. 16 D-RDRAM 64 D-RDRAM 64 D-RDRAM DDR2 DMC 16 DMC D-RDRAM 64 D-RDRAM DDR2 16 DMC DMC 64 D-RDRAM HPQ Alpha EV7 processor D-RDRAM Intel i925X system controller 16 D-RDRAM Two Channels: 64 bit wide per channel Two Channels: 64 bit wide per channel FIGURE 10.3: Systems with two independent memory controllers and two logical channels of memory. 412 Memory Systems: Cache, DRAM, Disk two DRAM controllers that independently control in an asymmetric mode and independently controls 64-bit-wide data busses.2 The use of indepen- the physical channels of DRAM modules. However, dent DRAM memory controllers can lead to higher since there is only one DRAM memory controller, the sustainable bandwidth characteristics, since the multiple physical channels of mismatched memory narrower channels lead to longer data bursts per modules cannot be accessed concurrently, and only cacheline request, and the various ineffi ciencies one channel of memory can be accessed at any given dictated by DRAM-access protocols can be better instance in time. In the asymmetric confi guration, amortized. As a result, newer system controllers are the maximum system bandwidth is the maximum often designed with multiple memory controllers bandwidth of a single physical channel. despite the additional die cost. A second variation of the single-controller- Modern memory systems with one DRAM multiple-physical-channel confi guration can be memory controller and multiple physical chan- found in high-performance FPM DRAM memory nels of DRAM devices such as those illustrated in systems that were designed prior to the emergence of Figure 10.2 are typically designed with the physi- SDRAM-type DRAM devices that can burst out multi- cal channels operating in lockstep with respect to ple columns of data with a given column access com- each other. However, there are two variations to the mand. Figure 10.4 illustrates a sample timing diagram single-controller-multiple-physical-channel con- of a column access in an SDRAM memory system.