Optimized MV Generator Power Plant Architectures for Large Data Centers White Paper 262 Revision 0 by Julien Moreau Malik Megdiche Daniel Radu Executive summary For large (>5 MW) data center applications, it is common to have large generator power plants connected directly to the medium voltage (MV) electrical distribution network. The principal way to cost optimize the system is to reduce the number of generators installed using N+1 redundancy instead of 2N. This paper introduces some optimized electrical distribution architectures for the generator power plant that meets the various Uptime Institute Tier levels. A cost comparison study and reliability calculations were made on a representative case study. The results are presented in this paper. Rec- ommendations for choosing an architecture based on the expected performance of the system are also provided. Schneider Electric – Data Center Science Center White Paper 262 Rev 0 2 Introduction For data centers larger than about 5 MW it begins to make sense to move more of the electrical distribution typically done at low voltage (LV) to the medium voltage (MV) level while also, perhaps, making the connection to the utility grid at the high voltage (HV) transmission level1. Data centers at these large power capacities have an opportunity to significantly reduce both capital and operating expenses by opti- mizing the design of both the electrical distribution network and how the backup power supply is connected to that network. The key optimization for doing this is going from a traditional 2N or N+1 LV generator architecture to a centralized MV generator power plant that is connected to the MV distribution network. This im- proved architecture is discussed in detail in this paper. Traditionally, generators are provided in a 2N configuration to yield a high level of availability for the backup power supply. But, this comes at a high cost, both CAPEX and OPEX-wise. And since these larger data centers are connected either to redundant MV distribution feeds or to the HV transmission grid itself, the question arises as to whether 2N generator plants are required since the reliability of the grid at MV and HV is very high. Additionally, generator power plants are typically run only periodically for testing and are not used as the continuous power supply for the data center. Of course, a highly available grid does not mean that a reliable backup power supply is not required. But a 2N-based generator architecture is un- necessary especially in light of its high cost. As a result, for large data centers, N+1 (or N+2) generator architectures are becoming the reference standard design. Since in a 2N design the generator power plant represents about 30% of the total electrical distribution investment for a data center of this size - from the connection to the grid down to the rack PDU - optimization has a significant and positive im- pact on the total CapEx by reducing the type and number of equipment including electrical switchboards and so on. It can also potentially improve OpEx, reduce the overall footprint, simplify the design, reduce operational risks, improve scalability, and lower carbon footprint. This paper will describe a globally-applicable electrical architecture using an N+1 generator power plant connected on a fully redundant MV electrical distribution (Figure 1). Note that the generator power plant distribution network (shown in the red box in the figure) is the focus of this paper. The overall electrical distribution network supporting the generators will be explained along with analyzing the relia- bility and cost of this proposed architecture. To do this, this paper will cover the fol- lowing topics: • High level description of cost and comparative cost differences of optimized MV generator power plant architectures based on the Uptime Institute’s Tier levels (i.e., Tier II, Tier III and Tier IV classification)2 • A comparison of these architectures in terms of CAPEX and reliability. 1 Most data centers below 1MW connect to the grid at the LV level. However, much larger data centers tend to connect to the grid at the MV and even HV level. Connection to the grid refers to where the customer-owned gear is connected to the utility-owned gear. 2 Turner, W.P., J.H. Seader. V. Renaud, K.G. Brill; “Tier Classifications Define Site Infrastructure Perfor- mance.” 2008, Uptime Institute, NM. Optimized MV Generator Power Plant Architectures for Large Data Centers Schneider Electric – Data Center Science Center White Paper 262 Rev 0 3 HV substation N+1 Generator power plant EG EG EG EG Figure 1 Primary MV distribution Generator MV Primary MV distribution Shows a large data center con- Path A distribution Path B nected on the HV transmission grid. This architecture is also applicable in the case of the Secondary MV Secondary MV connection to a MV distribution distribution distribution grid. The IT load distribution is Path A Path B in 2N but block or dynamic re- dundancy is also applicable. LV distribution LV distribution LV distribution LV distribution & UPS & UPS & UPS & UPS Path A Path B Path A Path B IT Load IT Load The architectures described in this paper are based on the Uptime Institute’s Tier Generator classification system. That classification system is described in Table 1. power plant architectures Tier Level Basic Description Impact to Generator Tier I Basic capacity, no redundancy Redundant capacity compo- One single failure or maintenance opera- Tier II nents tion can lead to unavailability of the gener- (e.g., chiller, UPS, Generator) ator power plant. Redundancy on the emergency generator and its electrical distribution equipment is needed. Table 1 Description of the Uptime Tier II + concurrently maintain- One single failure can lead to unavailability Institute’s Tier levels and able of the generator power plant, but the oper- their implication on Tier III (each capacity component can ator is able to manually reconfigure the generator power plant be maintained and/or removed system no matter the fault type. architecture without impact to IT) The electrical system (protection, monitor- ing, etc.) can be designed to help the op- erator manually reconfigure the system more quickly in the event of a failure Tier III + fault tolerant (no single point of failure; auto- The generator power plant is fault tolerant, matic response to any failure to Tier IV without any Single Point of Failure ensure service continuity to IT (SPOF)*. without any operator action needed) * To be considered a “Tier IV equivalent” generator power plant (fault tolerant), it is necessary to take into account all operating modes when designing the system. For example, a pre-existing failure before the generators start, a failure during the start- ing sequence, or a failure in normal operation must all be taken into consideration and accounted for when designing the system. Optimized MV Generator Power Plant Architectures for Large Data Centers Schneider Electric – Data Center Science Center White Paper 262 Rev 0 4 The Uptime Institute Tier classification system does not take into account the utility’s availability for evaluating designs, solely relying instead on the on-site backup equipment (i.e., emergency back-up generators, water storage, etc.). Tier II generator power plant architecture To meet Tier II requirements, we propose a simple N+1 generator power plant ar- chitecture with one common MV switchboard and a basic electrical protection sys- tem (based on overcurrent and directional protection3) (Figure 2). In this architecture, all generators are connected to a single switchboard. This “backup” switchboard supplies both paths of the data center. In the event of a ma- jor fault (e.g. something that requires switchboard or cubicle replacement) or dur- ing scheduled maintenance of the generator MV switchboard (i.e. busbar cleaning every 5 years), it is not possible to supply the data center through the generators. EG EG EG EG Utility 1 Utility 2 Figure 2 NO NO NO NO Single switchboard Tier II type architecture; Backup NOTE: During normal Loadbank operation with normal conditions, the switch gear is “NO” or “nor- Path A Path B mally open”. ATS NO NO ATS Note: The MV protection system can be designed to have discrimination between the generator incomers (input power feeds) to avoid a blackout, through use of di- rectional overcurrent or generator differential protection. The exception would be a failure on the generator switchboard busbar. Tier III generator power plant type architectures Double-fed architecture To meet Tier III requirements, we propose an architecture with redundant MV switchboards and a simple protection system (Figure 3). This architecture has each generator connected to both MV distribution switchboards (Gen-A and Gen-B) via two feeders. Only one of these two feeders is closed at a time to maintain a “simple” protection system (based on overcurrent and directional protection) and facilitate fault localization in case of a failure. Gen-A and Gen-B switchboards are intercon- nected by a coupling cable (i.e. a tie), as well as to Path-A and Path-B switch- boards to be able to supply both paths even if Gen-A or Gen-B are down. 3 Refer to the “Electrical Network Protection Guide”, http://www.schneider-electric.com/en/down- load/document/CG0021EN/ Optimized MV Generator Power Plant Architectures for Large Data Centers Schneider Electric – Data Center Science Center White Paper 262 Rev 0 5 With a Tier III system, maintenance operation of any part of the installation (i.e., generators and the surrounding electrical equipment) will not affect the availability of the generator power plant. In case of a failure on the power plant MV distribution (i.e. on switchgear G1 to G4, Gen-A and Gen-B, or one of the interconnection cables), the power plant will expe- rience a blackout: however, after the fault location process, it is possible to recon- figure and restart the power plant.