sign in sign up
<<
Home , Communication protocol

◆ and Design Principles for Smart Grids Kenneth C. Budka, Jayant G. Deshpande, Tewfik L. Doumi, Mark Madden, and Tim Mew

An integrated high performance, highly reliable, scalable, and secure network is critical for the successful deployment and operation of next-generation electricity generation, transmission, and distribution systems—known as “smart grids.” Much of the work done to date to define a smart grid communications architecture has focused on high-level service requirements with little attention to challenges. This paper investigates in detail a smart grid communication network architecture that supports today’s grid applications (such as supervisory control and data acquisition [SCADA], mobile workforce communication, and other voice and ) and new applications necessitated by the introduction of smart metering and home area networking, support of demand response applications, and incorporation of renewable energy sources in the grid. We present design principles for satisfying the diverse quality of service (QoS) and reliability requirements of smart grids. © 2010 Alcatel-.

Introduction The global electric power industry is entering a cells is being deployed in homes and enterprises. period of significant transformation. Generation, Introduction of alternate and renewable sources of transmission, distribution, and control infrastructure energy and new storage technologies is fundamen- are aging while energy consumption is increasing. tally altering the centralized power generation and Figure 1, which was developed using data from the distribution paradigm that predominates today. U.S. Department of Energy [18], illustrates the trend Furthermore, variations in the output power of renewa- in worldwide electricity consumption between 1980 ble sources caused by changes in weather and time and 2006. of day are driving the control of distribution networks Smart metering and other demand-side tech- to finer and finer time scales. niques have become increasingly necessary to control “Smart grid is a concept for transforming . . . [the] demand during peak and off-peak hours. Industrial- electric power grid by using advanced communica- scale wind and solar power plants are being connected tions, automated controls, and other forms of infor- to the grid as part of worldwide efforts to reduce car- mation technology. This concept, or vision, integrates bon emissions. Smaller-scale micro-generation in the energy infrastructure, processes, devices, , form of small wind turbines and photovoltaic (PV) and markets into a coordinated and collaborative

Bell Labs Technical Journal 15(2), 205–228 (2010) © 2010 Alcatel-Lucent. Published by Wiley Periodicals, Inc. Published online in Wiley Online (wileyonlinelibrary.com) • DOI: 10.1002/bltj.20450 Panel 1. Abbreviations, Acronyms, and Terms AC—Alternating current NASPInet—NASPI network ADR—Automated demand response NERC—North America Electric Reliability AMI—Advanced metering infrastructure Corporation BPL—Broadband over power line NIST—National Institute of Standards and CCTV—Closed circuit Technology CDMA—Code division multiple access OFDM—Orthogonal frequency division CPP—Critical peak pricing CS—Class selector OSI—Open System DER—Distributed energy resource P2P—Peer-to-peer DiffServ—Differential services PEV—Plug-in electric vehicle DSCP—Differential services code point PHEV—Plug-in hybrid electric vehicle DSL— PLC—Power line carrier EDGE—Enhanced data rates for GSM Evolution PMU—Phasor measurement unit EF—Expedited forwarding PRIME—PoweRline Intelligent Metering EMS—Energy management system Evolution EPRI—Electric Power Research Institute PTT—Push-to-talk GPON—Gigabit PV—Photovoltaic (cells) GPS—Global positioning system QoS—Quality of service GSM—Global System for Mobile RF— frequency Communications RFC—

GtCO2e—Giga (metric) tonne carbon dioxide RTO—Regional transmission organization equivalent RTP—Real time pricing HAN—Home area network RTU—Remote terminal unit HSPA—High-speed packet access SCADA—Supervisory control and data IEC—International Electrotechnical Commission acquisition IED—Intelligent electronic device SDH—Synchronous digital hierarchy IEEE—Institute of Electrical and Electronics SONET—Synchronous optical network Engineers TDM—Time division multiplexing IETF— Engineering Task Force TOU—Time of use (pricing) IntServ—Integrated services UMTS—Universal Mobile IP— System ISM—Industrial, scientific, and medicine UPS—Uninterruptible power supply ISO—Independent system operator VAR—Volt-ampere reactive L1, L2, L3—Layer 1, 2, 3 (of OSI model) VoIP—Voice over IP LAN— VPN— LMR—Land VVWC—Volt, VAR, Watt control LTE—Long Term Evolution WAMS—Wide area measurement system MDMS—Meter data management system WAN— MPLS—Multiprotocol Label Switching WiMAX—Worldwide for NAN—Neighborhood area network Access NASPI—North American SyncroPhasor Initiative

which allows electricity to be generated, dis- This paper addresses network architecture and tributed, and consumed more effectively and effi- design principles for an integrated smart grid commu- ciently” [13]. A high performance, reliable, and secure nication network. We examine some of the challenges communication network is one of the fundamental faced in supporting a diverse set of applications each building blocks to the introduction of smart grid appli- with varying requirements, relia- cations. bility requirements, and traffic characteristics, as well

206 Technical Journal DOI: 10.1002/bltj 18,000 Asia & Oceania 16,000 Africa Middle East Eurasia 14,000 Europe Central & South America 12,000 North America

10,000

8,000

6,000

Energy consumption in billion KWH 4,000

2,000

0 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 P2006 Year Electricity demand is increasing in Asian countries, and in China in particular, which saw demand for energy grow nearly tenfold over a 25 year span. The North American market experienced a twofold increase over the same period despite drastic reduction in the manufacturing industry and slow population growth. Along with the pent up demand, energy sources are becoming scarce and the cost of generating electricity is becoming prohibitive. Therefore, making efficient use of electric energy should, in theory, help reduce dependence on fossil fuel and combat carbon emissions.

Figure 1. Worldwide electricity consumption.

as the challenges faced with supporting legacy appli- applications including brief descriptions of the appli- cations and networks. While there are many legacy, cation examples listed above. A smart grid communi- new, and evolving applications, the following five cation network architecture is presented including the classes of applications (not necessarily mutually exclu- physical connectivity architecture, examples of logical sive) will be used as examples in presentation of connections, access network options, and the archi- communication network architecture and design prin- tectural implications of shared ownership of networks. ciples in this paper: We then address specific quality of service (QoS) and • Smart metering, also known as advanced meter- reliability design considerations for integrated smart ing infrastructure (AMI), grid communication networks. We illustrate the • Automated demand response (ADR), “green benefits” of a smart grid—and by implication • Teleprotection, those of the integrated communication network—and • Distribution automation, and offer our conclusions and recommendations on areas • Micro grid management. for future work. We begin with an overview of the evolution of a Complete treatment of smart grids requires dis- traditional power grid to the smart grid. Next, we cussion of a wide variety of technologies and topics. present a high-level characterization of smart grid Due to space restrictions, we have had to limit scope.

DOI: 10.1002/bltj Bell Labs Technical Journal 207 An essential topic not addressed in this paper is net- from consensus-gathering workshops attended by work security—a topic worthy of several papers on representatives from government agencies, regula- its own. Furthermore, details of local area networks tors, vendors, (communication) service providers, (LAN) or home area networks (HAN) are outside of academia, and standards organizations. Some of the the scope of this paper. earlier EPRI work on IntelliGrid* can be found in [6]. The following brief smart grid presentation will An Overview of Smart Grid be used to set context for network architecture and There is a wealth of information on the smart grid design. In a traditional power grid of an electric power concept and its evolution in the public domain. The system (or utility), electricity flows from bulk power most comprehensive smart grid overviews are found generators to consumers over a grid of transmission in the 2009 reports to U.S. National Institute of lines and distribution feeders, as shown in Figure 2. Standards and Technology (NIST) prepared by the A hierarchy of transmission lines is connected Electric Power Research Institute (EPRI) [5] as well as through a series of transmission substations leading the final NIST Framework and Roadmap document to distribution substations at the edge of the grid. [17]. These extensive reports draw on contributions (See [1] for this classification of substations.) Step-up

Alternate, renewable energy source To regional or national grid Transmission substation Storage Distribution Large scale Bulk power Transmission Large DER (PV, wind, generation substation substation business, diesel, UPS, industrial CHP, …) Transmission complex substation transmission lines Feeder Extra high and voltage Bulk power Transmission Distribution generation PV Residence substation substation

Transmission Medium voltage and substation (sub)-transmission lines

Thermal (coal, Feeder gas), hydro- Business electric, nuclear PHEV Residence Wind, DER PV, Alternate, Storage bio mass, renewable Micro-generation hydro, energy (PV,…) tidal, source fuel cell, …

Transformer(s) (Hierarchy of) micro grids Generator

CHP—Combined heat and power DER—Distributed energy resource PHEV—Plug-in (hybrid) electric vehicle PV—Photovoltaic (cell) UPS—Uninterruptible power supply

Figure 2. Generation, transmission, and distribution in smart grid.

208 Bell Labs Technical Journal DOI: 10.1002/bltj or step-down transformers at the substations are used the use of -aided relay-to-relay communication to change voltages to levels appropriate for the corre- between adjoining substations (i.e., substations con- sponding transmission lines and feeders. Finally, a dis- nected by a ). If protection equip- tribution transformer on a feeder (such as those ment at either end detects a fault, the other end is mounted on utility poles or underground) steps down notified, and protective actions such as tripping the voltage to the standardized level required at the (power circuit disconnect) are initiated in order to iso- consumer locations. Some industrial and large busi- late the fault. SCADA systems consist of remote ter- ness customers may connect to the grid at the feeder minal units (RTUs), programmable logic connectors, voltages or even at the sub-transmission voltages. and other intelligent electronic devices (IEDs) con- An example of voltages used in a typical power nected over communication networks. These sensors system [7] uses generation at 15–25 kV; hierarchy of and actuators are located at power stations, substations, transmission lines at 500 kV, 230kV, and 69kV (sub- distribution transformers, and other grid locations. transmission); distribution feeders at 12 kV; residential They communicate with their respective manage- customers at 115/230 V. Over time, the power grids of ment systems at the utility (centralized) many utilities have been interconnected to form or substations (distributed). In addition to grid opera- regional, national, and international grids improving tions, utility communication needs may also include energy management and transmission reliability. support for enterprise voice and data applications. With the advent of cost-effective generation of Many utilities have deployed private land mobile renewable and/or alternate sources of energy it is now radio (LMR) networks for their mobile workforce for possible to connect these energy sources of various group voice communications (push-to-talk) as capacities throughout the grid (see Figure 2). As a well as some peer-to-peer voice communication result, the direction of electricity transfer will fluctu- needs. ate based on local weather conditions, the position of Examples of Smart Grid Applications the sun, and other environmental effects. To com- The NIST/EPRI roadmap documents, [5] and [17], pensate for the variable nature of photovoltaic and divide the smart grid conceptual model into seven wind generation sources, for example, storage ele- domains that together represent the smart grid ments are deployed. Some of the new storage systems community of interest. These domains are bulk gen- (whether associated with power generation or stand- eration, transmission, distribution, customers (con- alone) include batteries, high-energy flywheels, sumers), grid operations, service providers (for (ultra) capacitors, pumped hydro, and compressed air services such as billing and third party providers), and energy systems. One important class of storage devices markets (wholesale, retail, and trading). While some that are expected to be prevalent in the future are of these domains are connected by the electric grid plugged-in (hybrid) electric vehicles (PEV, PHEV). In (generation transmission, distribution, and cus- addition to supporting transportation, when parked, tomers), all of them must communicate with each the vehicles with charged batteries can potentially be other. The five classes of applications listed earlier are used to supply electricity to the grid. briefly described in this section. These applications have been chosen for their diverse network require- Smart Grid Applications ments and together they incorporate many of the net- The requirements of smart grid applications drive work architecture elements covered in this paper. the design and architecture of an integrated smart grid Smart metering. Smart metering is one of the first communication network. new smart grid applications deployed by most utilities. Traditional Applications Smart metering encompasses much more than peri- Teleprotection and supervisory control and data odic energy measurement. Many new smart grid acquisition (SCADA) applications have been employed applications require frequent power (both active and for power grid management. Teleprotection refers to reactive) and power quality (e.g., voltage, frequency)

DOI: 10.1002/bltj Bell Labs Technical Journal 209 measurements. Such measurements (provided by of 99.95 percent should be reasonable (corre- smart meters) may be required as often as once every sponding to an average downtime of 263 min- 15 minutes to support energy management applica- utes/year). tions. Measurements provided by smart meters are 6. However, some smart grid applications may also used to support real time pricing (RTP), time of require data transfer from all linked meters over use (TOU) pricing, and critical peak pricing (CPP) fea- a relatively short timeframe (several minutes), tures for billing and demand response applications. which requires low latency for each of the indi- 1. Depending on the size of a utility, the number of vidual meters, even if higher latency may be smart meters in the network can vary from a few acceptable for billing purposes. thousand to several million. 7. While security issues are out of the scope of this 2. Regulatory requirements, lack of timely smart paper, it is important to note that smart meters metering standards, and cost considerations have are, perhaps, the weakest link in smart grid secu- led to deployment of vendor-proprietary smart rity. In addition to the security threat to electric- metering solutions based on neighborhood area ity usage data and unauthorized physical access to networks (NAN). These solutions can be readily the meter itself, threats attributable to deployed using wireless technologies deployed in connectivity (for meters thus connected) must be unlicensed spectrum or using power line carrier considered in the architecture and design of the (PLC) technologies. A meter concentrator con- network. nects to the meters over the NAN and is respon- Automated demand response. Demand response sible for aggregating data collected from the activity is an action taken to reduce electricity demand meters it serves. The number of meters served by in response to price, monetary incentives, or utility a concentrator can vary from a handful for a PLC- directives so as to maintain reliable electric service or based NAN to several hundred or even several avoid high electricity prices [20]. Demand response thousand for a (RF) mesh-based is a temporary change in electricity consumption in solution. The meter concentrator connects to the response to supply conditions or other events in the meter data management system (MDMS) over an grid [5]. The inclusion of new energy sources and Internet Protocol (IP) network. storage elements combined with the need to reduce 3. There are products with direct inter- peak loads and conserve energy has driven the intro- faces to a wireless service or interfaces to duction of distributed automated demand response connect to wireline services. For such deployment applications. ADR applications, for example, can be there is no meter concentrator and the meter used to reduce the amount of energy consumed by communicates directly with the MDMS over an IP appliances during peak power periods. While demand network. response has been used by utilities over the years 4. Smart meter connections to home area networks through scheduled load shedding and manually man- are fundamental to residential or building energy aged consumption reduction with a few large con- management. Such connections, for example, sumers, ADR is much wider in scope—bringing allow appliances to respond to pricing signals or dynamic load management directly to residential con- other triggers carried over the smart grid. sumers. ADR often works in concert with distributed 5. Under normal operating conditions, the accepta- energy resources (DER) closer to the point of con- ble response times for completing meter transac- sumption or other energy sources connected into the tions can be high. Thus, one-way packet latency grid. Thus, in some cases, ADR may not necessarily allowances can also be high—on the order of sev- reduce overall energy consumption, but only transfer eral seconds. The availability of an individual the source of some of the consumed energy to DER. meter may not be considered too critical to net- Such load shifting will result in reduced carbon emis- work operations; hence, an availability objective sions if the DER is a renewable energy resource.

210 Bell Labs Technical Journal DOI: 10.1002/bltj Dynamic pricing mechanisms such as CPP and TOU Teleprotection applications require extremely pricing through smart meters contribute to efficient high network availability: failure of such applications ADR implementation and possible cost reduction. may result in destruction of grid infrastructure and, 1. ADR is still evolving. The Demand Response potentially, loss of life. For this reason, utilities have Research Center OpenADR communication speci- deployed redundant communication links between fication is a data model for information exchange substations using a variety of options including pilot between the utility and consumer facility and is wires, leased voice grade lines, leased data lines, PLC, designed to automate demand response actions and fiber including Ethernet, synchronous optical net- at the customer location [12]. work/synchronous digital hierarchy (SONET/SDH), 2. The latency allowance between a utility (or an and microwave. To support the low latency require- independent service operator) and a single con- ments, connections are typically point-to-point along sumer’s premise should be less than one minute. the transmission line between the substations, which As discussed earlier, however, to span large is seldom longer than 300 km. numbers of locations (through their respective Distribution automation. Distribution automation meters), much smaller latencies will be required extends monitoring and control much deeper into the to support “sweeping” through the meters in a distribution network to encompass line reclosers, volt- “neighborhood.” age regulators, capacitor banks, sectionalizers, line 3. Electric vehicles offer another form of storage for switches, fault indicators, circuit breakers, load tap the smart grid, one with unique communication changers, and transformers. In addition to these ele- networking challenges. Even with the most con- ments, new IEDs will also need to be supported. servative estimates of PEVs and PHEVs, utilities To date, power utilities have been accustomed to are ill equipped to support the increased demand managing a limited number of monitoring and control with currently deployed bulk power generation. points, e.g., at hundreds of substations. New commu- P(H)EVs will add tens of thousands of mobile and nications technologies need to be introduced into their endpoints not only to the utility grid, but grid operations in order to connect tens of thousands of also to communication networks. A P(H)EV can endpoints encountered in distribution automation in be charged or discharged at an owner’s home, substations and feeders. Communications with these special charging stations (e.g., at parking lots), or widely deployed endpoints can be challenging depend- other locations. Thus vehicle mobility and energy ing on the available network access mechanisms. transfer must be accurately captured through the Micro grid management. The energy management relationship between the vehicle communication system (EMS) of a typical utility consists of multiple port and the electric port connected to the grid. centralized or distributed systems. The smart grid may Teleprotection. The IEEE 1646 standard [8] lists include and/or connect to “micro grids” possibly man- latency requirements for some of the substation opera- aged by individuals or organizations. Taking the sys- tions at as little as 1/4 of a cycle (which translates to tem approach presented in [11], a micro grid with its about 4 ms and 5 ms, for 60 Hz and 50 Hz AC fre- generation, storage, power lines, and loads becomes a quencies, respectively). For applications requiring subsystem of the larger utility grid. Micro grids can communication between substations, the latency be small and simple such as within a home or may requirements are relaxed to 1/2 cycle. (See Inter- span an interconnection of grid elements in a neigh- national Electrotechnical Commission [IEC] specifi- borhood, or over a single feeder-based system, or over cation 61850-5 [9] as well as [8].) Thus remote a collection of systems connected to a distribution sub- activation of a protection scheme at a substation is station (see Figure 2). In most cases, the micro grid is needed within 8 ms to 10 ms after a fault at that sub- an autonomous system since it may be disconnected station has been remotely detected at an adjoining (involuntarily or voluntarily) from the larger grid, and substation. still support its consumers adequately. We assume the

DOI: 10.1002/bltj Bell Labs Technical Journal 211 Table I. Qualitative comparison of application requirements.

Data Rate/ (One Way) Scope HS Application Data Volume Latency Reliability Security or P2P (at Endpoint) Allowance Smart metering HS Low/v. low High Medium High Inter-site rapid response (e.g., teleprotection) P2P High/low Very low Very high Very high SCADA P2P, HS Medium/low Low High High Operations data HS Medium/low Low High High Distribution automation HS, P2P Low/low Low High High Distributed energy management and control (including ADR, HS, P2P Medium/low Low High High storage, PEV, PHEV) Video surveillance HS High/medium Medium High High Mobile workforce (push-to-X) HS Low/low Low High High Enterprise (corporate) data HS Medium/low Medium Medium Medium Enterprise (corporate) voice P2P Low/v. low Low High Medium Micro grid management HS, P2P High/low Low High High (between EMSs)

ADR—Automated demand response EMS—Energy management system HS—Hub-spoke P2P—Peer-to-peer P(H)EV—Plug-in (hybrid) electric vehicle SCADA—Supervisory control and data acquisition existence of an EMS for the micro grid that can com- 3. “Micro grid management” refers to communica- municate with the EMS of the utility grid or other tion between a micro grid EMS and the EMSs in micro grids for maintaining grid stability and to sup- the micro grid hierarchy. port ADR and other applications. 4. Requirements for some of the applications may be 1. The communication network architecture should different from Table I under certain circumstances. be divided in a hierarchy consistent with micro For example, lower latency and higher reliability grids supporting local reliability if disconnected. may be needed for smart metering during ADR 2. It is possible that the owner of the micro grid and emergency load management activities. communication network is distinct from the owner of the utility network. Network Architecture Overview of Smart Grid Application Requirements The new grid in Figure 2 is much more than an Table I lists a few of the applications (or classes of interconnection of transmission lines and distribution applications) and their qualitative requirements. Note feeders for delivering electricity to homes and busi- that: nesses. Our working and descriptive definition of the 1. Some of the table entries are based on [8] and [9]. smart grid is a power grid, where its applications are man- 2. The quantified values of the requirements depend aged by -of-the-art information technologies over an on the specific nature of applications and associ- integrated high-performance, reliable, and secure commu- ated utility requirements. nication network.

212 Bell Labs Technical Journal DOI: 10.1002/bltj Mobile Communication network Video workforce surveillance Extranet Enterprise voice, data

Sensors and actuators

(Hierarchy of) micro grids Electrical power network

Figure 3. Communication network beyond smart grid control network.

Figure 3 shows the composition of the integrated Physical Connections communication network for a utility. Figure 4 represents the essence of networking The communication network supports communi- architecture for connecting most of the smart grid and cation between the sensors and actuators attached to other endpoints, their locations, and implied applica- the grid elements, and the smart grid applications tions. (Also see [17].) enforcing grid policies through the actuators based on As illustrated, where a network endpoint is these measurements. In addition to the smart grid con- shown to connect to more than one network, not all trol applications, the integrated network supports other connections may be applicable in an implementation. utility needs such as multimedia data transfer (e.g., For example, a building may connect only to one of closed circuit television [CCTV]) from substations four possible network connection options shown: a and voice and data applications for its “enterprise” and renewable energy source may connect to the wire- mobile work force. Furthermore, the network must less or wireline access network, while a substation connect to other utilities’ smart grid networks as well as may require connections to more than one network. to other entities such as the independent system opera- It is not possible to show all smart grid elements and tors (ISOs) and regional transmission organizations there will be differences from Figure 4 in connection (RTOs). Finally, for operational simplicity, it is prudent arrangements in an actual deployed network. For to implement a network hierarchy consistent with the example, a large traditional generating station in an micro grid hierarchy as shown in Figure 3. outlying area may connect through an access net- A smart grid communication network architec- work. Many control and management systems are ture is presented in this section including physical named generically: their implementation will depend connectivity architecture, examples of logical con- on the utilities and vendor products. nections, access network options, and architectural Despite the broad consensus that IP is a reasona- implications of shared ownership of networks. ble choice for a smart grid ,

DOI: 10.1002/bltj Bell Labs Technical Journal 213 Utility data and control center Power station RTO/ISO Extranet (large, traditional) Billing Distribution Voice/data/push-to-x EMS connectivity system man. sys. systems Utility office Video WAMS Meter data SCADA Voice/data surveillance man. sys. man. sys. man. sys. …

(IP/MPLS) core network

Substation Voice SCADA Meter EMS … data man. sys. PMU SCADA Protection concentrator CCTV Vehicle charging station

Storage Mobile Distributed alternate renewable power generation workforce (e.g., PV, wind, bio mass, tidal, microturbines)

Essential Building (residential, business, industrial, other) if present if present if present if present Vehicle Meter Distributed energy Storage EMS resources (PEV, PHEV) Wireless access network Wireline access network Utility “pole” Meter PV SCADA concentrator Neighborhood area network Micro grid EMS Power line communication network HAN/(enterprise) LAN

CCTV—Closed circuit TV ISO—Independent system operator PMU—Phasor measurement unit DER—Distributed energy resource LAN—Local area network PV—Photovoltaic EMS—Energy management system Man. sys.—Management system RTO—Regional transmission organization HAN—Home area network MPLS—Multiprotocol Label Switching SCADA—Supervisory control and data acquisition IP—Internet Protocol PE(H)V—Plug-in (hybrid) electric vehicle WAMS—Wide area monitoring system

Figure 4. Physical connectivity architecture.

NIST has stopped short of mandating IP [17]: to the core network are assumed to be over point- “Among smart grid stakeholders, there is a wide to-point (layer 1 or layer 2) connections. There may expectation that Internet Protocol (IP)-based net- be more than one data and control center for relia- works will serve as a key element for the smart grid bility and load sharing of the systems. It is expected information networks. . . . An analysis needs to be that many smart grid applications will require dis- performed for each set of smart grid requirements to tributed control and management systems, which determine whether IP is appropriate.” We do assume could be located at the substations and in the corre- here that IP is the networking protocol of choice for sponding micro grids. The network must also con- the integrated network. Additionally, for many utili- nect to the networks of other utilities in the ties it may be prudent to implement Multiprotocol regional/national smart grid. Also (extranet) con- Label Switching (MPLS) virtual private networks nectivity is required to the utility’s partner ISO/RTOs (VPNs), with each VPN supporting a set of applications or and corporate service providers for billing, installa- user communities. In Figure 4, direct connections tion, and other services.

214 Bell Labs Technical Journal DOI: 10.1002/bltj Every consumer location is expected to have a The IEC 61850 standard provides comprehensive smart meter that is connected to the network. A con- specifications for substation automation for connect- sumer building may have DERs, energy storage ele- ing substation systems that support grid operations. ments, and/or electric vehicles. DERs may be classified These systems include legacy analog and digital sys- as microgeneration (at residences and small busi- tems as well as new IEDs (e.g., for SCADA and pro- nesses) or large-scale (at large business or industrial com- tection). The resulting substation automation LAN plexes). Depending on their engineered capacity, DERs may be implemented as a hierarchy of Ethernet LANs provide between 3 kW and 10,000 kW power. DERs are where a station connects many process busses typically located in or close to a consumer building with each process bus connected to multiple modern [4]. When present in consumers’ premises, DER, stor- and legacy substation systems. For a typical substa- age, vehicles, and meters may connect over a HAN tion LAN see [2]. or a LAN, which is in turn linked to the communica- The substation may have a separate LAN for tion network—often through the meter for residential applications that do not directly contribute to automa- consumers. tion. The substation router may need to connect to The grid equipment at distribution points more than one network from the set of networks (e.g., utility poles) connects to the network over one shown in Figure 5. Depending on the network tech- or more of the connections shown in Figure 4. nology, network access adapters or gateways will be Standalone (and large scale) alternate/non-traditional/ needed. An application such as teleprotection may renewable energy sources, storage elements, and need network connections to an adjoining substation vehicle charging stations connect to the wireless in addition to or instead of the connections through and/or wireline access networks. Each micro grid has the substation router. Over time, different networking its own communication network similar (but smaller technologies have been used with network-specific in scale) to that shown in Figure 4. The micro grid gateways facilitating such connections. IEC is working EMS connects to the utility or other micro grid net- on extending the 61850 standard to support commu- work. nication between substations using multicast over It is expected that Voice over Internet Protocol Ethernet, which would allow substation automation (VoIP) will be used for peer-to-peer (P2P) voice and LANs to connect over an Ethernet network. Further, push-to-talk (PTT) communication of the mobile tunneling Ethernet over an IP connection through work force. It is also expected that mobile wireless the router is a possibility if latency objectives can be data applications will migrate to broadband. Thus, all met. (See [21] for an overview of teleprotection con- voice, push-to-x, data, and video needs can be satis- nectivity options, including Ethernet.) fied by the mobile access terminal connecting to the Logical Connection Models wireless broadband access network. Until VoIP com- Irrespective of their physical network connec- munication is available, gateways will be needed to tions, it is important to determine endpoints of an connect legacy voice systems to the smart grid com- application carried over the network. As an illustra- munication network. tion, logical connection models for a few of the appli- Depending on its location and size, a substation cations are shown in Figure 6, highlighting the may many smart grid and other systems that interdependence of the applications—particularly require communication with other endpoints. Only among ADR, SCADA management, and distribution a few of these systems are shown in Figure 4. Figure 5 management. is a more detailed schematic of a substation’s com- Depending on the grid elements under its con- munications infrastructure and its connectivity to vari- trol, an EMS communicates only with a subset of the ous possible wide area networks (WANs). This generation and storage elements shown in Figure 6. illustration depicts only the generic systems requir- The smart meter that plays a central role in many ing communication with the outside world. applications has been replicated (as entity M) for ease

DOI: 10.1002/bltj Bell Labs Technical Journal 215 Core network

Transmission Wireless Wireline Feeder PLC NAN line PLC access network access network

One or more network-specific adapters, or gateways

Router Meter Data concentrator CCTV

Substation automation LAN (of station bus and process bus)

SCADA ADR EMS … Voice Protection … management SCADA PMU management system system for teleprotection “Gateway” Additional network(s) Substation Ethernet, optical, PDH, SDH/SONET PLC, other

ADR—Automated demand response NAN—Neighborhood area network SCADA—Supervisory control and data acquisition CCTV—Closed circuit TV PDH—Plesiochronous digital hierarchy SDH—Synchronous digital hierarchy EMS—Energy management system PLC—Power line carrier SONET—Synchronous optical network LAN—Local area network PMU—Phasor measurement unit

Figure 5. Substation network.

of presentation. Newer distribution automation appli- lines, and the need to connect PLCs across trans- cations such as the Volt, VAR, Watt control (VVWC) formers (except when the PLC carrier frequency is application will increasingly use periodic as well as identical to the 50 Hz or 60 Hz line frequency.) While on-demand smart meter measurements. Additionally, traditionally data rates have varied from about 15 bits the smart meter is a centerpiece of ADR applications per second for line frequency carriers to about 3 to 4 and home energy management. (Also see [5].) kilobits per second (kb/s), new orthogonal frequency division multiplexing (OFDM)-based techniques will Networking Technologies allow for rates as high as 130 kb/s [15]. The current This section presents networking technologies for draft of the emerging IEEE P1901 Broadband Over the network segments shown in Figure 4. Power Line (BPL) standard indicates future support PLC networks. Communication over power line for much higher rates, i.e., in the tens of megabits per carriers in various bands between 10 Hz and 500 KHz second range. have been used by utilities for some of their commu- Neighborhood area networks. PLC is one example nication needs. The main advantage of PLC is the of NAN connecting buildings in a neighborhood. availability of power lines connecting every endpoint Another example is RF mesh over unlicensed spec- that needs to be connected to the communication net- trum, such as over the 900 MHz industrial, scientific, work. However, PLC shortcomings include low data and medicine (ISM) band or the 2.4 GHz band. While rates, interference over long distance high voltage NANs are predominantly used for smart metering,

216 Bell Labs Technical Journal DOI: 10.1002/bltj Distributed alternate renewable Power station Electric vehicle power generation RTO/ISO (large, charging (e.g., solar, wind, bio mass, traditional) station tidal, microturbines)

Increase, Increase, decrease, Increase, Billing decrease Store Release energy shut-down decrease Demand system shut-down energy energy Storage Store energy Measurements Release M Building (periodic) or other energy Pricing energy (Utility) EMS Meter consumption entity signals Demand Pricing Measurements (policies) Demand Demand Power quality Measurements Micro grid measurements Distribution (periodic, Increase, Electric (periodic, man. sys. DER Meter data on-demand) decrease vehicle on-demand) man. system ADR On- SCADA On-demand demand Micro grid man. sys. (Processed) man. sys. measurements Regulate EMS Pricing signals, Building periodic poll, Status, Storage Meter on-demand WAMS Regulate measurements, M Regulate Measurements To other man. sys. incidents (periodic, utilities, on-demand) control centers, SCADA etc. Measurements SCADA (actuator) (sensor) M Meter PMU SCADA systems

Schedule, stop Status

Home appliance

ADR—Automated demand response P(H)EV—Plug-in (hybrid) electric vehicle CCTV—Closed circuit television PMU—Phasor measurement unit DER—Distributed energy resource RTO—Regional transmission organization EMS—Energy management system SCADA—Supervisory control and data acquisition ISO—Independent system operator WAMS—Wide area measurement system Man. sys.—Management system

Figure 6. Logical connection models for a few smart grid applications.

other applications may also be carried such as SCADA Access 2000 (CDMA2000), Enhanced Data Rates for RTUs and IEDs located at the neighborhood trans- GSM Evolution (EDGE), Universal Mobile Telecom- formers. Lately Wi-Fi has also become a viable NAN munications System (UMTS), High Speed Packet technology for these applications. Access (HSPA), Worldwide Interoperability for Micro- Wireless access. Wireless broadband access is one wave Access (WiMAX), and Long Term Evolution of the catalysts for successful deployment of the smart (LTE). Depending on the technology and configura- grid. In many countries, with the exception of unli- tion, data rates between 500 Kb/s and 3 Mb/s or more censed spectrum, most of the spectrum is owned by are possible (1 Mb/s to 6 Mb/s or more on downlinks). service providers and available for broadband services With little spectrum available for their exclusive use, over technologies such as Code Division Multiple utilities may subscribe to wireless broadband services

DOI: 10.1002/bltj Bell Labs Technical Journal 217 if service providers offer the required coverage, satisfy between the utility pole and the central office is often the security and reliability requirements, and support a fiber connection). preferential treatment for critical utility applications As noted before, direct SDH/SONET and direct when needed. As an example, utilities in the United Ethernet networks are also used for connecting adjoin- States have dedicated narrowband spectrum, essen- ing substations for applications like teleprotection. tially to carry voice services over their private mobile Core network. Depending on the number of com- , with little to no prospect for future munication endpoints of a utility, their locations, their broadband allocations. In some cases, utilities may communication requirements, and other require- be able to acquire spectrum for their smart grid net- ments, a core network may consist of a router at the work if such spectrum is available with the corre- utility data center, or routers connected over an opti- sponding product support. The recent allocation of 30 cal ring or metro-Ethernet in a metropolitan area, or MHz of spectrum in the 1.8 GHz band to electric utili- a mesh of routers connected over point-to-point links. ties in Canada is a prime example where spectrum was For the general mesh, the point-to-point links are indeed assigned but in a band not considered by wire- either utility-owned or leased from layer 1 (L1) (pri- less standardization bodies at this time, thus requiring vate lines) and L2 (e.g., Ethernet, frame relay) ser- product customization. vice providers. If the performance, reliability, and While spectrum can be made available in a few security requirements are acceptable, the core net- countries, one possibility is for utilities to enter into a work can also be an MPLS VPN from an L3 service partnership with wireless service providers or a spec- provider with the utility routers connecting to the trum licensee, for sharing resources such as spectrum, provider edge routers of the MPLS VPN service [16]. towers, equipment cabinets, and/or even network The utility may itself provide MPLS VPN service so equipment with acceptable logical partitioning. The that groups of users, applications, and/or locations can relationship between a utility and a wireless service manage their networking needs in an autonomous provider, whether a partnership for resource sharing fashion. In a case where the core network itself is an or direct customer-provider access agreements, will MPLS VPN over a service provider network, the VPNs have the corresponding impact on network architec- within the utility can still be created independent of ture and design. and within the service provider VPN. Wireline access. Broadband wireline services like digital subscriber line (DSL), cable, Gigabit passive opti- Network Ownership: Utility-Owned Versus cal network (GPON), and BPL can provide the band- Public Carriers width needed for smart meters, SCADA equipment, Most utilities prefer exclusive end-to-end owner- and meter concentrators located at the distribution ship of the network though this may not always be transformers, or aggregation of substation-based appli- possible because of cost considerations, spectrum availa- cations traffic through substation routers. Some utili- bility, the need for deploying applications in an expe- ties may be averse to allow residential broadband dient manner, and other considerations. There are connections to be used for smart meter traffic since many advantages and a few serious drawbacks the connection is shared among other applications (e.g., costs) for utility-owned networks over shared own- of the homeowner. On the other hand, if the broad- ership of network segments with service providers. The band service and the utility are owned by the local fact that multiple parties ownership of the net- government of a (small or medium sized) commu- work segments within an integrated network does nity, the residential broadband connection may in fact affect the network architecture’s physical and logical be preferred by the utility. (Note that, in spite of its connectivity, routing, reliability, security, and other name—broadband over power line—BPL uses the factors. In addition to commercial and business con- power line only between the secondary of the distri- siderations, interoperability agreements between the bution transformer and home. The BPL connection utility and providers can influence

218 Bell Labs Technical Journal DOI: 10.1002/bltj the network architecture in its end-to-end network honored design methodologies for service provider operations, monitoring, and management challenges. networks and even many large enterprise networks. A Even if the utility owns the end-to-end network, few implementation and product development chal- the point-to-point L1 and L2 connections may still be lenges also will be identified, with suggestions for leased from service provides with acceptable capacity, workarounds where possible. reliability, security, and preferably with exclusivity. However, implementing exclusivity on L1 and L2 links over a wireless service provider network sup- From the earlier discussions on smart grid, utility porting utility requirements for their mission critical applications, and the placement of application end- applications can be challenging. points, the obvious choice for the network topology is Interconnection With Network of Synchrophasors predominantly a tree structure. The initial topology Widespread blackouts at the beginning of this cen- of an iterative design will be similar to the one shown tury have underscored the necessity of wide area mea- in Figure 7 before QoS, reliability, and other require- surement systems (WAMS) for regional or national ments are applied to determine the final topology grids across all the connected utilities’ power systems. design. WAMS employ phasor measurement units (PMUs) to In addition to the centralized destination of the measure voltage and current phasors (phase vectors) utility data and control center for a significant amount of the corresponding alternating current waveform of applications traffic, a substation router is perhaps and its harmonics. PMUs are considered to be the state the other most identifiable location of traffic aggre- of the art SCADA RTU and are expected to be gation as shown in Figure 5. Traffic from multiple deployed at a large number of locations in participat- (smaller) substations may be aggregated at another ing utility grids. PMUs (often called synchrophasors) (large) substation. Depending on the meter technolo- are synchronized to a common clock, usually derived gies deployed, the metering traffic may be aggregated from the Global Positioning System (GPS). This allows at the meter concentrators at substations, or the for time-stamped measurements shared among meters or concentrators may connect directly to utilities, regulatory bodies, and other organizations the core network. Finally, other smart grid endpoints through PMU gateways connected to a regional or such as energy sources and storage units connect to national network such as the one being developed and the substations or directly to the core network routers, deployed by the North America SynchroPhasor depending on their locations and/or the location of Initiative (NASPI) [14]. The NASPI network (NASPInet) the corresponding EMSs. will be a high performance, reliable, and secure com- In principle, peer-to-peer applications can be sup- munication network that connects PMU gateways ported over the tree topology, since connectivity is among utilities in a region to a distributed data bus, always possible through the core routers. However, allowing for PMU data sharing (almost instantaneously the latency requirements of some applications may for some applications) between utilities as well as orga- not be satisfied by routing that traffic through the core nizations such as the North America Electric Reliability network. Many of these low-latency applications have Corporation (NERC). endpoints in adjoining substations. Thus, it is prudent Therefore, a utility smart grid network will need to maintain direct communication link(s) between to connect to (and be a part of) a network like substations as shown in Figure 6 and described fol- NASPInet. lowing Figure 5. These direct inter-substation links may additionally provide the possibility of a preferred Network Design Principles to Facilitate path for other P2P traffic such as VoIP bearer. Smart Grid Applications The topology design is also affected by the fact This section deals with topology, QoS, and relia- that the choice of access network is driven more bility considerations that may differ from the time- by the coverage of a large number of endpoints than

DOI: 10.1002/bltj Bell Labs Technical Journal 219 Utility data and control center

(IP/MPLS) core network

Power station Storage Utility office (large, RTO/ISO Substation traditional)

Micro grid EMS Substation Substation Substation Substation Vehicle charging station Mobile Vehicle charging SCADA workforce Storage Micro grid station EMS Distributed alternate renewable Mobile Meter power generation (including DER) workforce concentrator Distributed alternate renewable power generation (including DER) Meter Meter Meter Building Building Building

DER—Distributed energy resource MPLS—Multiprotocol Label Switching EMS—Energy management system RTO—Regional transmission organization IP—Internet Protocol RTU—Remote terminal unit ISO—Independent system operator SCADA—Supervisory control and data acquisition

Figure 7. Network topology.

by the traffic volume. Further, for most centralized For a few analog applications including push-to-talk applications, the upstream traffic volume is greater voice, circuit emulation (time division multiplexing than the downstream traffic, requiring special design [TDM] over Ethernet or IP) will have to be provided considerations when carried over service provider until these applications migrate to IP (e.g., VoIP). networks that are generally optimized for higher Quality of Service downstream traffic than upstream traffic. The two important QoS factors considered here are Integrating Legacy Applications and Networks a wide range of latency requirements and (dynamic) While new applications and new smart grid sys- association of flow priority to applications consistent tems are expected to support IP connectivity, the inte- with smart grid operations. With smaller data vol- grated smart grid communication network will also umes, efficiency in allocation to applica- have to connect to applications at legacy systems for tions may not be the most important QoS objective in a period of time. In most cases, gateways to the legacy smart grid network design. systems will be required. Depending on the evolution Managing latencies. Throughout this paper, the of end systems, these gateways can be as simple as need for supporting applications with diverse latencies those providing serial-to-Ethernet conversion to those (from about 8 ms to 1 second or more) was empha- supporting full protocol conversion. sized. But lower latency does not always imply higher

220 Bell Labs Technical Journal DOI: 10.1002/bltj Table II. (Representative) latency requirements of smart grid applications.

Latency Application (only a few allowance example applications (assumed, considered) Application setting unverified) Comments Teleprotection All 8 ms, 10 ms For 60 Hz and 50 Hz, respectively 60 messages per second stipulated for Phase measurement unit Class A data service 16 ms Class A data service in [14] Push-to-talk signaling Incident-related 100 ms Connect to many Example: ADR within 1 minute for up Smart meter meters in a short 200 ms to 300 meters connected over a shared time medium SCADA data: poll response 200 ms See [8]. VoIP bearer 175–200 ms Includes P2P and all PTT VoIP signaling 200 ms Includes non-incident-related PTT Post event (latency value assumed).

Phase measurement unit Class C data service 500 ms See [14].

In the order of decreasing priority On demand SCADAc 1 second See [8]. Periodic meter Say, once an hour or lower frequency Smart meter Ն 1 second reading of reading

ADR—Automated demand response P2P—Peer-to-peer PTT—Push-to-talk SCADA—Supervisory control and data acquisition VoIP—Voice over IP

priority. For example, according to Internet Engineering QoS design approach. The QoS design should begin Task Force (IETF) Request for Comments (RFC) 4594 with listing all utility applications and their priority guidelines [3], network control traffic with low and latency requirements. Table II is a sample of latency (delay tolerance) characterization is given utility applications with their priority and latency higher priority with a differential services (DiffServ) requirements. class selector 6 (CS6) while the VoIP bearer traffic Using VoIP bearer traffic as a reference, applica- with very low latency is allocated to a lower priority tions such as teleprotection and PMU data transfer to expedited forwarding (EF) class. In most data net- NASPInet have much lower latency requirements working , as in RFC 4594, the VoIP than the latency allowance of up to 175 ms to 200 ms bearer traffic is given a higher priority than any class necessary for good voice quality under most condi- of applications (other than network control), often tions, yet voice traffic latency is considered very low assigning it to the strict priority egress queue at each in RFC 4594. As can be seen from Table II, there are router. The primary reason for such a high priority is smart grid applications with much lower latency to manage the jitter and packet loss of the bearer traf- requirements and very high priority. The only plausi- fic even if there may be real-time or business appli- ble design choice for satisfying 8 ms to 16 ms latency cation traffic that requires higher priority. Managing a requirements may be in directly connecting the appli- diverse set of application priority and latency require- cation endpoints such as two substations, thus elimi- ments for a smart grid network will require a different nating intermediate hops and reducing propagation

DOI: 10.1002/bltj Bell Labs Technical Journal 221 delay (also see Figure 7). Packet loss and jitter con- alongside a more granular smart grid application pri- siderations for some other applications (e.g., VoIP) ority hierarchy that is similar to Table II. carried over this inter-substation connection may Thus with DiffServ QoS, differential services code have to take a back seat, with lower priority. (But point (DSCP) allocation to smart grid applications will note that many new codecs do correct for packet loss have to be different from RFC 4594 guidelines. In and jitter.) Even though RFC 4594 lists many appli- addition, while preemption of a packet under trans- cation classes that may be implemented with judi- mission (either one partially transmitted or transferred cious choices of DiffServ values, for many practical to a very small line buffer just before transmission) is QoS implementations in service providers or enter- not allowed in most products or network implemen- prise data networks supporting typical multimedia tations, critical high priority smart grid applications applications, four or fewer classes of service are usu- may require the preemption feature. The use of inte- ally provided. It is up to the network customers to grated services (IntServ) QoS will have to be consid- map their applications to the pre-defined QoS classes. ered for some of the applications. Since existing One such typical classification is shown in Figure 8, networking products may not support some of these

Network control Network control

Teleprotection PMU (class A data service) PTT signaling (incident-related) Smart metering (access many meters in a short time) SCADA (poll response)

Active ADR VoIP bearer (including PTT) VoIP bearer Class 1 VoIP signaling VoIP signaling (including some PTT) Class 2 Video Video Decreasing priority Critical enterprise/operation data Critical data PMU (class C data service) On demand SCADA Non-critical Non-critical data Class 3 enterprise/operations data Smart metering (periodic meter reading)

Best effort data Best effort data Class 4

Smart grid application priorities Typical multimedia network application priorities and QoS classes

ADR—Automated demand response QoS—Quality of service PMU—Phasor measurement unit SCADA—Supervisory control and data acquisition PTT—Push-to-talk VoIP—Voice over IP

Figure 8. Smart grid application priorities.

222 Bell Labs Technical Journal DOI: 10.1002/bltj required features, clever workarounds will be required; between a pair of adjoining substations are essential. using per flow QoS is one such possibility. Future net- Depending on the connection option, availability of working product architects must seriously consider the corresponding IP network elements, regulatory developing features that support the smart grid com- requirements and standards, and/or a utility’s prefer- munication networks’ QoS requirements. ence, one or more of these inter-substation connec- Finally, the importance of respecting latency tions may not be included in the integrated IP requirements of some of the ADR applications per- network design. However, every effort must be made taining to renewable variable energy sources such as to include at least one of these links in the IP network wind and PV cannot be overstated. Longer transients (e.g., Ethernet connection [21]). In addition to incor- due to a lack of efficient QoS design can lead to per- porating the teleprotection application in the turbation in the power grid. integrated network, such between Managing dynamic flow priority. Latency tolerance substations provide increased reliability for all appli- and/or priority for smart grid applications may depend cations carried over that connection. If needed, a judi- on the context or setting of the corresponding grid cious design of the routing protocol and/or MPLS VPN operation or environment. For example (see Table II implementation will help limit the use of these mul- and Figure 8), periodic meter reading traffic may be tiple links to applications with high reliability require- given lower priority and liberal delay allowance, ments and for peer-to-peer applications. whereas traffic from meters located in an area with Conventional but critical reliability design ele- active demand response processes or outage manage- ments including substation communication link diver- ment must be given higher priority and a lower delay sity, redundant meter concentrators, and disaster allowance. Setting a higher priority than normal to recovery plans for data centers and other establish- PTT signaling and bearer traffic during an emergency ments must be considered to achieve the required net- and widespread blackout or to video surveillance traf- work availability goals. Since the communication fic after detection of an incident are other examples. network is used for managing the power grid, it One possible workaround is to treat an application is imperative that the network elements are not with varying latency and priority requirements as impaired by power outages. At a minimum, battery multiple distinct applications. If DSCP is used, depend- backups or an uninterruptible power supply (UPS) is ing on the application setting, different DSCP value necessary for communication systems as well as for should be assigned to the same application. Since some end systems. Further, PLC cannot be the sole workarounds may not always be possible, standards means of connectivity for many applications. and product capabilities are needed for a generalized mapping of the tuple Ͻapplication, priorityϾ to a QoS Green Benefits class. The introduction of smart grid applications does Reliability contribute to green benefits as highlighted in the It is extremely important that the (smart) grid Smart2020 report [10], which predicts a 14 percent reliability requirements are translated into consistent reduction by year 2020 for global carbon emission communication reliability requirements. attributable to smart grid evolution, corresponding to As noted earlier, communication network relia- a reduction of 2.03 GtCO2e, from the current emis- bility objectives for different applications can be very sion of 14.26 GtCO2e. It is believed that 24 percent different: 99.95 percent availability (with an average of the total carbon emission today is attributed to the downtime of 263 minutes/year) may be sufficient for power sector. Accordingly, the expected reduction in periodic meter reading, but 99.999 percent availabil- overall carbon emission due to the electric power sec- ity (with an average downtime of 5.3 minutes/year) should be about 3 percent. Of course, this is pos- may be low for teleprotection. To support the latter sible partly due to the use of renewable and alternate requirement, multiple point-to-point connections energy sources, peak power reduction, and energy

DOI: 10.1002/bltj Bell Labs Technical Journal 223 sources closer to load. The last item mostly refers to to teleprotection with extremely low latency require- reduction in transmission (I2R) losses, since a signifi- ments, to communication between autonomous cant amount of the electricity carried over transmis- micro grid networks. (including sion lines is generated by fossil fuels today. cyber security), an extremely important aspect of An integrated communication network helps network architecture, was not considered in this make a smart grid more efficient, thus indirectly con- paper. tributing to green benefits. It is clear that communication network design for There are also some green benefits, however the smart grid requires network topology, QoS, and small, that are directly related to the time saved in reliability considerations that may not be common- automated policy execution through communication place in designing service provider or enterprise data networks. For example, automated demand response networks. While it may not be possible to implement with effective communication can reduce the amount the optimal design with available product and net- of time needed for manually shifting from bulk elec- work technologies, workarounds may be used. tric sources to DER. For illustration, if it takes 20 min- Finally, the “green benefits” of the smart grid— utes for manual shifting of an energy source, for every and by implication, that of the integrated communi- kW of power shifted using ADR, 0.333 kWh less cation network—were presented in terms of carbon energy will be drawn from the bulk electricity source reduction. than with manual operation. Under the assumption that the bulk electricity source is coal, a 40 percent Recommendation for Future Work thermally efficient power plant could produce an We believe that the correlation between the smart average of about 0.83 kg of CO2 emissions per kWh of grid architecture and the corresponding physical and generated electricity [19]. Further, if the resulting logical connectivity of the network architecture must DER used with ADR is a renewable source of energy, be exploited in developing the smart grid architec- our assumption of a 20 minute savings in ADR opera- ture. This holistic view of the smart grid and its com- tion would yield a reduction of 0.277 kg in CO2 emis- munication network will facilitate an easier extraction sion for every kW of power shifted. Assuming that of the network architecture from the smart grid archi- such a shift of energy sources occurs once every day, tecture, including direct connection between their the average annual carbon savings achieved by the respective performance, reliability, and security re- use of ADR over a manual demand response opera- quirements. Even if such “greenfield” smart grid tion is about 100 kg of CO2 for every kW of power implementations may not be practical in most shifted. instances, ongoing development of smart grid archi- tecture and design, as well as new grid applications, Conclusions can facilitate the corresponding development in net- This paper presents a network architecture for an work architecture and design. For example, integrated high performance and highly reliable com- 1. Tools that help determine network configurations munications network for the successful deployment as an integral part of new application develop- and operation of a smart grid. The architecture frame- ment and deployment. work was driven by the smart grid applications— 2. Network protocols that help reduce power tran- mission-critical and otherwise—as well as other utility sients, particularly those attributable to variable applications that must be carried over the integrated energy resources connected into the grid. network that meet or exceed their individual require- 3. Automatic setting of QoS configurations when ments. Throughout the paper, a few representative application requirements change based on grid smart grid applications were used to illustrate the net- events. work architecture. These applications ranged from 4. Translating the self-healing grid to the self-healing smart metering with a very large number of endpoints, communication network.

224 Bell Labs Technical Journal DOI: 10.1002/bltj Additional recommendations for future work: [6] Electric Power Research Institute and 1. QoS management of applications traffic with a Electricity Innovation Institute, IntelliGrid large variety of performance requirements includ- Architecture Report: Volume 1, IntelliGrid User Guidelines and Recommendations, ing latency and priority. 1012160, Final Report, 2002. 2. Extending the smart grid architecture to specific [7] L. L. Grigsby (ed.), Electric Power Generation, micro grids such as a micro grid spanning a build- Transmission, and Distribution, 2nd ed., CRC ing, a feeder, and an electric vehicle charging Press, Boca Raton, FL, 2007. station. [8] Institute of Electrical and Electronics 3. Extending the architecture and design principles Engineers, “IEEE Standard Communication Delivery Time Performance Requirements for introduced in this paper to include network secu- Electric Power Substation Automation,” IEEE rity. It is important to note that security consid- 1646–2004, Feb. 25, 2005. erations must be incorporated at the beginning of [9] International Electrotechnical Commission, network architecture and design process. “Communication Networks and Systems in Substations,” IEC 61850-1–61850-10, 2003. Acknowledgements [10] C. Kruse and B. Singanayagam, “SMART2020: We want to thank Marc Benowitz and Sam ICT and a Low-Carbon Economy,” J. P. Samuel, co-editors of this special issue, and an anony- Morgan, Europe Equity Research, July 9, mous reviewer for their review and valuable sugges- 2008. [11] R. H. Lasseter and P. Paigi, “Microgrid: A tions. We also thank Joe Morabito for his comments Conceptual Solution,” Proc. 35th Annual IEEE on an early draft of the paper. Power Electronics Specialists Conf. (PESC ‘04) (Aachen, Ger., 2004), vol. 6, pp. 4285–4290. *Trademark Intelligrid is a registered trademark of Electric Power [12] Lawrence Berkeley National Laboratory Research Institute, Inc. Demand Response Research Center, Open Automated Demand Response References Communications Specification (Version 1.0), [1] R. Albert, I. Albert, and G. L. Nakarado, CEC-500-2009-063, California Energy “Structural Vulnerability of the North American Commission, Public Interest Energy Research Power Grid,” arXiv:cond-mat/ 0401084v, Jan. Program, Apr. 2009, Ͻhttp://drrc.lbl.gov/ 2004, Ͻhttp://arxiv.org/ PS_cache/cond- openadr/pdf/cec-500-2009-063.pdfϾ. mat/pdf/0401/0401084.pdfϾ. [13] Michigan, Department of Labor and Economic [2] L. Andersson, C. Brunner, and F. Engler, Growth, “21st Century Energy Plan,” “Substation Automation Based on IEC 61850 Alternative Technologies Workgroup, Meeting with New Process-Close Technologies,” Proc. Handout 1, July 27, 2006, Ͻwww.dleg.state. IEEE Bologna PowerTech Conf. (PowerTech mi.us/mpsc/electric/capacity/energyplan/alttech/ ‘03) (Bologna, It., 2003), vol. 2. smartgrid_draftreportoutlinejul19_2006.pdfϾ. [3] J. Babiarz, K. Chan, and F. Baker, [14] National Energy Technology Laboratory, “Configuration Guidelines for DiffServ Service “Specification for North American Classes,” IETF RFC 4594, Aug. 2006, SynchroPhasor Initiative (NASPI),” Ͻhttp://www.ietf.org/rfc/rfc4594.txtϾ. Attachment A, Statement of Work, May 2008, [4] California, The California Energy Commission, Ͻhttp://www.naspi.org/ resources/dnmtt/ “California Distributed Energy Resources naspinet/quanta_sow.pdfϾ. Guide,” 2009, Ͻhttp://www.energy.ca.gov/ [15] PRIME Project, “PHY, MAC and Convergence distgen/index.htmlϾ. Layers,” White Paper, v1.0, 2008, Ͻhttp:// [5] Electric Power Research Institute, Report to www.iberdrola.es/webibd/gc/prod/en/doc/ NIST on the Smart Grid Interoperability MAC_Spec_white_paper_1_0_080721.pdfϾ. Standards Roadmap, Contract No. SB1341-09- [16] E. Rosen and Y. Rekhter, “BGP/MPLS IP CN-0031—Deliverable 7, June 17, 2009, Virtual Private Networks (VPNs),” IETF RFC Ͻhttp://nist.gov/smartgrid/InterimSmartGridRo 4364, Feb. 2006, Ͻhttp://www.ietf.org/ admapNISTRestructure.pdfϾ. rfc/rfc4364.txtϾ.

DOI: 10.1002/bltj Bell Labs Technical Journal 225 [17] United States, Department of Commerce, JAYANT G. DESHPANDE is a member of technical staff National Institute of Standards and in the Network Performance and Reliability Technology, Office of the National Coordinator Department at Alcatel-Lucent Bell Labs in for Smart Grid Interoperability, NIST Special Murray Hill, New Jersey. He holds a B.E. Publication 1108, NIST Framework and degree from Nagpur University, India; Roadmap for Smart Grid Interoperability master’s degrees from the Indian Institute Standards, Release 1.0, Jan. 2010, of Technology, Kanpur, India, and Princeton University, Ͻhttp://www.nist.gov/public_affairs/ Princeton, New Jersey; and a Ph.D. from the University releases/smartgrid_interoperability_final.pdfϾ. of Texas at Austin. His professional interests are in [18] United States, Department of Energy, Energy smart grid architecture, design, and performance. He Information Administration, “International has spent the last 26 years at Alcatel-Lucent Bell Labs Energy , Total Electricity Net and AT&T Labs with a brief tenure at Cisco Systems. Consumption (Billion Kilowatthours),” He has worked on data and voice networking services Ͻhttp://tonto.eia.doe.gov/cfapps/ipdbproject/ development, network architecture, design, and QoS. iedindex3.cfm?tidϭ2&pidϭ2&aidϭ2&cidϭ&s Dr. Deshpande was a faculty member of yidϭ1980&eyidϭ2007&unitϭBKWHϾ. science and electrical engineering at the Indian [19] United States, Department of Energy, Energy Institute of Technology, New Delhi, India, from 1973 Information Administration, “Voluntary to 1982. After spending one year as a visiting faculty Reporting of Greenhouse Gases Program— member at Pennsylvania State University, he joined Bell Fuel and Energy Source Codes and Emission Labs in 1983. Coefficients,” Ͻhttp://www.eia.doe.gov/ oiaf/1605/coefficients.htmlϾ. [20] United States, Department of Energy, Federal TEWFIK L. DOUMI is a principal in the Network and Energy Regulatory Commission, 2007 Performance Reliability Department at Assessment of Demand Response and Alcatel-Lucent Bell Labs in Murray Hill, Advanced Metering, Staff Report, Sept. 2007, New Jersey. He holds a license in physics Ͻhttp://www.ferc.gov/legal/staff-reports/ from the University of Algiers, Algeria; an 09-07-demand-response.pdfϾ. M.S. degree in electrical engineering from [21] S. Ward, W. Higinbotham, E. Duvelson, and Stevens Institute of Technology, Hoboken, New A. Saciragic, “Inside the Cloud—Network Jersey; and a Ph.D. degree in electrical engineering Communications Basics for the Relay from the University of Bradford in England. Engineer,” Proc. 61st Annual Conf. for Dr. Doumi’s professional interests are in spectrum Protective Relay Engineers (CPRE ‘08) management and radio engineering techniques for (College Station, TX, 2008), pp. 273–303. next-generation wireless systems. He is a member of the Alcatel-Lucent Technical Academy and a senior (Manuscript approved March 2010) member of the IEEE.

KENNETH C. BUDKA is a senior director of the Network and Performance Reliability Department at MARK MADDEN is the regional vice president for Alcatel-Lucent Bell Labs in Murray Hill, Energy Markets in Alcatel-Lucent’s New Jersey. He received a B.S. degree Americas Region. He is responsible for (summa cum laude) in electrical engineering Alcatel-Lucent’s North American market from Union College in Schenectady, New strategy, strategic partnerships, and York, and M.S. and Ph.D. degrees in engineering science business development in the utility, oil, from Harvard University in Cambridge, Massachusetts. and gas markets. Mr. Madden joined Alcatel-Lucent in Dr. Budka’s professional interests are in the devel- 1996. He has over 25 years experience with leading opment of next-generation wireless and wireline companies in the information and communications communication technologies and their application to technologies industry and has been actively engaged mission-critical communications systems for public providing consulting to various customers within safety agencies and utilities. He is a senior member of the electric utility sector on mission-critical the IEEE and a former distinguished member of telecommunications technologies for the last technical staff at Bell Labs. He holds 18 U.S. patents five years.

226 Bell Labs Technical Journal DOI: 10.1002/bltj TIM MEW is a member of Alcatel-Lucent’s Global Services team, responsible for defining, developing, and managing complex systems integration services solutions for the energy and utilities sector. Prior to this, he was the head of the Solution Design and Innovation team in Australasia, focusing on railways, highways, oil and gas, and security services solutions. Mr. Mew’s generalist background has included architecture, technology planning, and service development disciplines in the technology areas of next-generation networks, VoIP, intelligent networks, PSTN, CTI, Internet and IP, CCTV, e-commerce, and wireless communications. Before joining Alcatel- Lucent, he held diverse roles ranging from architecture manager, senior engineer, to brand manager and CTO roles in a number of industries including carriers, ISPs, and e-commerce. ◆

DOI: 10.1002/bltj Bell Labs Technical Journal 227