Smart Grid: Challenges & Opportunities

INT8684_PECover2013_8.indd 1 4/4/13 5:11 PM You don’t have to be Thomas Edison to be a force for change. Just a PES member.

Joining the IEEE Power & Energy Society can provide a big boost to your career by enabling you to: • Tackle broad-reaching challenges • Become recognized as a thought leader by your industry peers • Develop contacts that will prove useful throughout all stages of your career • Be a part of the very active and engaged global PES Community

We help our members to be successful by providing: • Up-to-date information on current trends and the latest technology • Industry insight through Power & Energy magazine, technical reports and peer-reviewed publications • Compelling programs and networking opportunities at our conferences and events • Opportunity to meet, network and collaborate with local members via our vibrant chapters

Over 30,000 members of the IEEE Power & Energy Society recognize that their membership is an exceptional, cost-effective way to acquire the latest information about all aspects of the fast-changing electric power and energy industry. You can too, if you join us now!

To learn more about the IEEE Power & Energy Society, including the many other membership benefits, please visit www.ieee-pes.org.

IEEE Power & Energy Society 445 Hoes Lane Piscataway, NJ 08854 USA

IEEE_MemberAds.indd 1 4/1/13 3:53 PM magazine

www.ieee.org/power Smart Grid: Challenges & Opportunities ...a 2013 reprint journal from PES

on the cover features

5 Smart Grid — Safe, Secure, Self-Healing Challenges and Opportunities in Power System Security, Resiliency, and Privacy By S. Massoud Amin and Anthony M. Giacomonis

13 A Virtual Smart Grid Real-Time Simulation for Smart Grid Control and Communications Design By David Anderson, Chuanlin Zhao, Carl H. Hauser, Vaithianathan Venkatasubramanian, David E. Bakken, and Anjan Bose 22 Forward Pass 13 Policy Changes and Technical Opportunties on the U.S. Electric Grid By Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

30 DC, Come Home DC Microgrids and the Birth of the “Enernet” By Brian T. Patterson

41 Enhancing Grid Measurements Wide Area Measurement Systems, NASPInet, and Security By Rakesh B. Bobba, Jeff Dagle, Erich Heine, Himanshu Khurana, William H. Sanders, Peter Sauer, and Tim Yardley

49 Staying in Control Cybersecurity and the Modern Electric Grid By Julie Hull, Himanshu Khurana, Tom Markham, and Kevin Staggs 49 contents

ieee power & energy magazine 1 magazine

Editor in Chief IEEE Periodicals/Magazines Department Melvin I. Olken 445 Hoes Lane, Piscataway, NJ 08854 USA 245 East 19th Street #20K +1 732 562 3950, fax +1 732 981 1855 New York, NY 10003-2665 USA www.ieee.org/magazines +1 212 982 8286 (phone fax) Geraldine Krolin-Taylor, Senior Managing Editor [email protected] Janet Dudar, Senior Art Director Gail A. Schnitzer, Assistant Art Director Associate Editors Theresa L. Smith, Production Coordinator Gerald B. Sheblé, Business Scene Peter M. Tuohy, Production Director Felicia Spagnoli, Advertising Production Manager Carl L. Sulzberger, History Dawn Melley, Editorial Director Fran Zappulla, Staff Director, IEEE Publishing Operations Editorial Board IEEE prohibits discrimination, harassment, and bullying. For more information, S. Massoud Amin, L. Goel, A.P. Hanson, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html. N. Hatziargyriou, M.I. Henderson, S.H. Horowitz, P. Kundur, R. Masiello, IEEE Power & Energy Magazine K.M. Matsuda, A.P.S. Meliopoulos, IEEE Power & Energy Magazine (ISSN 1540-7977) (IPEMCF) is published bimonthly by the Institute of Electrical and M.I. Olken, M. O’Malley, A.G. Phadke, Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA. Responsibility for the R.J. Piwko, C.E. Root, H. Rudnick, contents rests upon the authors and not upon the IEEE, the Society, or its members. IEEE Operations Center (for orders, subscriptions, address changes): 445 Hoes Lane, Piscataway, NJ 08854 USA. Telephone: +1 732 981 0060, +1 800 678 4333. P.W. Sauer, M. Shahidehpour, Individual copies: IEEE members US$20.00 (first copy only), nonmembers US$77.00 per copy. Subscription Rates: B.R. Shperling, S.S. Venkata, Society members included with membership dues. Subscription rates available upon request. Copyright and reprint permis- B.F. Wollenberg sions: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 articles that carry a code at the bottom of the first page, Advertising provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. For other copying, reprint, or republication permission, write Barry LeCerf Copyrights and Permissions Department, IEEE Operations Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Copyright Bullseye International Group, Inc. © 2013 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New +1 913 663 1112, fax +1 913 663 1119 York, NY, and at additional mailing offices. Postmaster: Send address changes to IEEE Power & Energy Magazine, IEEE Operations Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188 [email protected] Printed in U.S.A. ieee power & energy society (pes) The IEEE Power & Energy Society is an organization of IEEE members whose principal interest is the advancement of the science and practice of electric power generation, transmission, distribution, and utilization. All members of the IEEE are eligible for membership in the Society. Mission Statement: To be the leading provider of scientific and engineering information on electric power and energy for the betterment of society, and the preferred professional development source for our members. Officers Standing Committee Chairs Technical Committee Chairs N.N. Schulz, President M. Crow, Awards & Recognition M. Sedlak, Electric Machinery M.M. Begovic, President-Elect N. Nair, Constitution & Bylaws R. Groves, Energy Development & Power M. Selak, Vice President, Chapters L. Bertling Tjernberg, Finance & Audit Generation J.H. Nelson, Vice President, Technical Activities A.C. Rotz, Nominations & Appointments J. Smith, Insulated Conductors P.W. Sauer, Vice President, Education K. Butler-Purry, Power Engineering Education G. Ballassi, Nuclear Power Engineering S. Rahman, Vice President, Publications W.K. Reder, Scholarship Plus S. Carneiro, Jr., Power System Analysis, W. Rosehart, Vice President, Meetings Computing, & Economics H. Louie, Vice President, Chapter Representatives D. Nordell, Power System Communications Membership & Image B. Allaf, J. Ammentorp, A. Bakirtzis, T. Van Cutsem, Power System R. Podmore, Vice President, New Initiatives/ J.G. Calderon, R. Cespiedes, C. Diamond, Dynamic Performance Outreach D. Drumtra, J. Fleeman, B. Gwyn, R. Arseneau, Power System Instrumentation L. Bertling Tjernberg, Treasurer Z.F. Hussien, I. Kuzle, N. Logic, J.C. Montero, & Measurements C. Root, Secretary P. Naidoo, T. Rajagopalan, P. Pabst, D. Sharafi, A. Conejo, Power System Operations A.C. Rotz, Past-President G.N. Taranto, E. Tobin, E. Uzunovic, M.L. Chan, Power System Planning D. van Hertem & Implementation IEEE Division VII Director R. Hedding, Power System Relaying C. Warren Chapter Committee Chairs L. Varga, Stationary Battery S. Chakravorti, Chapter Secretary M. Dood, Substations IEEE Division VII Director Elect N. Mariun, Chapter/Section Relations A.J. Surtees, Surge Protective Devices W.K. Reder M. Armstrong, Electronic Communications T.W. Olsen, Switchgear E. Carlsen, Awards & Resources W. Chiu, Transformers Region Representatives Y. Chen, Distinguished Lecturer Program W.A. Chisholm, Transmission & Distribution M. Chaganti, Y. Chen, T. Hiemer, F. Lambert, K. Hadzimahovic, Chapters Web site M. Nissen, J. Skillman, United States Coordinating Committees M. Armstrong, Canada Membership & Image B. Djokic, Emerging Technologies C. Vournas, Europe, Middle East, & Africa Committee Chairs S. Pullins, Intelligent Grid N. Segoshi, Latin America A. St Leger, GOLD Coordinator P. Bishop, Marine Systems L. Goel, Asia & Pacific A. Bonthron, Membership Development R.J. Piwko, Wind & Solar Power L. Fan, Membership Development Governing Board Open, Web Site Development Standing Committees Members-at-Large S. Bahramirad, PES WIE Liaison E. Gunther, M. Jensen, J. Giri, T. Prevost J. Randolph, Awards W. Bishop, Marketing S.S. Venkata, Technical Sessions PES Executive Director Technical Council S.S. Venkata, Meetings & Marketing Patrick Ryan, +1 732 465 6618, J.H. Nelson, Chair K. Edwards, Organization & Procedures fax +1 732 562 3881, e-mail [email protected] S.S. Venkata, Vice-Chair W. Bartley, Standards Coordination K.S. Edwards, Secretary D. Novosel, Past-Chair Additional Positions Digital Object Identifier 10.1109/MPE.2013.2239531 M. Maytum, Web Master D. Nordell, Editor in Chief of Conference Papers

2 ieee power & energy magazine Noel Schulz, President, IEEE PES

greetings from the IEEE Power & Energy leader’s corner leader’s Society (PES) March 15, 2013 To: Recipients of the 2013 IEEE PES Smart Grid Reprint Journal:

WELCOME TO OUR NEW COMPEN- Patricia Hoffman, assistant secretary ✔✔ cyber-attack vectors and vulner- dium of articles that appeared in last at the U.S. DOE in the office of Elec- abilities and the value of simula- year’s IEEE P&E Magazine. tricity Delivery and Energy Reliability, tion for developing solutions, If the past is prologue to the future, delivered a timely keynote on “Grid ✔✔ the fundamental value of obtain- you’ll find an array of current issues Modernization and Resiliency,” which ing accurate data on the grid’s treated in depth here in this useful digest. articulated a theme that underlies much operational state, made possible In fact, looking over past reprint issues, of our work and is reflected by the col- by wide area measurement sys- it’s clear that a handful of fundamental lection of articles you hold in your hands. tems, or WAMS, Wthemes continue to challenge the IEEE No nation is without power-related chal- ✔✔ and more on cyber security and PES as well as the power industry at large. lenges in supporting its economic vitality communication networks. Yet as our thinking advances, we’re see- and security and our international mem- Of course, applying our knowledge, ing these issues more clearly, discovering bership guarantees that we maintain a pursuing discoveries and sharing insights fresh approaches and our collective work broad perspective on these issues. is what makes IEEE PES a world-class at meeting these challenges continues. Together we’re working toward solu- hub for innovation. So read this digest, I saw many colleagues at the Febru- tions and sharing our findings across the share it with colleagues, explain the ideas ary 2013 IEEE PES Innovative Smart world. By solving our shared electricity- to family and friends (it’s a good practice, Grid Technologies (ISGT) North Amer- related challenges and sharing the fruits they’re energy consumers, too). Share ica conference in Washington, D.C. and of our work that we can advance grid your work by writing an article for IEEE I look forward to seeing many more modernization for every member of the P&E Magazine, visit the IEEE Smart of you at our upcoming PES meetings international community. Grid Portal (http://smart-grid.ieee.org/), across the globe including our ISGT That brings me to the content of this keep up with our IEEE PES and smart conference suite of meetings in Sao new compendium. You’ll glean the details grid tweets (@ieee_pes and @ieeesmart- Paulo, April 15-17, and Copenhagen, from the Table of Contents, but we’ve grid) and join IEEE Smart Grid LinkedIn Oct. 6-9. ISGT conference programs packaged here articles on the array of discussion group. reflect many of the issues presented here issues that remain relevant year after year. The sustainable, efficient provision and will undoubtedly continue to drive We bring you up-to-date explorations of: and wise use of electricity can pave the discussion, research and efforts to reach ✔✔ the fundamental drivers, chal- way for more productive and sustain- solutions in 2013. lenges and probable solutions of able societies. I invite you to partake I’d like to remind all of our members and for smart, modernized grids, of this volume’s thoughtful articles and that these conferences offer outstanding ✔✔ federal and state regulatory advance the search for solutions. networking opportunities to engage with issues that bear scrutiny and need your colleagues and broaden your hori- carefully considered changes, Sincerely, zons. The ISGT conference in Washing- ✔✔ the rebirth of interest in DC power ton, DC involved participants from over and how its efficiencies and other 30 countries. I had the opportunity to characteristics will make it an excit- recap PES-specific priorities, including ing area for innovative solutions, our progress on standards, and our chal- ✔✔ cost-effective, software-based lenge in persuading students to join not simulation models that save Noel N. Schulz only the power field but our Society as the time and cost of laboratory IEEE PES President, 2012-2013 well. I also announced a new PES publi- or real-world trial and error in [email protected] cation, Electrification. advancing power systems, http://www.ieee-pes.org/

ieee power & energy magazine 3 IEEE Transactions on Smart Grid

The IEEE Transactions on Smart Grid is intended to be a cross disciplinary and internationally archival journal aimed at disseminating the results of research on smart grid that relates to energy generation, transmission, distribution and delivery. The journal will publish original research on theories, technologies, design, policies, and implementation of smart grid. The Transactions will welcome manuscripts on design, implementation and evaluation of energy systems that include smart grid technologies and applications. Surveys of existing work on smart grid may also be considered for publication when they propose a challenging perspective on the future of such technologies and systems. Topical issues considered by the Transactions include:

> Smart sensing, communication and control in energy systems > Wireless communications and advanced metering infrastructure > Smart grid for energy management in buildings and home automation > Phasor measurement unit applications for smart grid > Smart grid for plug-in vehicles and low-carbon transportation alternatives > Smart grid for cyber and physical security systems > Smart grid for distributed energy resources > Smart grid for energy savings and ﬁ nancial management > Smart grid in interdependent energy infrastructures > Smart grid for intelligent monitoring and outage management

If you are interested in reviewing papers for this journal, please sign up as a reviewer on the Manuscript Central site at: http://mc.manuscriptcentral.com/pes-ieee.

The Transactions on the Smart Grid can be accessed via the drop down menu on the PES portal site. If you are interested in reviewing papers for our new Transactions and you are currently a reviewer for PES Transactions, you can access your account in Manuscript Central and add smart grid to your keywords or areas of expertise. If you have an account in Manuscript Central and are not currently a reviewer for PES Transactions and would like to become a reviewer for PES Transactions, access your account and you will automatically be given a reviewer center, then update your areas of expertise. If you do not have an account, create a new user account and complete all the required ﬁ elds, you will then be given an author center and a reviewer center.

About the Editor-in-Chief: If you are interested in participating in the publication activities, please contact the Editor-in-Chief, Dr. Mohammad Shahidehpour at: [email protected]. Prof. Shahidehpour (Fellow ‘01) has been afﬁ liated with IEEE for the last thirty years. His is currently the Carl Bodine Distinguished Professor of Electrical and Computer Engineering at Illinois Institute of Technology. Dr. Shahidehpour is an IEEE Distinguished Lecturer who has lectured in 30 countries on issues related to power system operation and control. He has served as the Vice President of Publications for the IEEE Power & Energy Society and an Editor of the Transactions on Power Systems. Reprinted from January/February 2012 issue of IEEE Power & Energy magazine Smart Grid— Safe, Secure, Self-Healing Challenges and Opportunities in Power System Security, Resiliency, and Privacy

THE EXISTING POWER DELIVERY system is vulnerable to both natural disasters and intentional attack. A successful terrorist attempt to disrupt the power delivery system could have adverse effects on national security, the economy, and the lives of every citizen. Secure and reliable operation of the electric system is fundamental to national and international economic systems, security, and quality of life. TThis is not new: both the importance and the difﬁ culty of protecting power systems have long been recognized. In 1990, the U.S. Ofﬁ ce of Technology Assess- ment (OTA) issued a detailed report, Physical Vulnerability of the Electric System to Natural Disasters and Sabo- tage. The report concluded: “Terrorists © BRAND X PICTURES could emulate acts of sabotage in several other countries and destroy critical [power system] components, incapacitating large

By S. Massoud Amin and Anthony M. Giacomoni

Digital Object Identiﬁ er 10.1109/MPE.2011.943112 Date of publication: 13 December 2011

january/february 2012 1540-7977/11/$31.00©20121540-7977/12/$31.00©2012 IEEE ieeeIEEE power & energy magazine 335 segments of a transmission network for months. Some of less telecommunications infrastructure. Transportation sys- enormous investment, including more than 15,000 genera- extremely diffi cult, even for a large, well-organized group these components are vulnerable to saboteurs with explo- tems, including military and commercial aircraft and land tors in 10,000 power plants and hundreds of thousands of of terrorists. sives or just high-powered rifl es.” The report also docu- and sea vessels, depend on communication and energy net- miles of transmission and distribution lines. With dimin- Data on terrorist attacks on the world’s electricity sec- mented the potential costs of widespread outages, estimat- works. Links between the power grid and telecommunica- ished transmission and generation capacity and with dra- tor from 1994–2004 from the Oklahoma-based Memorial ing them to be in the range of US$1 to US$5 per kWh of tions systems as well as between electrical power lines and matic increases in interregional bulk power transfers and Institute for the Prevention of Terrorism show that transmis- disrupted service, depending on the length of the outage, the oil, water, and gas pipelines continue to be the lynchpins of the diversity of transactions, the electric power grid is being sion systems are by far the most common target in terms types of customers affected, and a variety of other factors. energy supply networks. This strong interdependence means used in ways for which it was not originally designed. Grid of the total number of physical attacks. Figure 2 shows the In the New York City blackout of 1977, for example, dam- that an action in one part of an infrastructure network can congestion and atypical power fl ows have been increasing percentage of terrorist attacks aimed at each of the major age from looting and arson alone totaled about US$155 mil- rapidly create global effects by cascading throughout the during the last 25 years, while customer expectations of reli- grid components. lion—roughly half of its total cost. same network and even into other networks. ability and cyber and physical security are rising to meet the One possible means of increasing the physical security During the 20 years since the OTA report, the situation In the aftermath of the tragic events of 11 September 2001 needs of a pervasively digital world. of power lines is to bury them. A 2006 study by the Edison has become even more complex. Accounting for all criti- and recent natural disasters and major power outages, there Upgrading the control and communication systems for Electric Institute (EEI) calculated that putting power lines cal assets includes thousands of transformers, line reactors, have been increased national and international concerns the power grid will present many new security challenges underground would cost about US$1 million per mile, com- series capacitors, and transmission lines. Protecting all these expressed about the security, resilience, and robustness of that must be dealt with before extensive deployment and pared with US$100,000 per mile for overhead lines, making diverse and widely dispersed assets is impractical. Moreover, critical infrastructures in response to an evolving spectrum implementation of smart grid technologies can begin. The the idea fi nancially infeasible. cyber, communication, and control layers add new benefi ts of threats. There is reasonable concern that national and digitization of such systems may enable remote attacks to only if they are designed correctly and securely. international energy and information infrastructures have grow rapidly, potentially spanning countries or even con- Cyber Challenges reached a level of complexity and interconnection that makes tinents. Moreover, the number of threats against computer The number of documented cyberattacks and intrusions Electricity Infrastructure: Increasing them particularly vulnerable to cascading outages, whether systems is rapidly increasing due to the increased availabil- worldwide has been rising very rapidly in recent years. The Interdependencies initiated by material failure, natural calamities, intentional ity of highly sophisticated hacker tools on the Internet and results of a 2007 McAfee survey highlight the pervasiveness Energy, telecommunications, transportation, and fi nancial attack, or human error. The potential ramifi cations of net- the decrease in technical knowledge required to use them to of such attacks. For example, Figure 3 shows the percent- infrastructures are becoming increasingly interconnected, work failures have never been greater, as the transportation, cause damage. While the digitization of such systems will age of IT and security executives from critical infrastructure thus posing new challenges for their secure, reliable, and telecommunications, oil and gas, banking and fi nance, and present many new security challenges, it will also provide enterprises located in 14 countries around the world report- effi cient operation. All of these infrastructures are complex other infrastructures depend on the continental power grid the grid with increased fl exibility to prevent and withstand ing large-scale distributed denial-of-service (DDoS) attacks networks—geographically dispersed, nonlinear, and inter- to energize and control their operations. Despite some simi- potential threats. and their frequency. acting both among themselves and with their human owners, larities, the electric power grid is quite different from gas, DDoS attacks utilize networks of infected computers— operators, and users (see Figure 1). oil, and water networks: phase shifters rather than valves Key Smart Grid Security Challenges whose owners often do not even know that they have been Virtually every crucial economic and social function are used, and there is no way to store signifi cant amounts infected—to overwhelm target networks with millions of depends on the secure and reliable operation of these infra- of electricity. Providing the desired fl ow on one line often Physical Challenges fake requests for information over the Internet. structures. Indeed, they have provided much of the high results in “loop fl ows” on several other lines. The size and complexity of the North American electric Due to the increasingly sophisticated nature and speed standard of living that the more developed countries enjoy. power grid makes it impossible both fi nancially and logis- of malicious code, intrusions, and DoS attacks, human With increased benefi t, however, has come increased risk. Potential Route Ahead: A Smarter Grid tically to physically protect the entire infrastructure. There responses may be inadequate. Figure 4 shows the evolution As these infrastructures have grown more complex in order The key challenge is to enable secure and very high-confi - currently exist more than 450,000 mi of 100-kV or higher of cyberthreats over the last two decades and the types of to handle increasing demands, they have become increas- dence sensing, communications, and control of a heteroge- transmission lines and many more thousands of miles of responses that can be used to combat them effectively. ingly interdependent. The Internet, computer networks, and neous, widely dispersed, yet globally interconnected system. lower-voltage lines. As an increasing amount of electric- In addition, adversaries often have the potential to ini- our digital economy have all increased the demand for reli- It is even more complex and diffi cult to control it for optimal ity is generated from distributed renewable sources, the tiate attacks from nearly any location in the world. A July able and disturbance-free electricity; banking and fi nance effi ciency and maximum benefi t to the ultimate consumers problem will only be exacerbated; the U.S. Department of 2010 article in The Economist quoted one senior American depend on the robustness of electric power, cable, and wire- while still allowing all its business components to compete Energy (DOE) has concluded that generating 20% of all military source as saying, “If any country were found to be fairly and freely. electricity with land-based wind installations will require planting logic bombs on the grid, it would provoke the equiv- To achieve this goal, a new “megainfrastructure” is at least 20,000 square miles. Thus it is probable that a alent of the Cuban missile crisis.” Furthermore, currently emerging from the convergence of energy, telecommunica- well-organized, determined group of terrorists could take Excellent Power tions, transportation, the Internet, and electronic commerce. out portions of the grid as they have previously done in System Reliability In the electric power industry and other critical infrastruc- the United States, Colombia, and other locations around 13% Exceptional Power A Secure tures, new ways are being sought to improve network effi - the globe. Several such incidents in the United States have Quality Energy ciency by eliminating congestion problems without seriously been publicly reported during the last 30 years, includ- 14% Infrastructure Integrated diminishing reliability and security. Nevertheless, the goal ing saboteurs operating in the Pacifi c Northwest and those Communications of transforming the current infrastructures into self-healing using power lines and transformers for target practice on 11% 62% Compatible Devices energy delivery, computer, and communications networks the East Coast. Colombia, for example, has faced up to and Appliances with unprecedented robustness, reliability, effi ciency, and 200 terrorist attacks per year on its transmission infra- quality for customers and our society is ambitious. structure over the last 11 years, as reported in a recent This challenge is further complicated by the fact that IEEE Power & Energy Magazine article by Corredor and Generation Transmission the North American electric power grid may be considered Ruiz. Such attacks, although troublesome and costly to Substations All Others as the largest and most complex machine in the world: its the local region, affect only a small portion of the over- figure 1. A complex set of interconnected webs (source: transmission lines connect all the electric generation and all grid, however. To cause physical damage equivalent figure 2. Electric terrorism: grid component targets, EPRI, 2002–present). distribution on the continent. This network represents an to that from a small to moderate-size tornado would be 1994–2004 (source: Journal of Energy Security).

346 IEEEieee power & energy magazine january/february 2012 january/february 2012 IEEE power & energy magazine 35 enormous investment, including more than 15,000 genera- extremely diffi cult, even for a large, well-organized group tors in 10,000 power plants and hundreds of thousands of of terrorists. miles of transmission and distribution lines. With dimin- Data on terrorist attacks on the world’s electricity sec- ished transmission and generation capacity and with dra- tor from 1994–2004 from the Oklahoma-based Memorial matic increases in interregional bulk power transfers and Institute for the Prevention of Terrorism show that transmis- the diversity of transactions, the electric power grid is being sion systems are by far the most common target in terms used in ways for which it was not originally designed. Grid of the total number of physical attacks. Figure 2 shows the congestion and atypical power fl ows have been increasing percentage of terrorist attacks aimed at each of the major during the last 25 years, while customer expectations of reli- grid components. ability and cyber and physical security are rising to meet the One possible means of increasing the physical security needs of a pervasively digital world. of power lines is to bury them. A 2006 study by the Edison Upgrading the control and communication systems for Electric Institute (EEI) calculated that putting power lines the power grid will present many new security challenges underground would cost about US$1 million per mile, com- that must be dealt with before extensive deployment and pared with US$100,000 per mile for overhead lines, making implementation of smart grid technologies can begin. The the idea fi nancially infeasible. digitization of such systems may enable remote attacks to grow rapidly, potentially spanning countries or even con- Cyber Challenges tinents. Moreover, the number of threats against computer The number of documented cyberattacks and intrusions systems is rapidly increasing due to the increased availabil- worldwide has been rising very rapidly in recent years. The ity of highly sophisticated hacker tools on the Internet and results of a 2007 McAfee survey highlight the pervasiveness the decrease in technical knowledge required to use them to of such attacks. For example, Figure 3 shows the percent- cause damage. While the digitization of such systems will age of IT and security executives from critical infrastructure present many new security challenges, it will also provide enterprises located in 14 countries around the world report- the grid with increased fl exibility to prevent and withstand ing large-scale distributed denial-of-service (DDoS) attacks potential threats. and their frequency. DDoS attacks utilize networks of infected computers— Key Smart Grid Security Challenges whose owners often do not even know that they have been infected—to overwhelm target networks with millions of Physical Challenges fake requests for information over the Internet. The size and complexity of the North American electric Due to the increasingly sophisticated nature and speed power grid makes it impossible both fi nancially and logis- of malicious code, intrusions, and DoS attacks, human tically to physically protect the entire infrastructure. There responses may be inadequate. Figure 4 shows the evolution currently exist more than 450,000 mi of 100-kV or higher of cyberthreats over the last two decades and the types of transmission lines and many more thousands of miles of responses that can be used to combat them effectively. lower-voltage lines. As an increasing amount of electric- In addition, adversaries often have the potential to ini- ity is generated from distributed renewable sources, the tiate attacks from nearly any location in the world. A July problem will only be exacerbated; the U.S. Department of 2010 article in The Economist quoted one senior American Energy (DOE) has concluded that generating 20% of all military source as saying, “If any country were found to be electricity with land-based wind installations will require planting logic bombs on the grid, it would provoke the equiv- at least 20,000 square miles. Thus it is probable that a alent of the Cuban missile crisis.” Furthermore, currently well-organized, determined group of terrorists could take out portions of the grid as they have previously done in the United States, Colombia, and other locations around 13% the globe. Several such incidents in the United States have been publicly reported during the last 30 years, includ- 14% ing saboteurs operating in the Pacifi c Northwest and those using power lines and transformers for target practice on 11% 62% the East Coast. Colombia, for example, has faced up to 200 terrorist attacks per year on its transmission infrastructure over the last 11 years, as reported in a recent IEEE Power & Energy Magazine article by Corredor and Generation Transmission Ruiz. Such attacks, although troublesome and costly to Substations All Others the local region, affect only a small portion of the overall grid, however. To cause physical damage equivalent figure 2. Electric terrorism: grid component targets, to that from a small to moderate-size tornado would be 1994–2004 (source: Journal of Energy Security). january/february 2012 IEEEieee power & energy magazine 357 100 Multiple Occurrences Every Day Less Than Monthly Occurrences 80 Multiple Occurrences Every Week Less Than Annual Occurrences Multiple Occurrences Every Month 60

40 Percentage

20 Italy India Total Brazil Spain China Japan United Russia France Mexico Australia Kingdom Germany Middle East United States Saudi Arabia/

figure 3. Percentage of critical infrastructure enterprise executives reporting large-scale DDoS attacks and their frequency (source: McAfee).

more than 90% of successful cyberattacks take advantage of and lower costs) but without adding the technology needed known vulnerabilities and misconfi gured operating systems, to make them secure. Moreover, numerous types of commu- servers, and network devices. nication media and protocols are used in the communication The security of cyber and communication networks is and control of power systems. Within a substation control fundamental to the reliable operation of the grid. As power network, it is common to fi nd commercial telephone lines as systems rely more heavily on computerized communications well as wireless, microwave, optical fi ber, and Internet con- and control, system security has become increasingly depen- nections. The diversity and lack of interoperability among dent on protecting the integrity of the associated informa- the various communication protocols cause problems for tion systems. Part of the problem is that the existing control anyone who tries to establish secure communication to and systems, which were originally designed for use with propri- from a substation. etary, stand-alone communication networks, were later con- Electric power utilities also typically own and operate nected to the Internet (because of its productivity advantages at least certain portions of their own telecommunications

Class III Human Response: Impossible Flash Threats Automated Response: Unlikely Seconds Proactive Blocking: Possible

Class II Warhol Threats Human Response: Difficult/Impossible Automated Response: Possible

Blended Threats Class I Human Response: Possible E-Mail Threats Days Contagion Time Frame Macro Viruses

File Viruses Months

Early 1990s Mid 1990s Late 1990s 2000 2003 Time

figure 4. Cyberthreat evolution (source: EPRI).

368 IEEEieee power & energy magazine january/february 2012 systems, which often consist of a backbone of fi ber optic or patch only after SCADA vendors thoroughly test and vali- microwave links connecting major substations with spurs to date it, and this sometimes causes deployment to be delayed smaller sites. Increased use of electronic automation raises by several months. signifi cant issues regarding the adequacy of operational As a result, cybersecurity is just as important as physical security, if security provisions are not built in. security, if not more so. Due to the gravity of these threats, More specifi cally, the operation of a modern power sys- the Federal Energy Regulatory Commission (FERC) pol- tem depends on complex systems of sensors and automated icy statement on the smart grid states that cybersecurity and manual controls, all of which are tied together through is essential to the operation of the smart grid and that the communication systems. While the direct physical destruc- development of cybersecurity standards is a key priority. tion of generators, substations, or power lines may be the The DOE has also stated that the ability to resist attack by most obvious strategy for causing blackouts, activities that identifying and responding to disruptions caused by sabo- compromise the operation of sensors, communications, and tage is one of the smart grid’s seven crucial functions. Much control systems by spoofi ng, jamming, or sending improper work remains to be done, however, to create standards that, commands could also disrupt the system, cause blackouts, when implemented, will adequately protect the grid from and in some cases result in physical damage to key system cyberattacks. Emerging standards fall well short of achiev- components. ing this ultimate goal. Any telecommunication link that is even partially outside the control of the organization that owns and operates power Smart Grid Security Needs plants, supervisory control and data acquisition (SCADA) systems, or energy management systems (EMSs) represents Layered Security a potentially insecure pathway into the business operations In order to protect electric infrastructure from the threats of the company as well as a threat to the grid itself. The outlined above, several layers of security are needed to interdependency analyses done by most companies in the minimize disruptions to system operations. Layered secu- last 12–14 years (starting with the preparations for Y2K and rity (or “defense in depth”) involves strategically combining continuing after the tragic events of 9/11) have identifi ed multiple security technologies at each layer of a computing these links and the system’s vulnerability to their failure. system in order to reduce the risk of unauthorized access They therefore provide an excellent reference point for an due to the failure of any single security technology. It expo- analysis of cybervulnerability. nentially increases the cost and diffi culty of compromising While some of the operations on the system are automatic, a system by creating a much stronger defense than the use human operators in system control centers ultimately make of any individual component alone, thus reducing the likeli- the decisions and take the actions that control the operations hood of an attack. of the system. In addition to the physical threats to such cen- The trend of connecting electrical control systems to the ters and the communication links that fl ow in and out of them, Internet exposes all layers of a system to possible attack. one must be concerned about two other factors: the reliabil- Computing layers that must be considered include ity of the operators within the centers and the possibility that ✔ personnel insecure code has been added to a program in a center com- ✔ networks puter. The threats posed by “insiders” are real, as is the risk ✔ operating systems of a “Trojan horse” embedded in the software of one of more ✔ applications of the control centers. A 2008 survey by the Computer Secu- ✔ databases. rity Institute and the U.S. Federal Bureau of Investigation of The security features to be employed at each layer include data compiled from 522 computer security practitioners and examination, detection, prevention, and encryption. To pro- senior executives of U.S. corporations, government agencies, tect control systems, well-established information security fi nancial and medical institutions, and universities reported practices must also be utilized. that within a 12-month period, 59% of the respondents experienced an attack from a virus, 29% reported unauthorized use Deception of computer services, and 44% reported insider abuse. An additional defense mechanism is the use of deception. The threat of a “Trojan horse” embedded in the control Deception consists of two possible techniques: dissimulation center software can only be addressed by means of careful (hiding the real) and simulation (showing the false). McQueen security measures within the commercial fi rms that develop and Boyer describe several potential dissimulation and simu- and supply this software along with careful security screen- lation techniques that can be used for control systems. Three ing of the utility and outside service personnel who perform of the dissimulation techniques described are: software maintenance within the centers. Today, security ✔ masking the real by making a relevant object unde- patches often are not supplied to end users, or users are not tectable or blending it into background irrelevance applying the patches, as they fear they will affect system ✔ repackaging, which hides the real by making a rel- performance. Current practice is to apply an upgrade or evant object appear to be something it isn’t january/february 2012 IEEEieee power & energy magazine 379 Upgrading the control and communication systems for the power grid will present many new security challenges that must be dealt with.

✔ dazzling, which hides the real by making the identi- ✔ disrupting the load balance of local systems by sud- fi cation of a relevant object less certain by confusing denly increasing or decreasing the demand for power the adversary about its true nature. ✔ gaining control of millions of meters and simultane- Likewise, three of the simulation techniques described ously shutting them down are: ✔ sending false control signals ✔ inventing the false by creating a perception that a rel- ✔ disabling grid control center computer systems and evant object exists when it doesn’t monitors ✔ mimicking, which invents the false by presenting char- ✔ disabling protective relays. acteristics of an actual and relevant object As more utilities move toward using Internet Protocol ✔ decoying, which displays the false so as to attract at- (IP)–based systems for wide area communications and as tention away from a more relevant object. the trend of using standardized protocols continues through- Deception will need to play a key role in smart grid out the industry, maintaining the security of such devices defense mechanisms. Since existing control system archi- will be critical. AMI introduces serious privacy concerns, as tectures are not random and therefore response characteris- immense amounts of energy use information will be stored tics are reproducible, the strength of potential adversaries is at the meter. Breaches into this data could expose customer amplifi ed. Defense mechanisms using deception can greatly habits and behaviors. Such arguments have led to the recent increase the diffi culty of planning and conducting successful moratoriums on AMI installations in numerous northern attacks on a system by portraying control system response California communities and other areas throughout the characteristics as random to attackers. They can also alert country. As a result, several key privacy concerns need to be operators to possible threats before any systems are harmed. addressed, including those outlined by the Cyber Security Additional security needs include rapid containment, Working Group of the U.S. National Institute of Standards restoration, and recovery strategies for times when systems and Technology (NIST). These include: are inevitably compromised. Either software patching or the ✔ Personal profi ling: using personal energy data to ability to rapidly identify and isolate the exploited systems determine consumer energy behavioral patterns for must be enabled in order to minimize downtime. This is commercial purposes extremely important, since the consequences of an attack ✔ Real-time remote surveillance: using live energy are directly proportional to the length of time the service is data to determine whether people are in a specifi c fa- disrupted. cility or residence and what they are doing ✔ Identity theft and home invasions: protecting per- Advanced Metering Infrastructure sonal energy data from criminals who could use the information to harm consumers Vulnerabilities ✔ Activity censorship: preventing the use of energy for The implementation of advanced metering infrastructure certain activities or taxing those activities at a higher (AMI) is widely seen as one of the fi rst steps in the digi- rate tization of the electric grid’s control systems. Despite the ✔ Decisions based on inaccurate data: shutting off increase in the utilization of AMI, there has been very lit- power to life-sustaining electrical devices or provid- tle assessment or R&D effort to identify the security needs ing inaccurate information to government and credit- for such systems. Smart meters, however, are extremely reporting agencies. attractive targets for exploitation, since vulnerabilities can In addition, AMI systems will need to be defended be easily monetized through manipulated energy costs against more traditional cyberthreats such as mobile and and measurement readings. Currently, in the United States malicious code, DoS attacks, misuse and malicious insider alone it is estimated that US$6 billion is lost by electricity threats, accidental faults introduced by human error, and the providers to consumer fraud in the electric grid. Possible problems associated with software and hardware aging. threats to the electrical grid introduced by the use of AMI include: Security Needs ✔ fabricating generated energy meter readings In order to defend against the vulnerabilities described ✔ manipulating energy costs above, several security features need to be incorporated into

3810 IEEEieee power & energy magazine january/february 2012 the development of AMI, along with new privacy laws to extraordinary efforts from many, including alumni of the protect consumers. Current privacy laws in the United States University of Minnesota’s infrastructure systems engineer- are fragmented and vague and do not specifi cally address ing program. They incorporated a sensor network into the consumer energy usage. Data stored at the meter and trans- new I-35W bridge (at less than 0.5% of total cost) that pro- mitted over communication networks must also meet stan- vides full situational awareness of stressors, fatigue, mate- dard cybersecurity requirements, including confi dentiality, rial, and chemical changes, so as to measure and understand integrity, availability, and nonrepudiation. the precursors to failure and to enable proactive and a priori One security feature alone, such as encryption, will not corrective actions. be able to cover all the possible security threats. Since it is Analogously, customized and cost-effective advance- imperative that the industry maintain 100% uptime, both the ments are both possible and essential to enable smarter and physical security of the AMI system hardware and multiple more secure electric power infrastructures. For example, standard IT security features like encryption and authenti- advanced technology now under development or under con- cation must be provided for. Furthermore, since it will be sideration holds the promise of meeting the electricity needs impossible to protect against all threats, smart meters must of a robust digital economy. The end vision of the smart be able to detect even the most subtle unauthorized changes grid consists of a highly developed electrical platform that and precursors to tampering or intrusion. Additional consid- engages consumers, enhances effi ciency, ensures reliability, eration must also be given to the cost and impact the secu- and enables integration of renewable energy and electric rity features will have on AMI system operations. Smart transportation. meters will need to be cost-effective, since millions will One key money- and power-saving element of the smart need to be purchased and installed to replace antiquated grid is its ability to measure how and when consumers use analog devices. And they must also be robust as they will be the most power. This information allows consumers to be deployed in very insecure locations. charged variable rates for energy, based upon supply and demand. This variable rate will incentivize consumers to Current Security Initiatives shift their heavy use of electricity to times of the day when Since the terrorist attacks of 11 September 2001, several steps demand is low. have been taken and initiatives accomplished to enhance the The total cost of a stronger transmission system would security and reliability of the nation’s current electricity be about US$82 billion over the next decade. Additionally, infrastructure. These include the Complex Interactive Net- to create a smarter end-to-end power delivery system, we works/Systems Initiative (CIN/SI), a joint program spon- must invest between US$17 and US$24 billion over the next sored by the Electric Power Research Institute (EPRI) and 20 years. the U.S. Department of Defense (DOD); EPRI’s Enterprise Investment in a smart grid would nearly pay for Information Security (EIS) program; EPRI’s post–9/11 itself by reducing stupendous outage costs, a savings Infrastructure Security Initiative (ISI); and various North of US$49 billion per year, and improving energy effi- American Electric Reliability Corporation (NERC) initia- ciency, a savings of US$20.4 billion per year. Likewise, tives, such as its information sharing and analysis centers through smart grid-enhanced energy efficiency, by 2030 (ISACs), public key infrastructure (PKI), and spare equip- carbon dioxide emissions from the electric sector would ment database. Information security frameworks for electric be reduced by 58%. power utilities have also been developed by the International Americans should not accept or learn to cope with Council on Large Electric Systems (CIGRE). A security increasing blackouts, nor should we rest on the notion that framework is considered as the skeleton on which various the technical know-how, political will, or money to bring elements are integrated for the appropriate management of our power grid up to 21st century standards do not exist. security risk. The various elements considered by CIGRE The truth is that, as a nation, we must and absolutely can include security domains, baseline controls, and security meet the power needs of a pervasively digital society if processes. the United States wishes to maintain its role as a global economic and political leader. The best of American inno- Research and Development Needs vation is yet to come, and the smart grid must be part of our future. The potential exists to create an electricity The Smart Infrastructure: system that provides the same effi ciency, precision, and A Smarter, More Secure I-35W Bridge interconnectivity as the billions of microprocessors that Within less than a year after the August 2007 collapse of it will power. the I-35W bridge in Minneapolis, Minnesota, a city of sorts From a strategic viewpoint, long-term developments and on the south side of the former bridge took shape, complete research issues relating to the defense of cyber and physical with a host of heavy-duty equipment pieces, temporary interdependent infrastructure networks must also be con- on-site areas for casting and other tasks, and crews considered. The driving scientifi c motivation is to further our stantly at work. The days and months that followed required understanding of adaptive self-healing and self-organizing january/february 2012 IEEEieee power & energy magazine 3911 Reprinted from January/February 2012 issue of IEEE Power & Energy magazine

mechanisms that can be applied to the development of about such reports is mainly one portion of an early article: secure, resilient, and robust overlaid and integrated energy, “The response to the alert was mixed. An audit of 30 util- power, sensing, communication, and control networks. ity companies that received the alert showed that only seven In addition to the above, further research and develop- were in full compliance, although all of the audited com- ment needs include the following areas: panies had taken some precautions.” This is the reality that 1) Enabling technologies for an end-to-end secure needs to be addressed. system of sensing and measurement, leading to im- Finally, no matter how many layers of security or how much proved analysis and visualization and eventually to sophistication is used in defense mechanisms, it is essential automation and self-healing systems: that the industry hire qualifi ed people. Research fi ndings sug- • monitoring and analysis, automation and control, gest that human and organizational factors do affect computer materials science, power electronics, and integrated and information security performance in a multilayered fash- distributed energy resources (DERs) ion. Often vulnerabilities are not the result of a single mistake • sensing, communication, data management, and or confi guration error but of numerous latent organizational mathematical and theoretical foundations to support conditions, such as management support and decisions made a better, faster, and higher-confi dence understanding by designers that combine to create scenarios in which fail- of what is going on, leading to improved state and ures and weaknesses may occur. In many complex networks, topology estimation and fast look-ahead simulation. the human participants themselves are both the most suscep- 2) Enabling a stronger and smarter grid by means of tible to failure and the most adaptable in the management of complex dynamical systems, systems science, con- recovery. Thus, staff members must be well trained to respond trols, and applied mathematics: to a wide variety of emergencies since no amount of technol- • modeling, robust control, dynamic interaction in in- ogy can replace well-trained personnel. terdependent layered networks, disturbance propa- gation in networks, and forecasting and handling For Further Reading uncertainty and risk J. Clemente, “The security vulnerabilities of smart grid,” J. • overall systems science and dynamics (including in- Energy Security, June 2009. frastructure, ecology and environment, markets, and P. H. Corredor and M. E. Ruiz, “Against all odds,” data-driven policy designs). IEEE Power Energy Mag., vol. 9, no. 2, pp. 59–66, Mar./ 3) Strategic R&D: Apr. 2011. • digital control of the energy infrastructure G. N. Ericsson, “Information security for electric power • integrated energy, information, and communica- utilities (EPUs)-CIGRE developments on frameworks, risk tions for the end user assessment, and technology,” IEEE Trans. Power Delivery, • transformation of the meter into a secure, two-way vol. 24, no. 3, pp. 1174–1181, July 2009. energy and information portal P. McDaniel and S. McLaughlin, “Security and privacy • robust advanced power generation portfolio. challenges in the smart grid,” IEEE Security Privacy, vol. 7, Awareness, education, and pragmatic tool development no. 3, pp. 75–77, May/June 2009. in this vital area continue to remain challenges. Educating M. A. McQueen and W. F. Boyer, “Deception used for stakeholders and colleagues about the cyber and physical cyber defense of control systems,” in Proc. 2nd Conf. Hu- interdependencies has often been diffi cult, as those who are man System Interactions, Catania, Italy, 2009, pp. 624–631. distinguished members of the community and understand NIST, “Guidelines for smart grid cyber security,” The power systems well but are less aware of their cybervulner- Smart Grid Interoperability Panel—Cyber Security Work- abilities routinely minimize the importance of these novel— ing Group, NISTIR 7628, Gaithersburg, MD, Aug. 2010. and persistent—threats. S. M. Amin, “Securing the electricity grid,” Bridge, vol. 40, no. 1, pp. 13–20, Spring 2010 Conclusion S. M. Amin, “Energy infrastructure defense systems,” Cyberconnectivity has increased the complexity of the con- Proc. IEEE, vol. 93, no. 5, pp. 861–875, May 2005. trol systems and facilities it is intended to safely and reliably S. M. Amin, “Balancing market priorities with security control. In order to defend electric infrastructure against the issues: Interconnected system operations and control under impacts of cyber and physical attacks, signifi cant challenges the restructured electricity enterprise,” IEEE Power Energy must therefore be overcome before extensive deployment and Mag., vol. 2, no. 4, pp. 30–38, Jul./Aug. 2004. implementation of smart grid technologies can begin. Cyber- security and interoperability are two of the key challenges of Biographies the smart grid transformation. As for security, it must be built S. Massoud Amin is with the University of Minnesota. in as part of its design, not glued on as afterthought. Anthony M. Giacomoni is with the University of Regarding recent cyberthreat reports, it is fundamental to Minnesota. separate the “hype” from the truth. What is most concerning p&e

4012 IEEEieee power & energy magazine january/february 2012 Reprinted from January/February 2012 issue of IEEE Power & Energy magazine A Virtual Smart Grid

IT IS GENERALLY RECOGNIZED THAT A HIGH-BANDWIDTH Real-Time and highly available networked communication system should overlay the transmission system topology in order to enable the control and pro- Simulation tection envisaged today to make the grid more effi cient and more reliable. The specifi cations for such a communication system have been diffi cult for Smart Grid to develop, however, because it needs to support a great variety of applications, many of which have not yet been developed. Organizations Control and such as the North American SynchroPhasor Initiative (NASPI) are trying to build on this vision of a communication system that can utilize phasor Communications measurement data to initiate fast controllers, including fl exible alternating current transmission system (FACTS) devices. Design IA major hurdle in developing such fast, wide area controls has been the lack of design tools available to do so. In particular, the development

By David Anderson, Chuanlin Zhao, Carl H. Hauser, Vaithianathan Venkatasubramanian, David E. Bakken, and Anjan Bose

Digital Object Identiﬁ er 10.1109/MPE.2011.943205 Date of publication: 13 December 2011

january/february 2012 1540-7977/12/$31.00©2012 IEEE ieeeIEEE power & energy magazine 4913 GridSim can represent a large portion of a grid and runs in real time so that various components running at different sampling rates can be tested together.

of controls that depend on communications to carry the rithms. Using actual power system artifacts is important input and output signals and complex software to process for two reasons. First, it allows the artifacts to be tested in these signals requires tools to simulate and analyze such the simulation environment, which is one way to increase controls. To accurately portray the behavior of such con- confi dence in a design. Second, it allows existing artifacts trols, design tools must integrate the dynamic behavior of such as the Grid Protection Alliance’s openPDC product the power system with the response of the communication and the GridStat communication framework to be used as and computation system. building blocks for GridSim, speeding its implementation. We describe here a simulator—GridSim—that can simu- From this decision comes another requirement: that Grid- late in real time the electromechanical dynamic behavior of Sim operate in real time so as to properly interface with the power grid, the IT infrastructure that overlays the grid, these artifacts. and the control systems taking advantage of that IT infrastructure. This simulator was devised for designing and test- The Overall Design of GridSim ing new wide area control and protection schemes. GridSim GridSim is a real-time, end-to-end power grid simulation is able to represent a large portion of a grid and runs in real package designed using a default sample rate of 30 samples time so that various components running at different sam- per second (per sensor). The goal of this project is to simu- pling rates can be tested together. late power grid operation, control, and communications at gridwide scale (e.g., the Western Interconnection) in order Background to give utilities the ability to explore new equipment and The use of time-synchronized, high-data-rate sensor tech- control system deployments. Possibilities include simulat- nology is widely viewed as a critical enabler for increasing ing large-scale PMU installations and power applications the reliability of the power grid while allowing the integra- able to utilize the vast quantities of data generated in such tion of many more stochastically variable renewable energy a situation. With the objective of providing tools to simulate sources such as solar radiation and wind. For example, the real-world equipment usage and the ability to be used in con- deployment of phasor measurement units (PMUs) is becom- junction with readily available utility industry equipment, ing more commonplace. PMUs are capable of sampling GridSim uses the IEEE C37.118 data format standard for all frequency, voltage, and current thousands of times per sec- streaming measurement data. ond and outputting accurate, time-stamped measurements The GridSim platform consists of a number of compo- 30–120 or more times per second. It is diffi cult, however, nents falling into four groups: power system simulation, for utilities to take full advantage of these devices due to a substation simulation, communication and data delivery, and lack of tools for designing and evaluating the control sys- control center applications (see Figure 1). We fi rst describe tems that exploit them. Furthermore, the behavior of such the overall relationship between these groups and then look control systems will also depend on the performance of the at each of them in detail. wide area communications systems that connect the sen- The power system simulation calculates the electro- sors, control logic, and actuators—wide area communica- mechanical dynamics in real time. Sensor data from the tions systems whose design and specifi cations are them- simulated power system are fed in C37.118 format to the selves still evolving. substation simulation processes at a rate of 30 samples Simulation is historically one the principal tools used per second. In the substations, data are optionally pro- in the design of power system controls. No existing simu- cessed by substation applications and published, along lation framework, however, can model at the scale of the with the outputs of the substation-level applications, to power grid the combined behavior of the power system, the data delivery component through simulated substa- the communications system that overlays it, and the con- tion gateways. Delivery to control center applications trol system that relies on the latter to monitor and control and other substations occurs via the data delivery system. the former. GridSim is intended to address these issues Note the design choice here: the wide area data delivery by providing a very fl exible simulation framework that system is not involved in connecting simulated sensors incorporates power system simulation, data delivery, fl ex- within the simulated substations where they are located. ible sensor deployments, and the ability to incorporate Although the substation-level processing of the data is actual power system components, protocols, and algo- simulated, the data communication within the substation

1450 IEEEieee power & energy magazine january/february 2012 Powertech TSAT Simulator

Simulated Power Measurement Static Data System Generator Generator

C37.118 Generator GridStat FE FE

Substation Substation N FE FE O Substation Su OM Substation SE FE SubOM Substation Substation SE FE Simulation Sub OM Substation SE Sub OM Substation SE Substation Gateway SE Substation 1 OpenPDC

Control State Oscillation Estimator Center Monitor Applications

figure 1. GridSim architecture. is assumed to be negligible for the current goals of wide An off-line transient stability simulation such as TSAT area control design. does not perfectly meet the needs of GridSim. To obtain The data delivery component of GridSim is GridStat, real-time performance, the simulator was modifi ed so that a publish-subscribe, wide area data delivery framework simulation time progresses no faster than wall-clock time. designed from the ground up to meet the emerging needs of This is accomplished by pausing after computing each set electric power grids. Once data are published, the fl exibility of measurements (30 sets per second) until the correct wall- provided by the GridStat data delivery middleware allows clock time arrives for that set to be published. To extract subscribing applications to be easily integrated into the sys- the measurement sets at the time they are produced by the tem without massive reconfi guration. simulation, certain TSAT functions are used. They directly In the current GridSim implementation, published data implement simulated PMUs attached to particular points in are used by the two control center applications included in the power system topology where they measure frequency, this project: the hierarchical state estimator and the oscilla- voltage, and current 30 times a second. These sensor data tion and damping monitor. from the simulated PMUs are sent to the measurement generator for postprocessing (see Figure 2). Power System Simulation Power system simulation in GridSim is provided by a mod- Substation Simulation ifi ed version of TSAT, an industry-proven transient stabil- The measurement generator also bridges the gap between ity simulator produced by Powertech Labs, Inc. Unmodi- the bus-branch power system model supported by TSAT and fi ed TSAT accepts power system topologies, initial values, the more detailed bus-breaker model that represents the sub- and dynamic simulation variables (such as faults at spe- stations. To do this, GridSim’s static data generator creates cifi c times) as inputs. On execution, the simulator loads tables that map the FromBus/ToBus/EquipmentID measure- the input values, then as quickly as possible computes the ment identifi cation information used in TSAT to the unique state of the system over time; on completion it writes the CircuitBreaker/BusID/PMUID numbers used throughout results to a fi le. the rest of GridSim. Data from the static data generator also

january/february 2012 ieeeIEEE power & energy magazine 1515 Communication System and Data Delivery Static Data Generator Generates Data delivery latency and loss Equipment in Each Substation rate are important factors in the performance of wide area control and protection applications, but Real-Time Data Generator: the data delivery infrastructure Acquire Data from TSAT Output that will ultimately support those applications is still evolving. Grid- Sim’s data delivery component, GridStat, is a publish-subscribe middleware framework that has Generator or Load Type of Data? infl uenced the NASPInet effort led by NERC and the U.S. Depart- ment of Energy (DOE). Its design Branch or Transformer centers on the fact that sensor measurements are digitally repre- Calculate Complex Voltage from Calculate Complex Voltage and sented as a periodic stream of data Current from Magnitude and Magnitude and Angle. Active and Angle Given in TSAT Output Reactive Power Are Given Instead points. Working from this data of Current Magnitude and Angle, model, GridStat was designed Thus Calculate Complex Current to allow for effi cient, wide area, by Solving the Equation of Complex Power encrypted multicast delivery of Assign Complex Voltage and data. GridStat as a component Current as Measurement to Respective Equipment of GridSim is a realistic model for emerging power system data delivery services and at the same time provides great fl exibility for No TSAT Output Ends? confi guring and evaluating potential wide area control and protec- Ye s tion applications. GridStat is designed to meet the requirements of emerging Assign Measurements to Circuit Breakers control and protection applications that require data delivery latencies on the order of 10–20 ms over hundreds of miles with Real-Time Data Generator Ends extremely high availability. The GridStat architecture consists of two communication planes: figure 2. Measurement generator logic. the data plane and the management plane (see Figure 3). The allow the measurement generator to synthesize additional data plane is a collection of forwarding engines (FEs) measurements, such as breaker currents, from the TSAT designed to quickly route received messages on to the outputs. Noise and other real-world attributes can be added next FE or termination point. The FEs are entirely dedi- within the measurement generator, if desired. Once these cated to delivering messages from publishers to sub- operations have been performed, the PMU measurements scribers. Routing configuration information is delivered are sent to a C37.118 encoder and then to the substation sim- to the FEs from the management plane. The forwarding ulation processes. latency through an FE implemented in software is on The substation simulation processes host substation- the order of 100 µs, and with network processor hard- level power applications and substation gateways. Power ware it is less than 10 µs. We believe that the perfor- applications perform computations—both the applications mance of a custom hardware implementation of an FE described below have substation-level processing—and sub- could match or exceed that of a general-purpose Internet mit results to the substation gateway. Measurement genera- router. Thus, in a typical wide area configuration, Grid- tor output for each substation is also published to the data Stat would not add more than 1 ms over the speed of the delivery component by the substation gateway. underlying network while providing quality-of-service

1652 IEEEieee power & energy magazine january/february 2012 GridStat allows for virtual substations to be created or reconfigured and additional subscribers and power applications to be added with minimal changes.

(QoS) guarantees tailored to rate-based control and protec- QoS Broker tion applications. The management plane is a set of controllers, called QoS bro- QoS Broker kers, that manage the FEs of the data plane. The QoS brokers are organized in a hierarchy to refl ect Leaf QoS Broker the natural hierarchy in power grids. When a subscriber wishes to receive data from a publisher, GridStat it communicates with a QoS FE Pub1 broker that designs a route for FE the data and delivers the routing information to the relevant FEs, Pub2 FE creating the subscription. Since FE path computations are done out Sub1 of band from data delivery, even FE heavy loads of new subscription creation do not adversely affect Sub3 the performance of the data plane. Sub2 Beyond this, QoS brokers have a privileged view of routing performance and the router graph figure 3. GridStat architecture. that allows them to create optimal delivery paths. QoS brokers also implement policies for called rate fi ltering: only forwarding an update on an outgo- resource usage, cybersecurity, aggregation, and adaptation. ing link at the highest rate that any subscriber downstream Because the entire purpose of GridStat is the effi cient via that link requires. Some kinds of data place additional delivery of data, it includes features providing confi gurable restrictions on the rate fi ltering. GridStat’s rate-fi ltering QoS per subscription while attempting to minimize data algorithms are coordinated across multiple PMU streams in delivery costs. A subscriber can request quality-oriented order to ensure that subscribers receive sets of updates from parameters such as data delivery rate, temporal redundancy different PMUs taken at the same instant. For example, con- of data packets, and spatial redundancy of data streams sider PMUs that send updates at a rate of 120 Hz. While such (delivery over multiple independent delivery paths, each of a high rate would be useful for a few application programs, which meets the end-to-end delay requirements). The QoS many applications would not need such frequent updates. brokers ensure that each subscriber gets the resources it needs For an application subscribing to two different PMU streams while preserving the needs of existing subscriptions. To con- at a rate of 20 Hz, fi ve-sixths of the updates will be dropped serve network resources, the management plane identifi es before reaching it. But GridStat ensures that the same one- any shared data paths between a publisher and two or more sixth of the updates are delivered from the two PMUs, so subscribers. If there is any overlap in these paths, the manage- they can be used as a global snapshot. This synchronized ment plane ensures that data are only sent once for that leg of rate fi ltering is set up when subscriptions are being added the journey before being duplicated at the split. and is based on time stamps in the updates, so it does not GridStat supports multicast delivery of a given sensor require any inter-FE coordination when updates are being update stream whereby different subscribers can subscribe delivered. So scalability is not harmed by this strong deliv- to different rates yet no update message is ever sent over a ery property. network link more than once and it is not forwarded on a link When used as the data delivery layer component of Grid- at all if not needed. FEs implement this via a mechanism Sim, GridStat allows for virtual substations to be created or

january/february 2012 ieeeIEEE power & energy magazine 5317 reconfi gured and additional subscribers and power applica- set is an open-source software system that collects PMU tions to be added with minimal changes. This contrasts starkly measurements from multiple sources, aligns them according with the current situation in the power grid, where even mini- to their time stamps, and processes them with user-defi ned mal changes to the number of sources or consumers of data functions. The openPDC applications also provide numer- can require the data delivery system to be completely re-archi- ous advanced functions, such as cybersecurity and device tected. Conversely, GridSim also allows for potential deploy- management, that are necessary for industry use. Thus far, ments of GridStat to be tested with real-world volumes of data however, GridSim uses only the C37.118 protocol parser and and with different network and power system topologies. the time-alignment functionality. The openPDC applications contain three kinds of adapt- Control Center Applications ers: input adapters, action adapters, and output adapters. Continuing the theme of using existing artifacts as compo- GridSim’s applications, however, use only two of these. nents of the GridSim environment, we now describe two Input adapters read data and parse them. Although the open- control center applications that have been incorporated into PDC applications provide many built-in input adapters that GridSim thus far. can read data from fi les, databases, or the network, none of One of the main objectives of GridSim is to allow exper- them supports the publish-subscribe communication pattern imentation with and testing of wide area control and pro- used in GridSim. New input adapters were therefore devel- tection applications using PMU and other high-rate, time- oped supporting the GridStat publish-subscribe system. stamped data streams. Thus far, two prototype applications Action adapters receive time-aligned measurements and have been included in GridSim: a linear, hierarchical state process them. In GridSim, all of the power system calcula- estimator and an oscillation monitoring system. tions, including substation-level and control center–level state Both applications were built using components of the estimation as well as oscillation detection, are implemented Grid Protection Alliance’s openPDC product. Thus, one using custom action adapters. These new functions embed- benefi t of incorporating these applications in GridSim is that ded in the openPDC applications are not only useful in the other openPDC-based applications can easily be brought simulation environment but can also be run in the real indus- into the GridSim environment. The openPDC application try environment. Since the openPDC applications were primarily designed and implemented for fi eld usage, which has different technical requirements from GridSim, work was performed to Start adapt them for the simulation environment. For example, the openPDC applications provide a user interface for confi guring devices, phasors, and measurements. Since GridSim is intended to simulate a variety of systems that may change Read Data frequently, manual confi guration is too cumbersome and from PDC error-prone. A program was therefore created to read the power fl ow fi le for TSAT and confi gure the whole system automatically, saving a lot of effort and simplifying the inte- Yes Event? gration of the openPDC and simulation software.

No The Oscillation Monitoring System The oscillation monitoring system (OMS) application has FDD Analysis Prony Analysis for been developed at Washington State University for real- Damping for Ambient Postdisturbance Event time monitoring of problematic electromechanical oscilla- Monitor Data Data Analysis Engine Engine tions using wide area PMU measurements. OMS combines advanced signal-processing algorithms with heuristic expert Moving Window Moving Window system rules to automatically extract the damping ratio, fre- Cross-Check Cross-Check quency, and mode shape of poorly damped electromechanical oscillations in a power system from power system measurements. A prototype OMS has been implemented as part of the phasor data concentrator at Tennessee Valley Authority (TVA) Poorly Yes Alarm since 2007. It is also currently being implemented at Entergy Damped Mode Detected? Controller in conjunction with a smart grid investment grant project. Trigger In our GridSim project, the OMS is being used as a real- No time application example, both serving to illuminate what GridSim must provide in order to incorporate actual applica- figure 4. Flowchart of an oscillation monitoring system. tions and demonstrating how executing an application with

1854 ieeeIEEE power & energy magazine january/february 2012 simulated real-time test data can help validate the application. The OMS engines are integrated into an action adapter module 5.15 of the openPDC applications. Thus, the OMS receives real-time 5.14 simulated PMU data streams from TSAT, via the measurement 5.13 AmbientAmbAAmAbmmbbient NNoiseoise AAnalysisnalysis generator and the data delivery system, which are buffered onto 5.12 1.2 Hz at + 1.8% DDamping.amping. Local ModeMode.. the internal signal-processing engines of the OMS. Results from 5.11 EventEEvev nt the OMS can be exported to a custom SQL database that can be 5.1 AnalysisAnan lysis visualized and set to trigger alerts or alarms whenever damping 5.09 1.21 2 Hz at + 1.5% Damping. Local Mode.Mode. levels of oscillatory modes fall below prespecifi ed thresholds. 820 840 860 880 900 920 940 The operator can then take manual action to bring the damping Time (s) back to acceptable levels. Unlike the real power system, where the actual modal figure 5. Illustration of analysis results from OMS engines. characteristics of the system are unknown values, the modal properties of the test system in TSAT can be accurately determined from model-based small-signal stability every computational run. As with the event monitor engine, analysis. Comparing the outputs of the OMS engines with the analysis is then repeated over moving time windows the respective model-based modal values is useful for test- and over different signal groups to verify the consistency of ing and tuning the OMS engines for target power systems. modal analysis results. Since GridSim includes communication models, such stud- Figure 5 shows the results from the two engines for a ies also reveal the effects of communication delays, the loss recent event near a major generating plant. The system of PMU channels, and network congestion on the resulting encountered a routine event at about 830 seconds. The event OMS modal estimates. We plan to use GridSim to test auto- analysis engine of the OMS carried out moving time-window matic control action by the OMS, although such closed-loop analysis of the PMU measurements using real-time Prony feedback will require further modifi cation of TSAT. analysis and concluded at 838 seconds (the vertical dotted The OMS includes two engines, as shown in the fl ow line in Figure 5) that the oscillation was from a local 1.2-Hz chart in Figure 4. The event analysis engine, shown on the right side of the fl ow chart, carries out an expert system– based Prony-type ringdown analysis of system responses following disturbances in the system. The objective for this Control Center engine is fast detection of sudden changes in the damping of Static Maintenance oscillatory modes from large disturbances in a power sys- Database tem, so that mitigating control actions can be initiated before CB/ND Equipment the damping problems degenerate into widespread black- Connections Parameters ND/Equipment outs. Typical analysis uses 5–10 s of PMU data at a time, Connections and the calculations are repeated over moving time windows and over different PMU signal groups to ensure the con- Topology State Processor System Estimator sistency of results. The event monitor engine can typically Topology detect oscillatory problems by using 10–15 s of PMU data, starting from the instant the oscillations begin to appear in Digital Analog a power system. Status Measurements The complementary damping monitor engine, shown on the left side of the fl ow chart, estimates the damping, fre- Real-Time quency, and mode shape of poorly damped oscillatory modes Database from ambient PMU measurements. Unlike the event monitor engine, which only works when the system is subject to disturbances, the damping monitor engine is applicable all the time. By using natural power system responses to rou- SCADA tine random fl uctuations from load variations and generation changes, the damping monitor engine continuously tracks damping levels and mode shapes of poorly damped oscillatory modes. The damping monitor engine uses an extension Substation RTU Substation Substation of a frequency-domain algorithm called frequency domain RTU decomposition (FDD). This engine is aimed at preventive RTU detection of poorly damped oscillations. The damping monitor engine uses about four minutes’ worth of PMU data in figure 6. The two-level linear state estimator. january/february 2012 IEEEieee power & energy magazine 5519 mode (i.e., one involving mainly one PMU or a few nearby application for testing in the GridSim environment. It is based PMUs) with a damping ratio of +1.5%. Subsequently, the on PMU data and requires algorithmic processing at the sub- damping monitor engine analyzed the real-time ambient station level, fast communication of the substation results PMU data and estimated the dominant oscillatory mode to to the control center, and synchronization of the data at the be the same local mode at 1.2 Hz, with a damping ratio of control center before it fi nally calculates a state estimate (SE) +1.8%. Thus the results of ringdown analysis and ambient for the whole system. The power system simulation produces noise analysis match well for this example. The two engines PMU measurements 30 times per second, and the fi nal SE is serve as complementary techniques for identifying the domi- also calculated at the same rate. Thus errors in the simulation, nant poorly damped oscillatory modes of a power system communication, synchronization, and SE calculation can all whenever such modes exist. be checked during the testing of this application on GridSim. The processing of this two-level SE is shown in Figure 6 State Estimator for both the substation level and the control center level. At A two-level linear state estimator has been developed at each substation, the local PMU data are processed using Washington State University that is an excellent candidate linear estimation algorithms for both current and voltage

TSAT Bus Voltage 1.2 1.0 0.8 0.6 0.4 Bus 1 Bus 2 Bus 3 Bus 4 Bus 5 Bus 6

Value (p.u) 0.2 Bus 7 Bus 8 Bus 9 Bus 10 Bus 11 0.0 01234567 8 9 10 11 Time (s) (a)

Generated Bus Voltage 1.2 1.0 0.8 0.6 0.4 Bus 110 Bus 150 Bus 220 Bus 260 Bus 330 Bus 410

Value (p.u) 0.2 Bus 440 Bus 500 Bus 770 Bus 880 Bus 990 0.0 01234567 8 9 10 11 Time (s) (b)

Substation Bus Voltage 1.2 1.0 0.8 0.6 0.4 Bus 11 Bus 12 Bus 21 Bus 22 Bus 31 Bus 32

Value (p.u) 0.2 Bus 41 Bus 42 Bus 71 Bus 81 Bus 91 0.0 01234567 8 9 10 11 Time (s) (c) Control Center Bus Voltage 1.2 1.0 0.8 0.6 0.4 Bus 11 Bus 12 Bus 21 Bus 22 Bus 31 Bus 32

Value (p.u) 0.2 Bus 41 Bus 42 Bus 71 Bus 81 Bus 91 0.0 01234567 8 9 10 11 Time (s) (d)

figure 7. GridSim results for an 11-substation system using the two-level linear state estimator.

2056 IEEEieee power & energy magazine january/february 2012 phasor measurements. This processing has the advantage of esti- SE Bus Voltage Curve 2 mating and eliminating errors 1.2 from noise, bad analog data, and 1.0 bad circuit breaker status data on 0.8 a small set of measurements. The topology, current, and voltage esti- 0.6 mates from each substation are 0.4 Bus 11 Bus 12 Bus 71 Bus 22 Voltage (p.u) then sent through the communica- 0.2 Bus 81 Bus 91 Bus 42 Bus 41 tion network to the control center. Bus 32 Bus 31 Bus 21 0.0 At the control center, the data are 012 345 6 7 8 9 10 11 synchronized for the same time Time (s) stamp, and the whole system states are linearly estimated. figure 8. State estimator results with jitter in the communication system. Figure 7 provides some results for this test as carried out on Grid- Sim for an 11-substation power system. For a small system GridSim, and the signiﬁ cant changes needed in the power like this, the simulation and communication speeds were system simulator to accomplish this are being developed. not a problem, so the test’s purpose was mainly to check the computation processes and data delivery. When the SE Acknowledgments was running perfectly, the ﬁ gure shows that the bus voltages We gratefully acknowledge the assistance of Powertech Labs (a) calculated by the TSAT simulation, (b) generated by the and the Grid Protection Alliance in adapting their TSAT and PMU data generator, (c) estimated at the substation level, and openPDC products, respectively, for use in GridSim. This (d) estimated at the control center all compare quite well 30 research was supported by a grant from the U.S. Department times a second for about eight seconds after a fault on the sys- of Energy (Award #DE-OE0000032). tem. Many things can go wrong, however, as demonstrated in Figure 8 by introducing some jitter in the data delivery For Further Reading between the substation and the communication level, thus [Online]. Power Tech Labs. TSAT—Transient Security As- producing erroneous SE results at the control center. sessment Tool. 2011. Available: http://www.powertechlabs. com/software-modeling/dynamic-security-assessment- Conclusions software/transient-security-assessment-tool A fast communication and computation system overlay- D. Bakken, A. Bose, C. Hauser, D. Whitehead, and G. ing the power grid is a key enabler for applications taking Zweigle, “Smart generation and transmission with coherent, advantage of PMUs and FACTS controllers to achieve the real-time data,” Proc. IEEE (Special Issue on Smart Grids), smart grid of the future. The tools needed to develop and test vol. 99, no. 6, pp. 928–951, June 2011. these new applications do not exist today, however. We have [Online]. Grid Protection Alliance. The Open Source described such a tool—a simulation platform called Grid- Phasor Data Concentrator. 2011. Available: http://openpdc. Sim—that can be used to develop and test wide area control codeplex.com and protection schemes. G. Liu, V. M. Venkatasubramanian, and J. R. Carroll, We have developed this platform to simulate the power “Oscillation monitoring system using synchrophasors,” in grid in real time for electromechanical dynamics and to Proc. IEEE PES General Meeting, Calgary, Canada, July generate and stream PMU data in standard format. It also 2009, pp. 1–4. includes the ability to deliver measurements and processed T. Yang, H. B. Sun, and A. Bose, “Transition to a two- data over a high-bandwidth networked communication sys- level linear state estimator, part I: Architecture, part II: Al- tem called GridStat. Finally, we have used GridSim to simu- gorithm,” IEEE Trans. Power Syst., vol. 26, no. 1, pp. 46–62, late and test two new applications—oscillation monitoring Feb. 2011. and linear state estimation—that are quite different from each other but both utilize PMU streaming data in real time. Biographies We show that platforms such as GridSim can successfully David Anderson is with Washington State University. and rapidly prototype new “smart” applications. Chuanlin Zhao is with Washington State University. We should note that closed-loop control is not illustrated Carl H. Hauser is with Washington State University. in this article. Both the OSM and the linear state estimator Vaithianathan Venkatasubramanian is with Washing- are real-time but open-loop applications, which means that ton State University. the outputs are used by the operator to initiate manual control David E. Bakken is with Washington State University. if necessary. Closed-loop control will be incorporated into Anjan Bose is with Washington State University. p&e

january/february 2012 ieeeIEEE power & energy magazine 2571 Reprinted from May/June 2012 issue of IEEE Power & Energy magazine

By Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

PUBLIC POLICIES AT BOTH THE STATE AND FEDERAL LEVELS in the United States and a variety of technological and economic changes Policy Challenges and are poised to signifi cantly alter both the demand for and supply of electricity in the country over the next several decades. These changes will yield Technical Opportunities a wide range of new challenges and opportunities, including incorporating variable energy sources like wind and solar radiation; adjusting distribu- on the U.S. Electric Grid tion systems to accommodate small-scale, distributed generators; accom- modating the charging of electric vehicles and other changes in electricity demand; making the best use of new technologies to ensure reliability and effi ciency under changing conditions; responding to threats presented by the vast increase of data communications within the grid; and meeting chang- Ping workforce needs. A variety of technologies exists today that can help meet the emerging challenges effectively in the United States. In a recently completed two-year study on the future of the U.S. electric grid that we performed with a dozen other economists and engineers, however, we found that the promise of these new technologies will only be fully realized if a number of regulatory policies are changed, if necessary research and development is performed, and if important data are compiled and shared. Maintaining system reliability, keeping electricity rates at acceptable levels, and achieving state and federal policy goals will depend to a large degree on a few key choices made—or not made—at the state and federal levels and within the industry over the next few years. In this article, we fi rst discuss the performance of the U.S. grid today. Then, we describe several of the most important challenges and opportunities that are likely to face the U.S. grid over the next several decades.

The U.S. Grid Today Physically, the U.S. electric grid consists of approximately 170,000 mi of high-voltage electric transmission lines (i.e., lines rated at >200 kV) and associated equipment and nearly 6 million mi of lower-voltage distribution lines. In aggregate, the U.S. grid serves about 125 million residential customers, 17.6 million commercial customers, and 775,000 industrial customers that account for 37%, 36%, and 27% of electricity use, respectively. At the highest level, the electric power system of the continental United States consists mainly of three independently synchronized grids: the Eastern Interconnection, the Western Interconnection, and the Electric Reliability Council of Texas (ERCOT). The three grids are linked by only a few low-capacity dc lines. Within these broad areas are 107 balancing authorities, responsible for balancing the supply and demand for power in speciﬁ ed zones.

Digital Object Identiﬁ er 10.1109/MPE.2012.2188669 Date of publication: 19 April 2012 ©ARTVILLE, LLC.

By Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

Digital Object Identiﬁ er 10.1109/MPE.2012.2188669 Date of publication: 19 April 2012 ©ARTVILLE, LLC.

30 IEEE power & energy magazine 1540-7977/12/$31.00©2012 IEEE may/june 2012 may/june 2012 ieee power & energy magazine 23 As a result of the layering of historical policy decisions An important measure of the performance of a transmis- and the lack of a comprehensive, shared vision of system sion and distribution system is the fraction of energy gener- structure or function, the U.S. electric power system today ated that is lost due to heating of transmission and distribu- operates under a fragmented and often inconsistent policy tion lines and of other components. That fraction has fallen regime. For instance, organized wholesale markets for signifi cantly over time in the United States. As Figure 1 power play a central role in some areas, but in others the shows, losses in transmission and distribution decreased traditional vertically integrated utility model remains domi- from more than 16% in the late 1920s to less than 7% today. nant. Generation facilities are variously owned by investor- This refl ects investments in transmission and distribution owned utilities, rural cooperatives, municipal utilities, fed- systems, the development and deployment of more effi - eral government entities, and independent power producers. cient transformers and other equipment, and transmission at Subsidies of various sorts for public and cooperative entities higher voltages. are important in some regions but not at all in others. Trans- Reliability is also an important dimension of perfor- mission and distribution voltage levels also vary regionally. mance. Increases in transmission voltage and many other, Several hundred entities currently own parts of the trans- less visible, technological advances have contributed to mission or bulk power system and, at the distribution level, improved reliability over time. Protective relaying enabled approximately 3,200 organizations provide electricity to the detection and isolation of system faults, for instance, and retail customers. high-speed reclosing circuit breakers and relaying allowed Assessing the performance of a system as complex as the transmission lines to be reenergized after a fault automati- U.S. electric grid is not a simple task. International com- cally and in only a few seconds. Lightning arrestors allowed parisons and even comparisons within the United States are the effects of lightning strikes to be contained automatically. diffi cult because of differing geography, rates of growth, At the bulk power level, data on major disturbances and and defi nitions of performance measures. Systems that have unusual occurrences have been reported to the U.S. Depart- grown more rapidly in recent years, for instance, will on ment of Energy (DOE) since the 1970s and to the North average have newer equipment. Comparisons over time may American Electric Reliability Corporation (NERC), which reveal nothing more than the advance of technology driven has responsibility for the reliability of the bulk power system, by vendor R&D. Moreover, because there are diminishing since 1984. These data are not consistent, complete, or nec- returns to investing to increase effi ciency and reliability and essarily accurate, however, and they cannot reliably be used because perfection is unattainable at any cost, it is possible to assess changes in the reliability of the bulk power system not just to underinvest but also to overinvest in these and over time. Most outages in the United States occur within other dimensions of performance. distribution systems, but a recent study from Lawrence Berkeley National Lab found that only 35 U.S. states require utilities to report data on the impact 18 of all outages on consumers, and 16 reporting standards and practices differ. It is accordingly impossible 14 to make comprehensive comparisons across space or over time. In 12 particular, the treatment of very 10 short interruptions varies among U.S. states and among different 8 countries, so outage counts can- 6 not be usefully compared. Nonetheless, the data that are 4 available suggest that U.S. reliability is on a par with that of T & D Losses (% of Total Generation) 2 other industrialized countries. 0 According to a recent study from the Electric Power Research 1926 1929 1932 1935 1938 1941 1944 1947 1950 1953 1956 1959 1962 1965 1968 1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 Year Institute (EPRI), U.S. customers can expect to experience between figure 1. U.S. transmission and distribution losses, 1926–2009. Losses are measured one and a half and two power as the difference between energy generated and energy delivered to customers and interruptions and between two thus in practice include losses due to theft. Theft is not considered to be important in and eight hours without power the United States today, but it is significant in some other nations. each year. This is on a par with

2432 IEEEieee power & energy magazine may/june 2012 most European countries, where customers generally expe- mance specifi cations, such as low-voltage ride-through and rience from less than one interruption per year to almost inertial response, appropriate for operation in the high-VER three, based on data from the Council of European Energy future they are likely to encounter. Regulators. Of course, there is great variation in reliability In the United States, many of the most attractive wind between urban and rural areas. Such comparisons cannot resources are located in the “wind belt” that stretches north reveal whether U.S. reliability is too low, too high, or opti- from Texas through the Dakotas to the Canadian border mal, given the benefi ts of reducing outages and the costs and offshore on both coasts. While the offshore resources of doing so. are closer to major load centers, the costs of offshore wind A fi nal dimension of performance involves the use of new installations are generally considerably greater than those technology to increase productivity. The U.S. electric utility for onshore facilities in good locations. Similarly, the prime industry has historically devoted a very small fraction of its locations for solar power are in the nearly cloud-free and revenues to R&D, instead relying primarily on its suppliers sparsely populated desert Southwest. for innovation. U.S. utilities have sometimes collaborated Exploiting these resources will require building more with vendors on R&D activities and have participated in col- transmission than if fossil-fueled or nuclear generating laborative research through EPRI. In recent years, however, plants built relatively close to load centers were supporting utilities have shifted away from longer-term, collaborative system expansion. The use of very long transmission lines projects and toward shorter-term proprietary efforts. More- can cause technical issues and compromise system stability; over, investor-owned utilities, which account for almost all such issues will have to be monitored carefully in the years nonfederal utility R&D spending, reduced their R&D bud- to come. In addition, adequate planning tools that can deal gets beginning in the 1990s, spending on average less than with complex networks and that take uncertainty rigorously 1% of their revenues on R&D. The decrease in utility R&D into account do not exist today, and research to develop them funding refl ects, in part, reluctance among utilities to incur is needed. For such research to be most productive, detailed (and regulators to approve) R&D expenditures as U.S. fed- data covering the major interconnections—data that are now eral and state policies pursued more industry competition deemed proprietary—must be made appropriately available during those years. to researchers. As VER penetration increases, more of the new trans- New Challenges mission lines will cross state borders or the 30% of U.S. and Opportunities land managed by federal agencies. Cost allocation and siting have been particularly contentious for these transmis- Grid-Scale Variable Generation sion facilities. When boundary-crossing lines are proposed Owing to strong federal and state policy support, wind and today, they tend to be evaluated in isolation rather than as solar generation are almost certain to become more impor- part of a wide-area planning process, and allocation of the tant in the United States over the next several decades— costs involved is often done via facilities-specifi c nego- though perhaps not as important in many U.S. regions as tiations. Under current law, the siting of all transmission they already are in some European Union countries. Effi - lines is a matter for the states rather than FERC. Lines ciently increasing the penetration of grid-scale renewable that cross land managed by federal agencies also need the generation while maintaining reliability will require modifi - approval of those agencies. Consequently, the construction cations to power system planning and operation. of interstate transmission facilities requires the consent of At high penetration levels, the variable and imper- multiple state regulators and, sometimes, of one or more fectly predictable power output of these “variable energy federal agencies. FERC Order No. 1000, issued in July resources,” or VERs, causes the demand minus VER gen- 2011, should signifi cantly increase wide-area planning of eration—that is, the net load that must be met by other gen- transmission systems, make routine the allocation of the erators—to become noticeably more variable and diffi cult costs of boundary-crossing transmission facilities, and, to predict. To maintain reliability despite this variability, by explicitly adopting the “benefi ciaries pay” principle, the system and its operation must be modifi ed at some cost. rationalize the allocation of those costs. Establishing per- Wind and solar forecasts will have to be fully integrated into manent and collaborative planning processes at the inter- system operations and planning. Indeed, utilities and system connection level and a single cost allocation procedure operators in many U.S. regions are already actively working for boundary-crossing projects in each interconnection on improving their forecasting capabilities. Power system would further enhance the ability of the United States to fl exibility will also become more important, and incentives effi ciently and reliably achieve its renewable energy goals. for investments that add generation fl exibility or for operat- Expanding FERC’s authority for siting boundary-crossing ing generation resources in a fl exible manner may be needed transmission facilities would help facilitate transmission in regions with organized markets. Full or virtual consolida- expansion for the integration of VERs. Even with this tion of small balancing areas would facilitate VER integra- change, however, siting transmission facilities will remain tion, as would requiring new VER generators to meet perfor- a diffi cult challenge to manage. may/june 2012 IEEEieee power & energy magazine 2335 Distributed Generation require investment by the distribution utility. Current regu- Policies at both the U.S. state and federal levels favor distrib- latory frameworks may not provide adequate incentives for uted generation from low-carbon sources, and these policies such investments, however, as growth in distributed genera- seem likely to continue. At the federal level, personal and tion will often reduce utilities’ sales and profi ts. Transition- corporate tax incentives encourage distributed generators. ing away from recovering the largely fi xed costs associated Most states have programs that subsidize distributed gen- with transmission and distribution networks through volu- eration. The DOE has also established the goal that all new metric (US$/kWh) charges would help alleviate this incen- commercial and industrial construction should be energy- tive misalignment and could have a signifi cant impact on the neutral by 2030. That is, such buildings must generate as growth of U.S. distributed generation. much energy as they use. Furthermore, “net-metering” programs in 46 states and the District of Columbia compensate Changes in Electricity Demand end users for generating their own energy at the retail elec- Unlike some other regions of the world, electricity demand tricity rate rather than the wholesale cost of energy. Cus- growth in the United States is not likely to emerge as an tomers who generate electricity on-site in these programs important source of disruption in the next few decades. save both the energy charge, or the wholesale cost of energy, Based on data from the U.S. Energy Information Admin- and the distribution charge for that electricity. The utility, istration (EIA), between 1949 and 1973 U.S. electricity use however, saves only the corresponding energy cost. In this grew at an average annual rate of 8.3%. The system was able way, recovering network costs through per-kWh charges to meet that demand growth with only sporadic diffi culty. provides an additional subsidy to distributed generation that With rising prices after 1973, electricity use grew at an aver- can encourage its uneconomic penetration. age annual rate of 2.5% between 1973 and 2006. In contrast, At low levels of penetration, distributed generation sim- EIA’s most recent reference case projection is for growth to ply reduces the load at individual substations. At high levels average only about 0.9% per year between 2010 and 2030. of penetration, however, distributed generation can exceed U.S. electricity demand has changed, however, and is load at the substation level, causing unusual distribution likely to continue to change in ways that pose challenges fl ow patterns. In some cases, power many even fl ow from to the system. Over the past several decades, due in part to the substation into the transmission grid. Many distribution the increased penetration of air conditioning and the rela- systems are currently not designed to handle such reverse tive decline of industrial loads, there has been a substan- fl ows, however, and customer power quality can sometimes tial increase in the ratio of system peak loads to average suffer. High levels of penetration can also add to the stress loads. Because power systems need to be sized to meet peak on electrical equipment, such as circuit breakers, and com- demand with a margin for reliability, the peakier demand plicate the operation of the distribution system, particularly becomes (all else being equal), the lower capacity utilization during emergencies. Additional monitoring and new sys- becomes, and thus the higher rates must be raised to cover tems for the operation, protection, and control of distribu- all costs. tion systems will be necessary if U.S. distributed generation Figure 2 illustrates this change, showing load dura- penetrations grow signifi cantly. And since much of this dis- tion curves for New England and New York expressed as tributed generation will be in the form of VERs, there will percentages of peak hour demand. The fi gure shows, for be an impact on the control of central generating resources. instance, that in the 1980–84 period in both New York and Enabling such penetration in a cost-effective manner will New England, demand exceeded 80% of its peak for only about 1,000 hours—about 11.4% of the time. By the 2005– 09 time frame, demand in both New York and New England 1 exceeded only 70% of its peak for about 1,000 hours, so that 0.9 more than 30% of capacity was in use less than 12% of the 0.8 time. This trend raises average costs because of the need to 0.7 pay for capital that is idle most of the time and, by increasing 0.6 capacity requirements,worsens the problem of siting genera- 0.5 tion plants and transmission lines. 0.4 New England Average 1980–1984 Electric vehicles (EVs)—including plug-in hybrids 0.3

Normalized Load New England Average 2005–2009 and pure electric vehicles—could exacerbate these trends. 0.2 NY Average 1980–1984 Although their penetration is generally projected to be slow 0.1 NY Average 2005–2009 at the national level, EVs are expected to achieve high levels 0 1 2,001 4,001 6,001 8,001 of penetration quickly in some high-income areas with envi- Hours of Year ronmentally conscious consumers. If EVs are charged when commuters return home, as seems most likely under current figure 2. Normalized load duration curves for New policies, they could add signiﬁ cantly to system peak loads, England and New York. worsening the problem of increasing peakiness of demand.

2634 ieeeIEEE power & energy magazine may/june 2012 Due to a variety of recent policy changes and technical innovations, the U.S. electric grid will encounter significant opportunities and challenges over the next several decades.

On the other hand, measures that encourage overnight charg- mature, however, and further research into the behavior of ing could increase demand when it would otherwise be low, U.S. residential consumers faced with dynamic pricing is thus tending to fl atten load duration curves. needed. In this later regard, although there have been many Making other loads similarly responsive to system con- dynamic pricing pilot programs, few have been structured ditions could also shift demand off-peak, helping to slow so as to produce reliable data, and results have been highly the trend depicted in Figure 2. Dynamic pricing—in which variable. retail prices vary over short time intervals to refl ect changes in the actual cost of providing electricity—could induce such Innovative Technologies for responses. U.S. demand-response programs have grown sub- Increased Reliability and Efficiency stantially in recent years. Most demand-response programs Innovative technologies that can improve system perfor- in place today, however, use other approaches and focus on mance, offering enhanced reliability, increased capac- responding to occasional emergencies rather than system- ity, and the ability to better accommodate new resources atic load leveling. In some regions with organized wholesale (VERs, EVs, and so on) are poised for signifi cant growth markets that include a capacity market, demand response in the United States. The integration of such technologies has been allowed to bid in as a proxy for capacity, illustrating into comprehensive networks of sensors, communications its potential value to the economic effi ciency of the system. infrastructure, control equipment, and intelligent manage- A variety of new and emerging technologies, including ment systems will be a major focus of the U.S. electric power advanced metering systems, can receive price information industry over the next several decades. These technology based on the real-time cost of providing electricity and can opportunities, often referred to as the “smart grid,” are likely transmit usage information every few minutes. This makes to provide signifi cant benefi ts. it possible to provide real-time incentives to reduce system In the transmission system, phasor measurement units peaks caused by central air-conditioning, vehicle charging, (PMUs) are powerful devices that provide rich streams of and other loads, resulting in more effi cient use of grid assets frequent, time-stamped data on transmission system con- and thus lower rates. Many large commercial and industrial ditions. PMUs with appropriate analysis tools that turn customers already operate under dynamic pricing. Such the measured data into actionable information could allow pricing regimes likely will also be widespread options—if system operators to anticipate contingencies, reduce the not the default—for residential U.S. consumers by 2030. risk of wide-area blackouts, enhance system effi ciency, and Existing studies suggest that regulators can achieve sub- improve system models. While PMU hardware exists and is stantial load shifting—and perhaps overall demand reduc- currently being installed more widely in the United States as tion—when dynamic pricing is combined with the use of a result of ARRA funding, the software and analysis tools technology to automate responses to price changes. Residen- necessary to fully capitalize on this investment are yet to be tial dynamic pricing also requires substantial investment in developed and deployed. More widespread PMU data shar- advanced metering infrastructure (AMI) to measure usage ing among utilities, system operators, and researchers will over short time intervals. Substantial AMI investments have also be essential for the development and effective use of recently been funded through the American Recovery and these tools. Reinvestment Act of 2009 (ARRA), and some state regula- In addition to PMUs, fl exible alternating current trans- tors have mandated universal AMI deployment. But thus far, mission system (FACTS) devices based on advances in there has been little if any movement toward the dynamic power electronics will provide greater control of voltages pricing regimes that AMI enables. As long as the results are and power fl ows throughout the bulk power system and could shared widely, ARRA-supported and regulator-mandated allow more power to be transmitted on existing lines with- investments in AMI will provide important learning oppor- out increasing the risk of failure. Historically, deployment of tunities to develop effi cient paths to universal dynamic pric- the most versatile FACTS devices has been limited by their ing. Where wholesale electricity markets exist, effective relatively high cost. Costs are falling, however, and higher competition in the retail sales of electricity might stimulate penetration of VERs is likely to increase the value of deploy- innovation in ways that make dynamic pricing both accept- ing these technologies within the U.S. transmission system. able to consumers and regulators and effective in modify- Furthermore, integrating FACTS devices with PMUs and ing demand. Response automation technologies are not yet emerging wide-area measurement systems will allow their may/june 2012 IEEEieee power & energy magazine 3527 control capabilities to be leveraged so as to provide even est increases in utility R&D budgets. It will also likely be greater benefi ts. Ongoing research into new system control necessary for the industry to reverse the downward trend algorithms, software, and communication systems that fully in cooperative R&D spending and make appropriate use of utilize PMUs, FACTS devices, and other new transmission cooperative funding through EPRI, one or more independent system technologies is likely to create a high payoff and system operators, and project-specifi c coalitions. could accelerate the deployment of these technologies. Finally, utilities and regulators in the United States and Many technologies are also available to enhance the reli- elsewhere have historically tended to avoid investments in ability and effi ciency of distribution systems. Coping effi - unfamiliar technologies perceived to have uncertain payoffs. ciently with the integration of distributed generation, elec- The tendency of traditional regulatory systems to encourage tric vehicles, and demand response will require signifi cant excessively conservative behavior is likely to become more investments in new and emerging technologies, including and more expensive over time if the increasingly attractive distribution management software systems, equipment that opportunities to enhance effi ciency and reduce cost through is capable of more accurately monitoring and controlling the deployment of unfamiliar technology are not exploited. voltages, automatic reconfi guration of distribution circuits, Regulatory innovations are necessary in the United States and advanced metering. to provide adequate incentives for investments in unfamiliar The benefi ts of deploying these technologies are less well technologies while also ensuring that the returns on these known and may be more diffi cult to quantify relative to most investments are shared appropriately with ratepayers. This is recent investments in distribution systems; they will aim to an important problem—but one without an obvious solution. provide new capabilities, not just expand capacity. To reduce perceived uncertainties and make possible better system-spe- Data Communications, Cybersecurity, cifi c decisions, it is critical that detailed information about and Information Privacy Challenges the results of technology pilots and early deployments is The increasing use of communications systems, sensing shared as widely as possible. The DOE has recently funded and control equipment, AMI, and distribution automation a variety of smart grid demonstrations and technology pilots. technologies will enhance reliability and effi ciency but will These projects provide an important opportunity for learning. also give rise to new challenges. As the U.S. grid evolves, The electric utility industry has traditionally relied pri- increasing amounts of data will be exchanged among meters, marily on its suppliers for the innovation that has driven its other sensors, and various computers and control facilities productivity growth. Supplier R&D has naturally focused on through complex communications systems. The National equipment that can be sold to utilities. Therefore, although Institute of Standards and Technology (NIST), with sub- research in the several non-equipment-related research areas stantial industry input, is overseeing the critical process of mentioned above is likely to bring substantial payoffs, these developing the interoperability standards that are needed to are unlikely to attract equipment vendors. The electric utility ensure these systems are compatible not only with each other industry itself should be able to support the efforts required, but also with future generations of technology. In addition, however, even if federal support does not materialize. For there are ongoing debates in the United States about the use this to happen, regulators will need to recognize that techni- of spectrum and the roles of public and private networks. cal progress benefi ts consumers broadly and to permit mod- Since no communications system can be completely free from errors, the future grid must be designed to mitigate the consequences of data errors. More chilling is the possibility of deliberate sabotage via computers and data communications, the sort of cyberattacks that other industries Assess Vulnerabilities, have experienced. The existence of more communications Threats, nodes and channels facilitates the insertion of malicious data Impacts into the system; in addition, a greater reliance on automated Reduce responses to system conditions that may be misreported can Recover and Vulnerabilities, make it more diffi cult to prevent serious damage. Restore Threats, As illustrated in Figure 3, cybersecurity involves more Impacts than protecting against attacks. In fact, as communications systems expand into every facet of grid control and Mitigation operations, their complexity and continuous evolution will preclude perfect protection from cyberattacks. Response Prevent and recovery, in addition to preparedness, will therefore Respond Attacks, During Attack Incidents, be important components of cybersecurity, and it is impor- Other Outages tant for the government agencies involved to work with the private sector and publicly owned utilities in a coordinated figure 3. The cybersecurity life cycle. fashion, to support the research necessary to develop best

3628 IEEEieee power & energy magazine may/june 2012 practices for response to and recovery from cyberattacks on sary to mitigate the concerns of companies that operate in transmission and distribution systems, and to deploy those multiple jurisdictions and the concerns of their customers, as practices rapidly and widely. data on both companies and their customers regularly cross NERC is responsible for cybersecurity standards devel- state boundaries. opment and compliance for the U.S. bulk power system, but no entity currently has comparable nationwide respon- A Changing Workforce sibility for distribution systems. State public utility com- Even if it faced none of the challenges discussed above, the missions (PUCs)—which generally are responsible only electric power industry would need to rejuvenate its work- for investor-owned distribution systems—usually lack force in order to maintain current levels of performance. The cybersecurity expertise, and the same is true of munici- challenge of an aging technical workforce, a problem made pal utilities, cooperatives, and other public systems. While more serious by the decline in university power engineering the consequences of a successful attack on the bulk power programs, could have a signifi cant impact on the ability of system are potentially much greater than an attack at the the grid to meet the new challenges and seize the new oppor- distribution system level, the boundary between transmis- tunities described above. The IEEE U.S. Power and Energy sion and distribution has become increasingly blurred, and Engineering Workforce Collaborative (PWC) has reported distribution-level cybersecurity risks will require serious that approximately 45% of U.S. electric utility engineers will attention. Though NIST is facilitating the development of be eligible for retirement or could leave engineering for other cybersecurity standards broadly, it does not have an opera- reasons in the next fi ve years. While it is diffi cult to predict tional role, and no single agency currently has responsibil- exactly how many new engineers will be needed between ity for cybersecurity across all aspects of grid operations, now and 2030, there appears to be a signifi cant gap between including distribution systems. This is an unsolved prob- anticipated industry demands and both the pipeline of stu- lem, but one that the federal government is actively focus- dents entering power engineering and the faculty in place to ing on. The DOE and the U.S. Department of Homeland train them. Fortunately, U.S. industry workforce challenges Security recently announced an initiative to work together have received increasing attention in the past several years. with industry to develop a comprehensive approach to Despite these efforts, this will likely remain an important cybersecurity. But even if this joint effort proves workable area of focus in the years to come. or if a single agency is ultimately given appropriate regulatory authority, cybersecurity preparedness, response, and Conclusion recovery efforts across the electric power sector, including Due to a variety of recent policy changes and technical both bulk power and distribution systems, will be critical. innovations, the U.S. electric grid will encounter signifi cant A variety of federal government agencies, NERC, NIST, opportunities and challenges over the next several decades. state PUCs, utilities, public power authorities, and such As we have described above, various policy and system-level expert organizations as IEEE and EPRI will need to be issues will need to be addressed and new technologies will involved if these efforts are to be effective. need to be fully developed and used appropriately for the With the collection, transmission, processing, and stor- U.S. grid to evolve along an effi cient path with minimal dis- age of increasing amounts of information about customer ruption and to ensure electricity rates and levels of reliability electricity usage comes heightened concern for protecting remain acceptable. The journey to the electric grid of 2030 the privacy of those customers. As advanced metering is has begun, and there will be plenty of surprises along the implemented, information on personal habits will be avail- way. Much can and should be done now to smooth the road able to electric companies at a level never before envisioned ahead. by utilities or policy makers. Information about the operation of the electric grid itself will soon be available at a level For Further Reading of detail that will be of interest to those with both commer- Massachusetts Institute of Technology. (2011). The Future cial and malicious interests. of the Electric Grid. Cambridge, MA. [Online]. Available: Deciding who has access rights to these data and ensur- http://web.mit.edu/mitei/research/studies/the-electric- ing consumers’ privacy will be important considerations in grid-2011.shtml the design and operation of grid communications networks. Many governments have passed laws protecting the privacy Biographies of personal information, though this legislation as yet does Timothy D. Heidel is with the Massachusetts Institute of not specifi cally target electricity usage information. Utilities Technology. and related organizations will have to develop systems and John G. Kassakian is with the Massachusetts Institute procedures to protect the privacy of grid information so as of Technology. to satisfy the concerns of customers and their governments. Richard Schmalensee is with the Massachusetts Insti- The complex issues involved are being actively debated in tute of Technology. several U.S. states. Coordination across states will be neces- p&e

may/june 2012 ieeeIEEE power & energy magazine 2379 Reprinted from November/December 2012 issue of IEEE Power & Energy magazine DC, Come Home

By Brian T. Patterson

MOST DISCUSSIONS ABOUT AC VERSUS DC ELECTRICITY INCLUDE A RETELLING of the famous technical and commercial battle between Edison and Westinghouse/Tesla. It’s a story about everything from electrocuting elephants at state fairs to the ambitious work of electrifying both urban and rural America. It’s the tale of one of man’s greatest engineering feats. It tells of a centralized power generation system based on the dominant use of incandescent light bulbs and ac constant-speed motors. In the end though, it is a retelling of history—and unfortunately, it is a history that doesn’t project well into the future. This article is about making history in the power world. It’s about the rebirth of the earliest form of electrical power—dc power—and its potential to change the world once again. It is being reborn with the help of modern solid-state power electronics technology. The story is also about the work of EMerge Alliance (EA), a nonprofi t open industry association Mthat is creating and promoting new standards based on the contemporary use of dc technology for power generation, storage, distribution, and use. This quickly growing alliance—it already includes more than 100 organizations from industry, government, and academia—was conceived by and is populated with thought leaders motivated by the need for a phase change in the way we think about electric power. EA was born into a world searching for ways to move away from its almost exclusive dependence on synchronous DC Microgrids fossil-fueled centralized power generation and ac macro grid transmission and distribution toward a system that can adaptively and effi ciently include and the Birth of highly distributed, native dc electrical power generation and storage and deliver it an evolved predominance of natively dc loads. In the end, it’s about the “Enernet” a new energy network, or “enernet.” The future of civilized progress is increasingly underwritten by our use of electrons to do work. So their sourcing, distribution, and effi cient use is as fundamental as it is critical to our continued existence on this planet. While seeking better and cleaner ways of collecting and returning energy to and from the environment, it should be fundamentally recognized that electrons play a valuable role in utilizing energy from sustainable sources that can be used to do the vast majority of the work we desire The members of EA propose an expanded use of hybrid ac-dc power systems that are more akin to today’s adaptive and information-rich Internet than they are to yesteryear’s hard-wired party- © STOCKBYTE line telephone system. The application standards they are creating include a family of application area–specifi c dc microgrids that, when interconnected with the “soon to be smart” ac grid, will com- key enabler of today’s Internet.) Such a network should have by the dominant use of highly regulated central power gen- bine to form the aforementioned “enernet.” (This term was fi rst used in a presentation made at the the means to value and effi ciently utilize electrons produced eration and one-way distribution and moves us strongly Massachusetts Institute of Technology (MIT) by Bob Metcalfe, the well-known inventor of Ethernet, a by small private or community-owned renewable generators toward a more democratic, user-centric view that includes on an equal footing with those served up by huge private or distributed local generation and multidirectional networked

Digital Object Identifier 10.1109/MPE.2012.2212610 publicly owned and regulated utilities. This also liberates us distribution and use. Such systems are capable of reshaping Date of publication: 18 October 2012 from the constrained practice of required behaviors imposed the prevailing notion that quality of life around the world

3060 ieeeieee powerpower && energyenergy magazinemagazine 1540-7977/12/$31.00©2012IEEE november/december 2012 november/december 2012 ieee power & energy magazine 61 Reprinted from November/December 2012 issue of IEEE Power & Energy magazine DC, Come Home

By Brian T. Patterson

60 ieee power & energy magazine 1540-7977/12/$31.00©2012IEEE november/december 2012 november/december 2012 ieee power & energy magazine 3611 It’s about the rebirth of the earliest form of electrical power—dc power—and its potential to change the world once again.

will be constrained by the limits and harmful effects of our Even if one is not swayed by the desirability of the current electrical energy systems. It is a view that stimulates improved economics or ecology related to ZEBs—the U.S. innovation and investment in a far more resilient and flexible government, for example, has called for all new commercial network with far less impact on the environment in the short buildings by 2030 and 50% of existing buildings by 2040 to term—and one that seeks harmony with it in the longer term. qualify as ZEBs—perhaps with the addition of local power To accomplish this, EA’s vision includes a system topol- storage, the prospect of making buildings less vulnerable to ogy that links electrical elements starting at the chip level technical and external threats to our national electric grid sys- to electrical elements at the public utility’s generation plants tem is enough of a motivator. It should be noted that ZEBs and everything in between in different ways than they are are not necessarily islanded from the grid—in fact, they are currently configured. In this context, it thus redefines both typically connected to the power network with the concept of the physical topology (how things are connected) and the having the grid provide back-up power supply in the case that logical topology (how things behave). The essential new a ZEB can not meet net zero energy for some reason. Also, physical ingredients of this vision include the concept of during periods of excess generation to load at a ZEB, a point semiautonomous microgrids and the recognition that dc of interconnection to the grid is provided for the ZEB to sell power is the technologically preferred form of electricity to back to the grid under certain circumstances. be used within these grids. It seeks to minimize the wasteful And it’s not just the federal government that’s involved. impact of unnecessary power conversions and recognize that There’s a large movement in the architectural and engineer- the increasing majority of new sources and uses of electri- ing community, called the 2030 Challenge, that is focus- cal power are, for the most part, natively dc or, at the least, ing on building and renovating our way to climate-neutral are not constant-frequency ac and that they make use of dc- buildings by 2030. Many leading firms have already joined based power electronics. this effort, and it is supported by the American Institute of EA acknowledges the technical and social challenges Architects. Trying to combine forces, the U.S. Department of certain to be raised during the pursuit of its vision. When it Energy (DOE) has funded a Zero-Energy Commercial Build- comes to energy—and especially electrical energy and the ings Consortium (CBC) to bring industry leaders, building marvel of our existing 100-year-old ac electrical energy sys- owners, designers, and manufacturers together to identify the tem—many are tempted to disown the challenge of creating challenges and obstacles facing us on this path. a better future as represented by this more balanced vision The biggest aspect of this challenge is that we are not of the role dc can play. But what EA’s members envision is starting from scratch. We can’t just concern ourselves with no more (or less) dramatic, demanding, or risky an undertak- new buildings. Of all the commercial buildings that will exist ing than that associated with the recent transformations of in 2030, 85% are already built. So we need ways of taking our telephony, information, and computing systems during existing buildings and improving their energy use dramati- the creation of today’s Internet. In some ways it should be a cally. Some of these existing buildings are pretty old. And far less ominous transformational job, as the lessons learned nationally, more than 95% of our building stock is small: from crafting the Internet are still fresh in our minds. The under 50,000 ft2. In New York City, for example, the average reward: an electricity network that can enhance business and age of commercial buildings is 50 years. In the mid-Atlantic personal economic growth and ecological well-being in a region, nearly 50% are that old, and they tend to be small, less way that rivals the positive effects of the Internet. than 100,000 ft2. Many haven’t been renovated significantly, particularly for energy retrofits, in decades. Even in Califor- The Future of Zero-Net-Energy Buildings nia, a bellwether state for energy efficiency, there are no effi- The future starts today, so EA’s vision is directly connected ciency standards for existing buildings. This is particularly to the widely discussed contemporary goal of creating zero- problematic in office buildings, where 37% of all commercial net-energy buildings (ZEBs). ZEBs, at least in the context electrical energy is consumed. These statistics indicate the of this article, are buildings that “cleanly” generate enough challenge we face in transforming today’s building stock. energy on-site to equal the energy they use, thus creating a Fortunately, many individuals and groups are now begin- “net zero” balance at the building level. This further creates ning to focus on the challenge of existing buildings in terms the opportunity to lessen the overall impact of energy gen- of sustainability. One group is creating strategies for exist- eration on our economy, climate, and ecology. ing buildings in Philadelphia, where DOE has funded an

3262 ieeeieee powerpower && energyenergy magazinemagazine november/december 2012 We believe the dc-empowered “enernet” will be seen as the heart of what’s coming: a new electric energy age.

“innovation hub” for existing buildings, originally called It is also believed by a growing number of proponents the Greater Philadelphia Innovation Cluster (GPIC) and now that “smart” dc microgrids can help us make better use of the known as the Energy Efficient Buildings Hub (EEB Hub). energy generated, stored, and used at a local level. Whether Another effort, on the same campus as the EEB Hub, is the they are for new on-site energy generation (e.g., solar instal- GridSTAR Center, another DOE-assisted program that is lations) or adding smart devices to monitor energy use or coordinated in part by the Penn State Center for Sustainability. intelligently connecting power to electric vehicles and Several common approaches to designing for low- or zero- battery storage, such approaches give us added control of net-energy buildings, whether they’re new or existing, are energy use at the building level, thus making buildings bet- emerging from these and other similar efforts around the world. ter “partners” with the nation’s smart grid efforts. They also Lighting is often a primary target, both in terms of increas- provide a way to buy centrally generated energy at times of ing day lighting and making the remaining electric light more the day when it is more abundant, temporarily store it, and energy efficient. And mechanical and heating, ventilation, and then use it during peak demand periods. air-conditioning (HVAC) systems are seeing a range of new DC microgrids interconnect a localized grouping of design strategies, including revised ventilation schemes, the electricity sources and loads that predominately gener- use of new technologies like chilled beams and radiant panels, ates, distributes, and uses electrical power in its native dc and the expanded use of variable-speed drive motors for pumps form at low voltages (up to 1,500 Vdc) and operates either and air handlers. So-called “smart building” approaches add connected to the traditional centralized grid or func- controls and building automation. Another focus is on-site tions autonomously as physical and/or economic condi- power generation and storage, including using solar, wind, and tions dictate. Such microgrids are typically connected to other clean energy generation and more efficient power distri- and operate in conjunction with ac macro grids to form a bution throughout a building. In general, design strategies for smart grid. The macro grids are typically utility-operated, new building and deep renovation projects are changing, with centralized generation, wide-area transmission, and local a growing focus on the 2030 challenge. distribution electricity grids that predominately use electrical power in its alternating current (ac) form at high and What’s a DC Microgrid? medium voltages (above 1,500 V) that otherwise require One of the least publicized but most significant ways a build- waveform, phase, and voltage synchronization for mul- ing’s design can change is in the way it is powered. Chang- tiple power source interconnection. A pictogram of typi- ing basic infrastructure has never been the glamorous part cal macro grid-to-microgrid interconnection is shown in of any design challenge. But a building’s power infrastruc- Figure 1. ture is one of the key facets linking building design and Regarding the potential use of such dc microgrids, the renovation to the national electrical “smart grid” effort. A DOE-sponsored Zero Energy CBC has reported that dc new approach to the way we generate and use power in our power may hold the key. The consortium cited dc power buildings—using an infrastructure called dc microgrids—is and dc microgrids as a next-generation technology and linked to how we should make and distribute power at the application that could fundamentally change the way we national electrical grid level—the “macro grid.” power commercial buildings. They noted that dc power can The use of microgrids is partly motivated by the increas- reduce or eliminate ac-to-dc conversions at the equipment ing concern for the strain on and vulnerability of our elec- and building level so that we can save more of the energy trical macro grid system. Witness the 2011 blackout in we need. Southern California due to a utility worker’s mistake in But how much dc power is being used in commercial Yuma, Arizona, and the blackout in the northeastern United buildings? DC power is already used in most of the elec- States in 2003. And these are only the sensationalized events tronic devices you’re familiar with and use in your every- reported by the media; there are thousands of lower-level day work environment, from smartphones to computers and events, power disturbances, and failures recorded each day. printers to your iPad and even the lighting over your head. These random disturbances and linear dynamic failures in But it is also used in the racks and racks of equipment in data the power delivery system are putting their own emphasis on centers that support your information technology systems. creating independent, building-level power self-sufficiency And dc is fundamental to the variable-speed motor drives via such microgrids. that help deliver your heating and air conditioning and to

november/december 2012 ieee power & energy magazine 3633 The dc microgrid-enabled “enernet” vision represents a certain level of decentralization of the nation’s grid and is intended to facilitate the current smart grid overhaul.

system, the native dc power produced by the solar panels is inverted to ac power, Generation Transmission Distribution Smart Meters just so it can be distributed in the building. Then the ac power gets converted back to dc for specific device uses, such as lighting. This double conversion wastes even more Smart Buildings energy. After these double conversions, 15% or more of the solar energy generated Smart Grid is lost. Building The trend toward the use of dc Microgrids devices has been increasing for decades, Why Microgrids? Onsite Renewable and there’s no end in sight. Data center Increase Renewable Energy Availability Energy Generation growth alone approaches a compound Improve Reliability and Security average annual rate of nearly 30%. The Improve Availability in Underserved Markets simple reality is that almost everything Create Open Environment for Energy Innovation based on semiconductor electronics is Local Energy Storage also based on the use of dc power, not ac power. In fact, Virginia Tech’s Center figure 1. Pictogram of macro grid-to-microgrid interconnection. for Power Electronics Systems in Blacks- burg estimates that more than 80% of the electricity used in office buildings the electric vehicles you drive, or are planning on driving, passes through power electronics and experiences one or to and from your buildings in the future. More and more of more conversions between ac and dc electricity. And yet we what uses electricity is utilizing solid-state and semiconduc- don’t have comprehensive standards for how best to gener- tor power electronics based on dc. ate, distribute, and use dc power, the form of electricity The challenge is this: for those dc devices to use the ac most of these devices need. Such standards could provide electricity that is delivered to them, they have to convert ac the opportunity to reduce or eliminate unnecessary power to dc. Simply put, these conversions waste energy. conversions. They would also help simplify and improve the reliability of the electronic equipment involved, reduce The Plague of Wasteful the waste generated when these chargers and converters Power Conversion are put into landfills, and help make the “user experience” A telltale sign of these wasteful conversions from ac to dc simpler by eliminating the many different adapter plugs are the ubiquitous power bricks and chargers cluttering our now necessary. Defining common interfaces and standards work spaces. Every time you plug in your laptop charger, for our dc devices at multiple building levels could help us you’re converting the ac available in the building to the dc simplify how we use power while saving energy, offering power that your computer needs to run. The same thing the potential for 5–15% savings or more, depending on the applies to your smartphone and other personal electronic ac-dc conversions we reduce or eliminate. devices. When you feel these converters get hot, that’s the energy lost in the conversion process. The amount of energy The Critical and Clarifying lost differs with various devices, but is generally 10–25%. Role of Standards and Codes And what’s worse, many of these converters consume nearly Standards and codes play critical roles in moving us toward as much energy when the associated device they’re attached improved energy use. Organizations such as National Fire to is off as when it is on. Protection Association (NFPA), Underwriters Labora- There are other, less obvious ac-dc conversions going on tory (UL), National Electric Manufacturing Association in buildings. One is in the electrically ballasted fluorescent (NEMA), and newer ones (including EA) are working ceiling lights you see overhead. Another takes place within together and have established task groups to address criti- solar installations. For example, in a typical photovoltaic (PV) cal issues for alternative energy, including dc microgrid

3464 ieeeieee powerpower && energyenergy magazinemagazine november/december 2012 distribution systems and electric vehicle charging as well as dc distributed Power Sources electricity storage, natively dc generation systems, and other new dc elec- Utility Battery Fuel Cell trical uses. Model installation codes Solar PV Wind Gen Set Meter Storage Other such as the National Electrical Code (NEC) help assure safety and other Wind 380 Vdc important attributes of energy sys- MPPT Contr. Converter tems; they therefore become critically important to energy use improve- ments. These organizations have com- Facility Power Server and Common Distribution / Collector (380 Vdc Nom) Bus mitted to addressing these new issues proactively and aggressively. Already, Lighting Plug EV HVAC Electronic ICT new sections have been added to the Data Center Loads Loads Charger Loads Loads Desktop 380 Vdc NEC to cover small wind turbine elec- 24 Vdc 380 Vdc 380 Vdc 380 Vdc 380/24 Vdc 24 Vdc trical systems and solar PV systems in ways that minimize any associated safety risk. And for the next code Electrical Loads cycle, hundreds of proposals have been submitted and are being configure 2. New microgrid power distribution topologies in buildings. sidered regarding alternative energy systems, new battery technologies for distributed energy standards for “dc power distribution” that can transport and storage, electric vehicle systems, fuel cells, and low-voltage distribute energy safely and effectively between new energy dc power distribution systems. sources and uses? What are the likely use cases? Product and system standards also play an essential role EA formally—and enthusiastically—took on this chal- is supporting the effective deployment of products for alter- lenge just three short years ago. Based in California, with native energy equipment and systems. Proactive develop- more than 100 member organizations that include national ment of the requirements for appropriate application, design labs, universities, manufacturers, UL, NEMA, and other and test requirements, code compatibility, and the definition industry liaisons, EA has been identifying and creating tech- of standardized product interoperability, system attributes, nology application standards that promote the safe and effi- and usage outcomes are all a part of their clarifying roles. cient use of dc electricity for all types of applications within In the case of dc power distribution systems, UL and EA and around buildings. EA has set out to create open, nonpro- have directly teamed up in a number of formal and informal prietary dc application standards in each of four key areas in ways to develop these much-needed standards. Combining buildings as well as dc microgrid standards that interconnect UL’s extensive technical, research, and government collabo- all the pieces. Each application area is defined as a potential ration competencies with EA’s group of visionary and moti- microgrid that can be implemented by itself, much the way vated leaders in industry has been essential in helping define you can buy a laptop computer and not connect it to a data the preferred alternatives for beginning the fundamentally network but still enjoy improved productivity. In this way, transformative national shift to native dc electricity genera- any or all of the subgrids can be opportunistically created tion, distribution, and use. And together with NEMA and in whatever order makes sense for either new or existing Electric Power Research Institute (EPRI), they have begun buildings. to lay the groundwork for North American and global har- Figure 2 shows how the EA member organizations see monization activities. the potential for a larger common bus collecting and distrib- A good deal work has been done and yet more begun, uting dc power in buildings. It shows a common dc bus that while much work remains for standards organizations. can directly connect a variety of power sources such as solar, But thus far, many of the key standards organizations are wind, fuel cells, and rectified utility ac power, when needed, embracing the challenge before them. Maintaining this early to serve multiple electrical loads—at a number of different momentum and velocity in this regard is vital. dc voltages, high and low, throughout a building. The key application areas (shown in Figure 3) for stan- Getting from Here to There dardization of dc power use in buildings include: The challenge in doing this, of course, lies in the details ✔ interiors and occupied spaces where lighting and con- of defining what’s needed. Both standards and ecosys- trol loads dominate the need for dc electricity tem development rely heavily on use cases. What types of ✔ data centers and telecom central offices with their dc- energy generation should be used? What loads need to be powered information and communications technology addressed? How do we create a new architecture or new (ICT) equipment

november/december 2012 ieee power & energy magazine 3655 DC Interiors: The Occupied Space But buildings are not designed by engineers concerned with energy use alone. They are principally designed by architects, who are also focused on how all aspects of their build- Occupied Data ings will perform for the owners and occupants who are their Space Centers clients. It is important to appreciate that it is not just about energy and energy efficiency but about effective and productive spaces for working, learning, healing, and so on. The sacrifice of good design simply translates into inefficiency of dc Power Microgrids energy in a high order, i.e., poor productivity, under-utilized space, etc. Running a crane motor at a lower horsepower can be more efficient, providing the crane can still safely lift a Building prescribed load of the correct weight and articulation. The Outdoor Services case for properly designed buildings is similar in principle. Electrical system design strategies in ZEBs that implement new standards for power distribution should also help meet a building’s overall goals. A pictogram of EA’s dc standards as implemented in the building interior is shown in Figure 4. An example of an implementation of this standard figure 3. EA’s key dc microgrid building application is the headquarters of the U.S. Green Building Council segments. in Washington, D.C. Another is the new Sustainability Resource Center (SRC) at the University of California, San ✔ outdoor electrical uses, including electric vehicle Diego (UCSD), which was looking for innovation in green charging and outdoor light-emitting diode (LED) building strategies. As a leader in promoting new energy lighting approaches and a regular user of the solar power already on ✔ building services, utilities, and HVAC with vari- campus, the center decided to implement direct dc distribu- able-speed drive (VSD) and electronic dc motorized tion through a new array created just for this project. equipment. Figure 5 shows a solar array put in place for a new com- The thought leaders and major companies involved in mercial interior. The goal was to use this clean energy this groundbreaking work of setting new power standards source directly whenever it was available and not invert it to for buildings include power system and information tech- ac power, avoiding the typical 7–15% energy loss from the nology networking leaders, lighting and building products conversion process. The loads for the solar dc power were innovators, and electromechanical and solar companies. energy efficient but otherwise ordinary lighting and interior The collective focus of leaders across technology and appli- controls. This use of direct dc power led SRC to better-qual- cation areas has jump-started this broad effort, enabling it ity power and greater lighting efficiency. In fact, SRC won to quickly reach the kind of critical mass necessary to meet several awards and a U.S. Green Building Council Leader- our building efficiency and security challenges. ship in Energy and Environmental Design (LEED) Gold rat- Much of the focus is on using clean, renewable power ing under commercial interiors (CI) for the project, which generation (in its native dc generating form), whether that’s included an innovation credit for its “high-efficiency dc biofuel, solar PV, or wind, and on electrical power use, such microgrid.” as green IT and low-energy lighting schemes. Just as we’ve LEED has also started to recognize the importance of leveraged hybrid power systems for cars, we can leverage incorporating flexibility into interior design. Proposed 2012 hybrid power systems for buildings. The transformational credit areas include a specific credit focused on flexible design. coexistence of both ac and dc systems will let us focus on Although the credit is currently envisioned as relating partic- existing buildings as well as new buildings. It also seems best ularly to health care, the importance of design flexibility in to take a modular approach, as the timing of the opportuni- many types of buildings is being more generally recognized. ties to use hybrid power or dc power may differ in various areas and types of buildings. Some areas—such as data cen- DC Data Centers ters—represent a significant potential for dc use when they There are flexible dc power design strategies for other are new, significantly expanded, or considerably updated. spaces within buildings as well. Data and telecom centers Others—such as interior lighting—are already recognized are great candidates. “Green” data centers and “green IT” as big energy consumers that can be updated area by area to have become hot topics. Data centers are huge and grow- use dc. Still others—such as plug loads—may have to await ing energy users in buildings, and there are data centers in standards for the conversion of existing branch wiring to nearly every building, not just the huge server farms created reach all your small miscellaneous equipment uses. for organizations like Facebook and Google. In fact, 99% of

6636 ieee power & energy magazine november/december 2012 ac Branch Power Interiors 208-277 Vac

Optional dc Celling Onsite dc Power Grid HVAC Actuator

Occupancy and Daylight Sensors 380 Vdc Bus AV Devices 24 Vdc Bus and Security IT Wireless Access Device Occupied Space Infrastructure: Power Supply P1 = Ceiling Lights

P2 = Walls

P3 = Furniture Room P3 P1 Controls P4 P2 P4 = Floors P2

dc Loads: Lighting and Controls dc Source: Dedicated Solar Array

figure 5. Lighting and controls on a solar-powered dc microgrid at UCSD. (Source: Armstrong World Industries.)

november/december 2012 ieeeieee powerpower && energyenergy magazinemagazine 3677 Optional Onsite dc Power

100–600 Vdc

Optional Onsite dc Generator and/or Storage MPPT Optional Onsite dc Power

B 380 Vdc Busway (or Cabling) A ac-dc dc Converter ECC ac Input ICT Racks dc Point of Common UPS Connection > ac Flow > dc Flow (Native) Optional Onsite dc Flow (Converted) ac Generator Physical Data Center

figure 6. Pictogram of EA’s dc standards as implemented in a data center.

all data centers are considered “small.” But they contain the In particular, the experience of Duke Energy is instruc- majority of servers using power, according to EPRI. tive for those interested in looking at dc data center design. The challenge is that smaller data centers are operated Duke’s is a typical “small-to-medium-size” data center. The in organizations that often don’t have the internal resources owner has years of experience with ac-based data center to focus on best practices for power distribution and effi- systems. It worked with EPRI to set up a rigorous compara- cient energy use, as they are busy focusing on making sure tive study. the system performs the data management and processing A review of some highlights of the study follows; a full work it is intended to do. But the U.S. Environmental Pro- report is available on the EPRI Web site, along with a video tection Agency (EPA) has estimated that 6 billion kWh of that displays technical details. The big takeaway for Duke energy could be saved each year with only a 10% efficiency was a 15% increase in the electrical efficiency of this data improvement in these data centers. center when running on dc. In its report, EPRI noted that Again, there are new application standards starting to appear average reductions for other smaller data centers could fall for this dc power application. EA’s technical standards group, anywhere within a 10–30% range. led by EPRI and including such companies as ABB, Cisco, Delta, Emerson, Intel, Juniper, and others, is nearing completion Barriers: The Challenges of a new standard whose key elements are shown in Figure 6. of Increased DC Use in Buildings While these standards are being finalized, leading orga- The use of dc power is not without it challenges. These fall nizations and institutions have started to implement proto- into five major categories: type approaches for dc in data centers. These include Duke 1) lack of application and equipment standards for dc Energy, Lawrence Berkeley National Laboratory, and, once power distribution again, UCSD—a national pioneer in new energy research 2) lack of common understanding and basic application and innovation. knowledge of building distribution-level dc

3868 ieeeieee powerpower && energyenergy magazinemagazine november/december 2012 3) differences in safety and power protection device from—or indeed require—the operational duplicity that application comes with efficient electrical storage devices such as bat- 4) lack of a robust ecosystem to support the use of dc in teries and capacitors. building-level electrification But extensive employment of dc microgrids will not hap- 5) an unclear pathway for moving from ac-centric power pen without human intervention. The impediments to their distribution to dc-inclusive distribution schemes. full deployment, as outlined herein, must be dealt with. The first three challenges are being addressed with The standardization and ecosystem development work EA increasing resources by such standards and trade organiza- is doing with the help of others will continue in the areas of tions as EA, the European Telecommunications Standards dc microgrid–supported electric vehicle charging, building Institute (ETSI), the International Electrotechnical Com- services (HVAC, water and waste pumping, compressed mission (IEC), IEEE, NEMA, NFPA, the Power Sources air, and so on), and the definition of dc microgrid and smart Manufacturers Association (PSMA), the Smart Grid Interop- grid connectivity standards. And although this work can erability Panel (SGIP) of the National Institute of Standards be viewed as disruptive unto itself, it is motivated by the and Technology (NIST), UL, and others. As awareness of and desire to bring new order and logic to the very disruptive interest in the potential benefits of dc power use increases, technologies it intends to serve and optimize so do the resources each of these organizations is willing to The aggregated and continuously growing use of elec- dedicate to resolving these challenges. Currently, each of the tronic data and telephony; electric vehicles; solid-state and above-named organizations has a dedicated and clearly iden- electronically driven lighting, motors, and controls; and tified project or program addressing these needs. personal electronics—coupled with the increasing use of The fourth challenge, the lack of an ecosystem, is a clas- natively dc distributed “clean-tech” electricity production— sic “chicken or egg” issue. The power industry, following has already and hurriedly pushed us past a logical tipping Darnell Research and Pike Research, has begun formally point in the ac-dc electrical energy equation. It’s a time for forecasting and tracking the ecosystem growth opportunity true innovation, not the reiterative extension of our past associated with dc microgrids. The numbers they are begin- ways. For as surely as the digitally empowered Internet will ning to report suggest the egg is beginning to hatch. be viewed as the heart of the information age, we believe the The fifth challenge, the transformational path forward, is dc-empowered “enernet” will be seen as the heart of what’s perhaps the least clear of all. But EA, via its strategic plan, has coming: a new electric energy age. plotted a path with a layered approach that allows the transformation to be opportunistic, especially with respect to transforming For Further Reading existing building stock. Dividing building power applications H. Kakigano, Y. Miura, and T. Ise, “Low-Voltage Bipolar- into blocks of subdistribution microgrids, the plan calls for a Type DC Microgrid for Super High Quality Distribution, section-by-section approach over time. Each of the application IEEE Trans. Power Electron., vol. 25, no. 12, pp. 3066– “standards” lays out a subsection that can be converted if and 3075, Dec. 2010. when that part of the building is due for a renovation or updat- C. Marnay and S. Vossos. LBNL/DOE Webinar: Direct ing for other reasons, so the cost of the transformation is largely DC power systems for efficiency and renewable energy inte- offset by normal capital or leasehold improvement spending. gration [Online]. Available: http://efficiency.lbl.gov/news/ While this means that a complete transformation may lbnl_doe_webinar_direct_dc_power_systems_for_effi- take decades, early adopters with fast-churning buildings ciency_and_renewable_energy_integration_0 could be done much sooner. This timing and approach is B. Nordman. What the real world tells us about saving reminiscent of similar transformations seen with the Inter- energy in electronics. Lawrence Berkeley National Labora- net and with wireless telephony. tory Symposium [Online]. Available: http://eetd.lbl.gov/ea/ The dc microgrid-enabled “enernet” vision represents nordman/docs/e3s_nordman.pdf a certain level of decentralization of the nation’s grid and is P. Savage, R. R. Nordhaus, and S. P. Jamieson, “DC intended to facilitate the current smart grid overhaul. The dc Microgrids: Benefits and Barriers,” Yale School of Forestry microgrid changes the model from an almost exclusively cen- & Environmental Studies, 2010. tralized generation and distribution system of electrical power K. Shenai and K. Shah, “Smart DC mircro-grid for effi- delivery to one that is significantly more flexible and accom- cient utilization of distributed renewable energy,” in Proc. of modating of both new alternative sources of on-site electricity IEEE EnergyTech, Cleveland, OH, 2011, pp. 1–6. generation and storage and the new mix of loads that have M. Ton, B. Fortenbery, and W. Tschudi, “DC power increasingly become the norm. It better recognizes that future for improved data center efficiency,” Lawrence Berkeley electrical loads will be even more electronic, more distributed, National Lab, Report, Mar. 2008. and more essential to our economy and way of life. By designing electric power systems that focus better Biography on the needs of digital devices, we improve the networks in Brian T. Patterson is with Armstrong World Industries, which they operate (both power and control) so as to benefit Lancaster, Pennsylvania. p&e

november/december 2012 ieee power & energy magazine 3699 Meet Global Modernization Experts at the

Worldwide IEEE PES Innovative Smart Grid

Technologies Conference Series

The vision of a modernized electrical delivery system — The Smart Grid — promises to revolutionize the production, delivery and use of electricity worldwide.

Experts around the world gather annually at the IEEE Power & Energy Society's global ISGT Conferences to discuss state-of-the-art innovations in smart grid technologies. Each of the ISGT conferences feature special sessions and tutorials on wide ranging topics related to grid modernization, including: • Impact of Smart Grid on Distributed Energy Resources (electric cars, demand response, distributed generation, storage) • Smart Sensors and Advanced Metering Infrastructure Cyber Security Systems (intelligent monitoring and outage management) • Wide Area Protection, Communication, and Control in Energy Systems • Power and Energy System Applications (generation, transmission, distribution, markets, operations, planning) • Energy Management Systems (with applications to smart buildings and home automation) • Smart Grid Devices and Standards • And More...

Networking: Meet and speak directly with utilities, business decision makers, industry leaders, regulators, and entrepreneurs working in grid modernization Research: Noted academics and industry professionals come together to explore ways to make smart grid a cost-effective proposition Opportunities: International speakers report on real success stories and pitfalls – as well as current business opportunities in their region Results: Learn about real-practice technology, deployment experience, and customer acceptance related to grid modernization

The ISGT Conferences present the very best of smart grid technology to the global community with events held annually in North America, Europe, Asia and every other year in Latin America. For information on future ISGT events, plus other PES conferences, events publications and membership, please visit www.ieee-pes.org. 40 ieee power & energy magazine

ISGT2013_PE5.indd 1 4/4/13 2:09 PM Reprinted from January/February 2012 issue of IEEE Power & Energy magazine Enhancing Grid Measurements

DEREGULATION, MARKET TRANSACTIONS, CONGESTION MANAGE- ment, and the separation of functions have created increasing complexity that is making it difﬁ cult to maintain situational awareness and supervision of power system performance over large areas. Past reliability events (such as blackouts) have highlighted the need for better situational awareness and advanced applications to improve planning, operations, and maintenance. The deployment of a continent- wide wide area measurement system (WAMS) is an important part of the solution to these complex problems, but it faces challenges with respect to communications and security. DBy Rakesh B. Bobba, Jeff Dagle, Erich Heine, Himanshu Khurana, William H. Sanders, Peter Sauer, and Tim Yardley

Digital Object Identiﬁ er 10.1109/MPE.2011.943133 Date of publication: 13 December 2011

january/february 2012 1540-7977/12/$31.00©2012 IEEE ieeeIEEE power & energy magazine 4671 Wide Area Measurement System (WAMS) local utility and exchanged regionally with other utilities In its recent book A Century of Innovation, the National and reliability coordinators using the Inter-Control Cen- Academy of Engineering listed widespread electrifi cation ter Communications Protocol (ICCP). Typically, SCADA fi rst on its list of the top 20 engineering achievements of systems acquire sensor data every 2–4 s. Since the data are the 20th century. Although the highly interconnected North not time-stamped at the point of measurement or acquired American electrical power grid is rightly hailed as a great synchronously, they do not capture the state of the system engineering feat, managing and operating it in a reliable at a given moment in time. Rather, the data can provide a and safe way remains a challenge that involves many com- good estimate of the system state, assuming that the system plex technical tasks that must be accomplished at different is in quasi-steady state. While the grid operates in quasi- time and geographic scales. Such tasks include continuous steady state most of the time, increased stress on the sys- feedback control, protection and control mechanisms that tem means that operators’ views of it must be more fi ne- operate every few milliseconds at substations, state estima- grained and cover a wider area, moving across multiple tors and contingency analysis processes that operate every organizations in order to improve the reliability and stabil- few minutes, and generation dispatch decisions to bring ity of the grid. power plants online or take them off-line based on load or A WAMS can be defi ned as a system that takes measure- expected demand. In earlier years, control areas were ver- ments in the power grid at a high granularity, over a wide tically integrated in all respects and acted as quasi islands area, and across traditional control boundaries and then responsible for fl ow control. The interconnections among uses those measurements to improve grid stability through control areas enabled emergency fl ow paths and occasional wide area situational awareness and advanced analysis. economic benefi ts. Knowledge beyond control area bound- Certain power system measurements cannot be meaning- aries was limited and often depended on slow point-to-point fully combined unless they are captured at the same time. communications. Modern operations are far more complex, An important requirement of a WAMS, therefore, is that the as reliability constraints require extensive congestion man- measurements be synchronized. A high sampling rate— agement with signifi cant economic consequences. Further, typically, 30 or more samples per second—is particularly given that various parts of the system are owned and oper- important for measuring system dynamics and is another ated by many independent entities, reliable operation of the important requirement of a WAMS. Certain elements of a grid depends on those tasks being accomplished at a range WAMS have existed in rudimentary forms in the Western of geographic granularities and with a high level of coordi- Interconnection since the early 1990s, and the cascading nation among the various entities that manage and operate outage of 1996 provided the impetus for further WAMS the grid. With the passage of the Energy Policy Act of 2005 development. in the United States, the Federal Energy Regulatory Com- Many advanced applications can take advantage of the mission (FERC) and North American Electric Reliability measurement capability provided by a WAMS, including: Corporation (NERC) have been given additional authority ✔ Wide area monitoring: High-speed, real-time mea- to regulate electric power entities to ensure reliable opera- surement data and analysis are essential to achieve tion of the grid. wide area visibility across the bulk power system Historically, the grid has been very reliable. While minor for entire interconnections. Time-synchronized mea- outages have been fairly common, large-scale and wide- surements from geographically dispersed locations spread outages have been rare, and most customer inter- throughout a large region enable better operational ruptions occur within relatively localized distribution infra- awareness of the real-time condition of the grid and structure. An increasing demand for electricity has not been allow operators to make better-informed decisions. accompanied by increases in transmission capacity, how- ✔ Real-time operations: Real-time operations improve ever, putting growing pressure on the reliability and safety of operators’ understanding of how to take advantage of the grid. Recent large blackouts and outages, such as the 14 the newfound visibility of grid dynamics, including August 2003 blackout in the Northeast and the 26 February interarea oscillatory modes and methods for damping 2008 outage in Florida, stand as evidence. The fi nal report and stabilizing frequency oscillations. by the U.S.-Canada Power System Outage Task Force on ✔ Improved accuracy of models: Time-synchronized the August 2003 blackout pointed out that the job of main- wide area measurements continue to be very valu- taining the system reliably had become harder because of able for improving the accuracy of planning models reduced transmission margins. The report recommended the by precisely correlating simulation output with ob- development and adoption of technologies, such as WAMS, served system behavior under a variety of conditions. that could improve system reliability by providing better Improved planning models enable better assessment wide area situational awareness. of system behavior and will permit a more complete Traditionally, sensor readings from substations in utili- assessment of dynamic performance issues, such as ties are sent via a communication network to the supervi- disturbance response, voltage and frequency response, sory control and data acquisition (SCADA) systems in the and stability performance.

6842 IEEEieee power & energy magazine january/february 2012 It is crucial to secure a WAMS to ensure the availability and integrity of the data it carries since monitoring and control applications may rely on those data.

✔ Forensic analysis: Synchronized measurement data an effort under way at NASPI to develop such an infrastruc- collected at high sampling rates are also helpful for ture, known as the NASPI Network (or NASPInet); it is forensic analysis of blackouts and other grid distur- being designed to be secure, standardized, distributed, and bances. Because the data are collected at high speed capable of supporting future needs. One of the key require- and are time-synchronized, their analysis can lead to ments for this communication infrastructure is that it must faster and better understanding of precise sequences be able to support different classes of applications with of events. varying levels of latency, accuracy, availability, message rate, and time-alignment requirements. For example, one NASPInet class of applications, such as feedback control, places strict Phasor measurement units (PMUs), developed in the early requirements on the latency, availability, and accuracy of 1990s, were among the fi rst devices that could monitor the data, while another class of applications, such as post-event grid in a synchronized way and produce coordinated phasor analysis, values accuracy, availability, and sampling or measurements, also known as synchrophasors. A GPS clock message rate more than latency. The communication infra- signal is the most commonly used mechanism for providing structure should therefore be able to support different qual- the time reference needed for synchronizing PMU measure- ity-of-service (QoS) classes for traffi c and should be able to ments. Another distinguishing feature of PMUs, in addition prioritize one class over another. Conceptually, as shown to synchronized measurements, is their sampling rate, which in Figure 1, NASPInet is made up of two components: the ranges from 30 samples per second up to 120 samples per phasor gateway (PGW) and the data bus (DB). The PGW second in current implementations. Even the low end of that is envisioned as a utility’s or control center’s sole point of spectrum, 30 samples per second, is an order of magnitude access to the DB. It will let the utility or control center share higher than the sampling rate of SCADA systems, meaning its synchrophasor data and obtain synchrophasor data from that PMU devices are capable of measuring system dynamic other utilities or control centers. The idea is that the data performance in a manner that is not possible with traditional sharing will follow a publish-subscribe pattern, according SCADA systems. Their synchronized monitoring and high to which a gateway that wishes to share data will publish sampling rate make PMUs the ideal class of monitoring them so that authorized gateways may subscribe to the pub- device for a WAMS. Traditionally, PMUs were stand-alone lished stream and receive the data. Each PGW will need devices, but today many devices such as relays and digital to manage QoS and administer cybersecurity and access fault recorders (DFRs) also have the ability to produce syn- rights for the data it is sharing. The DB is envisioned as a chrophasors at high sampling rates. wide area network that connects all the PGWs and provides Realizing the need for wide area measurement, moni- the associated services for basic connectivity, QoS manage- toring, and control across the continent and the potential ment, performance monitoring, and cybersecurity. of synchrophasor technology to enable these functions, the U.S. Department of Energy (DOE), the National Electric NASPInet’s Cybersecurity Reliability Council (NERC), and a range of electric utilities Requirements and Challenges and other organizations formed the North American Syn- It is crucial to secure a WAMS in order to ensure the avail- chroPhasor Initiative (NASPI) in 2007. NASPI’s vision is ability and integrity of the data it carries, which in turn affect “to improve power system reliability through wide area mea- the reliability of the power grid, since monitoring and con- surement, monitoring and control,” and its mission is “to cre- trol applications may rely on those data. The core security ate a robust, widely available and secure synchronized data goals of a WAMS are to ensure the availability, integrity, and measurement infrastructure for the interconnected North confi dentiality of the data and the underlying computing and American electric power system with associated analysis communication infrastructure. Furthermore, the data secu- and monitoring tools for better planning and operation and rity should be ensured end to end, that is, from the time of improved reliability.” data origination at the sensor to the time of use by a control Realizing a continent-wide WAMS requires not only or monitoring application. Achieving these security objec- synchronized measurement but also a high-speed commu- tives is easier within a single organization (that is, from the nication infrastructure that enables secure sharing of syn- measurement sensor to the control center owning or manag- chronized monitoring data among control centers. There is ing the sensor) than it is for an infrastructure distributed over

january/february 2012 ieeeIEEE power & energy magazine 4693 keys among entities, a certifi - cate authority that issues digital NASPInet certifi cates that are trusted by members of NASPInet, or just a Data Bus simple secure and authenticated directory service in which entities like gateways can post their PGW PGW PGW public keys or digital certifi cates. Utility A Once an entity is authenticated, its authorization to access the data needs to be verifi ed. Access APPS Historian APPS Historian control lists (ACLs) associated Other with data are often used to spec- Utilities and ify the list of entities authorized PDC Monitoring Centers to access the data. In such a case, Monitoring authorization checking involves Center 1 ensuring that the authenticated PMU PMU PMU entity is listed in the ACL associated with the data. In addition PMUs to access control for data, which is enforced by the data owner’s PGW Phasor Gateway PDC Phasor Data Concentrator APPS Applications gateway, there must be an access control mechanism at the network level to limit access only figure 1. A continent-wide WAMS and NASPInet concept. to authorized entities. In the case of NASPInet, the network-level a wide area like NASPInet, which is envisioned as enabling access control is to be administered and enforced by the data sharing across organizational boundaries and helping DB function. to realize a continent-wide WAMS. Here we highlight the security requirements of NASPInet, the many security func- Integrity and Confidentiality tions and mechanisms needed to meet them, and the chal- of Measurement Data lenges of realizing them. When sending data to an authenticated and authorized entity, it is necessary to protect the data’s confi dentiality and integ- Authentication, Authorization, rity. It is important to protect measurement data confi denti- and Access Control ality from malicious eavesdroppers because such data may Owners of sensor data would not want anyone other than contain information sensitive for the market or reveal sen- authorized data-sharing partners to gain access to their data. sitive information about the grid that could be exploited to Toward that end, they need to be able to ensure that an entity disrupt grid operation. Encryption primitives are commonly with which they are communicating is what it claims to be used to protect data confi dentiality. Similarly, it is important and that it is an authorized data-sharing partner. In other to protect measurement data integrity as inadvertent or mali- words, they need to be able to authenticate the entity with cious modifi cation of measurement data could lead opera- which they are communicating and verify that it is an autho- tors or applications to make catastrophic decisions. A typical rized entity before they share their data. Similarly, a data approach is to use symmetric-key-based cryptographic mes- receiver may want to authenticate the entity from which it sage authentication (or integrity) codes to detect data tam- is receiving data to make sure that the incoming data are pering and to ensure that only legitimate data are accepted legitimate. for use. Another notion, closely related to data integrity A naive strategy for authentication is to create an out- protection, is that of data origin or source authentication, of-band security and communication context and use it to which assures a receiver that data indeed originated at the establish communications and perform authentication. A entity from which the receiver was expecting data. As the more dynamic and scalable approach is desirable, how- symmetric key used to compute the cryptographic message ever, and could include leveraging a trusted third-party authentication code is shared only between the sender and service to establish trust and long-term cryptographic receiver, in a two-party setting (one sender to one receiver), keys among the WAMS entities, such as their PGWs. verifi cation of a message authentication code assures the The third-party service could be a Kerberos-like service receiver that the data were not tampered with in transit and that helps establish long-term symmetric cryptographic that the data originated at the expected sender. Thus, in a

4470 IEEEieee power & energy magazine january/february 2012 two-party setting symmetric-key- based cryptographic message authentication codes provide both PGW PGW data integrity protection and data 1 1 origin authentication. Encrypt{Data, K1} Since a data owner might PGW2 PGW share the data with multiple enti- 2 PGWS PGWS ties at the same time, for effi - Encrypt{Data, K2} ciency reasons, a WAMS should Encrypt{Data, KG} support not just unicast or two- Encrypt{Data, Kn} party data sharing (that is, one sender to one receiver), but also multicast or multiparty data shar- PGWn PGWn ing (that is, one sender to multiple receivers). So multicast integrity and confi dentiality issues PGWS Sending Phasor Gateway PGW1 Receiving Phasor Gateway must also be addressed. Whereas the cryptographic key is shared K1 Through Kn: Pairwise Keys KG: Group Key between two parties in two-party (a) (b) or unicast data sharing, in a multicast setting, the cryptographic keys used for encryption and figure 2. (a) Unicast versus (b) multicast data sharing. message integrity protection are shared among a group of entities. Support for multicast data sharing can make data sharing guarantees is to use digital signatures, which use asymmetric effi cient, as shown in Figure 2. With support for multicast keys (public-private key pairs), instead of symmetric-key- data sharing, the sender only needs to encrypt the data and based message authentication codes. The data sender would compute the message authentication code once using the digitally sign the data using a private key. When the signature group keys; the sender then transmits the data only once, is verifi ed as valid using a public key, which corresponds to using the underlying multicast primitive. In contrast, with- the private key and is distributed to all group members, the out support for multicast data sharing, a sender will have to receivers can be sure that the data originated at that sender, as encrypt the data and compute the message authentication only that sender had access to the private key used to gener- code separately for each receiving entity, with different ate the signature. Unfortunately, digital signatures are expen- keys for each receiver; then the sender must transmit the sive in terms of both computation and communication, and data as many times as there are receivers, increasing both it is a challenge to meet real-time requirements when every communication and computation costs. measurement is digitally signed. While multicast data sharing reduces communication Schemes to amortize the signature cost over multiple and computation costs, it adds additional complexity for key measurements exist and could reduce the overhead associ- management and for data origin authentication. Specifi cally, ated with digital signatures. By defi nition, however, those in a multicast setting, when a symmetric-key-based message schemes provide data source authentication for a group of authentication code is verifi ed as valid by the receiver, the measurements, and the group size must be picked care- receiver is assured that the data haven’t been tampered with fully to reduce the costs per measurement while provid- by anyone outside the multicast group. The receiver cannot be ing data source authentication at a meaningful granular- sure, however, that the data originated at any particular mem- ity. Furthermore, loss of one or more measurements in the ber of the group, as the symmetric key used to compute the group might mean that the signature cannot be verifi ed. cryptographic message authentication code is shared among To pursue this approach, it would be necessary to design all the multicast group members and any one of the members mechanisms to prevent or deal with loss of measurements. is technically capable of generating a valid message authen- An alternative to schemes that rely on asymmetric-key- tication code. As a result, the receiver may have to rely on based (or public-key-based) cryptographic primitives would other means of data origin authentication. In a secure, well- be schemes that use symmetric-key-based cryptographic confi gured, and well-monitored network, the receiver may be primitives but use time synchronization between entities to able to rely on the network layer to provide assurances about create the asymmetry necessary for data origin authentica- the origin of data packets. tion. But such schemes often introduce a great deal of key One straightforward way to achieve data origin authenti- management complexity and, like the amortized signature cation in a multicast setting without relying on network-layer schemes, result in verifi cation delays.

january/february 2012 ieeeIEEE power & energy magazine 4715 Nonrepudiation delays in data delivery. That further complicates multicast Reliability coordinators and other regional entities may security solutions, especially for message source authentica- need to make decisions based on data from a WAMS that tion and integrity protection, and could suggest a need to will have economic consequences for their members. They deploy sophisticated key management solutions that allow may be held accountable for those decisions, and they might timely source authentication of each data packet, utilizing have to defend them. They may therefore need to use an symmetric-key solutions that are signifi cantly more effi cient approach that not only protects data integrity but also pre- than asymmetric-key-based digital signatures. vents the data source or sender from denying having sent the data. In other words, they need a nonrepudiation property. Data and Infrastructure Availability Digital signatures are commonly employed to provide non- To ensure data availability, it is necessary to ensure the repudiation. But as mentioned earlier, digital signatures are integrity and availability of the underlying computing and expensive in terms of both computation and communication, communication infrastructure. While a carefully thought- and it is diffi cult to meet real-time requirements when every out fault-tolerant design will help, such a design by itself measurement is digitally signed. While signature amorti- will not be suffi cient. As part of a critical infrastructure, zation schemes could perhaps be applied here as well, it is NASPInet will be an attractive target and must be resilient in general harder to provide nonrepudiation via alternative against cyberattacks and intrusions by adversaries rang- schemes that use symmetric-key-based cryptographic prim- ing from novices to nation-states. There should be mecha- itives but rely on time synchronization to create asymmetry. nisms in place to protect against cyberattacks, to monitor for and detect cyberattacks and intrusions, and to respond Key Management to and recover from cyberattacks and intrusions in a timely An important aspect of NASPI’s network security solutions will manner. Network access control (NAC) is an example of be key management: the ability to generate, distribute, revoke, a network-layer protective mechanism that prevents any- and update cryptographic keying material among NASPInet one other than authenticated and authorized devices and entities. The cryptographic keying material might be used to entities from accessing the measurement communication provide various security properties such as entity authentica- infrastructure. Secure logging, along with the associated tion, data confi dentiality or integrity protection, and nonrepu- auditing or monitoring functions, is an example of a mech- diation. Long-term cryptographic keys established between anism that can help with investigation and recovery from entities, either with the help of a trusted third party or using an intrusions. out-of-band mechanism, are often used for secure distribution While it is clear that data and infrastructure need to be pro- of keys for confi dentiality and data integrity protection. tected, the level and kind of protection depend on each situa- Key management is more complex in multicast set- tion’s relevant threat model and risk assessment. For instance, tings than in unicast settings, as the cryptographic keys are must data be kept confi dential from everyone other than the shared among a group of entities. When the group composi- intended recipients, or is it suffi cient to keep the data confi - tion changes (that is, when a member of the group leaves or dential from anyone outside the measurement network? The a new member joins), group keys need to be updated in a latter scenario might require simpler multicast security solu- timely manner. Existing group keys need to be revoked and tions than the former. Furthermore, even if data must be kept new group keys distributed rapidly, without disrupting the confi dential from everyone besides the intended recipients, real-time measurement streams. Furthermore, in a multicast does the trust model assume that NASPInet organizations are setting, a sending gateway may have to maintain a group key honest, or does it assume that they are potentially malicious? for every data stream that it is sharing; that would not be The former scenario might lead to simpler security solutions necessary in a unicast or pairwise setting, in which a pair- than the latter. Likewise, depending on the kind of security wise key between the sending and receiving gateways might services available from the underlying network layer or the be suffi cient to protect all data shared between them. level of trust in the underlying network layer, security solu- While multicast networks and their associated security tions at higher layers such as the application layer may end challenges are common to several problem domains (such as up being simpler. For example, if the network layer is able to audio and video conferencing, mobile ad hoc networks, and provide data origin authentication, then symmetric-key-based wireless sensor networks), NASPInet presents more strin- schemes may be suffi cient to provide data confi dentiality and gent real-time requirements on data delivery. For certain integrity protection, thereby reducing complexity at the appli- control applications, latency requirements can range from cation layer. The requirements, threat model, and risk must ten to a few hundred milliseconds for continent-scale appli- therefore be carefully analyzed, as they have major implica- cations. Such real-time delivery requirements have a signifi - tions for the security design of the system, including the poli- cant impact on security solutions. As discussed above, it is cies, components, and tools needed for an appropriate solu- not feasible to digitally sign every data packet for data source tion. That said, once a security solution has been deployed, authentication and integrity protection, as the computation it is far easier to relax its security requirements than to make and communication overhead could lead to unacceptable them more rigorous.

4672 IEEEieee power & energy magazine january/february 2012 The NASPI community is making steady progress toward achieving the vision of a continent-wide WAMS.

NASPInet: Current Status North American SynchroPhasor Initiative. (2009, May). and Future Directions Data bus technical specifi cations for North American The NASPI community is making steady progress toward Synchrony-Phasor Initiative network. [Online]. Available: achieving the vision of a continent-wide WAMS. Through https://www.naspi.org/site/Module/Team/dnmtt/naspinet/ its Smart Grid Investment Grant (SGIG) awards, the DOE naspinet_databus_fi nal_spec_20090529.pdf is investing signifi cantly in the deployment of hundreds of North American Synchro Phasor Initiative. (2009, May). PMUs, along with the associated communications infra- Phasor gateway technical specifi cations for North Ameri- structure, across the United States. can synchro-phasor initiative network. [Online]. Available: The realization of the vision of a continent-wide NASPInet https://www.naspi.org/site/Module/Team/dnmtt/naspinet/ will not be trivial, however. It faces many challenges, both naspinet_phasor_gateway__fi nal_spec_20090529.pdf technical and business-related. Potential options for creat- D. Novosel, V. Madani, B. Bhargava, K. Vu; and J. Cole. ing such a network range from leveraging the public Internet (2008, Jan.–Feb.). Dawn of the grid synchronization. IEEE to leased multiprotocol label switching (MPLS) circuits to Power Energy Mag. [Online]. 6(1), 49–60. Available: http:// utility-controlled fi ber networks to completely isolated high- ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4412 speed optical networks. Using the public Internet or other 940&isnumber=4408514 shared media poses QoS and security challenges. On the A. G. Phadke and R. M. de Moraes. (2008, Sept.–Oct.). other end of the spectrum, one can provision and manage a The wide world of wide-area measurement. IEEE Power completely isolated, private, high-speed network, but doing Energy Mag. [Online]. 6(5), 52–65. Available: http://ieeex- so could be prohibitively expensive and still retain security plore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4610295& and QoS issues such as the identifi cation of a trustworthy isnumber=4610275 entity that would own and/or manage that network. In rec- R. Bobba, E. Heine, H. Khurana, and T. Yardley. (2010, Jan.). ognition of the challenges, many of the SGIG awardees with Exploring a tiered architecture for NASPInet. Presented at the PMU projects are focusing on increasing PMU deployment Innovative Smart Grid Technologies Conf. (ISGT) [Online]. and utilizing the data from those deployments at a regional pp. 1–8, 19–21. Available: http://ieeexplore.ieee.org/stamp/ level. The idea is to grow these regional systems into a conti- stamp.jsp?tp=&arnumber=5434730&isnumber=5434721 nent-wide WAMS enabled by NASPInet in the future. D. E. Bakken, A. Bose, C. H. Hauser, D. E. Whitehead, As part of the ongoing PMU data infrastructure devel- and G. C. Zweigle. (2011, June). Smart generation and trans- opment and deployment efforts, several key cybersecurity mission with coherent, real-time data. Proc. IEEE [Online]. requirements will be addressed. In the preceding section 99(6), 928–951. Available: http://ieeexplore.ieee.org/stamp/ there was a progression from basic security and functional stamp.jsp?tp=&arnumber=5768095&isnumber=5768087 requirements to more advanced ones, e.g., from unicast security to multicast security and from data origin authenti- Biographies cation provided or supported by the network layer to applica- Rakesh B. Bobba is with the University of Illinois at tion-level data origin authentication. Correspondingly, solu- Urbana-Champaign. tions that address the basic requirements are less expensive Jeff Dagle is with the Pacifi c Northwest National Labora- and better understood than those that meet more advanced tory. requirements. Since the infrastructure is at an early stage of Erich Heine is with the University of Illinois at Urbana- development, there is an opportunity to carefully consider a Champaign. wide range of threats and security requirements so that secu- Himanshu Khurana is with Honeywell Automation and rity solutions can be built in from the ground up. This will let Control Systems Labs. the WAMS be realized as a resilient critical infrastructure William H. Sanders is with the University of Illinois at that can withstand sophisticated, targeted cyberattacks. Urbana-Champaign. Peter Sauer is with the University of Illinois at Urbana- For Further Reading Champaign. G. Constable and B. Somerville, A Century of Innovation: Tim Yardley is with the University of Illinois at Urbana- Twenty Engineering Achievements That Transformed Our Champaign. Lives. Washington, DC: National Academy Press, 2003. p&e

january/february 2012 ieeeIEEE power & energy magazine 7347 Life with Power & Energy.

Life without it.

Join the IEEE Power & Energy Society and Don’t be Left in the Dark

Your membership in We help our members to be IEEE PES enables you to: successful by providing: • Tackle broad-reaching • Up-to-date information on current challenges trends and the latest technology • Become recognized as a • Industry insight through Power & thought leader by your Energy magazine, technical reports industry peers and peer-reviewed publications • Develop contacts that will prove • Compelling programs and useful throughout all stages of networking opportunities at our your career conferences and events • Be a part of the very active and • Opportunity to meet, network and engaged global PES Community collaborate with local members via our vibrant chapters

To learn more about the IEEE Power & Energy Society, including the many other membership benefits, please visit www.ieee-pes.org.

48 ieee power & energy magazine IEEE Power & Energy Society 445 Hoes Lane Piscataway, NJ 08854 USA

IEEE_MemberAds.indd 2 4/1/13 3:53 PM Reprinted from January/February 2012 issue of IEEE Power & Energy magazine

By Julie Hull, Himanshu Khurana, Tom Markham, and Kevin Staggs

THE USE OF SUPERVISORY CONTROL AND DATA acquisition (SCADA) became popular in the 1960s due to the expense of manual monitoring and control and an increase in Cybersecurity the complexity of the systems. The blackout of 1965 in the northeastern United States prompted the U.S. Federal Power and the Modern Commission to urge passage of the Electric Power Reliability Act of 1967, which would have mandated closer coordination Electric Grid among regional coordination groups. The National Electric Reliability Council was formed in 1968. These events also drove the development of large energy management systems for transmission SCADA. Early SCADA protocols were built Ton electromechanical telephone switching technology. At that time, the goal of communications security was to ensure that

Digital Object Identiﬁ er 10.1109/MPE.2011.943251 Date of publication: 13 December 2011

january/february 2012 1540-7977/12/$31.00©2012 IEEE ieeeIEEE power & energy magazine 4419 There is an increasing amount of evidence showing that attackers are now focusing on control systems, operating with varying motivations and intentions.

the command got to the mechanism for control (this secu- Even with this type of layered protection, the system is rity was typically implemented through repetition). still vulnerable. The National Electric Sector Cyber Secu- Subsequently, SCADA moved to digital communica- rity Organization (NESCO) has published a white paper, tions, and the use of parity bits and checksums became “DNS as a Covert Channel Within Protected Networks,” that prevalent for error checking and is still common today in demonstrates DNS data exfi ltration techniques that do not the fi eld. Many protocols were in use; typically, each manu- require direct connectivity to any external resource from the facturer created its own, and some end users did the same. targeted device. An attacker can get information from the The network architecture was typically hierarchical, with RTU out through the corporate fi rewall and create a com- the substations isolated. In the 1980s, a number of groups munication path back to that device, highlighting the need to began working toward a common set of standards for proto- watch outgoing fi rewall data. cols. The introduction of master stations and RTUs neces- There is an increasing amount of evidence showing sitated local area networks (LANs) and wide area networks that attackers are now focusing on control systems. They (WANs), both of which can utilize more than one linking are operating with varying motivations and intentions, technology (e.g., satellite, telephone, wireless, power line including cybercrime, extortion, and warfare. In the area of carrier, fi ber optics, or microwave) to connect RTUs to mas- cyberextortion, for example, we have been warned for years ter stations. The RTUs typically perform actions requested about the increased cyberextortion being practiced on elec- by the master station and report out-of-bounds conditions; tric utilities in Africa, Europe, India, and Mexico, where some also perform local control, logging, and reporting. criminals threaten to cut off power if they are not paid. This diversity of communication media and protocols has In a recent paper published by McAfee and the Center for left its legacy in the fi eld and has made it diffi cult to secure Strategic and International Studies (CSIS), “In the Dark, the infrastructure. Crucial Industries Confront Cyberattacks,” 200 industry More recently, there has been a merging of the automa- executives from critical electricity infrastructure enter- tion and business networks, with a linking of the automa- prises in 14 countries were surveyed. The survey group was tion WAN to the corporate network and, in some cases, an composed of IT executives in the energy, oil and gas, and extension of these networks into customer sites. The use of water sectors whose primary responsibilities include IT intelligent electronic devices (IEDs) has also become com- security, general security, and industrial control systems. mon and has caused yet another shift in the communica- According to the paper, “One in four survey respondents tions architecture. Traditionally, the system was serial and have been victims of extortion through cyberattacks or hierarchical in nature: users communicated with the sub- threatened cyberattacks.” And it follows that once a crimi- station through an RTU or data concentrator (which then nal fi nds an avenue of attack that works, the attacker tends communicated with meters, relays, equipment, and so on), to use it again and expand the list of victims. Nation-states or users communicated directly with feeder devices (reclos- have also been accused of using cyberattacks on control ers, switch controllers, and other equipment). With the systems; such intrusions include the Russian cyberattack advent of IEDs, there is much more networked information, on Georgia’s pipelines and the alleged 2007 Russian attack which then fl ows up to substations and/or feeder devices on Estonia. In Kenneth Geers’s paper “Cyberspace and the using serial, direct-connect, wireless, and packet-switched Changing Nature of Warfare,” the author outlines the stra- circuits. The substation communication is often through a tegic reasons why cyberwarfare is on the rise with respect router on a LAN, along with the human-machine interface to the electric power sector, including the fact that the Inter- (HMI), data concentrator, equipment, and relays and may net is vulnerable to attack. Many may argue that the elec- offer remote access to feeder-level devices. Figure 1 illus- tric power system is not on the Internet. In many cases it trates a typical architecture for modern SCADA systems. is, however. Even more common is the scenario in which Since many electric grid systems are now built using tra- a device without a direct Internet connection is connected ditional IT hardware and software, their attack surface is to the Internet at some point in its life cycle for software much larger, making them more vulnerable to cyberattack. or fi rmware updates, confi guration, or maintenance. Or With that in mind, deployed systems use a layered protec- the device may interface with another device (e.g., a laptop tion approach, with multiple levels of fi rewalls and “demili- or USB drive) that has been on the Internet and carries an tarized zones,” as seen in Figure 1. infection or malicious code.

4250 IEEEieee power & energy magazine january/february 2012 The methods used for a cyberattack vary depending on the realizing secure control systems and some approaches that attacker and the motivation. Some attackers are physically might work. We discuss control system security in general able to access a site through local surveillance, by browsing and use the example of modern SCADA systems to illustrate wireless networks within close physical proximity or even certain ideas. Finally, we review some key ongoing efforts in by accessing the site physically as part of the cyberattack; the control system security area involving the U.S. govern- some perform the entire attack from a computer that could ment, industry, and academia. be 10,000 mi away. In any case, typically the fi rst step is to gather as much information as possible through publicly available sources Legend (say, from the Internet). The Internet can provide names, phys- Denotes Attack Point ical layouts, installed equipment, Note: There Are Many Attack data useful for social engineering, Vectors Not Noted on This and port scanning for other data. Diagram, Including Drivers, Etc. Internet After this reconnaissance, adversaries target specifi c components and systems using malware that Corporate exploits vulnerabilities to gain Workstation Server access to the system. There are Firewall/DMZ many attack vectors for obtaining Business/ access to a SCADA system, from Corporate a brute-force attack through the Network business network to intercepting Control Firewall nonencrypted communications Center and playing them back, either to mimic control actions or to mask from the operator’s view the control actions that are really being SCADA performed. Attacks can vary from Server the relatively simple—such as that of the disgruntled former contrac- tor who used existing privileges Front-End and gained access to the control Processor system of a sewage treatment facility in Australia, then fl ooded the surrounding area with mil- WAN SCADA Network Feeder lions of liters of untreated sew- Devices age—to the Stuxnet worm, which was purportedly an attack on the Iranian nuclear industry using highly sophisticated malware and Local HMI Network Local HMI Network several zero-day vulnerabilities. Interface Interface Input/ In the rest of this article we Output look at cybersecurity objectives Points and properties and discuss meth- LAN ods for minimizing cyberattacks RTU RTU as well as detecting and respond- Equipment Remote Monitor Feeder ing to attacks that do succeed. We Access Devices Input/ Input/ then describe some cryptographic Output Output Meters Relays protocols commonly used to real- Points Points ize desired security properties such as confi dentiality and integ- Subdivision A Subdivision B rity. With this background in mind, we explore the challenges of figure 1. Typical security architecture for SCADA systems.

january/february 2012 ieeeIEEE power & energy magazine 5431 What Are the Goals and attacks. For example, encryption tools help provide con- Objectives of Cybersecurity? fi dentiality, cryptographic message authentication tools Cybersecurity tools and techniques are aimed at achieving help provide integrity, and redundancy helps provide avail- three primary properties, namely, confi dentiality, integrity, ability. Secure software and hardware development tech- and availability (CIA). Confi dentiality is the property that niques are also an essential form of protection. Given the ensures that only authorized entities have access to sensitive complexity of today’s systems, vulnerabilities are likely to information. For example, electricity market data and trans- remain after development that can be exploited by adver- action information are considered sensitive and should only saries despite the use of advanced protection systems. To be accessible to authorized market agents and not to other deal with this, detection tools observe network and system entities such as system operators. Integrity is the property behavior to identify malicious activities and attacks. For that ensures that any unauthorized modifi cations to data and example, intrusion-detection systems may look for mal- information are detected. For example, an adversary should ware signatures on the network. Finally, response tools are not be able to modify sensor data without detection. Avail- employed to enable administrators to deal with detected ability is the property that ensures that critical systems and attacks and activities. For example, such tools may allow information must be available when needed. For example, dynamic changes in fi rewall policies in order to limit infor- communication networks supporting wide area measure- mation fl ow to and from adversaries to contain an attack. ment systems must be available to deliver data and informa- Collectively these protection, detection, and response sys- tion (e.g., synchrophasor measurements) even in the pres- tems create an ecosystem in which secure and trustworthy ence of malicious activity such as an adversary launching operations can be executed. Typically, these technical solu- a denial-of-service (DoS) attack . For critical infrastructure tions are used in conjunction with appropriate training for such as the electric grid, availability and integrity are typi- people and the use of well-defi ned processes to form a com- cally considered to be more important than confi dentiality. prehensive solution. Other security properties of interest to control systems include nonrepudiation and privacy. Nonrepudiation What Are Some Common involves assurances that a particular command or message Security Components? was actually sent, as the receiving entity claims, and is typi- Earlier, we discussed the three objectives of security, namely, cally realized using digital signatures. Privacy, as a special confi dentially, integrity, and authentication. Cryptography is form of confi dentiality, refers to adequate protection of per- used to provide confi dentiality and integrity. sonally identifi able information and functions so that only The workhorse of secure communications systems is authorized entities have access to this data. For example, symmetric cryptography. This is often called secret-key consumer energy consumption data need to be kept private cryptography because the keys, which are the same at both as AMI systems are realized. Achieving these properties for ends of the communications link, must be kept secret. These all computing and communication systems supporting the algorithms are frequently identifi ed by the length of their electricity grid is a major research, development, deploy- keys, e.g., the 128-bit Advanced Encryption Standard (AES). ment, and maintenance challenge. They can be thought of as codebooks that take a block of A common approach to achieving these properties is input data and encrypt it in a unique way based on the secret to design, develop, and deploy cybersecurity technologies key. Figure 2 illustrates how a symmetric cipher could be for protection, detection, and response. Protection sys- used to protect data moving from a control center to a substa- tems devise security components such as key management, tion. The process unfolds as follows: authentication and authorization, and perimeter defense 1) The secret keys are generated, transported to the ends that help ensure the CIA properties against a range of of the communications link, and loaded into cryptographic devices (often part of a larger computing device) so that they are only known to the authorized sender and receiver. If attackers are able to obtain a Plaintext Plaintext copy of this key, they could also decrypt the data, ren- “Set 247 On” “Set 247 On” Key Key dering the system insecure. Codebook Internet Codebook 2) The sender’s plaintext message is then passed through the codebook algorithm, where it is transformed into ciphertext. The output of the codebook is a function of Ciphertext Ciphertext both the key and the plaintext. “k3>A+zLcb+” “k3>A+zLcb+” 3) The ciphertext is transmitted over the communication Eavesdropper link. 4) An eavesdropper listening in on the communications figure 2. Symmetric key cryptography provides is able to intercept the ciphertext, but without the key confidentiality. the eavesdropper cannot decrypt the data and recover

4452 IEEEieee power & energy magazine january/february 2012 the plaintext. Thus, the symmetric cryptography pro- Cryptography is helpful in addressing many security vides confi dentiality. issues. But the use of cryptography within the power grid is 5) The receiver passes the ciphertext through the code- challenging for the following reasons: book algorithm in reverse, using the secret key. The ✔ Legacy systems often lack the computing power and output of the codebook is the original plaintext. bandwidth necessary to support strong cryptography. Securely distributing the keys for symmetric-key cryp- SCADA systems often remain in the fi eld for years, tography is cumbersome, so asymmetric-key (also called making it impractical to support the newer, more “public-key”) cryptography, a newer form, is used to trans- computationally intensive algorithms required as the port the secret keys and perform other types of authentica- attacker’s computing power increases over the years. tion. Three common public-key systems are RSA, El-Gamal, ✔ Cryptography often relies on random number genera- and elliptic curve cryptography (ECC). The underlying tors with high entropy. Many embedded devices lack mathematics of these algorithms are signifi cantly different. the means to produce good random numbers. All three, however, have a private key used to encrypt or ✔ The key distribution and revocation process can be sign a message and a related public key used to decrypt or labor-intensive and prone to errors. This is especially verify messages, as shown in Figure 3. The originator of a true when multiple organizations are involved in the message (e.g., a control center) signs the message with its process. Mistakes made in the key management pro- encryption key, which is kept private. It then distributes its cess may reduce the ability to communicate, which public key to everyone, including potential attackers. The affects availability. legitimate receiver (e.g., a substation) uses the public key There are many other security functions used to enhance to verify that the message indeed came from the claimed the integrity and availability of systems. Antitamper mecha- source. An attacker could also use the public key to verify nisms are frequently used to protect hardware accessible to the message. But if an attacker attempts to forge a message, potential attackers (e.g., smart meters). These mechanisms the verify operation will fail. Thus, public-key cryptography deter the reverse-engineering of devices to recover crypto- can be used to provide integrity and nonrepudiation. Nonre- graphic keys or fi rmware that would disclose how a device pudiation lets a third party verify that a message came from operates. the entity holding the associated private key. Public-key cryptography may also be used to provide confi dentiality Why Is Cybersecurity for for small messages (e.g., a key for symmetric encryption) Control Systems Challenging? by encrypting them with the public key and then having the There are several contributing factors that make cyberse- intended recipient decrypt them with its private key. curity of control systems a challenge. Three of these chal- Hash functions, such as the Secure Hash Algorithm with lenges are: 256-bit output (SHA-256), are used to produce a mathemati- ✔ the clash between the operations team and IT team cal fi ngerprint of a message or fi le. The hash function takes cultures in a fi le of arbitrary size (often quite large) and produces ✔ the porting of legacy control software to common off- a fi xed-length output. Hash functions have the following the-shelf (COTS) platforms properties: ✔ the long life cycle of control systems. ✔ Given a fi le and its corresponding hash, it is very dif- The fi rst is a cultural issue. The SCADA system engi- fi cult to fi nd another fi le that will produce the same neers are responsible for the confi guration and operation of hash output. any process. This includes a requirement to assure that cer- ✔ It is very diffi cult to produce two fi les that when tain control systems, such as SCADA systems, are always hashed will yield the same hash output. available. In many cases, a control system is expected to The hash output may then be signed using asymmetric cryptography. The resulting signed hash lets a receiver check the integrity of a large fi le by recalculating the hash Open Breaker #3 Open Breaker #3 and comparing it with a hash signed with the private key of Key the sender. Key Sign = Encrypt Verify = Decrypt Certifi cation authorities are organizations that verify Internet X the credentials of a user, device, or software and then use asymmetric cryptography together with a hash function to Public Key issue the entity a digital certifi cate (e.g., under the X.509 Open Breaker #1 standard) that may then be used for authentication over a network. Public-key infrastructure, using certifi cation authori- Attacker ties, hash functions, and of course public-key cryptography, is often used to build authentication and key management figure 3. Asymmetric-key (public-key) cryptography can systems. provide integrity. january/february 2012 IEEEieee power & energy magazine 4553 Many control devices will require security devices in the network that act as compensating controls to assist in securing them.

operate a plant over periods of many years with no shutdown trol the generation and distribution of power and has created or reduction in product manufactured by that control system. the NERC-CIP standards, which help guide the owners and This means that availability is one of the most important operators of critical SCADA power systems. requirements for any control system. Today’s modern con- The migration from proprietary control systems to open trol systems are built using open-standard IT technologies systems–based control has also contributed to some of the such as Microsoft Windows–based computers and Ethernet challenges. The IT industry and the control industry have networks that include commercial routers, switches, and evolved at different rates. While the IT industry was moving fi rewalls. Because the SCADA system engineers are respon- to PCs and servers, the control industry was still produc- sible for the operation of the process, they feel responsible ing proprietary systems on proprietary networks. The con- for all of the equipment required to run the process. Because trol industry’s shift to open systems followed that of the IT IT systems are now part of the equipment required to run industry by approximately seven years, and the control sys- the process, the IT department feels it is responsible for the tem industry is approximately that far behind in understand- IT equipment running that process. This leads to a clash ing how to develop and deploy secure systems. Many secu- between the IT department and the process engineering rity issues that existed in IT systems six or seven years ago department. Among the factors contributing to this clash are are now just starting to appear in control systems. One rea- items related to the management and maintenance of those son for this is that the way the migration of control systems IT assets. One example concerns the installation of secu- to open systems occurred was to port as much of the propri- rity updates in the IT equipment. IT typically pushes out etary software to open system–based platforms as possible. security updates shortly after they are available, and most Because the proprietary control systems had an implicit trust security updates require a reboot of the computers being in the communications among devices in those systems, very updated. These reboots are usually done at a time controlled few checks were performed in the code. Once ported to an by the IT department. A reboot of a control system computer open system, an application or device may become compro- can severely affect a process operator’s ability to operate a mised by invalid input. Control device protocols were also process safely, and so the process engineering team wants developed with implicit trust, meaning that as they were more control over when the updates are installed. moved to Ethernet, there was no attempt to add such things Another example results from migration to Ethernet net- as authenticated and authorized communications. works. Many modern control systems integrate the status of Users of control systems expect them to last for a long Ethernet components such as switches and routers into the time. It is not unusual for a control system to operate a plant overall system status displays. IT wants to manage and mon- for a period of 20 years or more. Most operators don’t expect itor the Ethernet equipment, and this can result in a loss of to have to change the control system during that period. view of that equipment status to the SCADA operators. One This period far exceeds the life cycle of any modern piece way to sum up the clash is that IT is focused on the protec- of open-systems hardware or software. The IT industry has tion of the intellectual assets of the company while SCADA a turnover rate of new systems every three to fi ve years, system engineering focuses on the protection of the physical while the turnover rate for control systems has traditionally assets and manufacturing capabilities of the company. The exceeded 20 years. As the control industry evolves further, priorities of the two can easily confl ict, leading to a clash the turnover rate will have to decrease. This will be a signifi - between the two organizations. cant challenge for the industry as we move forward. Standards organizations such as the ISA99 standards development committee have recognized the unique secu- How Does One Design Secure Systems? rity management needs of SCADA and control systems There are several steps that can be taken to design secure and are drafting security standards for those systems. The control systems. First, consider procuring components that intent of ISA99’s proposed standards is to complement the were designed with security in mind. Designing with secu- IT standards that already exist while addressing those areas rity in mind means, for example, that the vendor of those that need special attention for control systems. The North components can demonstrate that it has integrated a security American Electric Reliability Corporation (NERC), the suc- development life cycle (SDL) into its development process. cessor to the National Electric Reliability Council, has also The SDL will include security steps at all phases of devel- realized the need for standards for control systems that con- opment. This means there are security requirements for the

5446 IEEEieee power & energy magazine january/february 2012 table 1. Representative efforts in the area of best practices for control systems security.

Type Description Title and URL Organization DHS Industrial Control Systems Joint Working Group (ICS JWG); Cross Sector Cyber Security Working Group (CSCSWG); IT Sector Coordinating Council (IT SCC); Communications Sector Coordinating Council (CommSCC) Organization with NERC Cyber Attack Task Force (CATF) and several related task forces enforced standards http://www.nerc.com/filez/catf.html Security guidelines: NERC 1300, CIP-002-1 through CIP-009-1 http://www.nerc.com/page.php?cid=2%7C20 http://www.nerc.com/docs/standards/sar/Draft_Version_1_Cyber_Security_ Standard_1300_091504.pdf Publication National Institute NIST Special Publication 800-53, Revision 3 of Standards and http://csrc.nist.gov/publications/PubsSPs.html Technology (NIST) SP800-53R3 Publication NISTIR 7628 NIST publication on guidelines for smart grid cybersecurity Publication DOE-supported Roadmap to Secure Control Systems in the Energy Sector and industry-led http://www.oe.energy.gov/DocumentsandMedia/roadmap.pdf roadmap Working Groups/ DOE Office of Electricity Delivery and Energy Reliability; Control Systems Security; Research Cyber Security for Energy Delivery Systems (CEDS) http://www.oe.energy.gov/controlsecurity.htm National SCADA Test Bed (NSTB) http://www.oe.energy.gov/nstb.htm http://www.sandia.gov/ccss/home.htm Draft road map: http://energy.gov/oe/downloads/roadmap-achieve-energy-delivery-systems- cybersecurity-2011 Working Group NIST Smart Grid NIST Smart Grid Interoperability Panel, the Cyber Security Working Group Interoperability (CSWG) Panel (SGIP) http://www.nist.gov/smartgrid/ Working Group UCA International Open SG Security Working Group’s Advanced Security Acceleration Project Users Group OpenSG (ASAP-SG) Standards/ International Electro- Data and Communications Security; focused on security for protocols 60870- Working Group technical Commission 5, 60870-6, 61850, 61970, and 61968 (IEC) Technical Committee 57 Working Group 15 Standards AGA 12 Cryptographic Protection of SCADA Communications Part 1: http://www.aga.org/our-issues/security/Documents/0603REPORT12.PDF Part 2, Performance Test Plan: http://cipbook.infracritical.com/book3/chapter8/ch8ref4.pdf Standards API 1164 Pipeline SCADA Security http://engineers.ihs.com/document/abstract/BPZBGBAAAAAAAAAA Standards FIPS 140-2 Security Requirements for Cryptographic Modules Standards IEC 62210 Power System Control and Associated Communications—Data and Communication Security http://webstore.iec.ch/preview/info_iec62210%7Bed1.0%7Den.pdf Standards IEC 62351 Power Systems Management and Associated Information Exchange—Data and Communications Security, Part 1 (there are seven parts, all of which can be found on the IEC Web site): http://webstore.iec.ch/preview/info_iec62351-1%7Bed1.0%7Den.pdf Standards IEEE 1686, Standard for Intelligent Electronic Devices (IEDs) Cyber Security Capabilities IEEE 1402 IEEE Guide for Electric Power Substation Physical and Electronic Security Standards ISA-99 Manufacturing and Control Systems Security http://www.isa.org/MSTemplate.cfm?MicrositeID=988&CommitteeID=6821 Academic Trustworthy Cyber Trustworthy Cyber Infrastructure for the Power Grid Research Infrastructure for the http://tcipg.org/ Power Grid

january/february 2012 IEEEieee power & energy magazine 4755 This article provides an introduction to relevant cybersecurity concepts and issues pertaining to emerging modern electric grid systems.

product. Roles are defi ned for the confi guration, operation, cal digital control system, the two classic types of control and administration of control systems. These roles should systems. include privileges for each role and identifying how the device responds when a user attempts to perform an opera- What Is Being Done to Secure tion on the device that the user does not have privileges to Control Systems Today? perform. Providing a role with only those privileges neces- It is important to note that if the attackers and attack vectors sary to perform the associated functions is commonly called are studied, a common set of high-ranking vulnerabilities least privilege. The device should be deployed with least can be created that will signifi cantly affect the success of privilege already confi gured, so that the end user or inte- the attack. There are many good studies that can be found grator does not have to perform any additional steps for the on common vulnerabilities and recommendations. Here are device to be secure. several: Many control devices will require security devices in ✔ “Common Cyber Security Vulnerabilities Observed in the network that act as compensating controls to assist in Control System Assessments by the INL NSTB Pro- securing them. When this is the case, the device speci- gram” (November 2008, U.S. Department of Energy) fication should define the compensating control, how to ✔ “Catalog of Control Systems Security: Recommen- configure it, and an explanation of why it is required. dations for Standards Developers” (June 2010, U.S. The device vendor should be following secure coding Department of Homeland Security; www.us-cert.gov/ practices. Finally, the device vendor should have pro- control_systems) cesses in place to respond to a security vulnerability ✔ “Common Cyber Security Vulnerabilities Observed disclosure if one ever occurs for its product. These are in DHS Industrial Control System Assessments” (July just some of the steps required. There are many good 2009, U.S. Department of Homeland Security). examples of SDLs available, including Microsoft’s Secu- Many organizations and governments have spent millions rity Development Lifecycle, the Open-Web Application of dollars and years’ worth of effort in studying and rec- Security Project, and the Common Lightweight Applica- ommending good practices for control systems security. In tion Security Process. addition, most vendors today are actively including security Once components are procured, system integrators also in the design of their products. Table 1 provides examples of need to have methodologies for developing and confi gur- representative work in this area, rather than an exhaustive ing control systems for end users. The system integrator list of the many activities currently taking place. is responsible for integrating all of the pieces that together form a control system. As a control system is integrated, it Conclusions will consist of multiple devices connected to multiple areas This article provides an introduction to relevant cyberse- of a process with multiple functions. A model for how a curity concepts and issues pertaining to emerging modern control system is to be confi gured and information is to electric grid systems. We looked at the history of these sys- fl ow within it exists within the international ISA-95 stan- tems, the objectives of cybersecurity, challenges in address- dard. This model provides a topology to be applied while ing security for control systems, common security tools and designing and confi guring a control system. This topology components, processes for designing secure grid systems, provides a natural defense-in-depth approach to help pro- and some key efforts under way today. tect the more vulnerable components of a control system. In addition to ISA-95, the International Society of Auto- Biographies mation (ISA) standards committees have formed the previ- Julie Hull is with Honeywell ACS Research Labs. ously mentioned ISA99 standards development committee, Himanshu Khurana is with Honeywell ACS Research which is developing the security requirements for indus- Labs. trial automation and control systems. The ISA-99 standards Tom Markham is with Honeywell ACS Research Labs. build on the reference models in ISA-95 and create security Kevin Staggs is with Honeywell ACS Research Labs. reference models for a typical SCADA system and a typi- p&e

4856 IEEEieee power & energy magazine january/february 2012 NEW

IEEE Electrification Magazine Launching in 2013!

Join PES by July 3, 2013 and receive the first issue FREE

Limited Time Offer. First Issue: $0.00

www.ieee-pes.org/electrification

IEEE Electrification Magazine is the only publication dedicated to disseminating information on all matters related to microgrids onboard electric vehicles, ships, trains, planes and off-grid applications.

Published quarterly starting in mid-2013, each issue will provide: • News, analysis and insights on electric vehicles, electric ships, electric trains and electric planes • Feature articles that allow you to stay current and connected to the challenges and opportunities for electrification in remote parts of the world • Access to comprehensive, in-depth technical analysis from engineers in the field of advanced electrification • Industry insights, public sector programs and case studies on electric transportation This is a limited time offer. To ensure you are among the first to sample this exciting new publication, be sure to join the IEEE Power & Energy Society by July 3, 2013 and we will send you the first issue absolutely FREE. There are a lot of exciting things to come from IEEE Electrification Magazine, so be sure to join PES now and reserve your issue TODAY!

www.ieee-pes.org/electrification

IEEE8691_ElectraAd_2.indd 1 4/18/13 2:57 PM Smart Grid: Challenges & Opportunities

INT8684_PECover2013_8.indd 1 4/4/13 5:11 PM