USING ADVANCED METERING INFRASTRUCTURE DATA FOR SMART GRID DEVELOPMENT by FRANKLIN L. QUILUMBA-GUDIÑO Presented to the Faculty of the Graduate School of The University of Texas at Arlington in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY THE UNIVERSITY OF TEXAS AT ARLINGTON May 2014 Copyright © by Franklin L. Quilumba-Gudiño 2014 All Rights Reserved ii Acknowledgements This work research was funded by the DOE’s Smart Grid Investment Grant for the Consolidated Edison Company of New York, Inc., and sponsorship of the author’s graduate studies was provided by the Fulbright Commission, the Escuela Politécnica Nacional (Quito, Ecuador), and the Energy Systems Research Center at the University of Texas at Arlington. February 12, 2014 iii Abstract USING ADVANCED METERING INFRASTRUCTURE DATA FOR SMART GRID DEVELOPMENT Franklin L. Quilumba-Gudiño, PhD The University of Texas at Arlington, 2014 Supervising Professor: Wei-Jen Lee Identifying and using Advanced Metering Infrastructure (AMI) data to improve customer experience, utility operations, and advanced power management is one of the most important challenges in the smart grid development. Smart meters, capable of capturing frequent interval customer consumption (and possibly other parameters) using communication networks, are vital components of smart grid technology. Thus, smart meters expand the available range of data and functionality. Making the most of information from smart meters and smart grids increasingly requires dealing with Big Data. Big Data is a game changer, enabling utilities to transform the ways they interact with and serve their customers. Today, many utilities are deploying smart meters as a vital step moving towards smart grids. Going from one meter reading per month to one meter reading at a sub- hourly rate (one minute, fifteen minutes, or thirty minutes) immediately poses a great technical challenge that can be overwhelming if not properly managed. AMI is becoming the standard in today’s utility industry, making it possible to transform the performance of the grid and dramatically improve customer experience, utility operations, and advanced power management. To attain the maximum benefits from AMI, it is of utmost importance that utilities perform large-scale data analysis and transform them into information. iv Consequently, this dissertation addresses the efforts involved in turning smart meter raw data into actionable information. Algorithms are developed to utilize data collected from AMI system for three main purposes: 1. To develop accurate customer daily load profiling for load estimation and network demand reconciliation to improve the efficiency and security of the utility grid. 2. To enhance the performance of load forecasting which impacts operating practices and planning decisions to build, lease, or sell generation and transmission assets and the decisions to purchase or sell power at wholesale level. 3. To investigate a nonintrusive load monitoring method for discerning individual appliances from a residential customer. v Table of Contents Acknowledgements .............................................................................................................iii Abstract .............................................................................................................................. iv List of Illustrations ............................................................................................................... x List of Tables .....................................................................................................................xiv Chapter 1 Introduction......................................................................................................... 1 1.1 Background ............................................................................................................... 1 1.1.1 Smart Grid Initiative ........................................................................................... 1 1.1.2 Smart Grid Programs ........................................................................................ 1 1.1.3 Recovery Act - Smart Grid Investment Grant Program ..................................... 2 1.1.4 Advanced Metering Infrastructure Projects ....................................................... 2 1.1.5 Advanced Metering Infrastructure ..................................................................... 3 1.2 Motivation ................................................................................................................. 5 1.3 Contribution .............................................................................................................. 6 1.4 Dissertation Outline .................................................................................................. 7 Chapter 2 AMI Data Preprocessing .................................................................................... 9 2.1 Data Preparation..................................................................................................... 10 2.1.1 Familiarizing with AMI Data ............................................................................. 10 2.1.1.1 Initial error checking ................................................................................. 10 2.1.1.2 Visualization ............................................................................................. 11 2.1.1.3 Smart meter data format .......................................................................... 11 2.1.2 Smart Meter Data Resolution and Grouping ................................................... 15 2.2 Smart Meter Data Cleaning .................................................................................... 15 2.2.1 Inconsistencies ................................................................................................ 15 2.2.2 Missing Data .................................................................................................... 20 vi 2.2.3 Duplicate Data ................................................................................................. 20 2.2.4 Outlier Detection .............................................................................................. 21 2.3 Data Preprocessing Software Design Criteria ........................................................ 22 Chapter 3 AMI Data for Load Profiling .............................................................................. 26 3.1 Model Variables ...................................................................................................... 27 3.1.1 Data Meters Variables ..................................................................................... 27 3.1.2 Calendar Variables .......................................................................................... 29 3.1.2.1 Day of the week variables ........................................................................ 29 3.1.2.2 Holiday variables ...................................................................................... 30 3.1.2.3 Weekday and weekend variables ............................................................ 30 3.1.2.4 Season of the year variables ................................................................... 31 3.2 Load Profile Development Based on Stratification Customer Information .................................................................................................................... 31 3.3 Load Profile Development Based on Customers’ Behavior Similarities ................. 42 3.3.1 Introduction to Data Clustering ........................................................................ 43 3.3.2 Clustering Definition ........................................................................................ 44 3.3.2.1 Proximity measures ................................................................................. 44 3.3.2.1.1 Proximity measures for continuous variables ................................... 45 3.3.2.2 Clustering algorithms ............................................................................... 46 3.3.2.2.1 Hierarchical clustering ...................................................................... 46 3.3.2.2.2 Partitional clustering ......................................................................... 48 3.3.2.2.3 Fuzzy clustering ................................................................................ 49 3.3.2.2.4 Affinity propagation clustering .......................................................... 50 3.3.3 Clustering Validity Indices ............................................................................... 51 3.3.3.1 External criteria ........................................................................................ 52 vii 3.3.3.2 Internal criteria ......................................................................................... 52 3.3.3.2.1 Cophenetic correlation coefficient .................................................... 52 3.3.3.3 Relative criteria ........................................................................................ 53 3.3.3.3.1 Davies-Bouldin index ........................................................................ 53 3.3.3.3.2 Dunn’s index ..................................................................................... 55 3.3.4 Load Profile Development by Means of Clustering Analysis .......................... 55 Chapter 4 AMI Data to Enhance the Performance of Load Forecasting .......................... 67 4.1 A Review on Load Forecasting Techniques ..........................................................