Electricity Retailing Decision Making Based on Data Mining Techniques

Home , Electricity retailing

ELECTRICITY RETAILING DECISION MAKING BASED ON DATA MINING TECHNIQUES

JIAJIA YANG

B.E. M.E.

A thesis in fulfilment of the requirements for the degree of

Doctor of Philosophy

School of Electrical Engineering and Telecommunications

Faculty of Engineering

April 2018

PLEASE TYPE THE UNIVERSITY OF NEW SOUTH WALES

Thesis/Dissertation Sheet

Surname or Family name: Yang

First name: Jiajia Other name/s:

Abbreviation for degree as given in the University calendar: PhD

School: School of Electrical Engineering and Faculty: Faculty of Engineering Telecommunications

Title: Electricity Retailing Decision Making Based on Data Mining Techniques

Abstract 350 words maximum: (PLEASE TYPE)

With the continuous development of Smart Grid, especially the emergence of Energy Internet, there is an increasing amount of measurement data available collected from power system end-users. Through data mining techniques, these measurement data can enable a better understanding of the load composition and end-user consumption behaviours, and therefore would provide great potentials for developing more flexible and targeted or even customized pricing schemes for electricity retail.

This research starts with a comprehensive literature survey on decision-making for electricity retailers. Publications on electricity retailing in the last two decades are surveyed and discussed in detail. Then, key business framework of electricity retailers is studied. It elaborates the typical business process of electricity retailers and its procedure of creating a new sales agreement. Considering the drawbacks of existing load data mining methods, a new non-intrusive load monitoring method is proposed which is able to cope with the big load data in the Smart Grid environment. After obtained the status of all identified appliances, a statistical residential load model is developed. With this load model, the appliance identification results can be conveniently used in demand-side management and developing electricity retailing strategies.

Next, this research proposes the idea of using the results of residential appliance identification and end-user behaviour analysis to help retail pricing. The problem of designing customized pricing strategies for different residential users is investigated based on the identification results of residential electric appliances and classifications of end-users according to their consumption behaviours. A novel framework of customizing electricity retail prices is proposed.

When to customize retail prices through appliance identification, load data at least sampled at every minute is needed. Differently, this research explores another data mining technique to customize electricity retail prices using the half-hourly sampled electricity consumption data. A model of customizing electricity retail prices based on load profile clustering analysis is developed. Electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms.

Declaration relating to disposition of project thesis/dissertation

I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation.

I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only).

…………………………………………………………… ……………………………………..……………… ……….……………………... Signature Witness Signature …….… Date

The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research.

FOR OFFICE USE ONLY Date of completion of requirements for Award:

ORIGINALITY STATEMENT

‘I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project's design and conception or in style, presentation and linguistic expression is acknowledged.’

Signed ……………………………………………......

Date ……………………………………………......

‘I hereby grant the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstract International (this is applicable to doctoral theses only). I have either used no substantial portions of copyright material in my thesis or I have obtained permission to use copyright material; where permission has not been granted I have applied/will apply for a partial restriction of the digital copy of my thesis or dissertation.'

Signed ……………………………………………......

Date ……………………………………………......

AUTHENTICITY STATEMENT

‘I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis. No emendation of content has occurred and if there are any minor variations in formatting, they are the result of the conversion to digital format.’

Signed ……………………………………………......

Date ……………………………………………......

Abstract

With the continuous development of Smart Grid, especially the emergence of Energy Internet, there is an increasing amount of measurement data available collected from power system end-users. Through data mining techniques, these measurement data can enable a better understanding of the load composition and end-user consumption behaviors, and therefore would provide great potentials for developing more flexible and targeted or even customized pricing schemes for electricity retail.

When to customize retail prices through appliance identification, load data at least sampled at every minute is needed. Differently, this research explores another data mining technique to customize electricity retail prices using the half-hourly sampled electricity consumption data. A model of customizing electricity retail prices based on load profile clustering analysis is developed. Electricity usage data collected by the Smart Grid, Smart City (SGSC) national demonstration project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms.

In memory of my father

To my beloved mother

Acknowledgements

Thanks to the guidance, supervision, and advice from my supervisors, my PhD research and study can always be kept on the right track and carried out in an efficient way. My supervisors have been a constant source of academic support, encouragement and innovative ideas for my research.

My thanks and appreciations firstly go to Professor Zhao Yang Dong. It is extremely lucky for me to join the research group led by Professor Zhao Yang Dong. Thanks Professor Zhao Yang Dong for your prompt response no matter when I have a request, and for all the support related to the funding of my PhD project. You always care so much about our study and life. Your serious scientific attitude, rigorous scholarship, open-mindedness, and humble character will always motivate and inspire me in my future work.

My sincere thanks also go to Associate Professor Junhua Zhao working at Chinese University of Hong Kong (Shenzhen) now. Professor Junhua Zhao has ever paved the way for me to get into my PhD research topic. I have been inspired by his perceptive insight and creative ideas on research. Thanks Professor Junhua Zhao for your expertized responses to my research questions and queries, patient revisions on my papers, and considered suggestions about my future career development.

My thanks and gratitude is extended to Professor Fushuan Wen at the Zhejiang University who is one of the most rigorous scholars I have ever met. Professor Fushuan Wen have ever guided me into the world of research and supervised my master research. I was deeply influenced by his rigorous attitude. My appreciation is expressed to Professor Fushuan Wen for his assistance in revising my papers and providing materials to help me keep up with the latest research trends.

Without the friendly, trusting and cooperative atmosphere in our research group, I would not have carried out my PhD research smoothly. I would also like to express my thanks to every colleague in our group for being such a great team and influencing me in a productive way.

The completion of my PhD witnessed my two transfers due to the job changes of my supervisor. I transferred first from the University of Newcastle to the University of Sydney, and then transferred to the University of New South Wales. All these universities deserve my thanks for providing the excellent environment for me to undertake my research and funding

III me to attend academic conferences to broad my eyesight and meet so many interesting people.

Finally, my special recognition goes to my families, they are always on my side during the ups and downs in my life. Your endless love, support, and encouragement will motivate me in my future life journey. I love and thank you all.

List of Publications

A complete list of publications during my PhD project is provided below.

1. Jiajia Yang, Junhua Zhao, Fushuan Wen, and Zhaoyang Dong, “A model of Customizing Electricity Retail Prices Based on Load Profile Clustering Analysis,” IEEE Trans Smart Grid, accepted for publication on Mar. 24 2018.

2. Jiajia Yang, Junhua Zhao, Fushuan Wen, and Zhaoyang Dong, “A Framework of Customizing Electricity Retail Prices”, IEEE Trans Power Systems, DOI: 10.1109/TPWRS.2017.2751043, accepted for publication on Sept. 1 2017.

3. Jiajia Yang, Junhua Zhao, Fengji Luo, Fushuan Wen, and Zhao Yang Dong, “Decision-Making for Electricity Retailers: A Brief Survey”, IEEE Trans Smart Grid, DOI: 10.1109/TSG.2017.2651499, accepted for publication on Jan. 4 2017.

4. YANG Jiajia, ZHAO Junhua, WEN Fushuan, MENG Ke, and DONG Zhaoyang, “Key Business Framework and Purchase/Sale Decision-making for Electricity Retailers”, Automation of Electric Power Systems, vol.41, no.14, pp. 10-23, Jul. 2017.

5. YANG Jiajia, ZHAO Junhua, WEN Fushuan, DONG Zhaoyang, XUE Yusheng, “Residential Appliance Identification and Load Modeling Based on Big Data Mining in Smart Grid Environment,” Electric Power Construction, vol.37, no.12, pp. 11-23, Dec. 2016.

6. Jiajia Yang, Junhua Zhao, Fushuan Wen, and Zhaoyang Dong, “Mining the Big Data of Residential Appliances in the Smart Grid Environment,” Proceedings of the 2016 IEEE PES General Meeting, 17-21 July 2016, Boston USA.

7. Jiajia Yang, Junhua Zhao, Fushuan Wen and Zhaoyang Dong, “A Risk Management Model for Carbon Constrained Coal Inventory Optimization,” Proceedings of the 2015 IEEE PES APPEEC, Brisbane, Australia, November 2015.

Co-authored papers:

8. HUANG Y., WEN F., YANG J., LIU W., YU M., ZENG P., “A Preliminary Investigation on Power Network Planning Evaluation System with High-Penetration Intermittent Generation”, Electric Power Construction, vol. 36, no.10, Oct. 2015.

9. J. Qiu, Z.Y. Dong, J.H. Zhao, Y. Xu, F.J. Luo, and J.J. Yang, “A Risk-Based Approach to Multi-Stage Probabilistic Transmission Network Planning,” IEEE trans Power Systems, vol. 31, no. 6, Nov. 2016.

10. F.J. Luo, Z.Y. Dong, K. Meng, J. Qiu, J.J. Yang, and K.P. Wong, “Short-term operational planning framework for virtual power plants with high renewable penetrations”, IET Renewable Power Generation, vol. 10, no. 5, May 2016.

Contents

ABSTRACT ...... I

ACKNOWLEDGEMENTS ...... III

LIST OF PUBLICATIONS...... V

CONTENTS ...... VII

LIST OF TABLES ...... X

LIST OF FIGURES ...... XI

LIST OF ABBREVIATIONS ...... XIII

CHAPTER 1 INTRODUCTION ...... 1

1.1 PROBLEM STATEMENT ...... 1 1.2 RESEARCH CONTRIBUTIONS ...... 2 1.3 THESIS OUTLINE ...... 3

CHAPTER 2 DECISION-MAKING FOR ELECTRICITY RETAILERS ...... 5

2.1 INTRODUCTION ...... 5 2.2 DECISION FRAMEWORK ...... 6 2.2.1 Retail Energy Forecasting ...... 7 2.2.2 Portfolio Evaluation ...... 8 2.2.3 Risk Management for Electricity Retailer ...... 8 2.3 LONG-TERM LOAD FORECASTING FOR ELECTRICITY RETAILERS ...... 8 2.3.1 Literature survey on long term load forecasting methods ...... 9 2.3.2 Industry applications of long term load forecasting methods ...... 12 2.3.3 Time resolution in various load forecasting techniques ...... 14 2.4 ENERGY PROCUREMENT STRATEGY FOR ELECTRICITY RETAILERS ...... 15 2.4.1 Sources of Electricity Supply and Involved Markets ...... 16 2.4.2 Factors Influencing Energy Procurement ...... 16 2.4.3 Procurement Optimization Approaches ...... 19 2.5 RETAIL PRICING STRATEGY FOR ELECTRICITY RETAILERS ...... 22 2.5.1 Dynamic Pricing Schemes ...... 23 2.5.2 Static Pricing Schemes ...... 26 2.5.3 Advanced Metering Infrastructures in Electricity Retail Markets ...... 26 2.6 RISK MANAGEMENT IN RETAIL MARKETS ...... 28 2.6.1 Considering Risks in Retail Pricing Models ...... 28 2.6.2 Risk Premium Determination ...... 29 2.6.3 Establishment of a Derivatives Portfolio ...... 29

VII

2.7 EVOLUTION OF RETAIL STRATEGY AND FUTURE DISCUSSION ...... 30 2.7.1 Long-term Retailer Load Forecasting ...... 30 2.7.2 Structure Optimization of Electricity Retail Prices ...... 31 2.7.3 Customized Electricity Retail Plans ...... 31 2.7.4 Management Strategy for Large-Scale Flexible Loads ...... 32 2.7.5 Data-Driven Retail Pricing Algorithm ...... 34 2.8 CHAPTER SUMMARY ...... 36

CHAPTER 3 KEY BUSINESS FRAMEWORK OF ELECTRICITY RETAILERS ...... 38

3.1 INTRODUCTION ...... 38 3.2 ESTABLISHED ELECTRICITY RETAIL MARKETS IN DEVELOPED COUNTRIES ...... 39 3.3 TYPE OF RETAIL COMPANY ...... 41 3.4 STRUCTURE OF ELECTRICITY RETAIL COMPANY AND ITS BUSINESS PROCESS ...... 42 3.4.1 Australia Electricity Retail Market ...... 42 3.4.2 Finland Electricity Retail Market ...... 45 3.5 RELATIONSHIP BETWEEN ELECTRICITY WHOLESALE AND RETAIL MARKETS ...... 48 3.6 CHAPTER SUMMARY ...... 49

CHAPTER 4 IDENTIFICATION OF RESIDENTIAL APPLIANCES AND LOAD MODELLING THROUGH MINING BIG LOAD DATA IN THE SMART GRID ...... 50

4.1 INTRODUCTION ...... 51 4.2 DYNAMIC TIME WARPING MATCHING ...... 53 4.3 DTW BASED APPLIANCE IDENTIFICATION ALGORITHM ...... 55 4.3.1 Segmentation of Temporal Load Sequence ...... 58 4.3.2 Electric Appliance Identification Algorithm ...... 60 4.4 RESIDENTIAL LOAD MODEL ...... 61 4.5 EXPERIMENTS ON DATA AND DISCUSSIONS ...... 63 4.5.1 Electric Appliance Identification ...... 63 4.5.2 Parameter Estimation of Residential Load Model...... 72 4.5.3 Result Analysis ...... 74 4.6 CHAPTER SUMMARY ...... 75

CHAPTER 5 A FRAMEWORK OF CUSTOMIZING ELECTRICITY RETAIL PRICES ..... 77

5.1 INTRODUCTION ...... 77 5.2 APPLIANCE IDENTIFICATION AND END-USER CLASSIFICATIONS BASED RETAIL LOAD DETERMINATION ...... 79 5.2.1 Residential Appliance Identification ...... 80 5.2.2 End-user Classifications ...... 80 5.2.3 Residential Load Analysis ...... 81 5.3 PROPOSED FRAMEWORK OF CUSTOMIZING ELECTRICITY RETAIL PRICES ...... 82 5.3.1 The Pricing Model for Hourly TOU Prices ...... 83 VIII

5.3.2 The Segmentation Model of TOU Prices ...... 89 5.3.3 The Model for Customizing Retail Prices for Categorized End-users ...... 90 5.3.4 Mathematical Reformulation of the Proposed Bi-level Retail Pricing Model...... 92 5.4 CASE STUDY AND DISCUSSIONS ...... 94 5.4.1 Solution Methodology ...... 94 5.4.2 Data of the Case Study ...... 94 5.4.3 Results and Discussions ...... 97 5.4.4 Correlation between the case studies and models in the proposed framework ...... 110 5.5 CHAPTER SUMMARY ...... 111

CHAPTER 6 CUSTOMIZING ELECTRICITY RETAIL PRICES THROUGH CLUSTERING ANALYSIS ...... 113

6.1 INTRODUCTION ...... 1 1 3 6.2 LOAD PATTERN AND CONSUMPTION QUANTITY ANALYSIS ...... 115 6.2.1 Clustering Analysis of Residential Load Profiles ...... 115 6.2.2 Load Pattern and Electricity Usage Determination ...... 131 6.3 MATHEMATICAL FORMULATION OF CUSTOMIZING ELECTRICITY RETAIL PRICES ...... 134 6.4 CASE STUDY AND DISCUSSIONS ...... 141 6.4.1 Data in the Case Study ...... 141 6.4.2 Results and Discussions ...... 142 6.5 CHAPTER SUMMARY ...... 156

CHAPTER 7 CONCLUSIONS AND FUTURE WORK ...... 157

REFERENCE ...... 167

List of Tables

Table 2.1 Time resolution of load data in load forecasting publications ...... 15 Table 2. 2 The number of published papers on each discussed sub-topic ...... 37 Table 3.1 Years of establishing electricity retail markets in some countries ...... 40 Table 4.1 Pseudocode for the Proposed Load Profile Segmentation Algorithm ...... 59 Table 4.2 Pseudocode for the Proposed Appliance Identification Algorithm ...... 61 Table 4.3 Parameter setting and identification results in numerical experiments ...... 69 Table 4.4 Numerical Comparison between the Proposed Algorithm and PALDi ...... 70 Table 4.5 Values of parameters in the system load model ...... 73 Table 4.6 Results of the one-sample Kolmogorov-Smirnov test for all coefficients in the load statistical model ...... 73 Table 5.1 Classification of residential load caused by electric appliances ...... 82 Table 5.2 The residential appliance list and their operation features ...... 96 Table 5.3 The expected values of prices in the real-time and day-ahead markets and the nominal prices in price elasticity function ...... 96 Table 5.4 Electricity consumption of time-shiftable appliances finishing a work cycle ...... 97

List of Figures

Fig. 2.1 Schematic diagram of a demonstrated bootstrap series ...... 11 Fig. 2.2 Schedule from pricing time to the start of the delivery period ...... 29 Fig. 3.1 Departments of a typical electricity retail company ...... 43 Fig. 3.2 The business process of an electricity retail company ...... 44 Fig. 3.3 The process diagram of creating a new sales agreement ...... 47 Fig. 4.1 Schematic of the proposed appliance identification algorithm ...... 57 Fig. 4.2 Schematic of the DTW mapping relationship between two sequences ...... 60 Fig. 4.3 The reference load sequences of residential appliances ...... 66 Fig. 4.4 Segmentation result of a half day load sequence ...... 67 Fig. 4.5 Accuracies of appliance identifications under different sampling frequencies ...... 72 Fig. 5.1 Schematic of logical relationships between models in the proposed framework .... 83 Fig. 5.2 The lifestyle pattern of each end-user ...... 95 Fig. 5.3 Calculation results of hourly TOU prices under different scenarios ...... 98 Fig. 5.4 Optimal structure of TOU prices with different number of blocks ...... 98 Fig. 5.5 Customized price strategies when overall upper bound of retail prices are 0.16 $/kWh and 0.18$/kWh ...... 99 Fig. 5.6 Result of residential load when overall upper bound of average retail prices is 0.18 $/kWh ...... 101 Fig. 5.7 Normalized profits of the retailer under different average retail prices ...... 102 Fig. 5.8 Normalized profits of the retailer under different load determination methods .... 104 Fig. 5.9 Electricity consumption of end-users under different load determination methods ...... 104 Fig. 5.10 Profits of the retailer with different market sizes and participant numbers ...... 105 Fig. 5.11 The profit difference between customized and optimized TOU prices under various scenarios ...... 106 Fig. 5.12 Lifestyle patterns of end-users 1 to 5 in the case study ...... 106 Fig. 5.13 CVaR and profit in the retail decision with different weighting factors ...... 107 Fig. 5.14 Retail power supply purchased by forward contracts ...... 108 Fig. 5.15 Market share of each end-user with the change of weighting factor ...... 108 Fig. 5.16 Normalized profits of the retailer under different number of price blocks ...... 109 Fig. 5.17 The average retail price of each user under different price blocks ...... 110 Fig. 6.1 Procedure of load profile clustering ...... 120

Fig. 6.2 The histogram of mutual distances and the cumulative percentage of load profiles whose total amount of adjacent load profiles is smaller than x when eps=0.2 ...... 122 Fig. 6.3 Density based clustering results of residential load profiles ...... 130 Fig. 6.4 The percentage of each end-user’s historical profiles being clustered into different clusters ...... 131 Fig. 6.5 Statistical analysis of end-users’ daily electricity usage ...... 133 Fig. 6.6 Electricity prices in the forward contract and real-time market ...... 142 Fig. 6.7 Customized retail plans for each end-user ...... 145 Fig. 6. 8 Residential load under different retail pricing schemes ...... 147 Fig. 6.9 The electricity consumption of end-users under different electricity retail prices 147 Fig. 6.10 The retail risks of different retail pricing methods measured by CVaR ...... 148 Fig. 6.11 The component of retail price stemming from forward contract for each end-user ...... 149 Fig. 6.12 Topology of the IEEE 37-bus distribution system ...... 150 Fig. 6.13 CVaR and expected cost of end-users in the optimization results when congestion happens in the distribution network ...... 151 Fig. 6.14 CVaR and the expected cost of end-users in the optimization results corresponding to different weighting factors ...... 152 Fig. 6.15 CVaR and the expected cost of end-users in the optimization results under different scenarios ...... 153 Fig. 6.16 The quantity of electricity consumption in the distribution system under different scenarios ...... 153 Fig. 6.17 The average value of customized TOU retail prices under different scenarios .... 155 Fig. 6.18 CVaR and the expected cost of end-users in the optimization results under different scenarios ...... 155

XII

List of Abbreviations

TOU time-of-use CVaR conditional value at risk SGSC Smart Grid, Smart City MINLP mixed-integer non-linear programming problem KKT Karush-Kuhn-Tucker DBSCAN density-based spatial clustering of applications with noise RTP real-time pricing STLF short-term load forecasting LTLF long-term load forecasting ANNs artificial neural networks ELM extreme learning machine SVM support vector machine GSP gross state product GDP gross domestic product GMP gross metropolitan product CPI consumer price index VaR value-at-risk RAROC risk adjusted recovery on capital EDR expected downside risk FIR finite impulse response LMCS lattice Monte Carlo simulation DisCo distribution company MILP mixed integer linear programming IGDT information gap decision theory CPP critical peak pricing TSO transmission system operator EV electric vehicle SPT stepwise power tariff ARMA autoregressive moving average model ARIMA autoregressive integrated moving average model AFAM Australian Financial Market Association G-ELM generalized extreme learning machine E-ELM evolutionary extreme learning machine

XIII

ILM intrusive load monitoring NILM non-intrusive load monitoring MAUs measurement and actuation units PF) particle filter HMM hidden Markov models PALDi particle filter based load disaggregation PDF probability density function FHMM factorial hidden Markov model TCL thermostatically controlled loads ETP equivalent thermal parameter DGs distributed generators DESDs distributed energy storage devices AMI advanced measurement infrastructure MDP Markov decision process GGP guidelines of good practice NEM National Electricity Market AER Australian Energy Regulator ACCC Australian Competition and Consumption Commission ASIC Australian Securities and Investments Commission CER Clean Energy Regulator DSO distribution system operator DTW dynamic time warping HoH head of household

XIV

Chapter 1 Introduction

1.1 Problem Statement

The continuous development of Smart Grid, together with the emergence of Energy Internet, brings an increasing amount of measurement data collected from power system end-users. Meanwhile, with the integration of advanced information communications technology into power systems, electric power loads can be managed in a more efficient and smart way.

Through data mining techniques, these measurement data can enable a better understanding of the load composition and end-user consumption behaviors, and therefore would provide great potentials for developing more flexible, targeted and customized pricing schemes for electricity retail. Thus, highly efficient data mining algorithms are needed to extract hidden information from the big load data, such as appliance identification, accurate load modeling, and end-user behavior analysis.

In the Energy Internet environment, the electric power market is becoming more complicated by integrating renewable resources, distributed generators, energy storage equipment, internet technology, and electrical vehicles. These changes bring both opportunities and challenges to electricity retailers to survive the competition. Even though information exchange between retailers and end-users would be easier and more frequent with the help of communications technology, electricity retailers still need advanced decision-making techniques for developing diversified retail plans.

As an intermediary between electricity producers and consumers, the electricity retail company is usually operated as an entity that is independent of any generation or distribution company. The retailer purchases energy from the wholesale and futures markets and resells it to customers through various retail contracts. In order to gain profits, electricity retailers face the problem of making optimal decisions on electricity markets in both the supply and retail sides. Decision-making of electricity retailers is based on the estimation of customer load and forecasting in electricity forward and spot markets. It covers the whole process including

1 long-term retail energy forecasting, energy procurement strategies, retail pricing schemes, and risk management in the retail market. Besides, there are thousands of consumers with various electricity consumption behaviors in the electricity retail market. Therefore, how to make smarter decisions in the upstream wholesale market and obtain a better understand of end-users’ consumption behaviors are of great importance for electricity retailers.

This research aims to develop novel electricity retail decision-making framework which is driven by data-mining techniques. In the proposed decision-making framework, the retailer would be able to model its retail load at a granular level using proposed data mining algorithms. With such an accurate load modelling, optimal electricity procurement decisions can be made in the electricity future and spot markets. Also, the algorithm about data mining on load profiles to extract end-user’s consumption behavior features will be studied. Electricity retail can develop more flexible retail pricing schemes through capturing customer’s consumption behaviors. In these pricing schemes, both the time-of-use (TOU) price structure and price level will be customized by taking individual load features into account, which are different from existing predefined time-of-use (TOU) retail prices.

1.2 Research Contributions

This research contributes to the electricity retailing problem in the following aspects. Firstly, to help researchers and engineers have a better overview of the state-of-the-art on electricity retail decision-making schemes, this research comprehensively surveys the latest progress on this subject. Some critical and open issues in this field are also discussed. Secondly, since there have been no publications studying the key business framework and process of electricity retail companies, this research has made an initial attempt to tackle this problem. Also, considering the drawbacks of existing load data mining methods, a new non-intrusive load monitoring method is proposed which is able to cope with the big load data in the Smart Grid environment. Next, this research proposes the idea of using the results of residential appliance identification and end-user behaviour analysis to customize electricity retail prices. A novel framework of customizing electricity retail prices is accordingly proposed. Finally, as one of the contributions, this research explores the approach to customize electricity retail prices through using the clustering analysis data mining method. A model of customizing electricity retail prices based on load profile clustering analysis is developed. Actual electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is adopted to demonstrate the proposed model and algorithms.

1.3 Thesis Outline

In this chapter, the problem statement is briefly introduced and the research contributions are summarized. The rest of the thesis is organized as follows.

Chapter 2 presents a comprehensive literature survey on decision-making for electricity retailers. Publications on electricity retailing in the last two decades are surveyed and discussed in detail. The state-of-the-art of the long-term retailer load forecasting, energy procurement strategies, retail pricing schemes, and risk management in the retail market are studied respectively, which cover the entire decision-making process of the electricity retailer.

Chapter 3 studies the key business framework of electricity retailers. The development history of electricity retail markets around the world is briefly introduced, followed by a summary about the motivation factors that facilitate the development of electricity retail market. The main departments of an electricity retail company are categorized and analyzed. It also elaborates the typical business process of electricity retailers and its procedure of creating a new sales agreement. Besides, as closely linked markets, the correlation between electricity wholesale and retail markets are also analyzed.

In Chapter 4, a new non-intrusive load monitoring method is proposed which is able to cope with the big load data in the Smart Grid environment. Experiments on data and comparisons with the other research verify the effectiveness and efficiency of the proposed algorithm. After obtained the status of all identified appliances, a statistical residential load model is developed. With this load model, the appliance identification results can be conveniently used in demand-side management and developing electricity retailing strategies.

Then, Chapter 5 proposes the idea of using the results of residential appliance identification and end-user behaviour analysis to help retail pricing. The problem of designing customized pricing strategies for different residential users is investigated based on the identification results of residential electric appliances and classifications of end-users according to their consumption behaviours. A novel framework of customizing electricity retail prices is proposed, where the bi-level programming and the optimal clustering in a time sequence are adopted. Meanwhile, profit risk is considered by taking conditional value at risk (CVaR) as the risk measure. The proposed bi-level optimization model is finally reformulated into a mixed-integer non-linear programming problem (MINLP) by solving Karush-Kuhn-Tucker

(KKT) conditions. Case study is employed to demonstrate the feasibility and efficiency of the developed models and algorithms.

When to customize retail prices through appliance identification, load data at least sampled at every minute is needed. Differently, Chapter 6 explores another data mining technique to customize electricity retail prices using the half-hourly sampled electricity consumption data. A model of customizing electricity retail prices based on load profile clustering analysis is developed. The density-based spatial clustering of applications with noise (DBSCAN) is firstly applied to load profile analysis, in order to explore end-users’ inherent electricity consumption patterns from their historical load data. Then, statistical analysis of end-users’ historical consumption is conducted to better capture their consumption regularity. After extracting these load features, a mixed integer nonlinear programming (MINLP) model for customizing electricity retail prices is established. In this research, it is among the first that the optimization of TOU price structure is studied in electricity retail pricing research. Electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms.

Finally, the thesis is concluded by Chapter 7, where the potential future work is also discussed.

Chapter 2 Decision-Making for Electricity Retailers

With the continuous development of smart grid and further restructuring of power industry, modern power systems have been transformed to complex cyber-physical systems characterized with high renewable energy penetrations, distributed facilities, advanced metering and communication technologies, as well as ever-increasing customer awareness. These new development trends pose significant challenges for electricity retailers and call for innovative decision-making methods. To help researchers and engineers have a better overview of the state-of-the-art on electricity retail decision-making schemes, this chapter aims to survey the latest progress on this subject. Some critical and open issues in this field are also discussed.

2.1 Introduction

Since early 1990s, both the generation and retail sectors of the power industry have been progressively experiencing the restructuring processes around the world. As a result of market deregulation, the fixed pricing scheme under regulation has been considered to be a limitation for optimizing the grid operation. A considerable amount of research work therefore has been conducted to develop more flexible pricing schemes for electricity retailing in the context of liberalized electricity markets.

Depending on the involved market entities, electricity pricing can be categorized into two types: wholesale pricing and retail pricing. In this chapter, the focus is on the available publications of retail pricing. Currently, the retail pricing schemes can be broadly classified into two categories: static and dynamic pricing schemes. It is widely accepted that the elasticity of the demand side can be well utilized under real-time pricing (RTP) which follows the price variations in the wholesale market. The major disadvantage of RTP is the

5 exposure of end-users to the risks resulted from price fluctuations. On the other hand, the static pricing scheme, known as time-of-use (TOU) pricing, can achieve a better balance between protecting end-users from price risks and utilizing demand elasticity. Due to the advantages of TOU pricing over RTP at the demand side, it has been widely adopted in practice and also has attracted more attentions from researchers.

In the TOU pricing scheme, since the retail tariffs in the same time interval are fixed to customers, the price risks from the spot market are undertaken by electricity retailers. Therefore, it is important for electricity retailers to manage the risks by simultaneously trading in wholesale markets (such as the spot market and futures market), and then design proper retail pricing schemes which match the trading strategies in whole-sale markets. In more details, electricity retailers firstly need to perform long-term demand forecasting for end-users; secondly, they need to assess potential risks caused by the volatilities of customer demand and spot market prices. Moreover, innovative pricing schemes are also necessary by taking into account the emerging factors such as the increasing penetration of renew-able energy, widely deployment of distributed generation and storage devices, adoption of advanced information and communication technologies and rising customer awareness of switching among electricity suppliers.

To help researchers and engineers understand the state-of-the-art of electricity retailer pricing, this chapter presents a comprehensive survey of recent works in subject. The rest of this chapter is organized as follows. Section 2.2 presents the decision framework for electricity retail. Sections 2.3, 2.4 and 2.5 review the long-term demand forecasting, energy procurement and retail pricing strategies, respectively; in Section 2.6, the risk management strategies for electricity retailers are summarized; Section 2.7 discusses some potential future research directions in electricity retailing; finally, the chapter is concluded in Section 2.8.

2.2 Decision Framework

An electricity retailer is an intermediary between electricity producers and consumers. Currently, the electricity retail company is usually operated as an entity that is independent of any generation or distribution company. The retailer purchases energy from the wholesale and futures markets and resells it to customers through various retail contracts. In order to gain profits, electricity retailers face the problem of making optimal decisions on electricity markets in both the supply and retail sides. Obviously, the electricity consumption of

6 customers is the basis for electricity retailers running their business. Therefore, estimation of load demand should be the first thing in the decision-making process of retailers.

Many decision-making problems with the presence of uncertainty in energy markets are discussed in the well-known book [1]. In [1], firstly, a detailed introduction to the structure of the electricity market and the stochastic programming based decision-making methodology are presented. Then, the decision-making process of electricity producers and retailers is further studied, respectively. In addition, the book also provides backgrounds and models of the electricity market clearing process. In this chapter, the main procedures in electricity retailers’ decision-making are considered, including the retail energy forecasting, portfolio evaluation, and risk management.

2.2.1 Retail Energy Forecasting

The retailer runs its business by purchasing energy from the wholesale electricity market and then reselling it to end-users through retail contracts. In electricity retail markets, the retail contracts usually span from several months to several years. In order to gain profits and ensure the business going well, retail energy forecasting is of great importance in following aspects:

(1) Load forecasting is necessary for the retailer to reliably supply electricity to customers. Retailers need to forecast the energy demand of their customers so as to make electricity purchase decisions in the electricity market;

(2) Load forecasting is the prerequisite for making optimal purchasing portfolio. Accurate understanding of customers’ load can help retailers avoid making too loose portfolio decisions, which will result in unnecessary costs. On the contrary, if the decisions are too tight, it will also cause high risks on electricity supply and cost.

(3) Load forecasting is crucial to electricity retailers in the course of managing their risks. The retailer mainly faces two kinds of uncertainties: wholesale electricity price and client demand. Through improved understanding of customer loads, the retailer will be able to uncover and mitigate excess exposure to load and price volatility risks.

(4) Load forecasting is essential for developing competitive retail pricing schemes. When serving customers, the volume, shape and variability of load demand determine the final cost. Accurate load forecasting can help provide a beforehand understanding of the true cost and risk, which would result in pricing systems that improve retailers’ competitiveness and profitability.

Considering the significance of retail energy forecasting on electricity retailers’ decision- making, the long-term load forecasting techniques are firstly analysed in Section 2.3.

2.2.2 Portfolio Evaluation

A significant challenge for an electricity retailer is to accurately determine the value of its portfolios. The retailer is a middle person between generation companies and customers, and its core business is to purchase electricity from various sources and resell it to customers. An optimal energy procurement portfolio could thus help the retailer not only ensure a stable and reliable supply source, but also control the cost. On the retailing side, portfolio valuation of electricity retail contracts is necessary when developing retail plans. Competitive retail plans should provide stable revenue by attracting more customers. Moreover, because of uncertain spot market prices and random end-user demands, the retailer is exposed to significant risks. Therefore, risk management is a critical element for electricity retailers when making optimal portfolio.

As the main components of portfolio evaluation for electricity retailers, the existing researches on retailer’s energy procurement and retail pricing strategies are reviewed in Sections 2.4 and 2.5, respectively.

2.2.3 Risk Management for Electricity Retailer

Due to the volatility of spot prices and the stochastic nature of customer demands, the retailer faces profit risks when making retail decisions. Even though the retail profit is usually characterized by its expected value, the retailer needs proper risk measures to control the volatility of profit in case of suffering from extreme loss. Various risk measures and risk management strategies have been proposed and incorporated into the decision-making process of electricity retailers. In Section 2.6, the risk management in electricity retail market is discussed in detail.

2.3 Long-term Load Forecasting for Electricity Retailers

Load forecasting is a field in which extensive researches have been reported. Based on the forecast lead times, load forecasting problems can be generally categorized into two categories, namely short-term load forecasting (STLF) and long-term load forecasting (LTLF). STLF refers to the load forecasting with lead times of up to several weeks, which is

8 mainly conducted for short-term power system operations. LTLF aims to forecast the future load with a longer lead time up to a few years. LTLF is mainly used for long term system planning.

Most of the existing load forecasting methods focus on STLF, and much less researches are reported on LTLF. Various techniques for STLF have been proposed, such as artificial neural networks (ANNs) including the extreme learning machine (ELM) in [2, 3] and the SVM in [4], the linear regression in [5], the semi-parametric additive model in [6], and the statistical method in [7]. STLF in the smart grid context is studied in [8] where the smart data collected by advanced metering infrastructure are used to improve the forecasting accuracy.

In particular, in [9] a comprehensive review of the evolution of energy forecasting practices is presented, starting from the time when Edison founded his steam-powered station. Load forecasting approaches that are proposed from the pre-PC era to current smart grid ear are summarized.

In this section, the discussion will focus on LTLF. This is due to the fact that the retailer’s decision is usually made for a long period of time.

2.3.1 Literature survey on long term load forecasting methods

In traditional load forecasting, the aim is usually to forecast the total consumption of the whole power system or a specific node. After the market deregulation, the load forecasting of electricity retailers is becoming an emerging research topic. Comparing with the system/node level load forecasting which is less affected by the customer behaviors of changing their electricity retailers, retailer load forecasting is a more challenging problem due to the short history of electricity retail markets and the changing behaviors of customers.

Most of the existing load forecasting methods focus on STLF, and much less research reported on LTLF. To forecast the long term consumption of a retailer, the existing LTLF approaches can also be applied after modified by adding a module of forecasting the number of consumers with electricity provided by the concerned detailer. Therefore, the long term retailer load forecasting mainly consists of several sub-tasks: model establishment, variable selection, scenario simulation, and forecasting of the number of consumers supplied by the retailer.

1) Forecasting Model Establishment

Many STLF approaches are employed to solve LTLF problems with minor modifications. Multiple linear regression analysis and semi-parametric additive models are used for LTLF in [7] and [10], respectively. In [7], STLF models are modified for LTLF by adding a macroeconomic indicator. While in [10], to perform LTLF, the model is split into sub- models which represent annual effects (economic and demographic variables) and half- hourly effects (temperature and calendar variables) respectively; each sub-model is then estimated separately. In [11], a forecasting model library is established, including a logistic model, a time series approach based model, a straight model, a gompertz model, an exponential model, and a polynomial model. The model library is then combined with an expert system to accept user inputs and predefined production rules to choose the most suitable model for LTLF.

Some researchers directly apply STLF models to LTLF. This is based on the assumption that there is no significant change in the electricity consumption patterns of an ordinary residential customer over a few years. In [4], the authors used STLF techniques to do LTLF for residential customers. The only difference between their work and the traditional STLF methods is that they model the number of future customers as a factor which influences the long-term load of customers.

2) Variable Selection

Weather, economic, demographic, and calendar variables are the most important factors in LTLF models. Weather variables [10] could include the average temperature in last 7 days, maximum/minimum temperature in last 24 hours, temperature deviations of the same time period of previous days, temperatures of the same time period for last two days, and so forth. Economic variables [7] include macroeconomic indicators such as GSP (gross state product), GDP (gross domestic product), and GMP (gross metropolitan product). The values of these variables should be determined based on the territory covered by the LTLF. Other economic variables namely CPI (consumer price index), household sector per capita disposable income, average electricity price, and the proportion of households with an air-conditioning unit also have been used in LTLF [10]. Demographic variables could include the residential population, persons per household, and number of households [10]. Several calendar variables are usually considered, including the day of a week, holiday effect, and day of summer effect [10, 12].

In addition to the variables discussed above, other variables deriving from the interdependences among these factors could also be incorporated into LTLF models.

Nevertheless, more variables do not necessarily lead to more accurate forecasting results. For a specific application, the most suitable variables are usually selected through repetitive trials.

3) Scenario Simulation

For LTLF, probabilistic forecasting techniques are more preferred over point forecast techniques that are often used in STLF. In probabilistic forecasting techniques, various forward scenarios of weather, economic, and demographic need to be created. Usually, the long term economic and demographic information could be obtained from the supervisory authorities or purchased from third parties. Therefore, existing publications often focus on the simulation of forward weather scenarios.

For scenario based probabilistic LTLF, temperature data ranging from 20 to 50 years are usually used to generate weather scenarios. In [7], temperature data of the past 30 years are used to generate scenarios by combining with base, aggressive, and conservative macroeconomic scenarios, respectively. While for the density forecasting of LTLF, a large number of possible simulated series are necessary to generate accurate estimations. However, there are no enough historical data available due to the short history of the electricity retail market. To tackle this problem, a double seasonal block bootstrap method is proposed in [10]. In the method, the annual temperature data are firstly segmented into multiple blocks and each block consists of an integer indicating the number of days. Then, the simulated annual bootstrap series are generated in which each block is from the same time period of a randomly selected year. Therefore, both daily and annual seasonality of historical temperature data are preserved when generating weather scenarios. Fig. 2.1 [10] gives an example of the simulated yearly bootstrap series.

B1: 2007 B2: 2005 B3: 2002 B4: 2001 ··· B17: 1999 B1: 1999 B2: 2007 B3: 2002 B4: 1999 ··· B17: 1998 B1: 2002 B2: 1997 B3: 2003 B4: 2001 ··· B17: 2004

 B1: 2003 B2: 2003 B3: 2004 B4: 2006 ··· B17: 1997

Fig. 2.1 Schematic diagram of a demonstrated bootstrap series

4) Consumer Count Forecasting

A fundamental element of consumer count forecasting is to model the switching activities of consumers between electricity retailers and forecast the tenured customers. Survival analysis (also known as time-to-event analysis) was introduced to analyze the customer behavior [12].

By analyzing the data of the lifetime of each customer account, the LIFETEST procedure of SAS [12] was used to derive the hazard function represented by Eq. (2.1), which is the probability of having the disconnection at time t given no prior occurrence of the event. After estimating the parameters in the hazard function using the life table method, the conditional probability of a customer i stay with the retailer within Ti days to stay connected for additional k days can be expressed by Eq. (2.2).

HPTtTtT   (2.1)

k1 SH1 TkT Tji  (2.2) ii j0 Finally, the expected number of existing event-free customers that will stay connected for additional k days can be obtained by Eq. (2.3),

n k1 NHTenured 1 (2.3) Tji  i1 j0 where NTenured is the number of loyal customers for the retailer; n is the total number of event- free customers at the beginning of the forecast.

Different from the methods discussed above, a method is presented in [13] to estimate the future load of a retailer. By using the Monte Carlo simulation technique, multiple scenarios with different load levels (low, medium, and high) are generated. Then, various portfolios of strategies for estimating loads of retailers, including overestimating, underestimating, always using a median estimated value and always using the maximum estimated possible, are evaluated while taking into account the cumulative track record of the profit from previous hours. By assessing the profit distributions of the estimation strategies, the best strategies can be finally selected.

2.3.2 Industry applications of long term load forecasting methods

As discussed above, various load forecasting techniques have been proposed including regression, multiple regression, artificial neural networks (ANNs) based techniques, exponential smoothing, statistical method, iterative least squares techniques, and so on. The methods adopted in industry practice are also based on these methods.

In [7], a probabilistic forecasting approach with hourly data is proposed. The LTLF problem is dissected into three element three elements, namely predictive modelling, scenario analysis, and weather normalization. When selecting the predictive models, the previously proposed multiple linear regression models are modified by adding a macroeconomic

12 indicator, GSP (Gross state product). The proposed approach has been deployed to many large and medium size utilities in the USA including NCEMC (North Carolina Electric Membership Corporation).

An implementation of spatial load forecasting work at Madison Gas and Electric Company in the USA is presented in [14], but the spatial load forecasting method is originally proposed by [15]. In the proposed method, an automatic computerized land use based method to calculate horizon year load (HYL) is first introduced, where HYL describes the load of a fully saturated small area. In long term spatial load forecast, an estimate of the farthest forecast year load can be considered as HYL, although the land may be further developed or redeveloped after the forecast range. It is crucial to get a quality HYL for a useful forecast. Considering that the results produced by the automated computer program may lack common judgment. Therefore, as an enhancement to the land use based method, Ref [15] proposed a methodology to integrate the human planner(s) into the problem solving loop to provide heuristics and insights to correct or confirm the results from the computer. This iterative calibration process, in which human and machine work together and negotiate with each other to come up with a solution, is named as human machine co-construct intelligence (HMCCI) method. This program is then purchased and applied by the Madison Gas and Electric Company to develop accurate spatial load forecasting.

In Australia, AEMO (Australia Energy Market Operator) takes in charge of developing and publishing annual electricity demand forecasts for the electricity industry. The forecasts were produced in components, which were then aggregated at a regional level to produce regional forecasts for energy and demand. These components include: (a) residential and commercial load, (b) large industrial load, (c) transmission losses, (d) auxiliary losses, and (e) small non- scheduled generation. As presented in [16], the annual consumption forecasts were developed using econometric methods which estimated the relationship between historical electricity consumption and the key drivers that determine residential and commercial consumption (income, electricity price, weather, and population). The estimates, also known as coefficients, were then used in conjunction with forecast values for the key drivers, to derive electricity consumption forecasts. In term of long term load forecasting, the regression model is adopted by AEMO. The Dynamic Ordinary Least Squares (DOLS) approach is used to estimate the relationship between electricity consumption and a number of long-run drivers (such as income and electricity prices). AEMO adopted this approach because DOLS enables a valid and consistent approach to be applied across all NEM (national energy market) regions. Besides, DOLS provides an efficient estimator for the long-run relationship in the presence of variables with differing and higher orders of integration. In addition, the

DOLS method is known to be effective when working with small datasets where endogeneity may be present.

In the European energy market, the commercial software developed by AleaSoft for energy demand forecasting has been widely adopted by a variety of market operators in Europe, such as the Nordic Pool Spot Market, the Electricity markets in Central Eastern Europe – CEE (Czech Republic, Hungary, Poland, Romania, Slovakia, Slovenia), and the Spain and Portugal MIBEL electricity market (Iberian Electricity Market) [17]. The AleaDemandLong developed by AleaSoft for long term load forecasting can offer automatic long term demand forecasts up to 15 years ahead. The forecasting models are developed by AleaSoft specialists. The model selection is done using AleaSoft own modelling tools, based on Genetic Algorithms, Neural Networks and Statistics. The modelling process includes the selection and analysis of the suitable explanatory variables. There is also other load forecasting software, such as the Energy Load Planning module of the ABB Energy Manager software[18], and the Load for (Electricity load prediction of electrical power systems) developed by the EMD Company in Demark[19]. However, compared with the AleaSoft, their products are not so commonly adopted.

2.3.3 Time resolution in various load forecasting techniques

Load forecasting can be divided into long-term, medium-term, and short-term load forecasting according to the length of the forecasting duration. Long-term load forecasting typically refers to forecasts made 1 to 10 years in advance, which is often used as a basis for the determination of future energy demand and planning policies. The medium-term load forecasting often refers to forecasts made for a few weeks to a few months in advance, which is used to guide planning in the power system. Short-time load forecasting includes forecasts made for a few hours to a few days in advance, which can be used to supervise the operation and control of the distributed energy system.

The time resolution of load data in existing load forecasting publications is summarized in table 2.1. From table 2.1, it can be found that even though various load forecasting methods have been proposed, but most of these publications concentrate on the hourly load forecasting. There are several reasons that may lead to this result. Firstly, the load forecasting model may do not have a requirement on high-resolution load data. For example, in the regression model of load forecasting, the independent variables usually are the temperature, economic or some demographic indicators. These variables usually keep unchanged for a period of time. Therefore, the high resolution data on these variables are meaningless for improving the model performance. Secondly, as in the actual electricity

14 market, the settlement period is usually half – hourly, then the available load data will be restricted to half-hourly data, such as in [10].

In addition, with the fast development of the Smart Grid, some research has proposed to use the smart meter data to help improve the load forecasting accuracy [8]. At this point, the high resolution load data will be available, such as the every minute load data. Then this high resolution load data can be used for developing more accurate load forecasting algorithms. Even though using the smart meter data to develop novel load forecasting method is a promising direction, currently there is seldom such research being reported.

Table 2.1 Time resolution of load data in load forecasting publications Time resolution of load data Reference # and Method Hourly load data The extreme learning machine (ELM) [2], the RBF network [4], the fuzzy interaction regression approach [5], the statistical method [7], and the survival analysis based method [12] Half-hourly load data The semi-parametric additive models [10], and the extreme learning machine (ELM) [3] 15-min and 30-min load data The clustering analysis and neural network (NN) combined method [8] 10-min load data The semi-parametric additive model [6]

2.4 Energy Procurement Strategy for Electricity Retailers

When determining the optimal procurement strategy, the retailer first needs to identify available energy sources, which mainly include the electricity spot market and futures market. Various internal and external factors are incorporated into the decision process of retailers, such as price trend, risk management strategies, end-user demand, and market competition. Then the retailer establishes its optimal energy procurement portfolio after the calculation using preferred procurement optimization approaches. In particular, the energy procurement process consists of following parts:

2.4.1 Sources of Electricity Supply and Involved Markets

As a middleman between the Generation Company and customer, the electricity retailer purchases electricity from various sources and then sells it to customers. In [20-24], the retailer supplies consumer demands by purchasing from the spot market, forward contracts, call options, and self-production facilities. Interruptible load contracts that allow the retailer to interrupt part or all of the electricity supply to customers over some periods of time can also be introduced. In [25], the electricity procurement of a retailer is modeled as an investment problem where the retailer invests in both the whole sale electricity market and financial market. In most of the existing publications, energy storage units are rarely considered in the energy procurement decision-making process. In [20], energy storage units owned by the retailer are considered and scheduled in the procurement strategy optimization in the energy market.

Retailers can acquire the consumer demand from various sources, but they need to participate in the bidding process of the spot market through submitting their bidding curves. In [21], retailers submit hourly price-quantity bidding pairs to the spot market for purchasing and conduct real time balancing either through trading in the regulation spot market or using the self-production facility. Aiming at maximizing the profit, when offering retail plans for customers, spot market indexed contract and TOU price based contract are considered, respectively. In [22], it is supposed that the retailer also submits hourly price-quantity bids to both the day-ahead and intraday markets. Given the forecast of consumer loads, the bidding model is built by minimizing the purchasing cost at the sequential trading opportunities in the joint market.

2.4.2 Factors Influencing Energy Procurement

The energy procurement strategy of a retailer is influenced by several uncertain factors in electricity markets, such as spot prices, demand elasticity of customers, and the competition among retailers. These risks need to be properly measured to quantify their impacts. As the demand elasticity and competitions in the retailing market will determine the total load served by the retailer, a few approaches have been proposed to model their impacts on the procurement strategies.

1) Uncertain factors

In the retail market, uncertainties are mainly from the prices in the spot market and customer demand. Various methods have been proposed to model the dynamics of electricity prices in

16 the spot market by considering the price characteristics such as seasonality, time-varying volatility, mean-reversion, and jumps and spikes [26].

The probabilistic processes of the electricity price and customer demand are considered to be independent [26], [25] and correlated [27-29], respectively. Normal distribution is the most common one for modeling the electricity price and customer load due to its simplicity [23, 28]. Other models have also been used, including the GARCH-jump model [26], GARCH model [26], mean-reverting Ornstein-Uhlenbeck stochastic process [27], and envelope bound model [29].

However, the electricity price and customer demand are not always assumed to follow a particular probabilistic distribution. For example, when applying the robust optimization approach to develop optimal energy procurement strategy, it would be only necessary to know the estimated boundaries of the random variables.

2) Risk measurement

The problem of developing an optimal electricity retailing strategy can usually be formulated as a multistage stochastic optimization problem, which could be nonlinear, non-convex, and the stochastic variables are not always assumed to follow the normal distribution. This would lead to the failure of risk measures such as variance in measuring the risks faced by retailers. In contrast, as a coherent risk measures, CVaR (conditional value-at-risk) is an alternative to VaR (value-at-risk), which overcomes the disadvantages of VaR and has been widely used in risk management for electricity retailers [26, 27].

Other risk measures have also been used, including the risk adjusted recovery on capital (RAROC) and expected downside risk (EDR). In [30], a risk measures called RAROC which indicates the ratio between the expected investment return and economic capital is introduced to quantify the risk of signing different bilateral contracts for a retailer. When the loss function is convex, EDR is convex. Due to this merit, EDR is employed in [28, 29] to measure the risk associated with a decision by its failure to meet the predefined targeted profit.

3) Demand elasticity of customers

Several functions have been used to represent the price elasticity of demand, such as the linear function [24, 31], power function [21, 32, 33], and stepwise function [34]. Among these methods, the stepwise function namely a stepwise price-quota curve is the most commonly one.

In [23, 28, 29], demand elasticity is modeled through a stepwise price-quota curve, which describes the amount of electricity that a customer is willing to buy at a certain price. Meanwhile, this stepwise price-quota curve also describes the switching activity of customers among retailers when facing different retail pricing schemes.

In the above mentioned publications, demand elasticity is modeled as a function of the electricity price only. By taking into account external variables (e.g., outdoor temperature), the demand price elasticity is modeled by finite impulse response (FIR) models in [35] and [36]. In [36], both linear and nonlinear FIR models are derived to capture the demand price elasticity. In order to estimate the coefficients in the FIR models, recursive and adaptive estimation is employed.

In modelling the demand elasticity, there are difficulties related to collect information about the load schedule and utility function of an individual. It is even harder when end users have privacy concerns. To overcome this problem, in [37] the utility function of a responsive load is modelled by a parametric stochastic process and the parameters are estimated through statistical analysis of load variances under different RTP scenarios. However, the proposed statistical demand elasticity model is only useful for aggregated loads.

Unlike other studies in which an assumption about the demand elasticity function is often a prerequisite, data-driven methods that do not require such an assumption are proposed in [38] and [39].

As is well known, a typical optimization problem is a forward problem because it identifies the values of observable parameters (optimal decision variables), given the values of the model parameters (cost coefficients, right-hand side vector, and the constraint matrix). On the contrary, an inverse optimization problem is to infer the values of the model parameters (cost coefficients, right-hand side vector, and the constraint matrix), given the values of observable parameters (optimal decision variables) [40].

In [38], the inverse optimization is the first time adopted to infer the feasible region of price- responsive demand in aggregator’s optimal consumption decision model. The main merit is that it does not require any assumption about the expression of the price-responsive load. The deferrable demand is described by a series of quadruples, namely the lower and upper limits of consumption power and cumulative energy for each load. Using above mentioned deferrable demand model, a cost minimization model is built for aggregators to make optimal consumption decision. Through generating simulated price-consumption data, the inverse optimization method is used to solve the aggregator’s decision model to reveal the

18 feasible region of price-responsive load demand. Besides, a data-driven pricing model for the utility is formulated based on the estimated price-responsive demand.

Similarly, the parameters of price responsive demand are also estimated through inverse optimization in [39]. However, some external variables that can potentially affect the electricity consumption are considered, such as temperature, solar radiation, and wind speed. These external variables are incorporated into the model by using the affine functions.

4) Competition among electricity retailers

Except for the stepwise price-quota curve, the market share function introduced from the econometrics is another way to model the competition of retailers. Retailers are usually heterogeneous in their long-term strategies, risk attitudes and marginal cost of electricity procurement. Therefore, they may choose different pricing strategies. In this condition, each retailer would select a mixed strategy [26]. A mixed strategy can be interpreted due to retailer’s uncertainty about the pricing decisions of competing retailers. In other words, each retailer would generate a price distribution, i.e. equilibrium prices are defined by a probability density function, indicating the range of prices that the retailer may charge. The probability density function according to which the retailer sets its equilibrium prices is defined as the market share function. In electricity markets, the market share function shows the percentage of the overall load that can be served by the retailer at different prices. Several factors such as long-term strategy of the retailer, risk attitude, the behavior of customers and the behavior of the competitors must be considered in determination of market share function [41]. In [26], the market share function of the electricity retailer is constructed as a function of the sale price of electricity to customers. It indicates the percentage of the overall load that can be served by the retailer at different prices is adopted.

Different from the stepwise price-quota curve adopted in [23, 28, 29] and the market share function used in [26], in [30] the competition between electricity retailers is modelled through dividing consumers into loyal consumers and switching load / consumers. Retailers compete to supply the consumers through bidding. The retailer who has lower bids will win the switching load / consumers.

2.4.3 Procurement Optimization Approaches

Up to now, a variety of models and approaches on energy procurement optimization have been proposed, which can be categorized as follows.

1) Stochastic optimization models

A mixed-integer stochastic programming model is proposed in [26] to determine both the optimal retail pricing and procurement strategy for a given retailer by considering the risk and competitions among retailers. In [23], an optimal strategy composed of a medium term planning model and a short term planning model is presented. The medium term planning is modeled as a stochastic optimization problem where the energy purchased from forward contracts and electricity retail price are determined. Given decisions determined in the medium term model, in the short term model, the interruptible load contract and the amount of energy procured in the spot market are optimized.

Scenario based stochastic optimization models are proposed in [27, 28] and [42] to generate the random scenarios of electricity price and customer demand. Various scenario generation techniques, i.e., scenario tree [27], Roulette wheel mechanism and lattice Monte Carlo simulation (LMCS) [28], are employed to generate scenarios over the planning horizon, respectively.

In [42], stochastic programming is used to model the medium-term decision making problem faced by a distribution company (DisCo). The DisCo is equivalent to an electricity retailer, but it also takes in charge of the operation of distribution networks. Therefore, the studied DisCo not only deals with the portfolio evaluation and retail pricing, but also considers the economic operation of distribution networks. Scenario generation and reduction techniques are used to transform the stochastic model into a deterministic problem. The nonlinear terms are linearized through the piecewise linear approximation. Finally, the proposed model is formulated and solved as a mixed integer linear programming (MILP) problem.

In [43], the impacts of various retail prices on DisCo’s forward contract scheduling are studied. The stochastic decision making framework is established to simulate DisCo’s decision making processes. Different retail pricing schemes including flat, TOU, and RTP prices are considered, respectively. The results show that RTP rates provide the most benefits to DisCos, since RTP can directly pass DisCo’s risk to customers. TOU rates can also bring high profits but it has no significant impact on risk exposure. Besides, amounts of forward purchases required for risk management are almost identical when adopting flat and TOU schemes.

2) Bi-level optimization models

Unlike the research work reported in [23, 26-30] which study the medium-term or long-term procurement strategies for electricity retailers, the decision-making problem of the retailer is represented by a two-stage model in [20]. The decision variables in [20] include the day-

20 ahead retail prices, operational strategies of energy storage units, and energy contracts. In the first stage, a Stackelberg game is established and a bi-level programming problem is formulated, where the retail price determination problem and customers’ energy consuming pattern identification problem are modeled in the upper and lower levels, respectively. In the second stage, another bi-level model is formulated. The operations of energy storage unit, which are optimized both in the upper level and the lower level models, are the same with those in the first stage.

3) Metaheuristic optimization models

Generally there are two categories of optimization techniques, and both have been widely applied in the power system optimization [44]. The first category is mathematical programming methods, including linear programming, dynamic programming, mix-integer programming, branch-and-bound method, etc. The mathematical programming methods have some merits, such as time efficient and can ensure to converge to local optima, while their major limitation is the global optimization capabilities. The second category is the heuristic based optimization methods, such as genetic algorithm (GA) [45], differential evolutionary (DE) algorithm [46], particle swarm optimization (PSO) [47], etc. Heuristic based techniques perform stochastic searches in high dimensional problem spaces, and have advantages like robustness and global searching capability.

However, in existing researches on electricity retail pricing, there is no publications using metaheuristic optimization models haven been reported. Usually, the electricity retail pricing problem is modelled as mathematical programming models or modelled as stochastic optimization models and then solved using commercially available solvers.

In [26], the proposed mixed-integer stochastic programming model for optimizing procurement strategies of electricity retailers is solved by a two-stage recourse approach. In each stage, stochastic dynamic programming was used to calculate the associated variables. In [24] the decision-making problem of a retailer is formulated as a mixed-integer stochastic programming. It is solved by a decomposition technique, and the decomposed parts are solved by a branch-and-bound algorithm. In [27] and [32], the proposed stochastic optimization models are solved by the Dash Optimization software Xpress [48]. While the stochastic programming models in [34], [42] and [43] are solved via the general algebraic modelling system (GAMS) [49] with its embedded commercial solvers. Besides, the mixed integer linear program (MILP) model for the energy pricing and dispatch problem faced by a smart grid retailer in [20] is also solved by the commercial solvers in GAMS. In [50], the

21 proposed nonlinear programming model for PEV aggregator decision-making is solved using solvers within GAMS as well.

4) Models based on other theories

In [29, 51], the information gap decision theory (IGDT) based electricity procurement models are proposed. The envelop bound model and info-gap model are used to model the uncertainties of parameters, respectively. These models can produce both a robust strategy and an opportunity strategy depending on the risk-averse degree of retailers. The robust strategy is immune against losses or low profit due to unfavorable deviations of spot prices from the forecasted values. While the opportunity strategy enables a risk-seeker retailer to benefit from these favorable variations.

A mean-variance investment model is developed in [25] to seek optimal decisions for a retailer. The retailer is modeled to allocate a part of the associate wealth to purchase electricity from the utility company concerned for reselling and to allocate the remaining wealth in the financial market.

2.5 Retail Pricing Strategy for Electricity Retailers

Widely adopted retail pricing schemes in real world are investigated and summarized in [52]. Retail pricing schemes are categorized by their rate structures in which prices vary on an hourly, daily, or longer time-period basis, respectively. Specifically, hourly pricing includes basic hourly pricing which is directly indexed to hourly wholesale energy prices, block and index pricing, two-part RTP, and unbundled RTP with self-selected baseline load. Daily pricing includes the day-type TOU rate, variable peak rate, critical peak pricing (CPP), variable CPP, CPP linked to a standard tariff, and peak-day rebate. Other pricing schemes are also included, such as the fixed TOU pricing and seasonal flat pricing. For each category of retail prices, price description, principles for setting the prices, and examples of implemented versions are elaborated. Besides, the advantages and drawbacks of above mentioned time-based price structures are discussed in detail. Readers interested are advised to refer to [52] for further information.

With the fast development of smart grid technologies in recent years, some trial real-time retail pricing programs have been reported and viewed as a future trend [25]. One demerit of RTP is that it directly exposes end-users to the price fluctuation risks. Therefore, it is difficult for small electricity customers to accept the RTP scheme. As a pricing scheme falling in between flat pricing and RTP, time-of-use (TOU) pricing is widely adopted in

22 practice [53]. There are also other pricing schemes developed by combing RTP and TOU pricing schemes. For convenience, here the term ‘dynamic pricing’ is used to denote the pricing schemes which are time-varying, and the term ‘static pricing’ refers to the pricing schemes which are pre-determined. The state-of-the-art of researches on these two classes of pricing schemes is summarized as below.

2.5.1 Dynamic Pricing Schemes

Various dynamic pricing schemes with different update cycles, such as 5 minute, 15 minute and hourly, have been explored in existing publications. The market structure, volatility and trading mechanisms are the issues that need to be considered in designing new pricing schemes.

1) Architecture design of real-time pricing market

With the advent of smart grids, real-time customer participation and electricity pricing are proposed. In actual operating electricity markets, these changes can be incorporated if the pre-requirements can be met.

In [54], a new bid-free real-time electricity market structure is proposed. Consumers and producers respond to the real-time signals by properly adjusting their consumption and production comparing with the day-ahead schedule. Then the transmission system operator (TSO) dispatches the real-time adjustment as balancing power in the regulating power market. Different from the centralized optimization method used in [54], in [55] a decentralized mechanism is proposed. In a multiple micro-grid system, the pricing problem is initially modeled as a centralized social welfare maximization problem. Through dual decomposition, the initial model is transformed into a non-cooperative game. Players in the game make decisions to maximize their own profits after receiving real-time prices from the ISO. It is proved that an elaborately designed subsidy function can ensure the existence of the Nash equilibrium of the non-cooperative game, and the Nash equilibrium coincides with the solution of the original problem.

Similarly, a distributed dynamic pricing scheme is presented in [56] based on a bidirectional communication network in the smart grid. Smart end users timely communicate with the regional control center through the community gateway installed within each community network and used for electricity usage collection and price indication. All the exchanged data are encrypted to protect the privacy of end users. Community gateways receive pre-defined parameters from the regional control center to dynamically differentiate peak and off peak periods, calculate real-time prices, and send the price signals to end users. Through enhanced

23 encryption, the proposed scheme can protect the privacy of customers at a relatively high level.

2) Market volatility analysis under dynamic pricing

Exposing retail consumers to the real-time electricity pricing mechanism will create a closed-loop feedback system and may also increase the market volatility. The influence of real-time pricing on market volatility is studied in [57]. Based on several assumptions, a theoretical framework for modelling and analyzing the dynamics of suppliers, consumers and the ISO is proposed. Several stability criteria are selected based on the Lyapunov theory and contraction analysis to conduct stability, invariance and volatility analysis. The theoretical analysis suggests that market volatility is linked to the ratio between the price elasticity of consumers and that of producers. The electricity market will become more volatile as the ratio increases.

3) Bi-level real-time retail pricing models

Optimization problems constrained by complementarity and other optimization problem are appropriate for describing the interactions among market participants. Therefore, they are widely used to model the functioning of energy markets. Generally, if the lower-level optimization problems constraining the upper level problem are convex, they can be replaced by their corresponding Karush-Kuhn-Tucker (KKT) optimality conditions. All the mathematical models with complementary constraints are called complementarity problems. In [58], the applications of complementarity models in the energy market are comprehensively elaborated.

The problem of setting the day-ahead hourly retail price is studied in [20] and [59]. They both use the Stackelberg game to model the interaction between the retailer and its customers, and two- and single- stage games are established in [20] and [59], respectively. Different solving methods are adopted. In [20], the two-stage model is transformed into a mixed integer linear program, while in [59] the Pareto front of the consumer surplus vs. retail profit tradeoffs is generated.

In [60], a bi-level optimization model is proposed for determining the optimal dynamic retail electricity price, which balances the monetary benefits between utility companies and large industrial customers. The utility company at the upper level devotes to maximize the profit of electricity sale by tuning dynamic retail electricity prices, while large industrial customers at the lower level minimize their power consumption costs by optimally scheduling tasks and generation outputs of self-provided power plants. The proposed model is finally

24 reformulated into a mixed-integer linear programing (MILP) problem, and solved by a hybrid optimization algorithm by integrating genetic algorithm (GA) and MILP.

When using the bi-level optimization to set retail prices, the pricing model often focuses on the interaction between the retailer and customers, and the merits are also manifold. Firstly, bi-level optimization can properly model the hierarchical decision process faced by the retailer in the hierarchical electricity market, i.e., wholesale and retail electricity markets. Secondly, customer load demand is determined through solving the lower level model instead of adopting an aggregated random variable (such as the approach in stochastic optimization model). Therefore, components of residential load can be more accurate.

However, there are also drawbacks. In the bi-level optimization, since load demand is determined through optimization, it fails to reflect the risk of uncertain electricity consumption in the final retail price. What’s more, the lower level of the bi-level optimization needs to be properly modelled to be convex. If not, analytical tools for solving the complete hierarchical problem usually do not exist. If the lower-level problem is convex, it can be replaced by their corresponding KKT conditions. Then the bi-level optimization problem is transformed into a complementarity model.

Diverse linearization techniques are available to further transform the complementarity model into a mixed-integer linear programming problem (MILP), such as the Fortuny-Amat McCarl linearization [61], SOS1 and Penalty Function Linearization (Special Ordered Sets of Type 1 variables, SOS1 variables) [62], and other linearization methods based on exact algebraic transformations [58]. Only in a few literatures, the lower level is directly modelled as a linear and convex programming model like in [50]. In [20], the duality theory is used to linearize the lower level max-min problem. Non-linear complementarity constraints, which are derived from KKT reformulation, are linearized into linear disjunctive constraints through Fortuny-Amat McCarl linearization.

Besides, due to the flexible charging strategy and energy storage, an electric vehicle (EV) is viewed as a special load in the power system. Retail pricing for EV charging is studied in [50] and [63], the stochastic optimization and bi-level programming are used respectively to establish the retail pricing models. In [50], bilinear terms in the lower level model is linearized by discretizing the variable with reasonable granularity. After recasting the bi- level optimization into MILP, the problem can be easily solved using various off-the-shelf commercial solvers. In [64], the effect of CO2 emission on retail pricing is addressed. The

CO2 emission tax is incorporated into wholesale market clearing models. As dynamic retail

25 prices are linked to the wholesale market clearing price, two dynamic pricing strategies, i.e., critical peak pricing and RTP, are used to test the proposed method.

2.5.2 Static Pricing Schemes

There are various static pricing schemes, e.g., stepwise pricing, critical peak pricing, demand reduction programs and TOU. In [65], a stepwise power tariff (SPT) model for residential customers is proposed. However, the existing related publications are mainly devoted to the TOU pricing. The state-of-the-art on TOU pricing can be categorized as below.

1) Stochastic programming models

By modeling the uncertain factors with ARMA (autoregressive moving average model) [24], ARIMA (autoregressive integrated moving average model) [34], or scenario based method [32, 66, 67], stochastic programming models can then be built to maximize the profit of the retailer while minimizing the risk, or to maximize the total social welfare. The risk and expected profit are usually merged into one objective by adding a weighting factor. Moreover, constraints associated with forward contracts, elasticity of demand, and energy balance are also considered.

2) Equilibrium models

The pricing problem in the retail market is modelled as an equilibrium problem of supply and demand in [33] and [53]. In their works, the objective of the supply side is modelled to minimize the total operating costs of generation facilities. And on the demand side, the customer demand is represented by demand equations that use the electricity price and lagged demand as independent variables. The objectives of equilibrium models in [34] and [68] are to maximize the total social welfare.

3) Game-theoretic models

Unlike other publications where end-users have slow response, in [69] a game-theoretic model is proposed to optimize the TOU pricing strategies, where end-users are supposed to response immediately. In the model, the objective of the utility is to maximize its profit by taking into account the cost from the end user demand fluctuations. By using the backward induction, Nash equilibrium of the model is obtained.

2.5.3 Advanced Metering Infrastructures in Electricity Retail Markets

Advanced metering infrastructure (AMI) is an integrated system of smart meters, communications networks, and data management systems that enables two-way

26 communication between utilities and customers. As the foundation of the Smart Grid, AMI is responsible for collecting all the data and information from loads and consumers. AMI is also responsible for implementing control signals and commands to perform necessary control actions as well as Demand Side Management (DSM). A comprehensive survey on the AMI in the Smart Grid is carried out in [70]. Generally speaking, the key features of smart electricity meters can be summarized as follows: (1) Time-based pricing. (2) Providing consumption data for consumer and utility. (3) Net metering. (4) Failure and outage notification. (5) Remote command (turn on/off) operations. (6) Load limiting for Demand Response purposes. (7) Power quality monitoring including: phase, voltage and current, active and reactive power, power factor. (8) Energy theft detection. (9) Communication with other intelligent devices. (10) Improving environmental conditions by reducing emissions through efficient power consumption.

AMI is the basis for electricity retail pricing schemes, such as the TOU pricing and the real- time pricing. AMI will enable retailers to identify the timing of end-users’ electricity consumption. AMI allows for remote monitoring of the energy meter, automatic meter reading, automatic billing and accounting and energy auditing. In contrast, traditional meters are read at most once a month, so customers are charged irrespective of the timing of their energy use within that period. Especially, the AMI enable the implementation of real-time pricing in the electricity retail market, which can benefit both the consumers and the utility companies. Through AMI, the consumers could obtain a more organized data regarding their electricity consumption. This information reaches the consumer at regular intervals as programmed and is also accurate and error free, unlike the data obtained through manual meter reading. The power utility companies could also offer concession for electricity usage during load valley periods. The smart meter based real time pricing is introduced in detail in [71].

Smart meters have already been successfully employed in several countries. By the end of 2013, all homes and small businesses in Victoria, Australia have been equipped with advanced metering infrastructure (AMI), also known as smart meters [72]. The UK government has planned to make smart meter installation in domestic premises compulsory

27 in the year 2020 [71]. Germany requires its utility companies to provide consumers with TOU tariffs, since 2011. By 2014, around 43% of the US domestic consumers had installed smart meters. The number smart meter units currently installed has exceeded 50 million [73]. Smart meters are widely installed among end-users in Finland [74, 75]. The Norwegian government has made it mandatory for consumers consuming more than 100,000 units to install smart meters that report the energy consumption on an hourly basis [71]. By 2021, French grid manager Electricité Réseau Distribution France (ERDF) will oversee the deployment of 28 million of the uniquely French smart meter - the Linky. France has committed to have 95% of its power consumer to have smart meters installed by 2020 [76]. It is estimated that over the next 10 years the Indian government would spend around $21.6 billion on adopting smart grid technology in India and by 2021, 130 million smart meters are expected to be installed [71].

2.6 Risk Management in Retail Markets

It is generally agreed that the volatility of spot price and the stochastic nature of customer demand are two main risk sources for the electricity retailer. In addition, the risks arising from the sale contract maturity are considered in [77]. In most of the existing publications, after identifying the risk sources, CVaR is used to model the risk in the retail price determination process. The wide applications of CVaR are due to its satisfaction of properties of monotonicity, sub-additivity, homogeneity, and translational invariance [26]. Moreover, CVaR exhibits good mathematical properties and can be easily handled by using scenario based simulations [66].

To manage the risks faced by the retailer, several risk management strategies have been incorporated into the determination process of electricity retail prices.

2.6.1 Considering Risks in Retail Pricing Models

The objective of existing pricing models is often to maximize the sale profit of the retailer while minimize the risk. Therefore, the pricing model could be formulated as a multi- objective optimization problem. In [24, 34, 78], multi-objectives are transformed into a single objective by incorporating the CVaR into the expected profit function of the retailer through a weighting factor, which reflects the risk aversion degree of retailers. By assigning different values to the weighting factor, the trade-off between the expected profit and the risk can be made in the final pricing scheme.

2.6.2 Risk Premium Determination

Through incorporating CVaR into the expected profit function, the process of solving pricing models can be simplified. However, in many of existing works [24, 34, 78], the risk premium and retail price are coupled together. In [77], the risk premium from the contract- offer maturity is modelled, where the contract-offer maturity means the period from the time when the retailer sets the offer price for the consumer to the time when the consumer decides whether to accept or reject it, as shown in Fig.2.2 [77]. Firstly, for the contract-offer with a maturity of tm days, its difference with the zero-day maturity offer is calculated. Then, the risk premium Rp(tm) of the offered retail contract is derived by Eq. (2.4).

Fig. 2.2 Schedule from pricing time to the start of the delivery period

RtRt   Rt  CVaR0 CVaR m pm (2.4) EDd  t0 T where RCVaR(t0) / RCVaR(tm) is the CVaR applied to the offer with a maturity of zero-day /tm days; Dd is the expected electricity consumption in the delivery period forecasted at t0; T is the length of the retail contract.

2.6.3 Establishment of a Derivatives Portfolio

In [79] and [80], the employment of derivatives to hedge the risk for generators and retailers is studied. To evaluate the appropriateness of different forward price forecasting methods in the marking-to-market process for determining the value of the derivative portfolio, two criteria are proposed. Based on the two criteria, a time series model and consensus forecast issued by the AFAM (Australian Financial Market Association) are evaluated and compared in details. In [80], different types of electricity financial and physical instruments are reviewed, including the electricity forwards, futures and swaps, electricity options, structured bilateral transactions, and financial derivatives on electricity transmission capacity. Followed by this, various risk management applications are discussed when utilizing such instruments to mitigate risks in electricity market.

2.7 Evolution of Retail Strategy and Future Discussion

Due to the continuous restructuring and rising customer awareness, customers are able to switch among different retailers without any penalty. This allows customers to choose their most favoured retail contracts, which consequentially leads to a more competitive retail market. In the future electricity market, the retailer will see more opportunities, but also faces more technical and commercial challenges. Potential research directions related to electricity retailing are discussed in this section. Clearly, they are not restricted to the ones discussed hereafter.

2.7.1 Long-term Retailer Load Forecasting

Up to now, only a few publications are aimed at studying the long-term retailer load forecasting problem. LTLF remains a challenging problem in power systems owing to various uncertain factors that drive electricity demand variations. Some research has been carried out on commercial and industrial loads [7, 10, 11, 13]. However, in these publications STLF or modified STLF approaches are applied to solve LTLF. Long term load forecasting for residential users is studied in [12]. However, the research in [12] is based on the assumption that there is no significant change in electricity consumption patterns of residential customers. Therefore, in [12] the STLF techniques are directly used to forecast the long term demand of residential customers. The only difference between their works and the traditional STLF is that they model the forward customer count as a factor which influences the long term customer load.

As a powerful method in the learning system, artificially neural network (ANN) based methods are widely used for STLF and electricity price forecasting in power systems. Especially, the extreme learning machine (ELM) is a new and popular one. Various modified versions of ELM algorithms have been developed and used. The ensemble ELMs are leveraged in [2] and [3] to conduct STLF, where individual ELMs are incorporated into one predicting model following specific ensemble schemes to improve the overall forecasting accuracy. The ELM, generalized extreme learning machine (G-ELM), and evolutionary extreme learning machine (E-ELM) are adopted respectively in [81-84] to construct accurate electricity prices forecasting methods. In [4], a review of some known ANN for load prediction in electric power systems is presented. The efficiency and

30 performance of decay radial basis function neural network, support vector machine (SVM), and ELM in load forecasting are investigated.

Up to now, there is no novel forecasting method proposed for the long term residential load forecasting. With the emergence of smart grid technologies, there are increasing measurement data available from the low-voltage end-user side. These measurement data enable better understanding of the load composition and electricity consumption activities of end users. Since load profiles contain information about electricity consumption behaviour of end users, the development of behaviour analysis based methods will therefore be a promising research direction for long term residential load forecasting.

2.7.2 Structure Optimization of Electricity Retail Prices

In existing publications, the structure of TOU pricing is often assumed to be pre-determined in advance [24, 34, 69, 85]. Without optimization of the retail pricing structure, these approaches cannot appropriately utilize the temporal difference between the system load and supplied load of a retailer. However, the temporal difference plays an essential role in developing flexible pricing schemes. The research work in this area is still preliminary, and how to appropriately model and customize electricity retail prices in the smart grid environment still remains an open question.

2.7.3 Customized Electricity Retail Plans

With the deployment of advanced metering and information techniques in power systems, more valuable information about end users could be collected. In the smart grid context, smart meter data would make it possible to model customer load at finer granular levels. Electric appliance identification is one of the applications to make use of smart meter data for big data analysis and enhance load modelling in future grid.

Different from load disaggregation which only classify load into different load categories (such as resistive loads and three-phase constant and quadratic torque induction motors) by their load characteristics [86], electric appliance identification aims to exactly recognize what exactly the power consumption equipment is. Depending on the methods of collecting load data, there are two kinds of appliance identification methods in existing research, namely intrusive load monitoring (ILM) [87]and non-intrusive load monitoring (NILM) [88- 90]. In ILM, traces of appliances are firstly collected by the distributed measurement and actuation units (MAUs) and then stored in a database system. Secondly, the features of each appliance are extracted from the input traces, which include energy and power consumption

31 levels, the shapes of the load profiles and so on. Finally, after choosing a proper classifier and training it with the features extracted above, namely through supervised learning, the appliances can be identified.

On the contrary, NILM only relies on a single measurement unit for a household’s overall electricity consumption. Generally, NILM approaches include both supervised and unsupervised approaches. Supervised methods need the prior knowledge of electric devices to form the signatures for individual devices. Then optimization or pattern recognition based algorithms can be used for NILM. For unsupervised approaches, various methods based on particle filter (PF) and hidden Markov models (HMM) are proposed, such as the particle filter based load disaggregation (PALDi) in [91]. In [91], firstly, the probability density function (PDF) of appliance states, which may be on/off or multi-state, is estimated by PF according to previous observations. Then, each appliance is modeled by a hidden Markov model (HMM). Individual HMMs are combined together to form the factorial hidden Markov model (FHMM). The observation of the FHMM is namely the total household demand.

Through mining smart meter data, load modeling and forecasting can be more accurate. The retailer will also be able to capture detailed properties of customers such as their consumption characters, responsiveness to various retail strategies and even to conduct behavioral economic analysis of customers. Moreover, the interactions between the retailer and customers will be more frequent considering that the real time electricity price can be easily offered using the smart meters.

Offering a variety of pricing options to customers is an essential feature of competitive markets and a key method to attract end-users. Through utilizing the complementation between different customer types, customized retail plans can further motivate responsiveness of end-users. Up to now, less research has been reported on this topic, which could be thus considered as an open research direction in the next-generation energy market.

2.7.4 Management Strategy for Large-Scale Flexible Loads

The increasing penetration of intermittent renewable energy and installations of distributed generation and storage facilities are gradually changing the original single direction energy flow in the power system. Therefore, the retailer is experiencing a transition from an energy seller to an energy service provider. The retailer will get involved in business on both selling energy and offering services to local distributed facilities. In the retailing side of electricity

32 market, operational frameworks are also needed to help customers locally balance their energy need and manage an optimal portfolio of their own distributed devices.

Due to customers’ random consumption behaviours and the competition in electricity markets, uncertainties exist in the load demand and electricity price. If these uncertainties can be forecasted and managed well, it can help to reduce electricity cost and system peak demand. It’s also beneficial to make better planning of power systems. In [92], smart meter data are used to forecast customers’ consumption uncertainty. An additive quantile regression model for load forecasting is proposed and the model is estimated using the gradient boosting algorithm.

Flexible load is an ideal way to manage the uncertainties in power systems. In [93] and [94], flexible load and energy storage system are used to reduce electricity cost and balance power demand while considering the uncertainties of renewable energy. The electricity cost is reduced by shifting electricity consumption from high to low price periods when facing electricity uncertainties in [93]. In [94], the problem of power balancing in a renewable- integrated power grid with storage and flexible loads is studied. In order to improve computation efficiency and reduce communication overhead, a distributed power balancing solution is proposed.

With the increasing number of flexible loads in the smart grid, such as the electrical vehicles, thermostatically controlled loads, air conditioning loads, and water-heating systems, coordination strategies and pricing schemes are needed to optimally leverage their benefits to the power systems.

The mean field game theory studies the problem of strategic decision making in very large populations of small interacting individuals. In mean field games, the players are coupled through a mean field term which depends on the statistical information of all players’ decisions. When the number of players is large, the coupled decision making can be captured by the interactions between an individual player and the mean field term instead of the detailed interactions among all the players.

In [95] and [96], this theory has been proven to be efficient to model the behaviour of large scale agents when constructing control framework and pricing schemes for flexible loads.

In [95], the scheduling of electrical vehicles’ (EVs) charge is studied when the number of vehicles tends to infinity. It is presumed that all EVs are indistinguishable by having similar batteries and similar individual objectives. The objective of each EV is to determine its own

33 consumption rates that minimize its total cost, given the consumption rates chosen by all the other EVs. The interaction among these infinite EVs is modelled as a mean field Game. In the established model, all the EVs are coupled through the empirical distribution of battery state variables, which indicate the amount of energy stored in the battery of each EV. In order to get the mean field equilibrium, the fundamental differential equations describing the mean field equilibrium of the game, i.e., Hamilton-Jacobi-Bellman and Fokker-Planck- Kolmogorov equations are derived.

In [96], the pricing problem for a large population of thermostatically controlled loads (TCL) is studied. The thermal dynamics of TCLs are modelled using the first-order continuous-time Equivalent Thermal Parameter (ETP) model. Each TCL load is optimized to maximize its individual utility. All the TCLs are coupled through a pricing function, namely the mean field term. The price is modelled as depending on the average value of all TCLs’ decisions. Then the individual utility maximization problem forms a mean field game at the lower level. And also, a coordinator is modelled as a leader in the upper level with its objective to maximize the social welfare. The whole problem is formulated as a reverse Stackelberg game, and solved through connecting it to a team problem and the competitive equilibrium.

Besides, some other researchers study the operational planning strategies of customers in the presence of distributed energy resources. In [97], a retail electricity market framework with high penetration of distributed generation is proposed. In the proposed framework, residential customers locally operate and manage their distributed generators (DGs), distributed energy storage devices (DESDs), and dispatchable loads. The utility makes profits by selling electricity and providing ancillary services to residential customers. Even though some researchers have noticed this problem, there is still a significant room for studying the management strategy for distributed large-scale flexible loads.

2.7.5 Data-Driven Retail Pricing Algorithm

With the emergence of smart grid technologies, there are increasing measurement data available from the low-voltage end-user side. Therefore, big data analytics in smart gird is of great importance. Through data mining, the retailer can gain a better understanding of the load composition and electricity consumption activities of end users. Big data analytics can also help retailers to develop data-driven based pricing algorithms. In [98], the special section of big data analytics for grid modernization on IEEE transactions on Smart Grid is introduced in detail. Papers associated with big data analytics for modernizing the electric power grid are presented. The guest editor also pointed out that potential research opportunities lay in areas including data visualization, data-driven dynamic pricing,

34 predictive asset management, and customer analytics. In [99], an overview discussion of the big data management and analysis in the smart grid is presented. The basic requirements of the big data analysis and the emerging bid data analysis technologies in the smart grid are introduced.

Based on big data analysis, more sophisticated energy retail policies can be made through analysing the energy usage data of the users. A variety of cloud computing platform and the data-centric data storage technology could be utilized to aggregate the energy usage data of the users, which could be generated by the advanced measurement infrastructure (AMI) in a smart grid or the load monitoring techniques. The data mining techniques could be then employed by the retailer to extract the energy usage characteristics and patterns of end-users. In data mining, the cloud plat-forms proposed in [100], [101], and the Map Reduce framework in [102] can be employed to improve the mining performance. Then the retailer can design highly customized incentive schemes that mostly fit the individual users.

A reinforcement learning based retail pricing algorithm is developed in [103]. The dynamic pricing problem for the electricity provider is firstly formulated as a Markov decision process (MDP). In order to solve the MDP, the well-known Q-learning algorithm is then chosen to search for the optimal action-selection policy both for the electricity provider and customers. In the MDP, the action of the electricity provider is to choose the optimal retail pricing policy aiming at minimizing its total cost. For the customers, their action is to decide electricity consumption based on observed retail price while aiming at minimizing the expected long-term electricity cost. The main merit is that the customer can determine its energy consumption in a distributed manner without a priori information exchange with the electricity provider and other customers.

Another research trend is to employ game-theoretic machine learning to develop retail pricing schemes. In [104], a game-theoretic machine learning approach is proposed for learning the best auction mechanism for sponsored searches. The proposed approach combines machine learning and game theory and its mathematical formulation is a bi-level optimization model. In the lower level, the Markov process is used to model advertisers’ bidding behaviours. In the upper level, the objective is to maximize the revenue of the search engine. When learning the optimal auction mechanism for the search engine, advertiser’s future bids are predicted using the learnt Markov model.

To some extent, the pricing problem for an electricity retailer is similar to that in [104], where the electricity consumer’s consumption activity is the lower problem and the retailer’s

35 retail revenue is the upper problem. However, due to the bidding difference between the electricity market and that for search engine, special endeavours are needed to develop the corresponding machine learning approaches for electricity retailing.

2.8 Chapter summary

To help researchers have an overall understanding of the existing research work on decision- making of electricity retailers, researches on electricity retailing in the last two decades are surveyed and discussed in this chapter. The state-of-the-art of the long-term retailer load forecasting, energy procurement strategies, retail pricing schemes, and risk management in the retail market have been discussed respectively, which cover the entire decision-making process of the electricity retailer.

In Table 2.1, the number of published papers in recent years on each sub-topic discussed in this survey is presented. During the period from 2000 to 2018, the topic of load forecasting has the highest number of publications, but only a small portion of these researches are devoted to LTLF. Therefore, novel LTLF techniques are still needed until now. Among all the sub-topics, retail pricing is of great importance and also attracts the high research interests, which can be verified by the number of published papers. From the literature survey, we found that the electricity procurement problem and the retail pricing for the retailer are always coupled together. Electricity retailers usually optimize their purchasing strategies and develop optimal retail contracts simultaneously.

Risk management has always been an important topic in competitive electricity markets. Risk measurement is considered in almost every decision-making process of the retailer. Various methods in the surveyed literatures have been proposed for the retailer to control their profit risks, such as the futures contract, flexible load, dynamic retail pricing, and the risk premium in retail prices.

In terms of future research trends, the sub-topics concerning the machine learning in forecasting, load disaggregation, large-scale flexible load management, and big data in smart grids are reviewed. The open issues that should be addressed in this field are critically discussed as well.

Table 2. 2 The number of published papers on each discussed sub-topic from 2000 to 2018

Sections in Discussed Surveyed # of Published this chapter Sub-topics References Journal Papers [2],[3],[4],[5],[6], Load forecasting [7],[8],[10],[11], 887 Section 2.3 [12],[13] [40] Long term load [10],[11],[12],[13] 61 forecasting [20],[21],[22], [23], [24],[25],[26],[27], Retail electricity [28],[29],[30],[31], Section 2.4 53 procurement [32],[33],[34],[35], [36],[37],[38],[39], [42],[43],[51] [52],[53] ,[54],[55], [56],[57],[59],[60], Section 2.5 Retail pricing 78 [50],[63],[64],[65], [66],[67],[68], [69] Electricity risk [26],[24],[34], [66], Section 2.6 191 management [77],[78], [79],[80] Machine learning [81],[82],[83],[84] 93 in forecasting Load [86],[87],[88],[89], 25 disaggregation [90], [91] Section 2.7 large-scale [92],[93],[94],[95], flexible load 17 [96],[97] management Big data in [98],[99],[103] 47 smart grid

Chapter 3 Key Business Framework of Electricity Retailers

In the former chapter, the various aspects lie in electricity retailer’s decision-making process are discussed in detail, including retail energy forecasting, energy procurement strategies and retail pricing strategies for electricity retailers, and risk management in retail markets. In this chapter, the logistic relationships of these aspects in retailer’s business process are studied. Firstly, the development history of electricity retail markets around the world is introduced. The motivation factors that facilitate the development of electricity retail market are summarized. Then, the main departments of an electricity retail company are categorised and analysed. Next, it elaborates the typical business process of electricity retailers and its process of creating a new sales agreement by taking Australia and Finland as examples respectively. Besides, as closely linked markets, the correlation between electricity wholesale and retail markets are also analysed. Finally, this chapter is ended by conclusions.

3.1 Introduction

Since the power industry reconstruction that firstly occurred in developed countries, such as UK and the USA, the world-wide deregulation of power industry has experienced more than 20 years’ development. Most of developed countries and regions have established their own electricity retail markets which are fully open to competition. There are also other countries that are endeavoring to establish a competitive electricity retail market, such as China. In [105-107], reforms on the supply side of power industry in UK, Singapore, the Nordic countries, and the USA are discussed in detail. Another series of publications represented by Ref. [108] focused on the regulation in the electricity retail markets. However, all these

38 publications did not elaborate the key business framework and process of electricity retail companies, as well as the purchase and sale decision-making of electricity retailers.

Under this background, this chapter studied the construction of electricity retail markets in developed countries and carried out a detailed analysis on the organizational structure and business process of retail companies by taking Australia and Finland as examples. As the upstream market of electricity retail market, the electricity whole market determines retailer’s supply cost and is also one of retailer’s risk sources. When making retail decisions, the retailer needs to optimize its corresponding portfolio in the electricity wholesale market. Therefore, the correlation between electricity wholesale and retail markets needs further analysis.

3.2 Established Electricity Retail Markets in Developed Countries

The world-wide deregulation in power industry started in the 1990s. The countries of UK, the USA and Australia took the lead in the deregulation and establishment of competitive electricity markets. They aimed to separate the integrated power industry of generation, transmission, distribution and retail into unintegrated entities. Competition is introduced into the generation and retail sides respectively in order to establish a fully competitive electricity market. In the actual practice of deregulation, most countries introduced competition into the generation side first and established the competitive wholesale market. Then, the competition is extended into the retail side and a competitive electricity retail markets is established. The competitive electricity retail market is firstly opened to large customers and gradually expanded to ordinary end-users. The development of Japan’s electricity retail market is a typical on this point. In Japan, competitive electricity retail is firstly opened to large industry customers whose load level is above 2000kW. It is further opened to end-users with the load level above 500kW and 50 Kw in 2004 and 2005, respectively. End-users in the electricity retail market can freely choose their electricity suppliers. Finally, Japan open its electricity retail market to all end-users in the April of 2016.

After 20 years’ deregulation of power industry, most developed countries have already established a competitive electricity retail market. Table 3.1 shows the time when each country finished establishing their electricity retail markets [109, 110]。

Table 3.1 Years of establishing electricity retail markets in some countries

Country Year Country Year Australia 2002 Italy 2002 Austria 2001 Korea 2001 Belgium 2007 Netherlands 2001 Czech Republic 2006 New Zealand 1994 Denmark 2003 Norway 1997 Finland 1998 Poland 2007 France 2007 Portugal 2006 Germany 1998 Spain 2003 Greece 2007 Sweden 1996 Hungary 2000 Turkey 2003 Ireland 2000 UK 1999 USA 1996 Japan 2016 Alberta (Canada) 2001

Note: there were 22 states and the Washington, D.C. in USA that allow competitive suppliers to supply electrical energy and other services to retail electricity consumers. But 8 of these states have suspended or rescinded this form of retail competition.

In a competitive electricity market, end-users are entitled to choose their electricity supplier freely. Even though all above countries have already established a deregulated electricity retail market, they all experienced a unique reform process. Their actual electricity retail market has its own features. In general, the development of the electricity retail market is driven by the following factors [109]:

(1) The call for a lower electricity retail price. The power industry restructuration is originally motivated by the expectation that substantial benefits were available through increased competition at the wholesale level. These benefits include the improvement of generation and transmission as well as the distribution efficiencies of electric power. Different from the competition in the supply side, the competition in the electricity retail market cannot obviously lower the power supply cost. However, it is still expected that the retail market can be a supplement of the wholesale market, to further lower retail prices.

(2) End-user’s expectation for diversified service products of electricity retail. In principle, retail competition can bring end-users a wider range of retail products than is traditionally available. In the electricity retail market, retail products can be differentiated along several dimensions, including the energy source (generating from renewables, fossil or nuclear fuel), firmness of service, pricing plan, billing and payment arrangement, and so on. The

40 competition can potentially motivate electricity retailers to increase the differentiation of products, and result in a wider customer choice in service conditions.

(3) Promoting the development of alternative generation technologies. Retail choice will foster competition in the promotion of both electric power and energy management services for homes and businesses. Retailers can help manage their distributed energy resources (DER) or self-generation devices to reduce their electricity cost or maximize their profits. Retail choice can allow retail energy suppliers to offer DER as part of their portfolio of services. Therefore, retail choice could foster market-driven growth in distributed energy resources. Retail choice could also foster competition in the promotion of renewable energy which is generated environmentally. It will facilitate a transition toward less-polluting renewable energy.

During the construction of European electricity and gas retail markets, working group on energy retail market design was set up by the European Commission. The working group identified key elements of retail markets that should serve as a guide for future work. These Guidelines of Good Practice (GGP) are directed towards EU Member States, national regulators and market actors when designing and acting in national electricity and gas retail markets [111]. The guidelines focus on the supplier switching and billing processes, as well as aspects related to proving information to the customer. Besides, in Europa, the Agency for the Cooperation of Energy Regulators takes in charge of monitoring the internal markets for electricity and gas. The Agency prepares and releases an annual market monitoring report in close cooperation with the European Commission, National Regulatory Authorities, and other relevant organizations. Competitiveness of the electricity retail market is measured by several indictors covering structure, conduct, and performance of the retail market [112, 113]. Structure comprises number of suppliers, the market share of the three largest suppliers, ability to compare retail prices. Conduct comprises the annual net entry, customer switching, and number of offers per supplier. Performance comprises the price dispersion, whether the market meet expectations or not, and the average mark-up. The difference between the retail price and the wholesale component is used as a proxy for mark-up. The more competitive a market, the lower the mark-up will be.

3.3 Type of Retail Company

With the evolution of electricity retail market, business models of retail companies tend to be more diversifies. There are several different modes of retail companies. The first is the stand

41 alone retail company. This is a small scale company and can be further classified into two types according to their sponsor. One is the independent retail company founded by privates. They may already have electric load but do not have installed generation capacity. The other one is the retail company founded by non-traditional electric power corporations. They corporations can be communications companies or internet companies. They have incumbent clients and the capability to server customers as well as enough capital, so they also have a strong competitive potential.

The second is the generator-owned retail company. This type of retail company was usually a subsidiary of large generation company. Such companies mainly focus on their generation business, and have a relative low customer load.

The third is the vertically integrated retail company, where they have both generation capacity and electricity retail business. The business scales of generation and retail are comparable, since it too difficult to keep them exactly matched with each other. That is because the power plant to invest or to build is usually in a very large scale, and customer accumulation is a slow process. Sometimes, the buyout of another retail company could also bring a large amount of customers and break the original balance. All these are typical features of this type of retail company.

3.4 Structure of Electricity Retail Company and Its Business Process

3.4.1 Australia Electricity Retail Market

The deregulation of power industry in Australia started from 1998. Similar to Europa and North America, the reform in Australia also follows the idea of unbundling generation and transmission services first, then unbundling the transmission and distribution services, and next introducing competition into power supply. Finally, it is finished by constructing a liberalized retail market. The Australian National Electricity Market (NEM) started operations in November 1998. There are currently more than 50 retailers in NEM, of which AGL, Origin Energy and Energy Australia are the three largest retailers [114]。

Departments of the electricity retail company can be categorized into front, middle and back offices. The front office mainly is mainly responsible for transactions in wholesale and retail markets through market analysis. It also takes in charge of the trading about mid-term and long-term financial contracts. The middle office is responsible for regularly assessing the

42 risk exposure of electricity retail and conduct the internal financial management for the company. Additionally, the retail company needs to run and maintain its intelligent financial system, customer service system. Meanwhile, it also needs to provide suggestions to governments on energy policy making, market regulation and improvement of market rules. All these are routine business of the company’s legal department which belongs to the back offices [115]。

Department Types Corresponding Responsibilities Trading of contracts, Bidding and Front Office Market research

Risk management, Internal Middle Office Financial Management

IT, Legal affairs, Accounting, Back Office Administrative support

Fig. 3.1 Departments of a typical electricity retail company

The priority of electricity retailers is to attract customers as many as possible. After obtaining customers, the retailer will carry out load forecasting for customers and purchase electricity from the wholesale market for power supply. Considering fluctuation of electricity prices in the spot market and the uncertainty of customer load, the retailer faces profit risks and needs risk management strategies. The risk is management through developing optimal portfolios in the forward contract and financial markets as well as adding a risk premium in the retail price. Retailers determine retail prices for end-users by solving an optimization problem about the energy procurement and risk management. Also, retailers can customize retail plans based on end-user’s unique load features and then offer diversified retail plans to customers. The business relationship between retailer and customer is finally determined in the form of retail contracts. Fig. 3.2 shows the business process of an electricity retailer.

Customer Large users Judge user type Negotiation

Small Procurement Customer users portfolio data

Contract Retail pricing Risk evaluation data

Load & price Load data forecast

Plan selection Weather data

Customer Contract end & Bill management management update

Fig. 3.2 The business process of an electricity retail company

After customer accepted the chosen retail contract, the retailer also provides additional services to customers including account management, guidance on energy saving, integration of end-users’ distributed generation, developing internet based service platforms or service apps. All these are offered to help customers solve their encountered problems and to improve customer experience.

The operation of Australia’s national energy market receives regulation and supervision from the Australian Energy Regulator (AER), Australian Competition and Consumption Commission (ACCC), Australian Securities and Investments Commission (ASIC), Clean Energy Regulator (CER) and other utility supervisors in each state. AER takes in charge of enforcing the laws for the National Electricity Market and spot gas markets in southern and eastern Australia. AER monitors and reports on the conduct of market participants and the effectiveness of competition. AER also releases annual about the state of the energy market. When evaluating the operation of electricity retail market, the feedback from retail customers on service quality is an essential indicator of the retail market operation and retailer’s performance [114]。

3.4.2 Finland Electricity Retail Market

As an important component of the Nordic electricity market, the Finnish market started its deregulation in 1995 and the electricity retail market was opened to all participants [116]. The unbundling of electric power generation, transmission, distribution and electricity retail fosters competition and enables a more transparent electricity price in Finnish power industry. The Finnish power market has been functioning well. There are 72 retail suppliers of which 45 offer their retail products nation-wide. The Finnish Energy Authority has estimated that there are only four electricity retailers with a market share exceeds 5%. Market share of the three largest retailers for small and medium sized customers is estimated to be 35% to 40%. So, the market concentration in Finnish electricity retail market stays at a low level [117]. The rate of supplier switching among electricity users stays around 10% in the past years (10.1% in 2013, 9.8% in 2014, 11.4% in 2015, 11.9% in 2016) [74, 75], and smart meters are widely installed among end-users in Finland.

According to the Finnish Electricity Market Act, Fingrid as the electricity transmission system operator is responsible for developing the information exchange system which is required by electricity trading and imbalance settlement. Then in 2014, Fingrid carried out an investigative project to broadly assess the information exchange system and related needs for development on the electricity retail market. As a result of its investigative work, Fingrid proposed in December 2014 a centralized information exchange system for the electricity retail market, namely a datahub, as the electricity market information exchange solution for the future. In April 2015, the Ministry of Employment and the Economy assigned Fingrid to develop and implement a datahub to Finland. The datahub has been planned to be in operation in 2019 [75]. Fig. 3.3 shows the process diagram of creating a new sales agreement in the Finnish electricity retail market when adopting the Datahub system [118].

As shown in Fig. 3.3 [118], when retailer creates a new retail contract, the request will be firstly reported to the Datahub. During the creation of sales agreement, the retailer needs to get access to accounting point and customer information, which includes the address and ID number of accounting point, end-user’s ID, name, date of birth, and corresponding authorization. The existing of a current retail contract will be checked when retailer assesses end-user’s eligibility for a new sales agreement. If there exists an old contract, it needs to further confirm whether it is a fixed term contract, whether its end date is in 90 days, and if the contract is limited by special termination terms.

Datahub decides if a new grid connection agreement is needed based on data reported by the supplier and on the accounting point's customer and agreement status at the given time. The

45 new grid connection agreement needs the double confirmation from distribution system operator (DSO) and Datahub. Similarly, when retailer or DSO needs to revise the created retail contract or grid agreement, they need to report the revision information to Datahub. Then Datahub will take over the revision request and coordinate the task with concerned participants. When ending a retail contract, Datahub also send the ending information to concerned participants. Besides, for any accounting point, there will never exist two valid retail contracts or grid agreements. When a new one is created successfully, Datahub will expire the old one automatically and inform the corresponding retailer.

Request a No contract Receive contract Receive contract Customer has a new contract and new contract created confirmation confirmation the accounting point is connected Customer Access Start the accounting Receive creation of point and No a new sales Receive positive customer Create reply and Receive grid agreement information Yes negative Can a contract new sales reply send contract agreement be made? confirmation New supplier agreement information

Receive and validate request No Receive report and validate the Yes Save sales Ok? Call / Create customer contract using agreement information, and check customer’s history information whether customer has information. End the old grid agreement a contract at the Create new grid accounting point. Does agreement Yes Decide if new Update grid agreement accounting No Yes information New grid point have grid agreement information (status = Datahub (status= agreement an old is needed. “unconfirmed”) “confirmed”) contract? No Is the new Yes Update supplier same Receive and previous as the former? validate the sales grid agreement agreement No as ended Report end Send positive of supply reply

Distribution system operator (DSO) Send contract confirmation

Yes Receive Receive Create new grid report report agreement Connection Meter No notification Connected? is ready from DSO

Current supplier Receive report and end sales agreement

Fig. 3.3 The process diagram of creating a new sales agreement

3.5 Relationship between Electricity Wholesale and Retail Markets

Electricity retail and wholesale markets are two distinct but closely linked markets. On the one hand, they have different participants. Participants of the retail market are retailers and small end-users. While participants in the wholesale market are medium- and large- scale end-users, generation companies and retail companies. On the other hand, they have different pricing algorithms. The pricing methods in these two markets are different. Retail pricing strategy for electricity retailers is elaborated in Chapter 2. The electricity wholesale market is usually cleared uniformly through unilateral or bilateral bidding.

Electricity retail and wholesale markets are closely linked. Under the market environment, electricity retailer is an intermediary between the Generation Company and end-users. The retailer makes money mainly through purchasing electricity from the wholesale market (or can be called a spot market together with the real-time market) and then resell it to end-users. Therefore, the wholesale market is the upstream market of electricity retail market. The wholesale market and non-discriminatory access to transmission and distribution networks constitute the foundation for the establishment of electricity retail market. In addition, in actual practice, price in the wholesale market fluctuates, but the retail price is usually fixed time-of-use price or tiered electricity pricing mechanism depending on consumption amount. Thus electricity retailers take the risk of fluctuated spot market prices and avoid exposing end-users to the risk from wholesale market [115]. The fixed retail price and the fluctuated wholesale price of electricity bring challenges to retailers who make money through the difference of these two prices. On the one hand, retailers want to attract customers by lowering their retail prices, and on the other hand, they would try to avoid losses due to variation of the spot market price. Because of retailers’ different capability of forecasting future trends and cost control, as well as the competition to win customers, all these result in the diversification of retail products. Especially, in Ref. [119] a quantitative analysis model based on the system dynamics method is developed to study the interaction between electricity wholesale and retail markets. Their interaction mechanism is analyzed in detail under different allocation scenarios for transmission and distribution cost.

3.6 Chapter Summary

This chapter studied the key business process of electricity retailers and find that the key factors motivating the development of electricity retail market are: (1) The call for a lower electricity retail price. (2) End-user’s expectation for diversified service products of electricity retail. (3) Promoting the development of alternative generation technologies. Meanwhile, during the process of developing a competitive electricity retail market, proper market construction guidance, such as the guidance from Council of European Energy Regulators, CEER, and proper supervision of the market performance, such as the supervision from Australian Energy Regulator, AER, can help keep the retail market construction on track. In addition, the rising of end-user’s consumer awareness, and the open access of market information are also important for improving the performance of electricity retail market. The different types of electricity retailers and their typical department structure are also introduced. Especially, the process of creating a new sales agreement using their new market information exchange system namely Datahub for retailers in Finland is elaborated. Finally, this chapter also analyzed the distinct but closely related electricity wholesale and retail markets.

Chapter 4 Identification of Residential Appliances and Load Modelling Through Mining Big Load Data in the Smart Grid

The identification of residential electric appliances is an important application area of data mining in smart grid environment. Through data mining of the residential load big data, information about electricity consumption behaviour of residents and electric appliances can be extracted. The acquired information could be used to make precise and targeted demand-side management as well as customized electricity retailing strategies. Given this background, based on the dynamic time warping (DTW) matching method, a novel appliance identification algorithm for low frequency sampling load data is proposed. First, the residential load sequence is segmented into subsequences composed of the single appliance load profile and multi-appliance load profile. Then, reference load sequences of all given appliances are generated which have the same length of each query subsequence before conducting DTW matching. Finally, the reference sequence which has the minimum DTW distance with the query subsequence is assigned as the identification result. For a subsequence composed of multiple appliances, the best matched reference sequence is reduced after each DTW is matched, and then segmentation and DTW matching are carried on until all appliances are extracted. Given the status of all identified appliances, a statistical residential load model is developed. With this load model, the appliance identification results can be conveniently used in demand-side management and developing electricity retailing strategies. The proposed algorithm is coded in the R programming language and tested through a load dataset containing 500 households’ profiles. Simulation results show

50 that the proposed algorithm could identify both the single and multi- appliance load subsequences at a high accuracy level.

4.1 Introduction

In the Smart Grid environment, the emergence of intelligent measuring equipment makes it possible to collect high resolution load data for the better operation and management of power systems. Applying data mining techniques, information about consumers’ electricity consumption behaviours and electric appliances could be discovered, which can then be used to improve the efficiency and load management in power systems. As an important method of load data mining, electric appliance identification has become an important research topic which attracts significant attentions recently.

Different from load disaggregation which only classify load into different load categories (such as resistive load and three-phase constant torque induction motors) by their load characteristics [86], electric appliance identification aims to recognise what exactly the power consumption equipment is. Meanwhile, it also has a higher requirement for the load sampling data [87]. According to the way of load data acquisition, there are two kinds of electric appliance identification methods in existing research. They are intrusive load monitoring (ILM)[87] and non-intrusive load monitoring (NILM) [88-90].

For ILM, it needs to install distributed power measurement and actuation units between an appliance’s power plug and the wall outlet and thus to record the power consumption and switch events of each connected device. Then together with consumer’s electricity bill and other data collected by smart meters if available, electric appliances of a household are recognized. On the contrary, NILM only relies on a single measurement unit for a household’s overall electricity consumption. Because it has a low requirement on hardware and requires almost zero effort from users, NILM has become a prevailing approach. Some research work has already been carried out on NILM. They can be classified into two aspects: (1) Research on appliance load signature extraction [90] [120]; (2) Research on developing novel identification algorithms [86, 121-125].

After acquiring household’s high resolution overall load data, the majority of existing algorithms conduct identification by matching appliance’s electrical signatures with high frequency sampling load data. These signatures include current waveform, active and reactive power, instantaneous admittance waveform and switching transient waveform to name a few [120].

According to the mathematical principles behind data processing, these algorithms could be categorised into two:

(1) Optimization based method. By solving an optimization problem, a set of candidate appliances in the given appliance database will be extracted. Power combination of appliances in the set has the minimum difference with the known load data. Then appliances in the set are the identification result.

(2) Data mining based method. Decision tree, artificial neural network (ANN) based methods and support vector machine (SVM) are used in [87], [121] [123] and [122], respectively. First, teach the ANN or SVM to learn the above-mentioned appliance’s signatures, and then use it to identify appliances.

As the optimization problem might become complicated and will get worse when number of candidate appliances gets larger. Therefore, existing research mainly concentrate on ANN- based methods.

Through appliance identification, information about end-users’ behaviours and the running statuses of residential appliances will be available. Then, a more accurate residential load model can be developed through a statistical analysis of appliance identification results. Such an accurate residential model can be used used to enable targeted demand-side management, design customized electricity retail plans, or model customer load in finer granularity. Quite a few publications have been reported on this subject. In [126], the influence of dwelling and occupant characteristics on domestic electricity consumption patterns is studied. A multiple linear regression model was applied to four parameters: total electricity consumption, maximum demand, load factor and time-of-use (TOU) of maximum electricity demand for a number of different dwelling and occupant socio-economic variables, including dwelling type, number of bedrooms, head of household (HoH) age, household composition, social class, water heating and cooking type. Especially, up to now, no publications have been reported on establishing accurate residential load model through appliance identification.

Against this background, this chapter proposed a novel appliance identification algorithm for low frequency load data. Firstly, the daily load temporal sequence that is sampled at a frequency of one value per minute is segmented into sub-sequences. Then dynamic time warping (DTW) matching is conducted to measure similarity between each sub-sequence and the reference sequences. The best-matched reference sequence is selected as the identification result. For testing the proposed algorithm, experiments on data that is composed of 500 households’ load profiles are carried out. Then, a statistical residential load

52 model is proposed through statistical analysis of appliance identification results. Finally, conclusions are presented in the last part.

4.2 Dynamic Time Warping Matching

Dynamic time warping matching is a robust distance for measuring time sequences. Even if they are asynchronous, similar shapes could still be matched by allowing the sequences to stretch along the time axis. Because of its flexibility, DTW has more advantages over other distances like Minkowski distance, Euclidean distance, Manhattan distance, et al. Thus DTW has been widely used in the field of data mining.

Given two time sequences Q and C, their lengths are n and m, respectively.

Qqq 12,,,,, qin q (4.1)

Ccc 12,,,,, cj cm (4.2)

When using DTW to match these two sequences, an n-by-m distance matrix D needs to be constructed. Each element of the distance matrix di,j represents the distance or square of the distance between the two points qi and cj [127]. For convenience, here we suppose di,j represents the square of distance.

2 dqcij, () i j (4.3)

Definition of warp path [128]: a warp path W is a contiguous set of matrix elements dij that defines a mapping which begins at (q1, c1) and ends at (qn, cm) between Q and C.

Www 12,,,, wkK w (4.4)

wdkijk (), (4.5)

th Where wk represents di,j of the k matched points between Q and C.

However, there is more than one mapping between Q and C of which the beginning and ending points could be (q1, c1) and (qn, cm), respectively. Thus the mapping defined in warping path W needs to satisfy the following three constraints [128]:

(1) Boundary constraint. This condition requires the warping path W to start and finish at diagonal elements of the distance matrix D, namely wd11,1 , wdKnm , ;

(2) Continuity constraint. This constraint would restrict that steps in the warping path W are composed of adjacent elements in D. Given wdkij , , wdkij1',' thus requires ii'1 and

jj'1;

(3) Monotonicity constraint. Similarly, suppose that wdkij , , wdkij1',' where i, j and i′, j′ should satisfy ii'1, jj'1. This constraint would ensure that points in the warping path W are monotonically spaced in time.

The warping path with the minimum cumulative sum of all elements is defined as the dynamic time warping path and the cumulative sum is defined as the DTW distance between time sequences Q and C. To indicate the DTW distance by DDTW, then

K DDTW QC,min  wk (4.6) k 1

The process of solving DTW path could be seen as a multi-stage decision problem. Suppose decision variable of the kth step is d(i, j) and the cumulative sum is r(i, j), then

rij, dij , min1,1,1,,,1 ri  j ri j rij   kk        k 1 (4.7)

where r(i, j)k represents the cumulative distance between the two time sequences Q (Q=q1,

th q2,…, qi, …, qn) and C (C=c1, c2, …, cj, …, cm) in the k step of the DTW path when matching the point qi in Q and the point cj in C. d(i, j)k represent the square of distance

th between the two points qi and cj in the k step of the DTW path.

If the lengths of time sequences Q and C are not equal, the Euclidean distance will fail to measure the distance between Q and C. But the DTW distance as shown in Eqn. (4.7) can solve this problem. When calculating the distance between Q and C, the distance between the point qi (i=1, 2 … n) in Q and the point cj (j=1, 2 … m) in C is calculated in each step of the DTW path. The points are not necessarily to be paired by q1 to c1 and q2 to c2. They may be paired as q1 to c1 and q2 to c1, depending the calculation result of Eqn. (4.7). Eqn. (4.7) can determine the best matching relationship between points in Q and C by always looking one step backward when calculating. Through solving Eqn. (4.7) by dynamic programming, corresponding DTW path and distance between Q and C would be found.

Actually, Eqn. (4.7) only gives the situation when there are no global and local constraints being put on the searching space of the DTW path. These constraints limit how far it may stray from the diagonal of the distance matrix when determining the DTW path. Therefore, they could help to speed up the calculation. More importantly, they could prevent pathological warping when matching. For more details about these constraints, please turn to [128].

4.3 DTW Based Appliance Identification Algorithm

Appliance identification algorithms proposed by existing research usually rely on monitoring data sampled at least 1Hz [87], or tens Hz (17Hz in [122], 60Hz in [125]) and even higher [120]. After acquiring the high frequency load data, these algorithms normally assume that there is only one appliance will be switched during a short enough period of time (typically 1 second). By matching load data with appliance signatures mentioned before, appliance is recognized.

Obviously, the higher sampling frequency of monitoring data has a higher requirement for both the data acquisition devices and data process capability. For this reason, research on developing approaches for appliance identification by analyzing low-frequency load data have emerged. [9] and [10] proposed ANN-based appliance recognition algorithms using monitoring data sampled at one value per minute and one sample every two minutes, respectively. Notably, most of the already proposed methods are implemented by ANN or ANN based derivative algorithms [120] [123] [124].

This chapter proposed a novel appliance identification algorithm for low frequency load data that is sampled by one value per minute, as is shown in Fig. 4.1. Firstly, residential load sequence is segmented into subsequences, then DTW matching between each subsequence and reference sequences of appliances is conducted. The best-matched reference sequence is selected as the identification result. It is ideal to do so for the subsequence containing only one appliance. But for a composite subsequence generated by multi-appliance operation, we need a strategy to extract the appliances one by one. As we noticed that there is often a dominant appliance or a distinctive characteristic of load pattern belonging to a specific appliance in a composite sequence. While DTW matching could figure out the candidate sequence that has the most similar pattern, so it is usually not difficult to recognize the most obvious appliance in it. Then, by reducing the identified appliance sequence from the composite subsequence, its complexity is brought down. Next, the composite subsequence is segmented again into smaller sequences containing only one appliance or multi-appliance. 55

Repeat the identification and segmentation until the DTW distance of the last matching is less than threshold setting, which means no appliance is left in current sequence, then computation finish.

Segmentation of temporal load Start sequence Import temporal load sequence P

Initialize parameters

t=1

Whether P(t) Yes Mark as a satisfy the conditions to be a segmentation points segmentation point No No Is P(t) the last t=t+1 point of the temporal load sequence Yes Output the final segmentation result

Electric Import appliance segmentation result identification of load sequence

Initialize parameters

n=1

For the nth subsequence, conduct appliance identification

Reduce reference Is this a composite Yes subsequence containing multi sequence of the identified appliances appliance from it

No Segment it into smaller sequences No n=n+1 Is this sequence the last one Replace the nth subsequence by these smaller sequences, and return Yes back to appliance identification

Output appliance identification result

Finish

Fig. 4.1 Schematic of the proposed appliance identification algorithm

Details of the proposed load sequence segmentation and appliance identification algorithms will be elaborated further as below.

4.3.1 Segmentation of Temporal Load Sequence

Usually, load data is collected on a daily basis. In the proposed load segmentation algorithm, it is assumed the original load data is sampled every minute during a time period of 24 hours. Practically, the main electric appliances, which consume the most electricity, are normally operated from minutes to hours. Therefore, in order to recognize each appliance in the entire load sequence by matching with individual appliance reference sequence, the 24h length load data needs to be cut into sub-sequences. These sub-sequences are load sequences generated by single appliance running and multi-appliance running events.

When temporal load sequence is segmented, starting point of a sub-sequence is determined by power jump at each moment. It is because when power jump happens which is bigger than threshold value, there must be an appliance is put into working. Due to power jump also happens when low power consumption appliances are turned on or when there is a disturbances happened, and we are only interested in main power consumption appliances, so a threshold value ΔPset is set up. Only when delta P(t) exceeding the threshold value ΔPset is detected, then the moment would be marked as a starting or ending point. However, for the ending point, after a valid power dropping is detected, it still needs to judge the load sequence values in the following time period of Tset (a pre-set parameter). That is because the valid power dropping may be caused by mode switching of some appliance. Thus if after detecting a valid dropping, the load sequence values stay at a low level (smaller that pre-set threshold value Pset) and keep stable for Tset periods, then the last moment of the stable status is marked as an ending point.

The proposed algorithm of load sequence segmentation is coded in Matlab and its pseudocode is given in Table 4.1.

Table 4.1 Pseudocode for the Proposed Load Profile Segmentation Algorithm

Input: Residential load temporal sequence Output: Segmentation result of load sequence 1 Initialize parameters: set both starting and ending tag parameters to 1 be 0; 2 For t =1: T 3 Compute ∆P(t); 4 If Pt() 0and Pt() Pset 5 If the starting tag is 0 6 % There is an appliance being switched on; 7 Store the moment t, meanwhile, set starting tag to be 1 and the 7 ending tag to be 0; 8 End If 9 End If 10 If Pt() 0and P()tPset 11 % there is an appliance being switched off, or being switched to 11 % low power mode; 12 If P(t+1)≤Pset and in the following Tset periods P(t) stay stable 13 If the starting tag is 1 and ending tag is 0 14 Store the moment t, meanwhile, set starting tag to be 0 and the 14 ending tag to be 1; 15 End If 16 If the starting tag is 1 and ending tag is 1 17 %It indicates that some moment earlier is recorded as an ending point, 17 %but now there is another appliance being switched off or switched 17 % into low power mode; 18 Replace the previous ending point by the current moment t, 18 meanwhile, set starting tag to be 0 and the ending tag to be 1; 19 End If 20 End If 21 End For 22 Finish, output the result;

Note: P()tPt ( 1) Pt ()

Parameter setting has an important influence on implementation effect of the proposed algorithm. Because of the difference of electric appliances and their load features in different households, when conducting appliance identification, parameter values are closely related to the concrete residential load sequence. In this paper, the optimal parameters are selected by running the algorithm many times.

4.3.2 Electric Appliance Identification Algorithm

After acquisition of segmentation results, each subsequence is DTW matched with reference sequence of candidate appliance. The reference sequence that has the minimum DTW distance with load subsequence is selected as the identification result. Meanwhile, similarity distance and the mapping relationship in the most similar matching are stored for further use.

set Then judge whether the stored DTW distance exceeds threshold value DDTW. If not, it means that the current load subsequence merely contains one appliance, and identification ends. Otherwise, it indicates that this is a composite load sub-sequence containing multi-appliance. Then, the identified appliance is reduced from this composite subsequence using the stored mapping relationship between these two sequences. That is because DTW could figure out the appliance that has the most similar power pattern to the load subsequence and link corresponding points by building a mapping relationship. Through the reduce calculation, complexity of the composite is brought down. Similarly, this subsequence is segmented again into smaller sequences followed by once again identification calculation for each of them. After finite loop, calculation will finish when DTW distance of each matching is less

set than the threshold value DDTW. Fig. 4.2 shows a mapping relationship of a DTW matching between two sequences.

Fig. 4.2 Schematic of the DTW mapping relationship between two sequences

Table 4.2 gives pseudocode for the proposed appliance identification algorithm [129] [130].

Table 4.2 Pseudocode for the Proposed Appliance Identification Algorithm

Input: Segmentation result of temporal load sequence Output: List of appliance identification result 1 For (n in 1: N) { 2 L<- length of the nth load subsequence; 3 Generate a set of appliance reference sequences of which each has a 3 length of L; 4 For (m=1: M) { 5 Calculate the dtw distance between the nth load subsequence 5 and the mth appliance reference sequence; } 6 Select and store the matched reference sequence that has the 6 minimum DTW distance as identification result; set 7 If (DTW distance≥ D DTW ) { 8 Set multi-appliance tag parameter of load subsequence n to be 1; 9 Declare an empty list variable: Slist; 10 Store load subsequence n into Slist; } 11 Else {Set multi-appliance tag of load subsequence n to be 0; } 12 While (multi-appliance tag of load subsequence n is 1) { 13 For (i in 1: length (Slist)) { 14 For the nth subsequence stored in Slist, do: reduce reference sequence 14 of the identified appliance from it; } 15 Update Slist by all the subsequences after the reduce calculation; 16 For (i in 1: length (Slist)) { 17 Segment the ith subsequence stored in Slist, store corresponding list 17 segmentation result in Ui ; list 18 For (j in 1: length (Ui )) { th list list 19 For the j sequence in Ui , do: step 4-step 6 for Ui (j); list set 20 If ( DTW distance of Ui (j)≥ D DTW ) { list list 21 Store Ui (j) into an intermediate list variable Ii ; }}} list list 22 Update S by Ii ; 23 If (there is no multi-appliance in Slist) { 24 Set multi-appliance tag parameter of load subsequence n to be 0; } 25 Finish, output the result; }

Note: N is the amount of subsequences in load sequence segmentation result; M is the total amount of candidate appliances.

4.4 Residential Load Model

Using the proposed appliance identification algorithm, the running status of various appliances at each time will be known. And also, end-user’s behaviour habits of using these appliances can be derived through a statistical analysis of each appliance’s running status over a certain time period. Thus, appliance identification results can be used to develop a

61 residential load model, where the total load is a linear combination of running appliances at each moment.

app Let M denote the types of appliance to be identified. Pi (t) (i=1,2, …, M) indicates the

th sys average power rate of the i appliance during time period t. P (t) and βi,t represent the total load during time period t and the coefficient of the ith appliance in the model, respectively. Considering that the proposed model is based on the statistical analysis of running appliances,

βi,t will be a random parameter following an unknown distribution. Consequently, the proposed statistical residential load model can be expressed as follows.

sys app app PPPt0, t 1, t 1, t  Mt , Mt , (4.10)

sys where Pt indicates the total system load during time period t. β0,t is a random variable and represents the deviation between the combined load of identified appliances and the actual

th system load. β1,t / βM,t represents the statistical amount of the #1/M appliance which have

app app been running during the time period t. P1,t / PM,t denotes the average power rate of the #1 / Mth appliance during time period t.

As mentioned above, in Eqn. (4.10), load components are clearly indicated. Due to different electrical characteristics of each appliance, the proposed load model is ideal for developing targeted demand-side management. The proposed method can also be used to explore the behavioural regularity of end-users in order to design customized electricity retail plans. However, all these have already exceeded the scope of discussion in this chapter. They will be analysed in detail in Chapter 5 of this thesis.

In Eqn. (4.10), both β0,t and βi,t are random parameters derived from appliance identification results. Therefore, the distribution of these parameters needs to be inferred. For simplification, it is assumed that appliance usage is independent of each other, so the distribution test for each parameter in Eqn. (4.10) can be carried out separately. Since the distributions of β0,t and βi,t cannot be easily assumed in advance, they can only be estimated based on the dataset of appliance identification results. The above is a nonparametric test problem. Therefore, the famous statistical analysis software SPSS [131] is used here, and the distributions of these random parameters are analysed by One-sample Nonparametric Tests.

There are five different choices of One-sample Nonparametric Tests in SPSS, namely Chi- square test [132], Binomial test [133], Kolmogorov-Smirnov test [134], Wilcoxon signed- rank test [135], and Runs test [136].

The Chi-Square test is applied to nominal and ordinal fields. This produces a one-sample test that computes a chi-square statistic based on the differences between the observed and expected frequencies of categories of a field. It is used to discover if there is a relationship between two categorical variables. For data to be analysed by chi-square test, it needs to meet two assumptions: (1) The two variables should be measured at an ordinal or nominal level (i.e., categorical data); (2) The two variables should consist of two or more categorical, independent groups. If it does not, the Chi-Square test should not be chosen.

The Binomial test can be applied to all fields. This produces a one-sample test that tests whether the observed distribution of a flag field (a categorical field with only two categories) is the same as what is expected from a specified binomial distribution. In addition, it also supports the request of confidence intervals.

The Kolmogorov-Smirnov test is applied to continuous fields. This produces a one-sample test of whether the sample cumulative distribution function for a field is homogenous with a uniform, normal, Poisson, or exponential distribution.

The Wilcoxon signed-rank test is applied to continuous fields. This produces a one-sample test of median value of a field. Specify a number as the hypothesized median.

The Runs test is applied to all fields. This produces a one-sample test of whether the sequence of values of a dichotomized field is random.

In this research, the Runs test is firstly used to ensure the tested data is not random without following a certain distribution. Then, the Kolmogorov-Smirnov test is applied to the amount of running appliances in the appliance identification results during each time period.

Besides, how to determine the length of time period t in the proposed model is essential. If the length is too short, there will be no enough sample data for accurate distribution estimation. The length of time period t should be determined properly according to the specific application condition.

4.5 Experiments on Data and Discussions

4.5.1 Electric Appliance Identification

In [137], a data set comprised of more than 1000 power consumption traces from 31 different types of appliances is collected. And all collected traces are available to the

63 research community. Using the data acquired from [137], [138] proposed a rule based domestic load profile generator in a Smart Grid environment. In this section, a data set that is generated by the method in [138] is used. It contains 500 domestic load profiles and each profile is of 24h length.

(1) Selection of candidate appliance to identify

The original test data set in [138] is composed of 12 different types of appliances. They are Air Conditioner, Refrigerator, Water Heater, Washing Machine, Dishwasher, Microwave Oven, Cooking Stove, TV, Iron, Laundry Dryer, Freezer, Vacuum Cleaner, respectively.

According to their usage features in [138], some of these appliances will fall into the following categories: (1) Appliances normally plugged in: Refrigerator, Freezer, re et al. (2) Appliances normally in short-term use: Microwave Oven, Iron, Vacuum Cleaner, et al. (3) Appliances with a low power rating: TV, et al. All these above categorized appliances have a common feature, namely their only contribute to a minor proportion of a household’s total electricity consumption. Therefore, in this case study, another 6 types of appliances out of the 12 are selected as candidate appliances to identify. They are Air Conditioner, Water Heater, Washing Machine, Dishwasher, Cooking Stove and Laundry Dryer. The reference load sequences of these selected residential appliances are given in Fig. 4.3 in which the data is sampled every 30s.

AirCon 4000

2000

0 0100 200 300 400 500 600 700 800 900

Cookingstove 4000

2000

0 0510 15 20 25 30 35

Dishwasher 2000

1000

0 050 100 150 200 250

LaundryDryer 4000

2000

0 010 20 30 40 50 60 70 80 90 100

Fig. 4.3 The reference load sequences of residential appliances

Note: the time interval on x-axis is 0.5min. The y-axis indicates the power consumption rate of each appliance and the unit is watt. Aircon represents Air conditioner.

(2) Parameter setting and statistical identification result

Since the original load data is sampled at a frequency of 1 Hz, firstly, the data is converted into a lower frequency of one value per minute by taking the average load value during one minute and by taking the load value every one minute, respectively. Then, these load temporal sequences are segmented into sub-sequences. Next, the proposed appliance identification algorithm is tested by running on a desktop computer having an Intel Core i5- 2400 CPU at 3.10GHz, 4GB RAM. Fig. 4.4 shows the segmentation result of a half-day (12h) load sequence. Table 4.3 gives the parameter setting and statistical accuracy of experiments on data.

9000

8000

7000

6000

5000

4000

3000

2000

1000

0 0 100 200 300 400 500 600 700

Fig. 4.4 Segmentation result of a half day load sequence

The proper parameter setting is essential for obtaining an accurate appliance identification result. Table 4.3 shows the results under different scenarios of parameter setting, where Nsigl denotes the amount of subsequences that constitute a single running appliance, Nmult denotes the amount of subsequences that constitute multiple running appliances, ηsigl and ηmult represent the identification accuracy of single and multiple appliance subsequences, respectively, ηavg is the arithmetic mean of ηsigl and ηmult.

Since the data used in this case study stems from [138], the appliance status in [138] is used to calculated the identification accuracy through direct comparison. Specifically, for each appliance in the identification result, if its status is identified as the same as in the original data, it indicates that the appliance is accurately identified. Otherwise, it is incorrectly identified. The ratio between the cumulative correct identification and the total number of identifications is namely the final statistical accuracy. This method is used for both single

67 and multiple appliance subsequences to acquire ηsigl and ηmult. Then, ηavg can be obtained by calculating the arithmetic mean of ηsigl and ηmult.

Table 4.3 Parameter setting and identification results in numerical experiments

Parameter value Segmentation result Identification accuracy Overall accuracy Computation time Scenarios Data conversion avg Pset (W) Ts(min) N sigl N mult  sigl (%) mult (%)  (%) (min) 1 Averaged 200 3 11471 4958 88.79 78.64 83.72 5.27 2 Non-averaged 200 3 13857 4496 87.48 83.55 85.52 4.74 3 Averaged 300 3 12554 4821 90.43 79.50 84.97 4.77 4 Non-averaged 300 3 15436 3899 91.27 82.26 86.77 3.97 5 Averaged 400 3 13738 4549 90.95 79.86 85.41 4.38 6 Non-averaged 400 3 16951 3507 91.15 81.04 86.10 3.69 7 Averaged 400 4 13170 4641 91.60 80.07 85.84 4.41 8 Non-averaged 400 4 16163 3667 92.32 82.16 87.24 3.62 9 Averaged 400 5 12382 4757 93.64 78.72 86.18 4.56 10 Non-averaged 400 5 15292 3839 93.81 81.31 87.56 3.78 11 Averaged 500 3 14182 4257 93.41 78.96 86.19 3.91 12 Non-averaged 500 3 17363 3484 90.82 80.71 85.77 3.59 13 Averaged 500 4 13362 4660 92.36 80.59 86.48 4.36 14 Non-averaged 500 4 16513 3674 92.09 81.03 86.56 3.59 15 Averaged 500 5 12709 4759 93.50 80.52 87.01 4.39 16 Non-averaged 500 5 15597 3859 93.61 82.66 88.14 3.52

(3) Comparison between the proposed algorithm and other methods

There currently exists no accepted and approved evaluation test case that can be used to test and compare different algorithms. It makes the numerical comparison between different approaches complicated. Besides, almost all of existing researches did not mention the run- time performance of their proposed algorithms in the paper, which makes direct comparisons impossible. Therefore, various algorithms have to be coded again for comparison.

The particle filter-based load disaggregation (PALDi) approach proposed in [91] is selected and programmed in Matlab to compare with the proposed dynamic time warping based appliance identification (DTWAI) algorithm. PALDi is chosen because its run-time performance is analysed in [91], which is rarely discussed by other researchers. In the numerical comparison, number of identified appliances is 6 and the number of used particles in PALDi is 100. The calculation results in Table 4.4 are average values acquired by running the approach several times. In Table 4.4, the result shows that the proposed algorithm is efficient and faster.

Table 4.4 Numerical Comparison between the Proposed Algorithm and PALDi

Calculation Number of Overall Identification Name time [s] profiles accuracy [%] DTWAI 3.285 1 80.328 PALDi 846.968 1 73.854 DTWAI 4.113 2 82.441 PALDi 1694.639 2 78.296 DTWAI 6.413 5 82.449 PALDi 4244.442 5 79.181 DTWAI 7.475 15 83.725 PALDi 12714.689 15 80.444 DTWAI 8.022 20 84.532 PALDi 16945.573 20 81.477 DTWAI 9.406 25 83.736 PALDi 21178.271 25 80.779 DTWAI 11.169 30 84.321 PALDi 25419.352 30 81.461

Because the PALDi is trained using the given reference sequences of electrical appliances, its identification ability will not be affected by the quantity of load profiles for testing. On the other hand, the appliance identification computation for each profile is actually a repetitive work. The calculation time almost linearly increases with the increasing of amount of load profiles, especially for the PALDi.

Besides, it is the method proposed in [138] that is used to generate the simulation data for tests in this research, where end-users’ behavior of using electrical appliances is simulated randomly. When the quantity of load profiles increases, the percentage of occasions when end-users manifest simple usage behaviors may also increase, such as the end-user could only use one appliance sometimes. At these occasions it would be easier to identify the electrical appliances. Therefore, for both DTWAI and PALDi, the overall accuracy fluctuates when the amount of load profiles increases. It can also be found in table 4.4 that the overall accuracy of DTWAI and PALDi stays around 84% and 81% respectively. However, the advantage of DTWAI on computation efficiency is obvious and it is always faster than the PALDi approach.

Compared with the high-frequency sampling data, the decrease of sampling frequency will weaken characteristics of appliance’s power waveform, and then it affects the accuracy of appliance identification. However, the decrease of sampling frequency can greatly improve the computation speed since the amount of data to be processed will decrease at the same time. In order to minimize the effect on accuracy caused by decreased sampling frequency, other electrical signatures of the electric appliances such as their usage time, reactive power, current and voltage waveforms can be supplemented to improve the identification accuracy. At an acceptable level of accuracy, a faster appliance identification algorithm will be more conducive to the data mining on load information in the big data environment of the Smart Grid.

The proposed algorithm is tested on datasets with various sampling frequencies. The identification accuracy is given in Figure 4.5. As can be seen from Figure 4.5, as the data sampling frequency decreases, the difficulty of identification gradually increases, and the accuracy decreases. When the data is sampled every 3 minutes, the identification accuracy on multi-appliance subsequences will drop to 72.84%.

95 Identification accuracy of multi-appliance subsequences 90 Overall identificatin accuracy

75 70

65 Identificationaccuracy (%) 60 0.511.522.53 4 Time interval of load data sampling (min) Fig. 4.5 Accuracies of appliance identifications under different sampling frequencies

4.5.2 Parameter Estimation of Residential Load Model

As discussed above, the length of time period cannot be too short. In this case study, it is set to be 0.5h. In the former section of electric appliance identification, the identification results of 500 households’ profiles have been obtained. Using these results, parameter values of β0,t and βi,t at each time are calculated, as shown in Table 4.5. Since Eqn. (4.10) is a statistical load model, the values of β0,t and βi,t at each time are used to estimate their distribution. The nonparametric test on SPSS is adopted and the final result is shown in Table 4.6.

Table 4.5 Values of parameters in the system load model

Time β0 Time β0 β1 β2 β3 β4 β5 β6 β1 β2 β3 β4 β5 β6 t(min) (MW) t(min) (MW) 1 0.027 14 14 0 0 1 0 31 0.397 66 90 1 1 9 1 2 0.093 20 27 0 0 2 0 32 0.403 69 79 1 1 10 1 3 0.149 31 34 0 0 2 0 33 0.411 70 75 1 1 9 1 4 0.229 38 42 0 0 3 0 34 0.399 67 77 1 1 9 1 5 0.277 44 49 0 0 3 0 35 0.408 81 80 2 1 9 1 6 0.306 50 55 0 0 3 0 36 0.402 83 90 2 1 10 1 7 0.297 52 62 0 0 3 0 37 0.386 88 85 2 1 10 1 8 0.284 56 63 0 0 3 0 38 0.346 89 86 1 1 10 1 9 0.285 58 64 0 0 3 0 39 0.339 93 90 1 1 10 1 10 0.326 57 69 0 0 4 1 40 0.370 85 87 1 1 10 1 11 0.341 63 79 0 0 6 1 41 0.424 93 79 1 1 10 1 12 0.336 65 83 0 0 6 1 42 0.471 96 75 1 1 10 1 13 0.344 66 83 1 0 7 1 43 0.490 98 79 1 1 11 1 14 0.331 68 87 1 0 7 1 44 0.515 96 80 1 1 13 2 15 0.357 71 85 1 0 7 1 45 0.531 95 79 1 1 15 2 16 0.377 68 86 1 0 7 1 46 0.545 91 83 2 1 16 2 17 0.392 67 83 1 1 8 1 47 0.522 93 80 2 1 15 2 18 0.393 66 78 1 1 8 1 48 0.501 97 81 2 1 14 2 19 0.394 60 73 1 1 8 1 49 0.529 106 80 2 1 14 2 20 0.374 64 72 1 1 8 1 50 0.530 113 80 2 1 14 2 21 0.386 64 72 1 1 8 1 51 0.508 113 85 2 1 15 2 22 0.405 67 74 2 1 8 1 52 0.510 120 87 2 1 16 2 23 0.383 68 73 2 1 8 1 53 0.510 128 90 3 1 17 2 24 0.382 70 80 1 1 8 1 54 0.515 137 89 3 1 17 2 25 0.402 68 79 1 1 8 1 55 0.531 137 90 3 1 17 2 26 0.400 63 85 1 1 8 1 56 0.488 140 90 3 1 18 2 27 0.412 65 88 1 1 8 1 57 0.465 139 92 4 1 18 2 28 0.418 64 94 1 1 8 1 58 0.446 138 92 4 1 19 2 29 0.414 61 92 1 1 9 1 59 0.448 134 86 4 1 20 2 30 0.396 61 90 1 1 9 1 60 0.481 123 74 3 1 20 2

Table 4.6 Results of the one-sample Kolmogorov-Smirnov test for all coefficients in the load statistical model

Parameters SPSS nonparametric test result β0 β1 β2 β3 β5 β6 Sample Size N 30 30 30 30 30 30 Minimum 0.34 66.0 74.00 ̶ ̶ ̶ Uniform Parameters Maximum 0.55 140.0 92.00 ̶ ̶ ̶ Poisson Parameter Mean ̶ ̶ ̶ 1.9667 13.500 1.5667 Absolute 0.190 0.168 0.156 0.140 0.189 0.209 Most Extreme Positive 0.035 0.168 0.100 0.050 0.189 0.208 Differences Negative -0.190 -0.126 -0.156 -0.140 -0.079 -0.209 Statistics of the test Kolmogorov-Smirnov Z 1.040 0.918 0.852 0.766 1.034 1.143 (D-value) P-value Asymp. Sig (2-tailed) 0.229 0.369 0.462 0.600 0.235 0.146

The running status of appliances from the 31 min to the 60 min in Table 4.5 is selected for the SPSS nonparametric test. The corresponding results are shown in Table 4.6. Since the time length in Eqn. (4.10) is set to be 0.5h, totally there is 30 sample data in each tested period. Under the significance level of 0.05, the significance level of one-sample nonparametric test results of β0、β1 and β2, i.e., the P-values in Table 4.6 are 0.229, 0.369 and 0.462, respectively, which are all greater than 0.05. So the null hypothesis of uniform distribution is accepted. The one-sample nonparametric test results of β3, β5 and β6 are 0.600, 0.235 and 0.146, respectively, all of which were greater than 0.05. They also accepted their null hypothesis of Poisson distribution. For β4, since the corresponding identification result remains constant during the analyzed period, there is no need to perform statistical analysis for this parameter.

It should be pointed out that the power consumption rate of appliance usually changes due to the adjustment of the working mode, such as the cooking stove. Therefore, this research uses the average power consumption rate of the appliance when calculating the parameters in the load model. But in the appliance identification, it is the typical fluctuated load curves of appliances that are used as their reference sequence.

4.5.3 Result Analysis

Based on the value of the parameters given in Table 4.3, different scenarios are simulated. From the statistical identification accuracy in Table 4.3, it can be found that the proposed algorithm could achieve an overall accuracy (the arithmetic mean of ηsigl and ηmult) at a range from 83% up to 88%. Especially for the single appliance subsequences, its accuracy is over 90% and up to 93.8% at most. While for the multi-appliance subsequences, it also achieves accuracy above 83% in the best case.

Simultaneous operations of multiple appliances result in the difficulty of identifying multi- appliance subsequence. The overlaps of power consumption curves that belong to different appliances will weaken their electrical signatures. For example, the sum of power draws that belong to two different appliances is very possibly similar to the power draw of a third appliance. So when using real power as a single feature, incorrect identification would occur. On the other hand, when appliances in a multi-appliance subsequence have different usage times, it will be difficult to precisely extract out each individual appliance. In addition, parameters ΔPset, Tset and the set of reference sequences, which indicate the power draw characteristics of appliances, also affect the identification accuracy. Since power draw of

74 some appliances could be complicated and changeable in practical situation, a comprehensive reference set of appliances would help to improve the identification accuracy.

Since most of residential appliances consume mainly real power, quite a lot existing researches have tried to disaggregate load using real power as a single feature. And it works quite well for high-power appliances with distinctive power draw characteristics. However, due to the similar power draw characteristics of some appliances and simultaneous state transitions of appliances, erroneous results could be acquired occasionally. Therefore, the introduction of more signatures, such as current waveform and voltage waveform, would provide more information when conducting identification. Meanwhile they also put a higher requirement of data processing capability and increase the complexity of calculation.

When further developing the proposed algorithm, reactive power, voltage waveform or even the current waveform would be considered. The final selection of additional appliance signatures would be determined through testing and balancing the effect with the increased calculation burden.

Table 4.6 presents the parameter estimation results of the proposed load model. It can be found that these coefficients all follow a specific distribution. Through statistical analysis, the proposed load model can to some extent reduce the influence of appliance identification error. And also, the distribution of parameters can help to extract end-user’s behavioural regularity of using these appliances. More importantly, the established statistical residential load model specifies the component of the system load in each time period and can be conveniently used for demand side management.

4.6 Chapter Summary

In the Smart Grid environment, the installation of smart metering devices makes it possible to collect more and more load data. Through data mining on the residential load big data, it can not only be used to analyse the behavioural regularity of end-users, but also to develop accurate load model to enable targeted demand-side management, or customized electricity retail plans. Residential appliance identification is the foundation of load data mining. This paper proposed a DTW matching based appliance identification algorithm, in which low frequency load data (one value per minute) is used. And also, a statistical residential model is proposed using appliance identification results to develop targeted demand-side management strategies or electricity retail plans. The proposed algorithm is implemented with R and tested with a load dataset that contains 500 households’ profiles. Experiments show that the

75 proposed algorithm is accurate and fast. In terms of accuracy, it could identify single appliance load subsequence at accuracy above 93%. While for the multi-appliance subsequences which is much more difficult to identify, it could also achieve accuracy at about 83%.

For further study, how to improve the identification accuracy of composite load subsequence will be a key problem. Another direction of next research would be consumer’s behaviour analysis with the help of appliance identification results.

Chapter 5 A Framework of Customizing Electricity Retail Prices

The problem of designing customized pricing strategies for different residential users is investigated based on the identification results of residential electric appliances and classifications of end-users according to their consumption behaviours. This study is based on following assumptions: 1) each retailer purchases electricity from the forward contract market, day-ahead spot market and real-time market; 2) the competition among retailers is modelled by a market share function; 3) each retailer adopts fixed time-of-use (TOU) prices for end-users; 4) the price fluctuations in day-ahead and real- time spot markets as well as uncertainty of electricity consumption behaviours are considered as main sources of risk. Under these assumptions, a pricing framework for retailers is established based on the bi-level programming framework and the optimal clustering in a time sequence. Meanwhile, profit risk is considered by taking conditional value at risk (CVaR) as the risk measure. The proposed bi-level optimization model is finally reformulated into a mixed-integer non-linear programming problem (MINLP) by solving Karush-Kuhn-Tucker (KKT) conditions. The online optimization solvers provided by the NEOS (Network-Enabled Optimization System) server and the commercial solver AMPL/GUROBI are used to solve the developed models, respectively. Finally, a case study is employed to demonstrate the feasibility and efficiency of the developed models and algorithms.

5.1 Introduction

The large-scale deployment of smart metering devices in power systems makes it possible to collect more useful data from electricity end-users. By mining the customer data, an electricity retailer is able to conduct residential appliance identification and behaviour

77 analysis of end-users, and then to extract valuable information on residential load patterns. Since load patterns can directly affect the supply cost of electricity, thus how to develop innovative retailing strategies with the help of appliance identification and behaviour analysis of end-users is an important problem concerned by electricity retailers.

In traditional regulated power industry, a fixed uniform retail price is usually used for a specific kind of end-users, which fails to provide incentives for end-users to change their behaviours. In some deregulated electricity markets, large industrial customers may be equipped with real-time meters in order to curtail power consumption in the periods of price spikes. However, the real-time pricing (RTP) mechanism directly exposes end-users to market price fluctuation risks. Therefore, it is difficult for small customers to widely accept RTP. As a pricing scheme falling in between the uniform pricing and RTP, the time-of-use (TOU) pricing is more widely adopted in practices [53]. In [139], different retail pricing strategies are investigated and summarized. Various price structures with different time resolutions ranging from hourly to seasonally are elaborated. Ref. [140] surveys the researches on electricity retailing in the last two decades. It elaborates the problems that cover the entire decision-making process of the electricity retailer, including the long-term retailer load forecasting, energy procurement strategies, retail pricing schemes, and risk management in the retail market. Especially, all the existing methods on retail pricing are categorized and analyzed in detail.

The retailer discussed in this chapter refers to an independent third-party entity that purchases electricity from suppliers and resells it to end-users. Retailer that belongs to a power supply or electricity distribution company is not considered [31].

In the proposed framework, firstly, several assumptions are made: 1) the retailer supplies electricity to end-users by purchasing from forward contract market, day-ahead spot market and real-time spot market; 2) the competition between retailers are modelled using a market share function; 3) the retailer adopts TOU rates for end-users; 4) price fluctuation in day- ahead and real-time spot markets as well as uncertainty of electricity demand are considered as main sources of risk, and conditional value at risk (CVaR) is taken as the risk measure. These assumptions are all general assumptions underlying the existing retail pricing researches, such as in Ref. [24], Ref. [26] and Ref. [27]. These assumptions will not affect the implementation of the proposed techniques. Also, end-users are classified according to their daily consumption patterns. Residential load is determined through combining the results of residential appliance identification and end-user classification. Under above assumptions, a framework of customizing electricity retail prices is proposed, which consists

78 of three models, namely the pricing model for hourly TOU prices, the segmentation model for TOU price structure and the customized retail price model for categorized end-users.

The main contributions of this chapter are summarized below:

(1) It proposes the idea of using the results of residential appliance identification and end- user behaviour analysis to help retail pricing. On the one hand, with the installation of smart metering devices, high-resolution load data can be collected, which may be used to extract valuable information about end-users. On the other hand, certain principles exist behind the power consumption behaviours of end-users. These will be obtained by mining the high- resolution load data.

(2) A mathematical model for optimizing the TOU price structure is developed. In existing research, the structure of TOU prices is often given in advance. In this chapter, time intervals of TOU price is optimized by clustering single temporal price sequences.

(3) A retail pricing model based on end-user behaviour analysis is proposed. In this model, in addition to price elasticity of demand, complementarity between different load patterns is also utilized, which is not considered in traditional retail pricing models.

The rest of the chapter is structured as follows. Section 5.2 reviews and discusses the publications on residential appliance identification and end-user classification. Then, the problem of retail load determination is analysed. The proposed framework of customizing electricity retail prices is presented in Section 5.3. Section 5.4 provides case study results and discussions. Finally, the chapter is concluded in section 5.5.

5.2 Appliance Identification and End-user Classifications Based Retail Load Determination

In modern power systems, the large-scale deployment of smart metering devices makes it possible to collect high-resolution load data for improving the operation and management of power systems. Through data mining, information about end-users’ behaviours and the running statuses of residential appliances will be available. This information can be used to enable targeted demand-side management, design customized electricity retail plans, or model customer load in finer granularity.

5.2.1 Residential Appliance Identification

As an important problem in load data mining, electric appliance identification has attracted wide attentions. Different from load disaggregation which only classify load into different load categories (such as resistive load, three-phase constant and quadratic torque induction motors) by their load characteristics [86], electric appliance identification aims to exactly recognise what exactly the power consumption equipment is. Therefore, it has a higher requirement on load sampling data [87]. According to the methods of collecting load data, there are two kinds of appliance identification methods in existing research, namely intrusive load monitoring [87] and non-intrusive load monitoring [88-90].

In [87], an approach of appliance identification based on distributed load metering data is proposed. On the contrary, non-intrusive method only relies on a single measurement unit for a household’s overall electricity consumption. After acquiring household’s high resolution overall load data, most existing non-intrusive algorithms identify appliances by matching appliance’s electrical signatures with the high-resolution load data. These signatures include current waveform, active and reactive power, instantaneous admittance waveform and switching transient waveform, to name a few [120].

Comparing with intrusive approaches, non-intrusive load monitoring methods have lower requirements on the hardware and require almost zero efforts from end-users, which makes it a more attractive approach.

Residential electric appliances can be identified using any of the above algorithms. It is assumed that the identification result is already known. According to whether the working power of the appliance can be adjusted and whether it is possible to shift its usage time, residential appliances are categorized.

5.2.2 End-user Classifications

Clustering algorithm is the most commonly used method to classify load profiles in existing publications. Each group of load profiles is called a cluster. In the same cluster, all profiles have similar patterns, while profiles in different clusters tend to be different.

The application of various clustering algorithms in grouping end-users or load profiles is studied in [141-144]. In [145], the performances of various algorithms, including modified follow-the-leader clustering, hierarchical clustering, K-means, fuzzy K-means and the self- organizing maps, are compared. In addition to profile patterns, other features of load data can also be used as a similarity measurement. Considering the uncertainty in electricity

80 consumption, end-users are clustered by their energy demand distributions in [146]. Furthermore, the cost of electricity supply for different clusters is also analysed in order to develop targeted residential energy efficiency programs.

Unlike [141-146] in which clustering is conducted according to similar patterns among different load profiles, in [147] pure binary integer optimization models for clustering load values in the same temporal load sequence are developed. The proposed models can also be used to cluster other temporal sequences, which are not only limited to load sequences. The proposed model in [147] is also adopted in this paper to build a segmentation model for optimizing TOU price structure.

Using above discussed clustering algorithms, it is assumed that end-users have been classified into J clusters according to their electricity consumption regularities.

5.2.3 Residential Load Analysis

As the electricity price in spot market changes quickly over time, it will be beneficial to precisely measure the cost of electricity supply if the load profile can be figured out. Because a residential load represents the aggregated power of all working appliances in the household, if end-user’s behavioural characteristics could be obtained, together with the results of appliance identification, the load patterns of this household can be discovered.

However, up to now, only a few publications that study electricity consumption behaviors are reported. Actually, the behavior information of end-users is usually hidden in the pattern of a load profile. In [148], load profiles are decomposed into harmonics by applying discrete Fourier transform. Analysis result shows that the higher frequency harmonics (harmonics beyond the 6th) act as noise and have negligible effects on the load profile. Each main component of load profiles can be viewed as periodical electricity consumption behaviors of end-users. Considering the behavioral characteristics, frequency components from the past could be used to forecast frequency components in the future.

According to the behavioral characteristics of end-users with various appliances, residential appliances can be categorized. Except for appliances like refrigerator which work all day long, other appliances can be generally categorized into three classes:

(1) Appliances that are used at specific occasions closely related to the daily schedule of end- users. These include Air Conditioner, lighting and all appliances for cooking. Air Conditioner is used only when an end-user returns home. Lighting is used in early morning and night and after users get back from work. Cooking related appliances are used before

81 each meal. In this research, it assumed that Air Conditioner would keep on since end-user returns home.

(2) Appliances that are used when electricity price is low. Specifically, they include Laundry Drier, Washing Machine, Dishwasher, Water Pump and load of charging EV battery.

(3) Appliances that are randomly used during certain period of a day. They include PC, TV, Microwave Oven and Electric Kettle, which may be randomly used when users are at home.

Table 5.1 shows the classifications of residential loads according to appliance usage.

Table 5.1 Classification of residential load caused by electric appliances

Time Power Price Typical Usage type shiftable adjustability elasticity appliance Refrigerator, AirCon, No Inelastic No Elec C-Stove, Oven Yes Elastic Lighting Regularly W-Machine, L-Drier, No Elastic Yes Water-Pump, Dishwasher Yes Elastic EV TV, PC, MicOven, Randomly No No Inelastic Electric Kettle

Note: AirCon/ MicOven/ Elec C-Stove/ PC/ L-Drier /W-Machine/ EV represent Air Conditioner/ Microwave Oven/Electrical Cooking Stove/ Electrical Kettle/ Desktop Computer/ Laundry Drier/ Washing Machine/ Electrical Vehicles, respectively.

5.3 Proposed Framework of Customizing Electricity Retail Prices

In the proposed framework, firstly, the retailer designs an hourly TOU price scheme by taking into account the entire load to supply simultaneously without classifying end-users. From these hourly TOU prices, the retailer will know the trend of retail price when supplying all the customers without distinction. However, the hourly TOU scheme would be too sophisticated for end-users to schedule their energy consumption activities. In practice, one day will usually be segmented into several blocks in a TOU scheme. Typical examples are two block TOU scheme (peak and off-peak periods), or three block scheme (off-peak, mid-peak and peak periods). Therefore, the second model in the proposed framework is a TOU time segmentation model. In existing publications, time segmentation of a TOU scheme is often given in advance. Using the proposed segmentation model, the optimal structure of TOU prices can be obtained when the number of TOU price blocks is given.

Finally, based on the optimal structure of TOU price, a model for customizing retail prices for categorized end-users is established in the third step. Fig.5.1 shows the three steps of the framework and logical relationships between these models.

Model of hourly TOU Use the Establish a model of price acquires hourly segmentation model customizing TOU

prices for TOU to optimize prices with the optimal structure optimization structure of TOU prices TOU price structure

Fig. 5.1 Schematic of logical relationships between models in the proposed framework

5.3.1 The Pricing Model for Hourly TOU Prices

In this chapter, it is assumed that the retailer will purchase electricity by signing forward contracts and participate in the day-ahead spot market to deal with imbalance electricity between actual demand and the quantity of forward contracts. Besides, the retailer also participates in the real-time spot market to balance the random demand of end-users.

Before establishing the pricing model for hourly TOU prices, it is necessary to determine residential loads based on the result of appliance identification and further classification of these appliances.

After having classified end-users into J clusters in section II, each cluster is viewed as a large electricity consumer. For the jth (j∈J) large consumer, its load components are analysed as follows.

(1) Time non-shiftable and power non-adjustable appliances

This type of loads has no price elasticity. The usage time of appliances belonging to this type completely depends on the daily schedule of residents. Through behavior analysis of residents, the probability distribution of an appliance working in each time interval can be derived.

For the jth large consumer, as there is more than one type of appliances belonging to this type

non th of loads, it is assumed that this type of loads contains Nj types of appliances. For the n (n

non ∈ Nj ) type, the total number of appliances and the average power rating of individual

non app,non th appliances are denoted by Nj,n and Lj,n . Denote the probability that the n appliance is

non non working in time interval t by δj,t,n , then the corresponding load Lj,t resulting from these appliances can be calculated with Eqn. (5.1).

non N j non non non app,non (5.1) LNjt,,,,, jnδ jtnL jn n1

(2) Time non-shiftable but power adjustable appliances

ela th Similarly, it is assumed that this type of loads contains Nj types of appliances. For the n (n

ela ∈ Nj ) type, the total number of appliances and the average power rating of individual

ela app,ela ela appliances are denoted by Nj,n and Lj,n , respectively. Let δj,t,n denotes the probability that the nth appliance is working in time interval t.

For appliances belonging to this type of loads, since their power consumption rates are adjustable, they are price elastic. According to the price signal, the power of appliances can

ela eu be reduced or increased appropriately. Given price elasticity of demand denoted by fj (rt ),

ela the corresponding load Lj,t resulting from these appliances is calculated as:

ela N j ela ela ela app,ela ela eu (5.2) LNjt,,,,, jnδ jtnLfr jn j() t n1

eu where rt is the electricity retail price at time t.

Several different functions have been used to model the price elasticity of electricity demand, such as the linear function [24, 31], the power function [32, 33], and the stepwise function

ela eu [34]. In this paper, the linear function is chosen to simplify the model, which is fj (rt ) denoted by Eqn. (5.3). The difference between different end-users’ elasticity is considered

ela ela through assigning different coefficient values, namely β0,j and β1,j .

frela euβ elaβ elarrr eu  eu eu j tjjttt0, 1,  0,  0, (5.3)

eu where r0,t is the nominal retail price at time t.

(3) Time shiftable but power non-adjustable appliances

Among existing residential appliances, Washing Machine, Laundry Drier and Water Pump belong to this type of loads. For these appliances, a similar feature is that they usually conduct a task by finishing a specific working cycle. For Washing Machine and Laundry Drier, they both have cycle settings. For water pump, it is usually filling or draining a pool that measures its work cycles.

It is assumed that end-users will try to minimize their electricity bills by scheduling the

LS LS usage of appliances. Let Nj represent the types of appliances; Nj,n denotes the total number

th LS LS of the n (n ∈ Nj ) type appliance; Qj,n represents the average energy consumption of

eu individual appliances for finishing a single task. Then if the retail price rt at time t is given,

LS the corresponding load Lj,t resulting from these appliances can be determined through the following optimization model.

LS T N j LS eu (5.4) min Lrjtn,,  t jJ tn11

LS LS LS LS s.t. N j,,njnjtnLL ,, 0 nN j (5.5)

T LS LS LS LS LtNQnNjtn,,Δ jn ,  jn ,  j (5.6) t1

LS Eqn. (5.5) denotes the constraint on appliance power rate and Lj,n is the maximum appliance power.

LS N j Then LS LS (5.7) LLjJtTjt,,, jtn , n1

LS where Lj,t,n is the decision variable denoting the scheduled power rate of each appliance at time t.

(4) Time shiftable and power adjustable appliances

EV is a representative appliance that belongs to this type of loads. With the energy stored in the battery of EV, it can also discharge electricity to power systems if needed. However, in this paper, it is only viewed as a load with its discharge capability being neglected. For simplicity, all EVs contained in the jth large consumer are considered equivalent to a single EV.

EV th EV Let Nj denote the total number of EVs in the j large consumer and Qj,n denote the average daily electricity demand of individual EV. The battery charging power at time t is denoted by

EV Lj,t and calculated by minimizing the total charging cost.

T EV eu min  Lrjt,  t jJ (5.8) t1

EV EV EV ela eu s.t. LNPfrjt,  j j() t (5.9)

T EV EV EV LQNjt,, jn j (5.10) t1

EV EV EV Ljt,  0 and Ljt,  0 if tT (5.11) where TEV is the set of candidate periods when battery is charged and PEV is the power rating of each individual battery.

(5) Random loads

Under the fixed TOU price scheme, end-users will be able to consume electricity freely. Therefore, it leads to some random loads. In order to meet this kind of unpredictable loads, the retailer needs to purchase electricity from the real-time spot market.

It is assumed that there are Nran types of appliances which may be randomly used. For the nth

ran app (n ∈N ) type, the power rating of each individual appliance is Lj,h .

app ran

Let Hj denote the set of appliances and Tj denotes the set of periods when random usage of appliances may happen.

HLLLapp(,,, app app app ) T jJ jjj,1 ,2 jN, ran (5.12)

Ttttran(,,, ran ran ran ) jJ jjj,1 ,2 jT, ran (5.13)

Thus, the random demand can be modelled as follows:

ran app ran ran LLj,,,,tjhjthj δ jJ, tT (5.14) app hH j

ran th app where δj,t,h is the probability that the h (h ∈ Hj ) appliance is working at time t and it is a random parameter following a given distribution.

eu th At this point, the gross load Lj,t of the j large consumer can be calculated by

eu non ela LS EV ran Ljt,,LLLLL jt jt ,,,, jt jt jt j JtT, (5.15)

In Eqn. (5.15), the last term represents the part of residential load caused by end-user’s free consumption behaviour, using identified or un-identified appliances. Therefore, Eqn. (5.15) can include the effect of un-identified appliances. Moreover, this last term can also be set as the deviation between the load of identified appliances and the actual load. Thus, it represents both the effects of identification error and the un-identified appliances.

Since the aim of the proposed retail pricing framework is to customize electricity retail plans using the results of appliance identification, the accuracy of appliance identification is essential. In existing publications, the accuracy of appliance identification is already quite high, which is 95.5% and 96.3%~97.6% in [87] and [89], respectively. Therefore, in the proposed framework, we mainly consider the error caused by un-identified appliances, but not the error caused by wrongly identified appliances. Indeed, if the error caused by inaccurate identification can be integrated in the model, it will help improve the quality of customized retail plans. This is because in the proposed framework, a residential load is analysed based on behavioural characteristics of end-users in using various appliances. Each component of a residential load is determined by considering decision-making of end-users on electricity consumption. A more accurate appliance identification result can help calculate the residential load more accurately. Besides, a new method is demanded for load modelling which is different from Eqn. (15) in order to incorporate the identification error. The proposed framework can be further improved on this point, and will be carried out in our future research work.

To assess the competition between retailers, the market share function [149] is usually used, which denotes the percentage of overall demand that can be served by the retailer at different retail prices. The same market share function presented in [24] is employed but with a different parameter of the electricity price. In [24], it is assumed that end-users will immediately reschedule demand when given a new price at time t. However, because assuming the retailer can lose market share within a very short period is impractical. Thus, in this paper the market share of the retailer is modelled as a function of the average retail price during a period of T, which means end-users can switch between retailers within days at the soonest. Moreover, existing research has shown that the electricity consumption pattern of residential customers does not change much over few years [12]. Then the customized retail prices based on daily load pattern can also be offered to end-users as monthly or quarterly pricing plan, as long as there is no significant change on end-users’ load pattern. At that time, Eqn. (5.16) will represent the switching activity of end-users over a longer term. Let M (reu,ave) denote retailer’s market share when the average retail price is reu,ave.

β ms r eu,ave  μr Mr()eu,ave ααα 1  11  1erf 112r (5.16) 2 2σ

where α1 represents the share of the retailer from loyal customers; α2 represents the share of

ms r r rival retailers from loyal customers; β1 is a constant coefficient; μ /σ are the mean and standard deviation of the historical average retail prices.

When purchase electricity from the forward contract market, a retailer needs to build the best portfolio of different forward contracts. According to the delivery period, they can be categorized as: peak, off-peak and round-the-clock [24, 34]. Let NF represent the amount of

F F th forward contracts signed by the retailer; pi /ri are the level of quantity and price of the i contract.

At this point, the pricing model for hourly TOU prices is proposed for a time period of 24 hours, utilizing the daily behavioural characteristics of end-users. The first objective function is to maximize the expected profit of the retailer.

JT TNF JT T J ret eu,ave eu eu F F F eu,ave ran rea mc mc R Mr() Lj,,,,tt r  p i r i u it  Mr () L jtt r  r t  p jt jt11 ti  11 jt  11 t  1 j  1 (5. 17)

ret F where R is the expected profit; ui,t is a binary parameter, it means that time t pertains to the

th F F rea mc delivery period of the i contract when ui,t is 1; if not, ui,t is 0; rt / rt are prices in the real-

mc time and day-ahead markets at time t; pj,t is the imbalance power between actual demand and the quantity of forward contracts, and will be purchased or sold in the day-ahead market at time t.

When maximize the expected profit, the retailer also needs to minimize the risks at a given confidence level βCVaR. For a retailer, price fluctuations in both day-ahead and real-time spot markets and end-user’s demand uncertainty are the main sources of risk. Using CVaR as the risk measure, the risk faced by a retailer can be calculated as:

s 1 N  RCVaRα CVaR ()R retα CVaR CVaR s   (5.18) 1β N ns 1 where αCVaR represents the corresponding VaR value used in CVaR calculation. Ns denotes the number of samples.

Besides, the pricing model for hourly TOU prices needs to satisfy the following constraints.

(1) Bounds on retail prices

The upper bound of the average retail price reu,up which is set by the regulatory department, should be considered.

TT j eu, ave eu eu eu eu,ave eu,up rLrLjjttjt,, and rrj  (5.19) tt11

eu,ave where rj is the average retail price for consumer j.

(2) Constraint on quantity traded in forward contracts

As forward contracts are usually traded at some specific quantities, it is assumed that the quantities of forward contracts will be selected from a set of discrete possible values [24, 34].

F F th Let pi / ri denote the quantity / price traded in the i contract.

FFFFF riiiβ01β (Δ)mp  (5.20)

F th where mi is the integer decision variable of the i contract.

(3) Power balance

In a power system, supply and demand are balanced in real time. Power balance of a retailer can be expressed as follows.

JNJF eu,ave eu ran F F F mc M ()()rLLmpupj,,tjt i  Δ iitjt , , (5.21) jij111

(4) Range of power adjustment

To make the model more realistic, the power adjustment range of price elastic appliances is considered.

app,ela ela eu app, up ela eu LfrLjn,, j t jn and frjt   0 (5.22)

app,up th where Lj,n is the maximum power of the n appliance in large consumer j.

5.3.2 The Segmentation Model of TOU Prices

In practise, TOU prices that are segmented into blocks of time periods are widely adopted. Although determination of TOU price structure is also an important aspect of retail strategies,

89 to the best of our knowledge, no research on optimally designing TOU price structure has been reported. The structure of TOU prices is often given in advance without explanation.

As mentioned above, in [20] binary integer optimization models for clustering data in a temporal sequence are developed. Hourly TOU prices also form a continuous temporal sequence. Given the number of blocks in an hourly TOU price scheme, the model in [20] is employed to determine the TOU price structure. Let K denote the number of blocks. The proposed segmentation model of TOU prices can be written as follows, where the objective is to minimize the sum of the distances between all pairs of time periods sharing the same block.

TT min Z psvuvuDx,, (5.23) vu11

wTv1 s.t. xvTuTmod , mod 1 (5.24) vwT1 uw 

TT11 xvu,  K (5.25) vu00

vu dxxyyxTxyymin (  )22  ( ) , (  ) 22 ( ) If , then vu,  vu vu v u vu (5.26)

vu dxxyyxTxyymin (  )22  ( ) , (  ) 22 ( ) If , then vu,  v u v u v u v u  (5.27)

 uu difvuhm, ,   hvmh1 D  vu, uNuN (5.28)  difvu,   hTmTmod , mod  hvmh1

where xv,u is the decision variable of this integer programming model; If a cluster starts at period v and finishes at period u, xv,u =1; otherwise, xv,u =0; x/y are the values of the

eu horizontal and vertical axes of electricity retail prices, r t ; dv,u represents the Euclidean distance between point (xv, yv) of period v and point (xu, yu) of period u on the coordinate plane. v mod T represents the remainder when v is divided by T.

5.3.3 The Model for Customizing Retail Prices for Categorized End-users

The electricity end-users are usually categorized into residential, commercial and industry users in existing research. Further classification in each category is seldom considered.

Using smart metering devices, more data about users can be collected. Using data mining techniques, more features of each category of end-users can be extracted. The retailer will then be able to manage end-users more precisely.

This chapter mainly focuses on residential users. After the behaviour analysis, they are classified into J classes. For different classes of end-users, their load patterns have different characteristics. As discussed before, end-users are sensitive to electricity prices. By offering different retail price plans to them, the complementarity of different end-users can be utilized to maximize the profit for retailer.

The problem of customizing retail prices for categorized end-users is modelled as a bi-level optimization problem. In the upper-level model, the retailer determines customized retail prices to maximize its total profit while minimizing the risk. In the lower-level model, end- users respond to retailer’s strategies by adjusting their electricity usage scheduling. End- users may also switch to other suppliers if retail prices exceed their expectation, which is reflected by the retailer’s market share function.

The upper-level model:

JT TNF ret,cus eu,ave eu eu,cus F F F maxR Mrj (jjtjtiiit ) L,, r  p ru , jt11 ti  11

TJ NF mc eu,ave eu ran F F F rMrLLmputjjjtjtiiit()() ,,  Δ , tj11 i  1

JT eu,ave ran rea cus CVaR,cus Mrjj() L jtt, r β R (5.29) jt11 s.t.

s 1 N  RCVaR,cusα CVaR R ret,cusα CVaR CVaR s   (5.30) 1β N ns 1

eu,cus eu,cus rrjt,,1 jt , tt,( 1) tk (5.31)

K tTk  (5.32) k1

Eqn. (5.19), Eqn. (5.20), and Eqn. (5.22) (5.33)

eu F rt  0 , mNi  (5.34)

The lower-level model:

LS T N j (1) LS eu,cus (5.35) min Lrjtn,, jt , jJ tn11

s.t. Eqn. (5.5)-Eqn. (5.7) (5.36)

T EV eu,cus (2) min Lrjt,, jt jJ (5.37) t1

EV EV EV ela eu,cus s.t. LNPfrjt,, j j() jt (5.38)

Eqn. (5.10)-Eqn. (5.11) (5.39)

eu,cus th eu,ave where rj,t is the customized retail price for the j large consumer; rj is the average value eu,cus cus cus of rj,t ; β is the weighting factor between retailer’s expected profit and profit risk, β ∈ [0,

cus ∞ ) [34]; the higher value of β , the more risk averse the retailer; tk represents the length of

th eu,ave th the k time block in the TOU price scheme; Mj (rj ) denotes the percentage of the j large

eu,cus F consumer’s demand that can be served by the retailer; rj,t and mi are the decision variables

LS EV of the upper-level model; Lj,t and Lj,t are the decision variables of the lower-level model.

5.3.4 Mathematical Reformulation of the Proposed Bi-level Retail Pricing Model

In order to solve the proposed model for customizing retail prices, the bi-level optimization problem needs reformulation. Considering that the lower-level models in the proposed retail pricing framework are linear programming problems, their Karush-Kuhn-Tucker (KKT) conditions are both necessary and sufficient for optimality. Therefore, the lower-level models are replaced by their KKT conditions in the original bi-level optimization problem.

The KKT conditions of the lower-level model indicated by Eqn. (5.35) and Eqn. (5.36) are derived as follows.

LS j Jt,,  Tn  Nj

eu,cus rggutjt,,,,1,,,2,, jtn jtn jtn Δ0 (5.40)

LS gLjtn,, ,1 jtn ,, 0 (5.41)

LS LS LS gLNLjtn,, ,2 jtn ,, jn , jn ,  0 (5.42)

ggjtn,, ,1,0 jtn ,, ,2 (5.43)

Eqn. (5.5)-Eqn. (5.7) (5.44) where the expressions of Eqn. (5.5) Eqn. (5.6) and Eqn. (5.7) are given in section 5.3.1(3).

Similarly, the KKT conditions of Eqn. (5.37) - Eqn. (5.39) are derived.

j Jt,  T

eu,cus rjt,,,1,,2,λλξ jt jt jt 0 (5.45)

EV λjt,,1L jt , 0 (5.46)

EV EV EV ela eu,cus λjt,,2LNPfr jt , j j()0 jt ,  (5.47)

λλjt,,1,0 jt ,,2 (5.48)

Eqn. (5.10)-Eqn. (5.11) (5.49) where the expressions of Eqn. (5.10) and Eqn. (5.11) are given in section 5.3.1(4). Eqn. (5.40) and Eqn. (5.45) are the zero gradient conditions. Eqn. (5.41)-Eqn. (5.42) and Eqn. (5.46) -

Eqn. (5.47) are the complementary slackness conditions. gj,t,n,1, gj,t,n,2, uj,t,n / λj,t,1, λj,t,2, ξj,t are Lagrange multipliers introduced during the reformulation.

To this point, the bi-level optimization problem is transformed into a non-linear complementarity model. Diverse linearization techniques are available to further transform the complementarity constraints, such as the Fortuny-Amat McCarl linearization [61], SOS1 and Penalty Function Linearization (Special Ordered Sets of Type 1 variables, SOS1 variables) [62], and other linearization methods based on exact algebraic transformations [58]. For calculation convenience, the Fortuny-Amat McCarl linearization method is adopted

g  L LS in this paper. For each of the product in the complementary equations such as j ,,tn ,1 jtn ,, in Eqn. (5.41), a binary variable η is added. Then the complementary slackness constraint can be replaced by the following two constraints: 93

g jtn,, ,1η M (5.50)

LS Ljtn,, 1 η M (5.51) where M is a large positive constant.

Finally, due to the non-linearity of the objective function and the market share function, the original bi-level optimization problem is reformulated into a mixed-integer non-linear programming problem (MINLP).

5.4 Case Study and Discussions

5.4.1 Solution Methodology

For convenience, the proposed models are numbered as model 1, model 2 and model 3 by their appearing sequence in above discussion. Due to the non-linearity of the objective function and the market share function, both model 1 and 3 are finally reformulated into a mixed-integer non-linear programming problem (MINLP) and coded into General Algebraic Modelling System (GAMS) models. The online optimization solvers provided by the NEOS (Network-Enabled Optimization System) server are used to solve the MINLP problem. Model 2 is an integer linear programming model and is solved by AMPL with the solver GUROBI.

5.4.2 Data of the Case Study

In the case study, three kinds of end-users represented by larger users 1, 2 and 3 are considered. Their lifestyle patterns are described by the expected probability of using appliances at each period, as plotted in Fig. 5.2. Each one of users 1, 2 and 3 is made up of 100 individual households. Three users 1, 2 and 3 are employed to represent the most common daily schedules in real life. User 1 represents a kind of people whose electricity usage happens in early morning, midday and early evening. User 2 represents the people who begin their work in early morning and don’t return home during the midday break until early evening. User 3 is a typical night worker at home so its electricity usage mainly happens from late evening to early morning. For these users, to some extent, the pattern of their electricity consumption behaviour is given as an assumption. The actual conditions must be much more sophisticated, but it does not affect our purpose to test the performance of the proposed framework.

In practice, the probability that each appliance is working at a certain time interval can be obtained through statistical analysis of appliance identification results. The information about the electricity consumption behaviour of end-users is hidden behind their historical load data. Through appliance identification on historical high-resolution load data, the working status of all appliances at each time period can be obtained. For the same time period in different days, the probability distribution of using each appliance can be estimated employing various density estimation methods. Then the probability that each appliance is working at each time period of a day can be attained. The idea of using the commercial statistical software SPSS to conduct statistical analysis for appliance identification results has been demonstrated in [150] in order to establish the statistical residential load model.

0.8

0.6

0.4

0.2

Expected probability behaviour of (%) 0.0

2 4 6 8 10 12 14 16 18 20 22 24 Time t (h) 用User 1 User 2 User 3 Fig. 5.2 The lifestyle pattern of each end-user

The details of the usage time of each appliance are given in Table 5.2. Some of the data in table 5.2 are from [87] and [90]. In Table 5.2, the power ratings refer to the equivalent average power consumption of each appliance. The equivalent power rating of a refrigerator is calculated when its daily energy consumption is 1.5 kWh. Besides, the lighting load is calculated based on a 70 m2 house with an average luminance load of 5.5 W/m2 for a family of three people. And the lighting load can be adjusted through controlling the number of bulbs being switched on.

Table 5.2 The residential appliance list and their operation features

Power Power Time Type Typical Usage time rating [kW] adjustable shiftable Lighting 0.385 Early morning & night Yes No Refrigerator 0.0625 All day No No AirCon 1.20 After work No No MicOven 1.00 Before meals No No Elec C-Stove 2.00 Before meals No No Oven 2.90 Before meals No No Elec Kettle 2.10 Before meals No No PC 0.18 After work No No TV 0.08 After work No No L-Drier 4.50 Low price periods No Yes W-Machine 0.50 Low price periods No Yes Water Pump 0.735 Low price periods No Yes Dishwasher 0.90 Low price periods No Yes EV 3.00 Low price periods Yes Yes

The coefficients of the price elasticity function and the parameters of the market share function in [24] are adopted here. The expected value of electricity prices in the real-time, r rea mc eu t and day-ahead markets, rt as well as the nominal sale prices, r0,t in the price elasticity function are given in Table 5.3. In the case study, the standard deviation of prices is set as 15% of their expected values.

Table 5.3 The expected values of prices in the real-time and day-ahead markets and the nominal prices in price elasticity function

mc rea eu mc rea eu

t rt rt r0,t t rt rt r0,t 1 0.04059 0.05042 0.05074 13 0.04344 0.05319 0.0543 2 0.03858 0.04821 0.04823 14 0.04278 0.0553 0.05348 3 0.03732 0.04938 0.04665 15 0.04278 0.05448 0.05348 4 0.03617 0.04741 0.04521 16 0.04407 0.05321 0.05509 5 0.03524 0.04737 0.04405 17 0.05043 0.05463 0.06304 6 0.03535 0.04749 0.04419 18 0.05033 0.05468 0.06291 7 0.03568 0.04832 0.0446 19 0.04925 0.05361 0.06156 8 0.03922 0.0497 0.04903 20 0.04533 0.05264 0.05666 9 0.04169 0.05195 0.05211 21 0.04304 0.05083 0.0538 10 0.04108 0.05101 0.05135 22 0.04278 0.05074 0.05348 11 0.04136 0.05328 0.0517 23 0.04285 0.04959 0.05356 12 0.04261 0.05198 0.05326 24 0.04175 0.04863 0.05219

Note: The unit of electricity prices is $/kWh.

ran When determining the random load of end-users, the parameter δj,t,h which indicates the random usage behavior of end-users at time t is assumed to follow normal distribution and is randomly generated. The similar forward contract configurations in [34] are also adopted here. For time-shiftable appliances, their electricity consumption of finishing a typical work cycle is given in table 5.4.

Table 5.4 Electricity consumption of time-shiftable appliances finishing a work cycle

Laundry Washing Electrical Appliances Water Pump Dishwasher Drier Machine Vehicle Energy 9.0 1.5 1.47 1.8 15 consumption

Note: The unit of energy consumption is kWh.

5.4.3 Results and Discussions

The first step of the proposed framework is to calculate the hourly TOU prices. With the increasing of the upper bounds on the overall average retail price, the price level of hourly TOU prices also increases. And the model obtains the optimal solution on boundary of the price constraint. Until the bound increases to 0.2 $/kWh, then the optimal solution stays at the price level of 0.199 $/kWh, see Fig. 5.3. That is due to the competition between retailers, when the average retail price increases, market share of end-users that is served by the retailer will decrease. Therefore, retailer needs to control a trade-off between retail price and market share to acquire the maximum total profit.

In the second step, the calculation result of hourly TOU prices is used to optimize structure of the TOU prices with a given number of blocks. Fig. 5.4 shows the segmentation results of hourly TOU prices when its average price is 0.18$/kWh. In the TOU price, adjacent points in similar level are classified into the same price block and distinct trends of change fall into different blocks. Moreover, the 24 time periods are segmented into uniform clusters relatively.

Using the model 2, a reasonable optimization result of the TOU price structure can be attained. Then, customized schemes of retail price for users 1, 2 and 3 are calculated when TOU prices are segmented into three blocks, see Fig. 5.5.

0.30 0.27

0.24

0.21

0.18

0.15

0.12

0.09

Retail price of electricity ($/kWh) of electricity price Retail 0.06

0.03

0.00 2 4 6 8 10 12 14 16 18 20 22 24 Time t (h) eu,ave eu,ave eu,ave r  0.051 r  0.06 r  0.08 eu,ave r eu,ave  0.1 r eu,ave  0.18 r  0.199

Fig. 5.3 Calculation results of hourly TOU prices under different scenarios

0.30 0.27

0.24

0.21

0.18

0.15

0.12

0.09

Retail price of electricity ($/kWh) of electricity price Retail 0.06

0.03

0.00 2 4 6 8 10 12 14 16 18 20 22 24 Time t (h) Hourly price 2 blocks 3 blocks 4 blocks 5 blocks Fig. 5.4 Optimal structure of TOU prices with different number of blocks

0.27 0.40 User 1 0.24 User 2 0.35 User 3 0.21 2 ($/kWh) user for Retail price 0.30 0.18 0.25 0.15 0.20 0.12 0.15 0.09 0.10 0.06 Retail priceRetail for users 3 ($/kWh)1 and 0.03 0.05

0.00 0.00 2 4 6 8 10 12 14 16 18 20 22 24 (a) Time t (h)

Fig. 5.5 (a) Customized price strategies when overall upper bound of retail prices are 0.16 $/kWh and 0.18$/kWh

0.34 0.45 User 1 0.32 User 2 0.40 0.30 User 3 0.35 2 ($/kWh)Retail user price for 0.28 0.26 0.30 0.24 0.25 0.22 0.20 0.20 0.18 0.16 0.15 0.14 0.10 0.12 0.05

Retail price for users 1 and 3 ($/kWh) 0.10 0.08 0.00 0.06 2 4 6 8 10 12 14 16 18 20 22 24 (b) Time t (h)

Fig. 5.5 (b) Customized price strategies when overall upper bound of retail prices are 0.16 $/kWh and 0.18$/kWh

Comparing Fig. 5.5(a) with Fig. 5.5(b), it can be found that when there is a smaller price bound, for user 3, its price in the peak period of power system (from period 14 to period 21) is even smaller than in the mid-peak period (from period 6 to period 13). That is reasonable for user 3 because of its special load pattern. Energy consumption of user 3 in the peak period of system is quite small and is even less than in the mid-peak periods, see Fig. 5.6(a). The retailer needs to control retail prices of user 3 pertained to off-peak and mid-peak periods in order to control the usage payment from user 3. So the price may present the pattern in Fig. 5.5(a). If the price bound further increases, patterns of retail prices for all users will be consistent, as shown in Fig. 5.5(b). And Fig. 5.6(b) shows the expected value of residential load corresponding to the case in Fig. 5.5(b).

2000 User 1 1750 User 2 User 3 1500

1250

1000

750

500 Residential load (kW)

250

2 4 6 8 10 12 14 16 18 20 22 24 (a) Time t (h)

Fig. 5.6 (a) Result of residential load when overall upper bound of average retail prices is 0.16 $/kWh

100

2000 User 1 1750 User 2 User 3 1500

1250

1000

750

500 Residentialload (kW)

250

2 4 6 8 10 12 14 16 18 20 22 24 (b) Time t (h)

Fig. 5.6 (b) Result of residential load when overall upper bound of average retail prices is 0.18 $/kWh

Due to the different regularity of their electricity consumption behaviour, the retail strategies have different features. The lowest average price belongs to user 1, while the highest belongs to user 2. Among these users, it is user 1 act as the main load during the daytime and its load is also quite stable. By offering a low average price to user 1, the retailer wants to win more market share of user 1 to support its trading of electricity in daytime. In terms of user 2, one of its energy consumption behaviours happens at the peak periods of system and the other one happens in the early morning. So it means that load of user 2 can neither be used to level total load in the evening nor be used to increase retailer’s trading of electricity in the daytime. Then it leads to the highest average price and lowest market share. For user 3, it is better than user 2 as it consumes electricity mainly in the evening. Load of user 3 is valuable for load levelling at night, which also brings itself a relative lower price than user 2.

To further test the proposed techniques, more simulation tests are carried out and the results are given, including the benefits of implementing proposed techniques, the benefits brought by the appliance identification, the test results with different market sizes and numbers of participants, the sensitivity analysis against risk weighting factor for users, and the sensitivity analysis against TOU price blocks for tested users.

101

(1) Benefits of implementing the proposed techniques

Fig. 5.7 gives the normalized profits of the retailer when adopting different pricing schemes. In Fig. 5.7, retailer’s normalized profits fall into a narrow range after the average retail price rises to 0.05$/kWh. This is because that the benefit of implementing customized retail prices is affected by the load elasticity. With the increasing of constraint on average retail price, profits of the TOU pricing grow to approach that of the hourly pricing. Since the percentage of elastic load in the total residential load is fixed, the normalized benefits from pricing elastic load will keep stable once the load elasticity is fully utilized.

1.00

0.95 Average retail Hourly price price ($/kWh) profit ($) 0.045 75.83 0.90 0.05 283.182 0.055 493.735 0.06 703.797 0.85 0.065 913.105 0.07 1115.998 0.075 1322.928 0.80 0.08 1525.728 0.085 1728.879 0.09 1933.384 0.75 0.095 2085.113

Retail profits normalized by the hourly price profit (%) profit price hourly the by normalized profits Retail 0.70 0.045 0.05 0.055 0.06 0.065 0.07 0.075 0.08 0.085 0.09 0.095 Average retail prices ($/kWh) Hourly price profit Un-optimized TOU price profit Customized TOU price profit Optimized TOU price profit Fig. 5.7 Normalized profits of the retailer under different average retail prices

Notably, the hourly pricing scheme brings the highest profit in most of the cases. However, because the hourly pricing scheme offers the same hourly prices to all types of end-users, it cannot fully utilize the temporal complementarity between different load patterns. Therefore, the profit brought by the hourly pricing scheme is exceeded by that of the customized retail prices when the average retail prices are 0.06, 0.065, 0.075 and 0.08 $/kWh respectively. Besides, it’s apparent in Fig. 5.7 that the customized retail prices can always bring the

102 retailer a higher profit than other TOU pricing schemes, where the non-optimized TOU pricing has the lowest profit.

(2) Benefits brought by the appliance identification

Fig. 5.8 gives the normalized retail profits through determining residential loads in different ways. By utilizing the results of appliance identification, each electricity retailer could have a better understating of residential loads. In existing publications on retail pricing, electricity consumption of all appliances is usually aggregated together indiscriminately. The aggregated load is then incorporated into the retail pricing model after multiplying the price elasticity function. When treating all residential loads as an aggregated elastic load, the price elasticity of demand cannot be accurately estimated. The elasticities of some appliances are nearly neglectable, such as the refrigerator. Besides, the aggregated elastic load may also fail to explore the benefits of time-shiftable appliances. In the proposed methodological framework, residential loads are classified based on the behavioural characteristics of end- users in using various appliances. Each component of a residential load is determined by considering the decision-making on electricity consumption by a given end-user. Therefore, in Fig. 5.8, the retail profit based on appliance identification is normally higher than that of an aggregated elastic load.

On the contrary, if all residential loads are treated as non-elastic, the concerned retailer will fail to utilize the demand responsiveness. As shown in Fig. 5.8, when treating all residential loads as an aggregated non-elastic load, the corresponding retail profit will be at the lowest level before the average retail price rises to 0.08$/kWh. If the degree of constraint on the average retail price increases, the prices in all TOU blocks will increase accordingly. The elastic load at each period will reach its upper limit. After the average retail price rises to 0.08$/kWh, the benefit brought by the price elasticity slightly decreases and is lower than that of an aggregated non-elastic load.

Fig. 5.9 shows the electricity consumption of end-users corresponding to the scenarios shown in Fig. 5.8. In Fig. 5.9, the electricity consumption level based on appliance identification is always higher than that of an aggregated elastic load. This indicates that the proposed method can help a retailer to attain a higher level of retail profit without sacrificing electricity consumption of end-users.

It can also be found that because of the demand elasticity, the consumption levels of an elastic load before 0.055$/kWh and after 0.08$/kWh are all lower than that of a non-elastic load. However, as mentioned before, when the average retail price is lower, the decision

103 space for the elastic load is larger. The retailer can develop a more favourable portfolio of electricity procurement, and then obtain a higher retail profit, so the retail profit before 0.055$/kWh is higher than that after 0.08$/kWh.

2.0

1.9 Average retail Profit under aggregated 1.8 price ($/kWh) non-elastic load ($) 0.045 34.093 1.7 0.05 248.318 0.055 455.614 1.6 0.06 669.242 1.5 0.065 880.351 0.07 1090.982 1.4 0.075 1298.976 0.08 1509.452 1.3 0.085 1719.291 0.09 1929.289 1.2 0.095 2089.968 Retail profits normalized by by normalized profits Retail 1.1

1.0 the profit under aggregated non-elastic load (%) load non-elastic theprofit under aggregated

0.9 0.045 0.05 0.055 0.06 0.065 0.07 0.075 0.08 0.085 0.09 0.095 Average retail prices ($/kWh) Profit under appliance identification load Profit under aggregated elastic load Profit under aggregated non-elastic load Fig. 5.8 Normalized profits of the retailer under different load determination methods 54000

53500

53000

52500

52000

51500

51000

50500

Electricity consumptionquantity of end-users(kWh) 50000

49500 0.045 0.05 0.055 0.06 0.065 0.07 0.075 0.08 0.085 0.09 0.095 Average retail prices ($/kWh) Quantity under appliance identification load Quantity under aggregated elastic load Quantity under aggregated non-elastic load Fig. 5.9 Electricity consumption of end-users under different load determination methods

104

(3) Tests under different size of the market and number of participants

Fig. 5.10 and Fig. 5.11 give the simulation results under different market sizes and participant numbers, where the upper limits of average retail price and the number of price blocks are set to be 0.085 $/kWh and 3, respectively. Even though the average retail price of end-users is fixed, Fig. 5.10 indicates that the profit of retailer shows a constant growing with the increasing of market size and involved participants. The enlarging of market size and number of participants can help to increase retailer’s profit. Meanwhile, Fig. 5.11 shows that when there is an obvious temporal complementarity between load patterns, the advantage of customized TOU price shows an apparent increase with the enlarging of market size. However, after the type of classified end-users increases to 4 and 5, the profit gap between customized and optimized TOU prices becomes less obvious. It is shown in Fig. 5.12 that with the increasing number of end-user types, there are more overlaps (indicated by the hatched area) between their lifestyle profiles. Because when load pattern overlaps, their complementarity can also be utilized through optimized TOU prices. Then the advantage of customized TOU pricing schemes begins to decrease. The benefits of customized retail prices are obtained mainly by utilizing the complementarity between end-user load patterns. Thus, the retailer should try to classify end-users into the most distinct clusters when customizing TOU retail prices.

21000 2 types of users participated 3 types of users participated 18000 4 types of users participated 5 types of users participated 15000 Secnario # Market size 170 12000 2 100 3 500 9000 4 1000

6000

3000

Retail profits brought by customized TOU price ($) Scenario 1 Scenario 2 Scenario 3 Scenario 4 Market size (the number of individual households included in users 1 to 5) Fig. 5.10 Profits of the retailer with different market sizes and participant numbers

105

180 2 types of users participated 3 types of users participated 150 4 types of users participated 5 types of users participated

120 Scenario # Market size 170 2100 90 3500 4 1000 60

30 optimizedTOU prices ($)

0 Profit difference between customized and customized between difference Profit Scenario 1 Scenario 2 Scenario 3 Scenario 4 Market size (the number of individual households included in users 1 to 5) Fig. 5.11 The profit difference between customized and optimized TOU prices under various scenarios

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2

0.0 0.0

0 2 4 6 8 10 12 14 16 18 20 22 24 0 2 4 6 8 10 12 14 16 18 20 22 24

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2 Expected probabilitybehaviourExpected (%) of 0.0 0.0

0 2 4 6 8 10 12 14 16 18 20 22 24 0 2 4 6 8 1012141618202224 Time t (h) User 1 User 2 User 3 User 4 User 5

Fig. 5.12 Lifestyle patterns of end-users 1 to 5 in the case study

106

(4) Sensitivity analysis against risk weighting factor for users

The weighting factor in the proposed framework quantify retailer’s attitude towards the decision risk. The higher value of weighting factor, the more risk averse the retailer. Therefore, with the increasing of weighting factor, the CVaR in retail decision would decrease, which complies with the simulation results in Fig. 5.13. Because the more risk averse the retailer, the more conservative the pricing strategy, so retailer’s profit decreases as well. To manage retail risks, the retailer usually needs to purchase electricity by forward contracts. Consequently, in Fig. 5.14 the retail power supply from forward contracts increases. On the demand side, when develops retail plans with low risk exposure, the end- user with higher random demands will be excluded namely user 1, which is assumed to have the highest percentage of random load. Then in Fig. 5.15, the market share of user 1 shows a decrease trend. On the contrary, the market shares of user 2, whose load is the least volatile, increase slightly. Since the percentage of user 3’s random load lies between the other two users, its market share keeps constant.

(The CVaR under scenario 1 is 2004188.756) 105

Weighting Scenario # 90 factor value 10 21E-06 75 31E-02 41 60 51E+02 61E+05

CVaR 30 Retail revenue

0 Values of Cvar and retailrevenue inthe results ($) Secanrio 1 Secanrio 2 Secanrio 3 Secanrio 4 Secanrio 5 Secanrio 6 Secanrios Fig. 5.13 CVaR and profit in the retail decision with different weighting factors

107

900 Peak contract Off-peak contract 750 Round-the-clock contract

Weighting 600 Scenario # factor value 10 450 21E-06 31E-02 40 300 51E+02 61E+05 150

Retailpower supply purchased byforward contracts (kW) Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Scenarios Fig. 5.14 Retail power supply purchased by forward contracts

0.35

User 1 User 2 0.34 User 3

0.33

0.32 Weighting Scenario # factor value 10 21E-06 0.31 31E-02

Market(%)3 and 1,2 users share of 41 51E+02 61E+05 0.30 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Scenarios Fig. 5.15 Market share of each end-user with the change of weighting factor

(5) Sensitivity analysis against blocks of customized TOU price

When conducting sensitivity analysis against TOU price blocks, scenarios with different constraints on the average retail price are considered. Under each average retail price, the number of price blocks increased from 1 to 5. Fig. 5.16 shows the intuitive result that with 108 the increasing of price block number, retail profit of the retailer also increases. Because when the price block number gets larger, the TOU retail price becomes more flexible. A more flexible pricing scheme will bring the retailer more benefits since the price elasticity of demand can be fully utilized. Especially, if the number of price blocks increases to 24, it is equivalent to offer a customized hourly pricing scheme to each user.

In the case study, user 1 and user 3 are set to have the least and largest price elasticity parameter, respectively. It is shown in Fig. 5.17 that the average retail price of user 3, which has the most price sensitive load, decreased slightly. As the whole average retail price for all users keeps fixed, user 2’s average retail price increased. To summarize, it’s the retailer and the users with the largest price elasticity that will finally benefit from the increasing number of TOU price blocks. Because different end-users benefit differently from the customized retail prices, the retailer should properly design corresponding subsidy policies for end-users to motivate them to adopt the customized retail plan. Especially, for the least elastic loads, they need more incentives.

1.005

reu,ave=0.065 1.000 reu,ave=0.075 reu,ave=0.08 eu,ave 0.995 r =0.09 reu,ave=0.095

0.990

0.985 Average retail Profit at 5 price ($/kWh) price blocks ($) 0.065 917.031 0.980 0.075 1325.246 0.08 1528.551 0.09 1926.76 when price block number is 5 (%)

Retail profits nomalized by the profit profit the by nomalized profits Retail 0.095 2076.606 0.975

0.970 12345 The block number of customized TOU price Fig. 5.16 Normalized profits of the retailer under different number of price blocks

109

0.098 0.096 0.094 0.092 user 1 0.090

0.088 0.10 0.09 0.08 0.07

user 2 0.06 0.05

0.10 0.08 0.06 0.04 user 3 0.02 The average retail price of users 1, 2 and 3 ($/kWh) 3 and 2 1, users retail of The average price 0.00 12345 The block number of customized TOU price reu,ave=0.095 reu,ave=0.09 reu,ave=0.08 reu,ave=0.075 reu,ave=0.065 Fig. 5.17 The average retail price of each user under different price blocks

5.4.4 Correlation between the case studies and the models in the proposed framework

The calculation of hourly TOU prices shown in Fig.5.3 tests the proposed pricing model for hourly TOU prices as expressed by Eqn. (5.1) – Eqn. (5.22). Different test scenarios are set by changing the bounds of retail prices in Eqn. (5.19). The calculation results of hourly TOU prices are given in Gif.5.3 where there is an increasing of the upper bounds on the overall average retail price.

The segmentation model of TOU prices in the proposed framework, which is indicated by Eqn. (5.23) – Eqn. (5.28), is tested through adjusting the constraint of Eqn. (5.25). Then the optimal structure of TOU pries with different number of blocks is given in Fig.5.4.

An initial test is carried out for the model for customizing retail prices for categorized end- users. The model is expressed by Eqn. (5.29) – Eqn. (5.39) and the results are given in Fig.5.5 – Fig. 5.6. The model of customizing retail prices is further tested by changing the constraint Eqn. (5.31) which represents the different retail pricing schemes. The retail profit as expressed by Eqn. (5.29) is then given in Fig.5.7, and these results show the benefits of implementing the proposed techniques. Moreover, the impacts of the lower level model as indicated by Eqn. (5.35) –Eqn. (5.39) on the retail pricing results are also analyzed. The

110 corresponding results shown in Fig.5.8 and Fig.5.9, which shows that the retail profit when determining retail load using appliance identification is normally higher than that when treating retail load as an aggregated elastic load. It verified the benefits brought by the appliance identification.

The results in Fig.5.10 – Fig.5.12 are obtained when changing the market size and the number of participants in the proposed framework. The market size is measured by the number of individual households included in each kind of end-users. In the proposed model, the market size is related to the total number of the nth type appliance, namely the parameters

non ela LS EV app Nj,n in Eqn. (5.1), Nj,n in Eqn. (5.2), Nj,n in Eqn. (5.4), Nj in Eqn. (5.9), and Hj in Eqn. (5.12). The number of participants are measured by the parameter J namely the clusters of end-users in the model. Results in Fig.5.10 – Fig.5.12 show that there is a constant growing of retail profit with the increasing of market size and involved participants. Meanwhile, the benefits of customized retail prices are obtained mainly by utilizing the complementarity between end-user load patterns. The retailer should try to classify end-users into the most distinct clusters when customizing TOU retail prices.

The results in Fig.13 are obtained through adjusting the risk weighting factor in the proposed model, namely the parameter βcus in Eqn. (5.29). βcus measures retailer’s attitude towards retail risks in the final retail decision. The choose of values of the weighting factor is referred to Ref. [34]. It is elaborated in [34] that βcus∈ [0, ∞ ], and the higher value of βcus, the more risk averse the decision maker. In this research, multi tests are carried out for the weighting factor. The result shows that with the increasing of weighting factor, the risk in retail decision would decrease. Several scenarios in these tests are chosen to manifest the results as plot in Fig.5.13 – Fig.5.15.

Besides, the constraint of Eqn. (5.25), which indicates the number of price blocks, are also tested to analyze its impact on the final retail pricing results. Under various retail constraints on the average retail price as expressed by Eqn. (5.19), the results in Fig.5.16 show that with the increasing of price block number, retail profit of the retailer also increase.

5.5 Chapter Summary

Constructing price strategies for an electricity retailer is an extremely important problem in the electricity market environment. Especially after the emergence of smart metering devices in modern power systems, more useful load data will be collected, which can be used for appliance identification and behaviour analysis of electricity consumption through data

111 mining. All the information will be beneficial to make targeted or customized retail schemes based on a better understanding of the components and features of residential load.

In this chapter, a fixed TOU pricing framework for electricity retailer is proposed based on the bi-level programming theory and the optimal clustering in a temporal sequence. The contributions to the research of electricity retail are threefold. A case study is presented to demonstrate the feasibility of the developed model, and verify the effectiveness of the algorithm employed. Several conclusions are reached based on the case study. Firstly, the proposed framework can effectively customize retail plans for classified end-users with distinct load patterns. Secondly, considering the advantages of customized TOU prices, the retailer has enough motivations to adopt the proposed techniques. Thirdly, when customizing retail prices, the retailer should try to classify end-users into the most distinct clusters. Thus, the retailer can benefit significantly from the temporal complementarity of end-users. Besides, subsidy policies for end-users are needed when implementing customized retail plans, since the benefit of end-users from the customized retail prices is related to their price elasticity of demand. For end-users with less elastic loads, they would need more incentives to participate.

Our further research will focus on algorithms of identifying residential appliances accurately as well as conducting behavioural analysis of end-users. Besides, the proposed framework will also be further developed to take into account uncertainty of appliance identification results.

112

Chapter 6 Customizing Electricity Retail Prices through Clustering Analysis

The problem of customizing electricity retail prices using data mining techniques is studied in this chapter. The density-based spatial clustering of applications with noise (DBSCAN) is firstly applied to load profile analysis, in order to explore end-users’ inherent electricity consumption patterns from their historical load data. Then, statistical analysis of end-users’ historical consumption is conducted to better capture their consumption regularity. After extracting these load features, a mixed integer nonlinear programming (MINLP) model for customizing electricity retail prices is proposed. In the proposed model, both the structure of TOU retail price and the price level are optimized once given the number of price blocks. It is among the first that the optimization of TOU price structure is studied in electricity retail pricing research. The proposed model is mathematically reformulated and solved by online commercial solvers provided by the NEOS (Net-work-Enabled Optimization System) server. Electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms.

6.1 Introduction

With the large-scale installation of intelligent metering devices in the Smart Grid, more useful data from electricity end-users can be collected. By mining the customer data, the electricity retailer is able to have a better understanding of end-users’ electricity consumption activity, and then extract valuable information on residential load patterns. Considering the time-varying prices and real-time balancing features of electricity markets,

113 research on the development of customized retail strategies using typical load patterns will be a problem of great importance concerned by electricity retailers.

Since the world wide deregulation of power industry back to early 1990s, electricity retail pricing schemes have evolved from a fixed uniform price to a dynamic and even a real-time price (RTP). However, for residential customers, the main demerit of RTP is that it directly exposes end-users to the price fluctuation risks. Therefore, it is difficult for small electricity customers to accept the RTP scheme. As a pricing scheme falling in between flat pricing and RTP, time-of-use (TOU) pricing is widely adopted in practice [53].

Ref [140] surveys the researches on electricity retailing in the last two decades. Various retail pricing models have been proposed in existing literature. For convenience, the term ‘dynamic pricing’ is used to denote the pricing schemes which are time-varying, and the term ‘static pricing’ refers to the pricing schemes which are pre-determined. On dynamic pricing schemes, the architecture design of real-time pricing market is studied in [54-56]. Since exposing retail consumers to the real-time electricity pricing mechanism will create a closed-loop feedback system and may also increase the market volatility, the influence of real-time pricing on market volatility is studied in [57]. Retail pricing for electric vehicle

(EV) charging is studied in [50] and [63], while the effect of CO2 emission on retail pricing is considered in [64]. Except real-time pricing, [20] and [59] studied the day-ahead hourly retail price. They both use the Stackelberg game to model the interaction between the retailer and its customers, where two- and single- stage games are established in [20] and [59], respectively.

On static pricing schemes, a variety of static pricing schemes such as the stepwise pricing, critical peak pricing, demand reduction programs and TOU pricing are proposed. However, most of the existing research is devoted to the TOU pricing. In [52], different pricing strategies for electricity retailers are investigated and summarized. Price structures for different time-horizons ranging from hourly to seasonally are discussed in details. The TOU pricing models proposed in the literature can be categorized into three categories: stochastic programming models [24, 32, 34, 66, 67], equilibrium models [33, 53] and game-theoretic models [69].

In existing researches, the structure of TOU price is usually given in advance. Without optimization of TOU price structure, the temporal complementarity among end-users that have different load patterns would be neglect. It is the complementarity of end-users that plays an essential role in developing flexible retail pricing schemes. The research work in

114 this area is still preliminary, and how to appropriately model and customize electricity retail prices in the smart grid environment still remains an open question.

In this chapter, the joint optimization of TOU price structure and price level for categorized customers is studied. Firstly, residential load features including the typical load pattern, the statistical feature of consumption quantity, and the classification of end-users are acquired through data mining in historical load data. Then, a model of customizing TOU price plans for categorized customers is established. To the best of our knowledge, the proposed model is the first model which can optimize the TOU price structure and retail price plans simultaneously.

The rest of the chapter is structured as follows. Section 6.2 reviews the literature on clustering algorithms for residential load profiles, and then presents the clustering and statistical analysis of residential load data. The proposed model of customizing electricity retail prices is presented in Section 6.3. Section 6.4 provides case study results and discussions. Finally, the paper is concluded in Section 6.5.

6.2 Load Pattern and Consumption Quantity Analysis

Since power systems are instantaneously balanced, the electricity price in the spot market is time-varying. Meanwhile, end-users can consume electricity freely under existing predefined retail pricing schemes. Therefore, all these lead to retailer’s exposure to retail risk when supplying volatile residential loads. In this chapter, the retail price is composed of two parts: the price determined by retailer’s purchasing cost both in the forward and spot electricity markets, and the risk premium due to electricity price and end-users’ demand fluctuation. Firstly, before establishing the retail pricing model, the cluster analysis of residential load profiles and the analysis on end-users’ electricity consumption are carried out, respectively.

6.2.1 Clustering Analysis of Residential Load Profiles

On the demand side of power systems, typical load patterns of end-users contain the temporal and spatial information about their electricity consumption activities. These implied load characteristics can be used to determine demand-side management strategies or the billing of electricity usage. In existing literature, various clustering algorithms have been adopted to extract typical load profiles from historical load data, which include k-means [151, 152], adaptive k-means [153], the hierarchical clustering [153, 154], the finite mixture model

115 clustering [155], the fuzzy c-means and the self-organizing map [156], support vector clustering [143], the two-stage clustering [144, 157], the subspace projection method based clustering [158], and the clustering by fast search and find of density peaks [159]. Besides profile patterns, other features of load data can also be used as a similarity measurement. Considering the uncertainty in electricity consumption, end-users are clustered by their energy demand distributions in [146]. Furthermore, the cost of electricity supply for different clusters is also analyzed to develop targeted residential energy efficiency programs. In [145], the performances of various algorithms, including modified follow-the-leader clustering, hierarchical clustering, k-means, fuzzy k-means and the self-organizing map, are compared.

Instead of focusing on the performance of clustering algorithms, the effect of the temporal resolution of data on clustering results is studied in [160]. It is found that load data needs to be sampled at a frequency of every 30min and ideally 8-15min. And also, the clustering result will not be reliable when the sampling frequency is lower than 30min.

In table 6.1, the comparison between existing publications on load profile clustering analysis is given in a tabular form. These publications are compared from perspectives of sample frequency, computation time and the adopted indicators for clustering validity assessment. In existing research, clustering experiments have been performed over load data with various resolutions ranging from 15 min to 60 min. Especially, in [160], a wider range of data from 0.5 min to 240 min is tested. It can be found from table 6.1 that the most of these researches use the 15 min, half-hourly or hourly load data.

Currently, there is still no standard or guidance on the resolution of load data to be used for clustering analysis. In [160], it is only three clustering methods namely the k-means, hierarchical clustering, and the Dirichlet process mixture model algorithm are studied and the results show that the clustering result will not be reliable when the sampling frequency is lower than 30 min. The choosing of clustering methods and load data in existing research is mainly based on the performance assessment of clustering results using the performance metrics as listed in table 6.1 or other metrics. Besides, the availability of smart meter data is another reason that would restrict the choice of load data in the research. Since the research on clustering analysis of load profiles in the power system is still preliminary, endeavors are still needed for developing more efficient methods.

116

Table 6.1 Comparisons between existing publications on load profile clustering analysis

Sample Method Computation time Indicator of clustering validity assessment frequency

To measure the performance of the method, three statistics were chosen in [151], which measure the accuracy, the bias, and the variability of clustering results respectively. In [152], the plausibility of the clusters is analyzed in order to assess the clustering results. The entropy of household, which measures the distribution of household load shapes over cluster The k-means [151, 152], centers, is constructed and used to analyze the adaptive k-means [153], clustering results in [153]. hierarchical clustering [153], subspace clustering and projected Hourly data Not mentioned. The constructed index of F1 – value, the accuracy of clustering [158], clustering, the entropy of clusters and the coverage of fuzzy c-means (FCM) [161], objects in each sub-space cluster are used to evaluate self-organization mapping (SOM) [161]. clustering results and to select the most appropriate number of clusters in [158]. The cluster validity index (CVI) is discussed for evaluating and validating the results of load classification in [161].

117

In [155], using a dataset consisting of 10, 000 bootstrap samples, the proposed clustering method took less than 2 hours on a basic desktop computer (single core Through Bootstrap resampling, the clustering processor at 2.30 GHz and 4 GB RAM computation repeats in [155]. The robustness of the A finite mixture model- based memory). clustering results is evaluated as a measurement of clustering [155], performance. Not mentioned in [156]. fuzzy k-means [156], A stability index and a priority index are proposed the fast search and find of In [159], the SAX technique is used for the Half hourly data for determining the most suitable clustering density peaks method [159]. dimensional reduction of data. Then two algorithm and the optimal number of clusters in different versions of the proposed algorithm [156]. are tested. In [159], the approach of entropy evaluation is The centralized clustering takes 60.058 sec adopted for assessing the clustering results.. for 3.46 million daily load profiles. In the distributed clustering algorithm, the time ranges from 0.415 sec to 0.542 sec, with an average of 0.472 sec and the times needed for global modelling is only 0.226 sec. Not mentioned in [143-145, 157]. The clustering validity indicators used in these Two-stage load pattern clustering [157], publications include the Clustering dispersion In [162], the computation efficiency of adaptive k-means [144, 157], indicator (CDI), the Intra-cluster index (IAI), the several clustering methods is compared k-means [145, 162], Modified Dunn Index (MDI), the Scatter Index (SI), (excluding the data input and the fuzzy k-means [144, 145, 162], the Variance Ratio Criterion (VRC), the Davies- calculation of the clustering validity hierarchical clustering [144, 145, 162], Bouldin Index (DBI), the Inter-cluster index (IEI), Every 15 min indicators). The results consistently indicate follow the leader method [162], the Mean Index Adequacy (MIA), the Similarity that k-means is the fastest one, followed by support vector clustering [143], Matrix Indicator (SMI), and the Ratio of within the follow-the-leader method. For the the adaptive vector quantization [144], cluster sum of squares to between cluster variation hierarchical clustering and the fuzzy k- modified follow-the-leader [145], (WCBCR). the self-organizing maps [145]. means algorithms, the computational burden is higher.

118

In order to assess the performance of clustering computation, different cluster numbers are assessed in terms of the dissimilarity within each group. The Hierarchical clustering and k-means [154] Every 10 min Not mentioned dissimilarity within cluster is represented by the sum of squares of the errors (SSE) between load profiles within each group.

Assessing the quality of the clusters is an internal process, while assessing the consistency of the cluster membership requires an external reference with The dataset of 5000 load profiles of 197 which to compare. Load data with customers is used in this study. With the various temporal increasing of sample interval from 0.5min 1) Internal Evaluators: Six internal evaluators are resolutions, selected: the clustering dispersion indicator (CDI), The k-means, hierarchical clustering, to 240min, the calculation time of including (in 5 the Davies-Bouldin index (DBI), the modified Dunn The Dirichlet process mixture model hierarchical algorithms decrease from 10 minutes): 0.5, 1, 4 Index (MDI), the mean index adequacy (MIA), the algorithm[160]. ms to 10 ms. But the k-means was 1-2 2, 4, 8, 15, 30, orders of magnitude faster than the scatter index (SI), and the variance ratio criterion 60, 120, and hierarchical algorithms. (VRC). 240, is tested. 2) Four external evaluators are used in this study, namely the Rand Index, the Pair-counting precision, the Pair-counting recall, and the Pair-counting F- score which combines the precision and recall.

119

Besides, widely used clustering methods for load profile grouping are surveyed and briefly reviewed in [161] and [162]. In general, clustering of load profiles is composed of following steps:

Data Data pre-processing Preparation

Data normalization

Dimension reduction

Clustering Analysis Choose clustering algorithm

Parameter setting

Results Evaluation Clustering results assessment

Fig. 6.1 Procedure of load profile clustering

As there could be distinctive load patterns over the different time horizon, such as seasons, workdays, weekends and holidays. Therefore, load data is usually classified by types of day before clustering. The coding process is an optional step and is used to transform high dimension load time series into low dimension series, such as when the weekly load profile [159] is used for clustering. When performing clustering, all existing methods face the same difficulty of parameter setting. Usually, the optimal parameters are selected through multiple tests or are chosen by the user. To evaluate clustering results, various clustering validity indicators are summarized in [162].

Instead of specifying the cluster number as a prior, such as in k-means and hierarchical algorithms, density-based clustering algorithm groups together points that are closely packed together, marking points as outliers that lie alone in low-density regions. In this chapter, density-based spatial clustering of applications with noise (DBSCAN) is adopted considering that inherent unknown load patterns are expected to be extracted out through clustering analysis of load profiles. Besides, DBSCAN algorithm has never been used for load profile clustering by existing research.

In terms of DBSCAN algorithm, the difficulty lies in choosing proper values for parameters ε, which defines the radius when searching for nearby neighbors, and Nminpt, which defines the minimum number of objects required to form a cluster within distance ε. In this chapter, ε

120 and Nminpt are chosen through establishing the histogram of distances between objects and the quantity cumulative distribution of each object’s neighbors within a given ε. Given a searching radius ε, the number of each profile’s neighbors is firstly calculated using the distance matrix. Then the empirical cumulative distribution of the quantity of these neighbors is derived, which intuitively shows the overall distribution of all objects’ neighbor quantity. Nminpt is chosen through referring to this overall distribution curve. Electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to test the proposed methods [163]. It recorded the half-hourly electricity consumption data of 31 end- users during the period from 1/6/2013 to 31/8/2013. After pre-processing, 2771 daily profiles each with 48 values are selected for clustering analysis.

In the process of data preparation, the raw load data is cleaned by deleting records with missing values and then normalized through dividing it by daily total electricity consumption. Fig. 6.2 (a) shows the histogram of distances between the normalized load profiles. Let y denote the percentage of load profiles and x denote the amount of their adjacent profiles. Given ε as 0.2, Fig. 6.2 (b) gives the percentage of load profiles whose amount of adjacent profiles is no bigger than x.

121

Frequency

(a) Distance value 0.6 0.8 1.0

0.4 0.2 Empirical cumulative Empirical distribution 0.0 0 500 1000 1500 2000 (b) Amount of neighbours within ε Fig. 6.2 The histogram of mutual distances and the cumulative percentage of load profiles whose total amount of adjacent load profiles is smaller than x when eps=0.2

122

In Fig. 6.2 (a), distances between those 2771 daily profiles fall in the range from 0 to 0.7 while most of the profiles lie within 0.2 away from their neighbors. Fixing the searching radius to be 0.2, a point (x, y) on the scatter plot of Fig. 6.2 (b) indicates that there is y percent of all the 2771 profiles and their adjacent load profiles is less than x. However, in terms of DBSCAN calculation, only the object with no less than Nminpt neighbors would be eligible to be a cluster member. Therefore, the point (x, y) also means that if assign x to parameter Nminpt, 1-y percent of all the 2771 profiles would be considered to form a cluster. To ensure at least 60% (namely y=0.4) of concerned profiles would be clustered, the Nminpt should be smaller than 600. After multiple tests, the optimal values of ε and Nminpt are set to be 0.2 and 300, respectively. Fig. 6.3 gives the final clustering results and the sizes of these clusters are 933, 304, 316, 302, 306, 303, and 307 respectively.

123

Cluster 1

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.09 Cluster 1 Centroid 0.08

0.07

0.06

0.05

0.04

0.03 Cluster Centroid

0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (1) Density based clustering results of residential load profiles

124

Cluster 2

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour)

0.07 Cluster 2 Centroid 0.06

0.05

0.04

0.03

Cluster Centroid0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (2) Density based clustering results of residential load profiles

125

Cluster 3

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.09 Cluster 3 Centroid 0.08

0.07

0.06

0.05

0.04

0.03 Cluster Centroid

0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (3) Density based clustering results of residential load profiles

126

Cluster 4

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.08 Cluster 4 Centroid 0.07

0.06 d 0.05

0.04

0.03 Cluster Centroi 0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (4) Density based clustering results of residential load profiles

127

Cluster 5

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.07 Cluster 5 Centroid 0.06

0.05

0.04

0.03

Cluster Centroid0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (5) Density based clustering results of residential load profiles

128

Cluster 6

Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.06 Cluster 6 Centroid 0.05

0.04

0.03

0.02 Cluster Centroid

0.01

0 0:00 1:00 2:00 3:00 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 2 2 2 2 Time (hour) Fig. 6.3 (6) Density based clustering results of residential load profiles

129

Cluster 7 Normalized load profile

0:00 4:30 9:30 14:30 19:30 23:30 Time (hour) 0.07 Cluster 7 Centroid 0.06

0.05

0.04

0.03

Cluster Centroid0.02

0.01

0 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Time (hour) Fig. 6.3 (7) Density based clustering results of residential load profiles

130

Through clustering analysis, electricity consumption patterns hidden in historical load profiles are extracted out. The cluster centroids plotted in Fig. 6.3 represent the different lifestyles of end-users in practice. Clusters 1 and 3 indicate the typical load pattern in which the peak load happens during 20:00-21:00 pm of each day. On the contrary, Cluster 4 shows a typical load whose electricity consumption mainly happens in the early morning. Clusters 2, 5 and 7 represent load patterns with two peak periods, which are in the early morning and evening, respectively. But their highest peaks happen at different time period and the peak durations are also different. Besides, Cluster 6 indicates a steady load, where the electricity consumption keeps relatively stable over the whole day.

6.2.2 Load Pattern and Electricity Usage Determination

Through clustering analysis of historical residential profiles, 7 typical load patterns are found. Since the electricity consumption pattern of a certain end-user is usually hidden in their historical load profiles. Besides, a certain end-user may have several different consumption patterns depending on the time span of historical data adopted. Therefore, profiles from each end-user may fall into different clusters. The distribution of each end-user’s profiles over all clusters is used for determining end-user’s typical load patterns, as indicated by Fig. 6.4. Since load profiles belonging to a certain end-user may fall into different clusters. The cluster that dominates an end-user’s profiles is chosen as the typical customer load pattern.

0.8

0.6

0.4

0.2 profiles over all clusters (%)

The distributionof end-users’ 0 1234567 # of load profile clusters user-3 user-5 user-8 user-27 user-28 user-17 user-12 user-16 Fig. 6.4 The percentage of each end-user’s historical profiles being clustered into different clusters

A typical load pattern represents the temporal distribution of end-user’s daily electricity consumption. To better understand the actual residential load, it is necessary to conduct statistical analysis of their electricity usage.

131

The historical daily electricity usage data of the 31 end-users in the chosen dataset are used for statistical analysis. The mean and variance of their daily electricity usage are derived, as shown in Fig. 6.5. . In Fig.6.5, each dot in the green line represents a historical data of end- user’s daily electricity consumption. The short blue and red lines represent the mean and variance of daily electricity consumptions during a period of time for each of the end-users, respectively. In Fig.6.5, the number above the short or red blue line is the ID of the 31 end- users in sequential order from left to right. As the volatility of end-user’s demand leads to retailer’s exposure to retail risk, the statistical analysis of electricity usage is therefore necessary when determining the risk premium in the retail price for each end-user. As the volatility of end-user’s demand leads to retailer’s exposure to retail risk. Therefore, the statistical analysis of electricity usage is necessary when determining the risk premium in the retail price for each end-user. In section 6.3, the retail pricing model is proposed to customize retail plans for the concerned households using the determined load pattern and electricity usage statistical results.

132

Mean of daily electricity consumption Variance of daily electricity consumption Historical daily electricity consumption # ID of end-users

#5 #22 #27 #16 #12 #6 #8 #24 household (kWh) household #7 #9 #10 #18 #29 #11 #13 #15 #25 #1 #14 #26 #17 #19 #30 #2 #3 #4 #20#21 #23 #28 #31 0 50 100 150 200 250 Daily electricity consumption of each each of consumption electricity Daily 0 500 1000 1500 2000 2500 Sequence of consumption data in the tested dataset Fig. 6.5 Statistical analysis of end-users’ daily electricity usage

133

6.3 Mathematical Formulation of Customizing Electricity Retail Prices

As an intermediary between generation companies and end-users, the retailer runs its business through purchasing electricity both from the contract and spot markets and then reselling to end-users at predefined TOU prices. In the contract market, different kinds of forward contracts are available depending on the delivery period: peak, off-peak and round- the-clock [24, 34]. Even though the long-term contract is a reliable source of purchasing, the retailer still needs to participate in the spot market for balancing load in real-time. For making optimal retail decisions, the retailer needs to optimize the portfolio of different forward contracts and transactions in the spot market. Meanwhile, the market competition between retailers should also be considered, because all retailers will try to attract customers by applying more competitive retail prices. In this chapter, the proposed retail pricing model is to develop retail plans by minimizing customer’s payment under the constraint of the rate of return. With such an optimization objective, the developed retail plans would be the most competitive in that the retailer’s revenue is guaranteed by the rate of return constraint.

On the end-user side, TOU prices are widely adopted by retailers to bill end-users. TOU pricing is a method of offering more than one fixed price before actual use and each price is applied during different predetermined intervals of the day. In practice, one day is usually divided into two blocks (namely peak and off-peak periods), or three blocks (namely off- peak, mid-peak and peak periods). However, the structure of existing TOU prices is often given in advance [24, 34, 69]. And also, all residential load is aggregated together for retail pricing. There are several advantages if retail plans are customized for different customers while the retail price structure is optimized.

(1) The derived electricity retail price is more accurate. End-users are characterized by their typical load pattern, as well as the statistical analysis result of their electricity consumption quantity. The retail plan for each cluster of end-users is calculated based on their unique load pattern and electricity consumption quantity.

(2) The setting of retail prices is more explanatory. In the proposed model, the retail price consists of two parts: price determined by long-term purchasing contracts, and the risk premium. These two parts of the final retail price can be easily explained by the corresponding load pattern and quantity analysis result discussed above.

134

(3) Through customizing retail prices, the different price elasticities of individual end-users can be fully utilized. In the proposed model, both price elasticity and cross price elasticity of demand are considered. For end-users, because of their unique load composition, for example some end-users may have many time-shiftable appliances, while others have more time non-shiftable but power adjustable appliances. All these factors affect their price elasticity and cross-price elasticity of demand. The customized retail prices can help well utilize the responsiveness of different types of customers.

In existing electricity retail pricing models, the objective is usually to maximize retailer’s profit while minimizing risks resulted from wholesale market price fluctuation and demand uncertainty. However, the objective of proposed models is to minimize customers’ payment while satisfying the constraint on retailer’s rate of return. Because by doing so, the assumption of market function can be avoided, which depicts the competition between retailers. To denote the total payment of end-users by Stp, the objective function can be expressed as follows.

JT tp minSLfrr Ej,t j ,t j ,t j ,t (6.1) jt11 where J is the number of end-user classification. T is the length of decision-making period.

th th E(Lj,t) is the expected load of the j end-user at time t. rj,t is the retail price for the j end-user at time t. fj,t (·) is the price elasticity function of residential load.

(1) Determination of end-user’s dominant load pattern

When developing electricity retail pricing schemes, load patterns of each end-user need to be firstly determined. In section 6.2.B, the distribution of each end user’s profiles over all clusters is given in Fig. 6.3. For end-users whose profiles are mostly (for example 60% of the end-users’ historical load profiles) clustered into the same cluster, then the centroid of the cluster can be intuitively selected as the dominant load pattern of these end-users. The dominant load patterns of end-users will be used to calculate their expected load for customizing electricity retail plans.

However, for end-users with highly uncertain electricity consumption behaviours, such as the 8th and 27th end-users in Fig. 6.3, there are more than one dominant load patterns for them. Assuming that there are Ndlp dominant load patterns for the jth end-user, for each load pattern s ∈ Ndlp, customized electricity retail prices rj,t,s are calculated. The final retail prices can be determined as follows:

135

dpl rjt,  maxr j,t,s , s N (6.2)

Since TOU price is the most commonly used retail pricing scheme, it is also adopted here but with the price structure left to be optimized.

(2) Constraint on TOU retail price

When adopting the TOU retail price, the 24 hours of each day will be divided into several periods, and during each period the retail price is fixed. It is assumed that the customized TOU retail price for the jth end-user is divided into Npb blocks. For the ith price block, retail

pb pb pb price is fixed at rj,i , and the corresponding time length is tj,i . Therefore, if t ∈ tj,i all rj,t equal to

pb the same price rj,i of price block i.

To optimize the TOU price structure, the binary vector yi,j is introduced with the length of T.

th th yi,j indicates the coverage of the i price block. If time t of a day is covered by the i price block, the element yi,t,j of yi,j will be 1. If not, yi,t,j will be 0.

yi,j ()y,y,,y,,y i,,j12 i,,j i,t,j i,T,j (6.3)

N pb (6.4) j,t yi,t,j 1 i1

T (6.5) i, j yi,11 ,j y i,T,j y i,t,j y i,t ,j 2 t2

Eqn. (6.4) ensures that each time period t exclusively belongs to a particular price block. When segmenting the T time periods into several blocks, it is assumed that only consecutive periods would be segmented into the same price block, as shown by Eqn. (6.5).

pb pc Besides, the price rj,i is assumed to be composed of two parts: the price rj,i determined by retailer’s purchasing cost both in the forward and spot electricity market and the risk

rp th premium rj determined by the profit risk of serving the j end-user.

pb pc rp rrrj,i j,i j ,  i,j (6)

Therefore, retail price rj,t with an optimal TOU price structure can be expressed as follows.

N pb ryrrpc rp (7) jjjj i, ,i i1

136

N pb ryrrpc rp j,t  i,t,j j,i j (8) i1

(3) Cost of forward electricity procurement

For the retailer, when supplying electricity to end-users, the retailer would sign forward contracts to ensure a reliable power supply for end-users after load forecasting. According to the available forward contracts in the forward contract market, they can usually be categorized as: peak, off-peak and round-the-clock [24, 34].

Given the expected load profile of end-users, the retailer would ideally expect that the portfolio of available forward contacts can perfectly match the load profile. Thus, the cost of power supply can be locked in advance without any risk. However, due to the uncertain electricity consumption behaviour of end-users, there are always difference between the portfolio of forward contracts and the load profile. In order to balance end-users’ demand, the retailer would purchase electricity from the electricity spot market. Therefore, the power difference leads to the retailer’s exposure to electricity spot market risk. When developing the optimal portfolio of forward contracts, the retailer usually should consider the balancing between the purchasing cost in the forward contract market and the retail risk exposed to the spot market.

F F F Let N represent the number of forward contracts signed by the retailer; pm/ rm are the level of quantity and price of the mth contract. CVaR is adopted as the risk measure. When supplying electricity to the jth end-user, retailer’s decision in the forward contract market can be modelled as follows:

N F FFfcCVaR,fc min prj,m mβ VjJ j ,  (6.9) m1

N  CVaR,fc fc1 fc fc s.t. V jjα   R jjα (6.10) 1β N n1

TNF RLfrrprCfc  F F fc, sp (6.11) j j ,t j ,t j ,t j ,t j ,m m j tm11

N F fc,spFF sp CLfrputrj j ,t j ,t j ,t  m  m ,tΔ t (6.12) m1

norm LLj ,tj,t Q j (6.13)

137 where βfc is the weighting factor between retailer’s expected revenue and profit risk, βcus ∈ [0,

fc CVaR,fc ∞ ] [34]; the higher value of β , the more risk averse the retailer; Vj denotes the retail risk due to the difference between forward contract power and the expected value of end-users’

fc total load; αj represents the corresponding VaR value used in CVaR calculation; β is the

fc,sp given confidence level; N denotes the number of samples; Cj is retailer’s cost of purchasing

F electricity from electricity spot market; um,t is a binary parameter, it means that time t pertains

th F F sp to delivery period of the m contract when um,t is 1; if not, um,t is 0; rt is the spot market price.

th norm Lj,t is the simulated load of the j end-user at time t;Lj,t is the expected value of normalized residential load which is derived in Section 6.2; Qj represents the simulated daily total electricity consumption of end-user j.

(4) Determination of risk premium in the retail price

The variability of spot market prices and the random electricity consumption of end-users are two main sources of risk in the electricity retail market. Eqn. (6.10) calculates the retail risk faced by the retailer. In the customized retail plans, risk premium is considered to

rp compensate the retail risks. To denote the risk premium of retail price by rj , it can be calculated by Eqn. (6.14), similar with the method in [77].

CVaR,fc rp Vj rj  (6.14) EQj

where E(Qj) indicates the expected value of end-user’s daily electricity consumption.

(5) Expected retail revenue of the retailer

As the proposed retail pricing model is to develop retail plans by minimizing customer’s payment under the constraint of rate of return, the retailer’s revenue is guaranteed by the rate of return constraint. Let W and e denote the retail profit and expected rate of return for the retailer, respectively. Before actually supplying retail load, the retailer only knows the typical load profile of end-users and the expect value of their electricity consumption. Therefore, the retail profit W and rate of return e when developing retail plans can be expressed as follows.

JT NF F F fc, sp WLfrrprCEEj,t j,t  j,t j,t  j,m m j (6.15) jt11 m  1

JNF F F fc, sp WprCej,m mE j  (6.16) jm11

138

(6) Constraint on price elasticity of demand

A variety of price elasticity functions of demand have been proposed, such as linear function [24, 31], power function [32, 33], and stepwise function [34]. In this chapter, the linear demand elasticity function fj(rj,t) is adopted for simplification.

 rrj,t 0 ,t frj j,tββ01 ,j ,j  (6.17) r 0,t

where r0,t is the nominal retail price at time t. β0,j and β1,j are coefficients in the function.

Eqn. (6.17) calculates the elasticity of electricity demand on nominal retail prices. Because of the existence of time shiftable loads, the cross-price elasticity of demand should also be considered. It means that due to the relative price difference between different price blocks, end-user may change its electricity consumption from one period of the TOU price to

th another period. For the i price block, let ci,j denote the elasticity coefficient resulting from the relative price difference. If take the first price block as a reference, then c1,j will be 1. ci,j of other price blocks can be calculated as follows:

2 coef ci, jβ i11rr i, j i , j  1 (6.18)

Considering the structure optimization of TOU retail price, the constraint on price elasticity of demand can be modelled by Eqn. (19).

N pb fr y c fr j ,t j ,t  i ,t , j i , j j  j ,t (6.19) i1

(7) Distribution network constraints

Even though end-users are managed by independent retailers in an electricity market environment, the operation of distribution network which conveys electricity to end-users is under the operation of the Distribution Network Operator (DNO). In order to ensure the security of network operation, the distribution network is operated under the control of DNO. Therefore, the retailer needs to consider distribution network constraints when making retail decisions. Besides, since the detailed modelling of distribution power flow is out of the scope of this paper, the distribution network here is treated as a lossless network with radial power flow. For simplification, it is assumed that all the end-users are connected to the same phase and the distribution network is already three-phase balanced. Therefore, the distribution network constraints can be modelled in a single-phase way. It is also assumed 139

th th that the j end-user is connected to the k bus. Let ρl,k denote the power transfer distribution factor (PTDF) which is used to indicate the relative change of real power that occurs on a particular branch l due to real power change at bus k. Especially, for a lossless and radial network, its PTDF matrix will be a binary matrix. Then for each branch l in the distribution network, it needs to satisfy the transmission constraint.

max  Lfrj,t j,t j,tρ l,kPlL l (20) kK

max where Pl is the power limit of branch l. K / L is the set of buses / branches in the distribution network.

Eqn. (6.1)-Eqn. (6.20) give the proposed model for customizing electricity retail prices. It has absolute value constraints for TOU price structure optimization and non-linear expressions in the formulation of CVaR. To linearize the constraint of Eqn. (6.5), binary variables Ui,j and Vi,j are introduced.

Ui,j u,u,,u i,12 ,j i, ,j i,T,j ; Vi,j v,v,,v i,12 ,j i, ,j i,T,j  (6.21)

T  i, j uvi,11 ,j i, ,j uv i,t,j  i,t,j 2 (6.22) t2

yyuvi,111 ,j i,T,j i, ,j i, ,j 0  yyuvi,2122 ,j i, ,j i, ,j i, ,j 0  yyuvi,3233 ,j i, ,j i, ,j i, ,j 0 (6.23)    yy uv0  i,T,j i,T1 ,j i,T,j i,T,j

T UVi,j i,j 0 (6.24)

 i,t, j ui,t,j 0; vi,t,j 0 (6.25)

The auxiliary variable Mj,n is introduced to linearize Eqn. (6.10).

N CVaR,fc fc 1 V j α jj,n M (6.26) 1β N n1

fc fc MRj,n j α j and M j,n  0 (6.27)

140

Due to the non-linearity of price elasticity function of demand, after these mathematical transformations, the proposed model is transformed into a mixed integer nonlinear programming (MINLP) problem and coded into AMPL (A Mathematical Programming Language) models. The online optimization solvers provided by the NEOS (Network- Enabled Optimization System) server are used to solve the MINLP problem.

6.4 Case Study and Discussions

6.4.1 Data in the Case Study

In the case study, customized TOU retail prices for the 31 end-users are calculated using the proposed model. The peak and off-peak periods of forward contracts are set to be from 17-22 and from 1-7 out of the 24 hours length. The corresponding peak, shoulder, and off-peak period’s forward contract price as well as the expected value of real-time market price are plotted in Fig.6.6. The number of customized TOU retail price is set as 3. Electricity retailer’s expected rate of return is fixed as 0.1. β0 and β1 in the price elasticity function are chosen as 1 and -0.25. The coefficient βcoef in the cross-price elasticity function is fixed to 1. After several tests, the parameter βfc which represents a trade-off between expected retail revenue and profit risk is set to be 1.0×105. Especially, it can be found in Fig.4 that most of the 31 end-users have a dominant load pattern except end-user No.8 and No.27. Their dominant load patterns are clusters 3, 4, 5 and clusters 1, 7, respectively. Besides, electricity consumption quantities of end-users are simulated using the statistical results shown in Fig.6.5.

141

0.35

Expected Value of Real-time Market Price 0.30 Forward Contract Price

0.25

0.20

0.15

0.10 Electricity price ($/kWh)

0.05

0.00 04812162024 Time (hour) Fig. 6.6 Electricity prices in the forward contract and real-time market

6.4.2 Results and Discussions

Fig.6.7 shows the customized retail plans for each end-user. Compared with existing un- optimized TOU retail prices, these customized retail plans are different on the following aspects: Firstly, in terms of the price structure, the peak period is generally shorter while the off-peak and shoulder periods are longer.

Secondly, the relative price difference between peak and other periods are bigger. These two differences may result from the cross-price elasticity function incorporated in the model. Because a larger price differences between different periods will result in a better price elasticity of load. Consequently, the elastic residential load in the peak period is lower when adopting customized retail plans. Meanwhile, residential load in off-peak and shoulder periods increase slightly, as shown in Fig.6.8. In terms of total electricity consumption, Fig.6.9 shows that end-users’ electricity consumptions when adopting customized retail prices are also higher than that under the un-optimized TOU price schemes.

Thirdly, for end-users that has the same load pattern but with different statistical characteristics of electricity consumption quantity. Their corresponding customized retail plans can be different, namely the statistical results of end-user’s electricity consumption quantities also affect their optimal TOU retail price, which can be found in Fig.6.7.

142

0.5

user 1 0.4 user 2 user 3 user 4

0.3

0.2

0.1

Customized electricity retail prices ($/kWh) 0.0

04 8 12 16 20 24 Time (hour)

0.5

user 5 0.4 user 6 user 7 user 8 user 9 0.3

0.2

0.1

Customized electricity retail prices ($/kWh) 0.0

048 12 16 20 24 Time (hour) Fig. 6.7 (1) Customized retail plans for each end-user

143

0.5

0.4 user 10 user 16 user 11 user 17 user 12 user 18 user 13 user 24 0.3

0.2

0.1

Customized electricity retail prices ($/kWh) 0.0

048 12 16 20 24 Time (hour)

0.5

user 25 0.4 user 26 user 27 user 28 0.3

0.2

0.1

Customized electricity retailCustomized ($/kWh) prices 0.0

048 12 16 20 24 Time (hour) Fig. 6.7 (2) Customized retail plans for each end-user

144

0.5

user 19 0.4 user 20 user 21 user 22 0.3 user 23

0.2

0.1

Customized electricity retail prices ($/kWh) 0.0

048 12 16 20 24 Time (hour)

0.5

0.4 user 14 user 15 user 29 user 30 0.3 user 31

0.2

0.1

Customized electricity retail prices ($/kWh) 0.0

048 12 16 20 24 Time (hour) Fig. 6.7 (3) Customized retail plans for each end-user

145

0.6

Residential load under customized prices 0.5 Residential load under un-optimized prices

0.4

0.3

0.2

0.1 Expected value of residential load of (kW) Expectedresidential value

0.0 04 8 12 16 20 24 Time (hour) Load profile of user-2 Fig. 6. 8 (1) Residential load under different retail pricing schemes

2.5 Residential load under customized prices Residential load under un-optimized prices 2.0

1.5

1.0

0.5 Expected of value load residential (kW)

0.0 04 8 12 16 20 24 Time (hour) Load profile of user-7 Fig. 6. 8 (2) Residential load under different retail pricing schemes

146

3.5 Residential load under customized prices 3.0 Residential load under un-optimized prices

2.5

2.0

1.5

1.0

0.5 Expected value of residential load (kW)

0.0 04 8 12 16 20 24 Time (hour) Load profile of user-12 Fig. 6. 8 (3) Residential load under different retail pricing schemes

40 Consumption under customized TOU prices Consumption under un-optimized TOU prices 35

0 Electricityquantity consumption of (kWh) end-users

0 2 4 6 8 1012141618202224262830 # of end-users Fig. 6.9 The electricity consumption of end-users under different electricity retail prices

147

Under the same constraint on rate of return, retail risks measured by CVaR are calculated under different retail pricing schemes. The practically adopted TOU price structure in New South Wales, Australia, is taken as un-optimized TOU price structure. The peak period covers from 14pm-20pm, off-peak period covers from 10pm-7am, and all the rest belongs to shoulder period. In Fig.6.10, the CVaR of customized retail plans is obviously lower than that of un-optimized retail plans. To manage the retail risk, on the one hand, retailer needs to utilize the price elasticity of load though a high peak price. On the other hand, electricity retailer develops optimal procurement strategies in electricity wholesale market. Through customizing retail prices, the retailer can develop more flexible pricing schemes that fit its risk management strategies well. In other words, through optimizing TOU price structure, retail risks can be managed more efficiently.

0.40 CVaR under customized TOU prices 0.35 CVaR under un-optimized TOU prices

0.30

0.25 rent pricingrent schemes 0.20

0.15

0.10

0.05

Retail CVaR under diffe under CVaR Retail 0.00

0 2 4 6 8 1012141618202224262830 # of end-users Fig. 6.10 The retail risks of different retail pricing methods measured by CVaR

As mentioned above, through customizing retail prices, the retailer can develop more flexible pricing schemes. Therefore, it can help retailer develop electricity procurement strategies with a lower cost. For each end-user, Fig.6.11 shows the component of retail price that stems from forward contracts electricity procurement. It is clearly shown that under customized retail prices, the price is lower.

148

0.03

Results under customized TOU prices Results under un-optimized TOU prices

0.02

0.01

0.00 Retail price resulting from forward contracts ($/kWh) resultingcontracts forwardRetailfrom price

0 2 4 6 8 1012141618202224262830 # of end-users Fig. 6.11 The component of retail price stemming from forward contract for each end- user

(1) Analysis of Distribution Network Constraints on the Optimization Results The IEEE 37-bus distribution system [164] is adopted to study the effect of distribution network constraints on the optimization results. Fig.6.12 gives the topology of the tested distribution system. All the 31 end-users are assigned to different nodes in the distribution network. As discussed in Section III, the distribution system is treated as a lossless network in this research. Case studies are carried out for scenarios 1-5. In each scenario, the distribution congestion is assumed to happen on different feeders from feeder #1 to feeder #5, which is indicated by red lines in Fig.6.12. The feeder capacity constraints before/after congestion, as well as the amount of affected end-users in each scenario are given in Table 6.1.

149

Distribution substation 799 User 3

722

User712 12 User 11 User701 1 User 4 User 8 User 9 742 713Feeder #3704 Feeder #4

User 27 705 70User2 2 User 5 714 Feeder #1 User 6 User729 16 7User44 15 7User27 14 70User3 13 User 10 718 User 7 User 17 728 73User2 23 7User08 22 User 19 Feeder #2 709 73User1 20 Distribution transformer User736 24

733 User775 21

0 7User34 28 74User0 18 User 25 Feeder #5 User735 26 User737 29 User738 30 User711 31 741 Fig. 6.12 Topology of the IEEE 37-bus distribution system

Table 6. 2 The value of parameters in the distribution network

Studied Feeder capacity before / Amount of Scenarios feeders after congestion (kW) affected users 1 Feeder #1 13.0/12.7 18 2 Feeder #2 5.8/5.5 10 3 Feeder #3 13.7/13.4 8 4 Feeder #4 8.2/7.9 6 5 Feeder #5 2.2/1.9 3

Fig.6.13 gives the optimization results when congestion happens in the distribution system. It can be found that expected electricity costs of end-users all increase due to the congestion. After rearranging the sequence of scenarios, it shows that when congestion happens on the feeder with a higher load, the bigger the expected cost increases. In term of CVaR, the result shows that there is no much difference when only a small amount of end-users are affected by the feeder congestion, namely scenario 3 4 and 5. However, with the further increasing of the amount of affected end-users namely scenario 1 and 2, the CVaR tends to decrease

150 because the freedom electricity consumption behaviors of end-users are constrained by the feeder limit. Therefore, these optimization results reveal the possible influence that may be brought by distribution network constraints. Using these analysis results, the retailer may consider developing various investment strategies to help upgrade the distribution system according to their own attitude towards the retail risk. Expected cost of end-users in the optimization results ($) 1.95 19.5 Scenario 3 1.80 Scenario 1 Scenario 4 18.0 1.65 Scenario 5 Scenario 2 1.50 16.5

1.35 15.0 1.20

1.05 13.5

0.90 Expected cost of end-users after rearranging CVaR in each scenario 12.0 0.75 Expected cost of end-users in each scenario Expected cost of end-users without congestion 10.5

Value of CVaR in the optimization results ($) results optimization the in CVaR of Value 0.60 CvaR without cogestion

0.45 9.0 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenarios Fig. 6.13 CVaR and expected cost of end-users in the optimization results when congestion happens in the distribution network

(2) Sensitivity Analysis against Risk Weighting Factor for the Retailer The weighting factor βfc in the proposed model quantify retailer’s attitude towards the decision risk. The higher value of weighting factor, the more risk averse the retailer. Therefore, with the increasing of weighting factor, the CVaR in retail decision would decrease, which complies with the simulation results in Fig. 6.14. To manage retail risks, the retailer usually needs to purchase electricity by forward contracts at fixed prices. As shown in Fig.6.6, the forward contract price is set to be a bit higher than the expected value of real- time electricity market price. Consequently, in Fig. 6.14 the expected electricity consumption cost of end-users increase gradually in the optimization results due to the change of purchasing strategies of retailers.

151

50 (The CVaR under scenario 1 is 1383.654.) Expected cost of end-users 40 CVaR of the retailer # of Scenarios Value of weighting fact 10E+00 30 21E-05 31E+00 41E+01 51E+02 20 61E+05

0 Expected cost of end-users and CVaR of the retailer($) the of CVaR and of end-users cost Expected

Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Scenarios Fig. 6.14 CVaR and the expected cost of end-users in the optimization results corresponding to different weighting factors

(3) Sensitivity Analysis against Parameters in the Price Elasticity Function of Demand

The demand elasticity function fj(rj,t) depicts the percentage change of residential load in response to the changes in price, namely when the price rises, end-users will reduce their electricity consumption. The price change in fj(rj,t) is measured by the relative change between the retail price and the nominal price at time t. Since fj(rj,t) is expected to be 1 when retail price equals to the nominal price, which means that residential load equals to the nominal load if there is no price change, the parameter β0,j is usually fixed to be 1. Besides, as the parameter β1,j depicts the sensitivity of demand to the price change, the bigger the absolute value of β1,j is, the more sensitive to the price signal end-users will be. Simulations are carried out for different values of β1,j, and the results are given in Fig.6.15 and Fig.6.16.

152

20 Expected costof end-usersin the results ($) 1.8 18

CVaR 16 1.5 Expected cost of end-users 14

1.2 12

10 0.9 Scenario # β0,j β1,j 8 11-0.1 0.6 6 21-0.15 31-0.2 4 0.3 41-0.24 51-0.27 2 0.0 0 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5

CVaR of the retailer in the optimization results ($) results optimization the in retailer the CVaR of Scenarios Fig. 6.15 CVaR and the expected cost of end-users in the optimization results under different scenarios

445

440 Expected value of end-users' electricity consumptioin -users -users in d 435

430

425 Scenario # β0,j β1,j 11-0.1 420 21-0.15 31-0.2

the distribution system(kWh) distribution the 41-0.24 415 51-0.27

Electricity consumption of all en all of consumption Electricity 410 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenarios Fig. 6.16 The quantity of electricity consumption in the distribution system under different scenarios

Fig.6.15 shows that the value change of the parameter β1,j has a negligible effect on the expected cost of end-users compared with its effect on the retail CVaR. With the increasing

153 of the absolute value of parameter β1,j , the CVaR tends to increase in Fig.6.15 and the expected quantity of end-users’ electricity consumption tends to increase as well, as shown in Fig.6.16. There are several reasons that may lead to the negligible effect on the expected cost. Firstly, it is the rate of return that is considered as the constraint in the proposed model instead of an absolute value of profit. Secondly, the objective function is to minimize end- users’ total electricity consumption payment while minimizing retailer’s profit risks.

Through the results it can be found that for end-users, to be more price sensitive can benefit them from consuming more electricity without increasing their electricity cost. Besides, considering the increased CVaR due to the increasing of end-users’ price sensitivity, retailers should pay attention to end-users with high price sensitivities in their risk management activities.

coef The sensitivity analysis against parameter βi-1 in the constructed coefficient ci,j is also carried

coef out. In the case study, all the βi-1 for each of the TOU price blocks is assumed to equal to βcoef and scenarios with different values of βcoef are studied. As discussed in Section III, the coefficient ci,j is constructed to model the effect of time shiftable load. Therefore, the bigger value of βcoef is, the more obvious the differences between TOU price blocks will be, which complies with results in Fig.6.17. Since the customized retail prices are different for each end-user in the optimization result, the average retail price of 31 end-users is used in Fig.6.17. And also, the CVaR of retailers and the expected electricity cost of end-users are not affected much by the changes of βcoef. Results in Fig.6.18 show that the CVaR stays at 1.2 and the expected of end-users stays at 17.5, respectively. There are only minor changes of CVaR and the expected cost of end-users when βcoef changes from 0.1 to 1.8. In summary, the parameter βcoef needs be properly chosen when adopting the proposed model, since the bigger difference between TOU price blocks can guide end-users’ load shifting activity better.

154

0.12 Average retail prices under scenario 1 Average retail prices under scenario 2 0.10 Average retail prices under scenario 3 Average retail prices under scenario 4 Average retail prices under scenario 5 0.08

0.06

coef Scenario # β 0.04 10.1 20.4 30.9 0.02 41.5 The average electricity retail price ($/kWh) price retail electricity average The 51.8

Price 1 Price 2 Price 3 # of price blocks in the TOU retail price Fig. 6.17 The average value of customized TOU retail prices under different scenarios

3.0 Costof end-usersin the optimizationresults ($) 22 2.7 CVaR Expected cost of end-users 20 2.4 18 2.1 16 1.8 14 1.5 12 1.2 10 coef 0.9 Scenario # β 8 0.6 10.1 20.4 6 0.3 30.9 4

CVaR in the optimization results ($) results optimization the in CVaR 0.0 41.5 51.8 2 0 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenarios Fig. 6.18 CVaR and the expected cost of end-users in the optimization results under different scenarios

155

6.5 Chapter Summary

The problem of customizing electricity retail prices for residential end-users is studied in this chapter. Firstly, data mining technologies are adopted to extract end-users’ load features from their historical load profiles. In order to explore the inherent electricity consumption patterns of end-users, the density-based spatial clustering is used for load profile analysis. And also, the statistical analysis of end-users’ historical consumption quantity is conducted to better capture their consumption regularity. After acquiring these load features, the model for customizing electricity retail prices is proposed. In the model, both the structure of TOU retail price and the price level are optimized once given the number of price block, which has never been realized in previous researches. The electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to test the proposed methods. The contribution of this chapter is twofold: (1) It proposes a method of customizing electricity retail plans combining with data mining technologies. In the proposed model, the structure and price level of TOU retail price are optimized simultaneously the first time. (2) Through customizing retail prices, electricity retail price is determined in more explanatory way. Besides, the customized retail plans can help maintain end-user’s electricity consumption at a higher level and help manage retail risk more efficiently.

Further research will concentrate on the extraction of more load features through data mining technologies as well as developing diverse electricity retail plans.

156

Chapter 7 Conclusions and Future Work

The fast development of Smart Grid and the emergence of the next generation of energy systems, namely Energy Internet, have been restructuring the power industry. In particular, with the wide installation of intelligent metering devices, big data about electricity end-users is collected. Meanwhile, communications technology and other advanced energy technology are being integrated into modern power systems. Additionally, due to the rising awareness of electricity consumers, there is a tendency for electricity end-users to switch between electricity suppliers. Electricity retailers are facing a more severe market competition. All these changes push the electricity suppliers to develop new electricity retail products rather than relying on their old business models. This research focused on developing innovative decision-making methods for electricity retailers with the help of data mining techniques.

In this PhD project, the author firstly conducted a comprehensive literature survey on the decision-making problem of electricity retailers. Publications on electricity retailing in the last two decades are surveyed and discussed in detail. The author has also studied the key business framework of electricity retailers. The typical business process of electricity retailers and its procedure of creating a new sales agreement are elaborated. To improve the inefficiency of existing load data mining methods, a new non-intrusive load monitoring method is proposed which is able to analyse the big load data in the Smart Grid environment. After obtaining the status of all identified appliances, a statistical residential load model is also developed. Using the proposed load modelling method, the appliance identification results can be conveniently used to customize electricity retailing strategies. As one of the main contributions, this research proposed the idea of using the results of residential appliance identification and end-user behaviour analysis to assist retail pricing. The problem of designing customized pricing strategies for different residential users is investigated based on the identification results of residential electric appliances and classifications of end-users according to their consumption behaviours. Then, a novel framework for customizing electricity retail prices is developed.

157

To customize retail prices through appliance identification, load data at least sampled at every minute is needed. Therefore, this research explores another data mining technique, clustering analysis, to customize electricity retail prices, where the half-hourly sampled electricity consumption data is adopted. A model of customizing electricity retail prices based on load profile clustering analysis is developed. Besides, the actual electricity usage data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms.

The studied problems, contributions, outcomes as well as main conclusions in each part of the finished work of this PhD project are given in table 7.1.

158

Table 7.1 Details of already finished work in this thesis

Chapter Studied problems Contributions Outcomes Conclusions

In the future electricity market, the retailer will see more opportunities, but also faces more technical and commercial challenges. There are several potential research directions related to electricity retailing. 1. Up to now, there is no novel forecasting method proposed for the long term residential 1. The work of last two decades load forecasting. With the emergence of have been investigated and smart grid technologies, there are increasing This comprehensive literature discussed. The topics include measurement data available from the low- review would help researchers and retailer load forecasting, voltage end-user side. The development of engineers understand the state-of- energy procurement strategies, behavior analysis based methods for long A comprehensive the-art of electricity retailer retail pricing schemes, and risk term residential load forecasting will be a Chapter 2 literature review for pricing. Besides, the discussion on management in the retail promising research direction. decision-making of the critical and open issues in the market. electricity retailers. 2. In existing publications, the structure of TOU field of electricity retailing can 2. Some critical and open issues pricing is often assumed to be pre-determined help guide the researchers who in the research field of in advance. The research work in the interested in these topics. electricity retailing are optimization of TOU price structure is still analyzed in detail. preliminary, and how to appropriately model and customize electricity retail prices in the smart grid environment remains an open question. 3. Through utilizing the complementation between different customer types, customized retail plans can further motivate responsiveness of end-users. Up to now, less

159

research has been reported on this topic, which could be thus considered as an open research direction in the next-generation energy market. 4. With the increasing number of flexible loads in the smart grid, such as the electrical vehicles, thermostatically controlled loads, air conditioning loads, and water-heating systems, coordination strategies and pricing schemes are needed to optimally leverage their benefits to the power systems. 5. Based on big data analysis, more sophisticated energy retail policies can be made through analyzing the energy usage data of the users. Endeavors are needed on developing data-driven retail pricing algorithms in the Smart Grid environment.

160

1. There are several different modes of retail companies in the electricity retail market. The first is the stand alone retail company. The This chapter studied the key second is the generator-owned retail business process of electricity company. The third is the vertically retailers and find that the key integrated retail company, where they have factors motivating the development both generation capacity and electricity retail of electricity retail market are: (1) business. The call for a lower electricity 2. Departments of the electricity retail company retail price. (2) End-user’s 1. It surveyed the development of can be categorized into front, middle and expectation for diversified service electricity retail markets in back offices. The front office mainly is products of electricity retail. (3) countries around the world. mainly responsible for transactions in Promoting the development of wholesale and retail markets through market alternative generation 2. The typical structure and work The business models analysis. The middle office is responsible for technologies. flow of electricity retailers are Chapter 3 of electricity retailers regularly assessing the risk exposure of investigated and presented. Meanwhile, during the process of are studied. electricity retail and conduct the internal developing a competitive 3. The relationship between the financial management for the company. electricity retail market, proper electricity wholesale market While the back office is responsible for market construction guidance, such and the retail market is running and maintaining the company’s as the guidance from Council of explored. intelligent financial system, customer service European Energy Regulators, system. CEER, and proper supervision of 3. Electricity retail and wholesale markets are the market performance, such as two distinct but closely linked markets. They the supervision from Australian have different participants. The wholesale Energy Regulator, AER, can help market is the upstream market of electricity keep the retail market construction retail market. on track.

161

The proposed algorithm is implemented with R and tested with a load dataset that contains 500 households’ profiles. The case study results show that: 1. The proposed appliance identification algorithm is efficient and faster. The proposed algorithm is accurate and fast. In terms of accuracy, it 2. Compared with the high-frequency sampling can identify single appliance load data, the decrease of sampling frequency will subsequence at accuracy above weaken characteristics of appliance’s power 93%. While for the multi-appliance 1. It proposed a novel appliance waveform. Consequently, the difficulty of subsequences which is much more A non-intrusive load identification algorithm for identification gradually increases, and then difficult to identify, it can also monitoring method is low frequency load data. the accuracy of appliance identification achieve accuracy at about 83%. Chapter 4 proposed to handle decreases. Moreover, the proposed statistical the bid data from 2. A statistical residential load 3. It is feasible to model residential load using residential model is feasible smart customers. model is developed based on through using appliance appliance identification. the proposed statistical model. The parameters in the model can be estimated identification results. It can be using nonparametric test on SPSS. used to develop targeted demand- side management strategies or

electricity retail plans.

162

Several conclusions are reached Simulation tests are carried out, including the based on the case study. benefits of implementing proposed techniques, Firstly, the proposed framework test results with different market sizes and can effectively customize retail numbers of participants, sensitivity analysis plans for classified end-users with against risk weighting factor for users, and distinct load patterns. sensitivity analysis against TOU price blocks for tested users. Secondly, considering the advantages of customized TOU The results show that: 1. It proposes the idea of using prices, the retailer has enough 1. The customized retail prices can always bring the results of residential motivations to adopt the proposed the retailer a higher profit than other TOU The residential appliance identification and techniques. pricing schemes, where the non-optimized appliance end-user behaviours analysis to TOU pricing has the lowest profit. Thirdly, when customizing retail identification and help retail pricing. prices, the retailer should try to end-user behavior 2. The result indicates that the profit of retailer Chapter 5 2. A mathematical model for classify end-users into the most analysis is proposed shows a constant growing with the increasing optimizing the TOU price distinct clusters. Thus, the retailer for customizing of market size and involved participants. structure is developed. can benefit significantly from the electricity retail 3. When there is an obvious temporal temporal complementarity of end- prices. 3. A retail pricing model based complementary between load patterns, the users. on end-user behaviour analysis advantage of customized TOU prices shows is proposed. Besides, subsidy policies for end- an apparent increase with the enlarging of users are needed when market size. implementing customized retail 4. The result shows that with the increasing of plans, since the benefit of end- weighting factor, the risk in retail decision users from the customized retail would decrease. Meanwhile, the retail power prices is related to their price supply from forward contracts increases. elasticity of demand. For end-users with less elastic loads, they would 5. With the increasing of price block number, need more incentives to retail profit of the retail also increases. participate.

163

Practical data collected by the Smart Grid, Smart City (SGSC) project in Australia is used to demonstrate the feasibility and efficiency of the developed models and algorithms. Case study results show that: 1. Compared with existing un-optimized TOU retail prices, the customized retail plans are different on the following aspects: Firstly, in terms of the price structure, the peak period is generally shorter while the off-peak and Through customizing retail prices, 1. It is the first research that shoulder periods are longer. Secondly, the the retailer can develop more proposed a retail pricing model The problem of relative price difference between peak and flexible pricing schemes. which optimizes the TOU customizing other periods are bigger. Thirdly, for end- Meanwhile, through customizing price structure and retail price electricity retail users that has the same load pattern but with retail prices, electricity retail price Chapter 6 plans simultaneously. prices through load different statistical characteristics of can be determined in a more profile clustering 2. It introduced the idea of electricity consumption quantity. Their explanatory way. More analysis is studied. categorized customers into the corresponding customized retail plans can be importantly, the customized retail electricity retail pricing different. plans can benefit both end-users problem. and electricity retailers. 2. The retail risk under customized retail plans is obviously lower than that of un-optimized retail plans. In other words, through optimizing TOU price structure, retail risks can be managed more efficiently. 3. The optimization results when congestion happens in the distribution system show that the expected electricity costs of end-users will all increase due to the congestion. 4. The weighting factor of risk in the proposed

164

model quantify retailer’s attitude towards the decision risk. The higher value of weighting factor, the more risk averse the retailer. The simulation results show that with the increasing of weighting factor, the risk in retail decision would decrease. 5. The case study results show that for end- users, to be more price sensitive can benefit them from consuming more electricity without increasing their electricity cost. Besides, considering the increased retail risk due to the increasing of end-users’ price sensitivity, retailers should pay attention to end-users with high price sensitivities in their risk management activities.

165

Future Work

In Chapter 2, the state-of-art of research on electricity retail is surveyed and discussed in detail. Notably, future research trends on long-term retailer load forecasting, structure optimization of electricity retail prices, customized electricity retail plans, management strategy for large-scale flexible loads, and data-driven retail pricing algorithm are elaborated respectively. All these open issues need further research and the author plans to continue doing research in directions.

As presented in Chapter 4, a new non-intrusive load monitoring method is proposed which is able to cope with the big load data collected in the Smart Grid environment. Experiments and comparisons to alternative methods have verified the effectiveness and efficiency of the proposed algorithm. For further study, how to improve the identification accuracy of composite load subsequence will be a key problem. Another direction of future research would be consumer’s behaviour analysis with the help of appliance identification results.

In Chapter 5, since the aim of the proposed retail pricing framework is to customize electricity retail plans using the results of appliance identification, the accuracy of appliance identification is essential. However, in the current proposed framework, we mainly consider the error caused by un-identified appliances, but not the error caused by wrongly identified appliances. If the error caused by inaccurate identification can be integrated in the model, it will help improve the quality of customized retail plans. Therefore, a new method is demanded for load modelling in order to incorporate the identification error, and it will be carried out in our future research work, as well as to conduct behavioural analysis of end- users.

In the research of Chapter 6, clustering analysis and statistical analysis are used to extract the typical load patterns and fluctuation features of end-user’s electricity consumption, respectively. It is these two features that are used for customizing retail prices. Our future work will focus on the extraction of more load features through data mining technologies as well as developing diverse electricity retail plans.

166

Reference

[1] A. J. Conejo, M. Carrión, and J. M. Morales, “Decision Making Under Uncertainty in Electricity Markets,” Springer, New York, 2010. [2] S. Li, , P. Wang, and L. Goel, “A Novel Wavelet-Based Ensemble Method for Short- Term Load Forecasting with Hybrid Neural Networks and Feature Selection,” IEEE Trans. Power Systems, vol.31, no.3, pp. 1788-1798, May 2016. [3] R. Zhang, Z. Y. Dong, Y. Xu, K. Meng, and K. P. Wong, “Short-term load forecasting of Australian National Electricity Market by an ensemble model of extreme learning machine,” IET Generation, Transmission & Distribution, vol.7, no.4, pp. 391-397, Apr. 2013. [4] C. Cecati, J. Kolbusz, P. Różycki, P. Siano, and B. M. Wilamowski, “A Novel RBF Training Algorithm for Short-Term Electric Load Forecasting and Comparative Studies,” IEEE Trans. Industrial Electronics, vol.62, no.10, pp. 6519-6529, Oct. 2015. [5] T. Hong, and P. Wang, “Fuzzy interaction regression for short term load forecasting,” Fuzzy Optimization and Decision Making, vol.13, no.1, pp.91-103, Sept. 2013. [6] Y. Goude, R. Nedellec, and N. Kong, “Local Short and Middle Term Electricity Load Forecasting with Semi-Parametric Additive Models,” IEEE Trans. Smart Grid, vol.5, no.1, pp. 440-446, Jan. 2014. [7] T. Hong, J. Wilson, and J. Xie, “Long term probabilistic load forecasting and normalization with hourly information,” IEEE Trans. Smart Grid, vol.5, no.1, pp. 456- 462, Jan. 2014. [8] F. L. Quilumba, W. J. Lee, H. Huang, D. Y. Wang, and R. L. Szabados, “Using Smart Meter Data to Improve the Accuracy of Intraday Load Forecasting Considering Customer Behavior Similarities,” IEEE Trans. Smart Grid, vol.6, no.2, pp. 911-918, Mar. 2015. [9] T. Hong, “Energy forecasting: Past, present, and future,” Foresight, no.32, pp.43-48, 2013. [10] R. J. Hyndman, and S. Fan, “Density forecasting for long-term peak electricity demand,” IEEE Trans. Power Systems, vol. 25, no. 2, pp. 1142-1153, May 2010. [11] M. S. Kandil, S. M. El-Debeiky, and N. E. Hasanien, “Long-term load forecasting for fast developing utility using a knowledge-based expert system,” IEEE Trans. Power Systems, vol. 17, no. 2, pp. 491-496, May 2002.

167

[12] J. Xie, T. Hong, and J. Stroud, “Long-Term Retail Energy Forecasting With Consideration of Residential Customer Attrition,” IEEE Trans. Smart Grid, vol. 6, no. 5, pp. 2245-2252, Sept. 2015. [13] S. Gabriel, M. F. Genc, and S. Balakrishnan, “A simulation approach to balancing annual risk and reward in retail electrical power markets,” IEEE Trans. Power Systems, vol. 17, no. 4, pp. 1050-1057, Nov. 2002. [14] D. J. Barger, “MGE experience with INSITE spatial load forecasting,” Proceedings of Power and Energy Society General Meeting, 2011 IEEE, Detroit, MI, USA, USA, Jul. 24-29, 2011. [15] T. Hong, S. M. Hsiang, and L. Xu, “Human-machine co-construct intelligence on horizon year load in long term spatial load forecasting,” Proceedings of Power & Energy Society General Meeting, 2009. PES '09. IEEE, Calgary, AB, Canada, Jul. 26- 30, 2009. [16] Australian Energy Market Operator (AEMO), “Forecasting Methodology Information Paper: 2015 National Electricity Forecasting Report (NEFR)”, Jul. 2015. http://www.aemo.com.au/-/media/Files/PDF/2015-NEFR-forecasting-methodology- information-paper.pdf [17] AleaSoft Energy Forecasting. https://aleasoft.com/energy-demand-forecasting/ [18] ABB Industrial energy load planning software. http://new.abb.com/cpm/energy-manager/industrial-energy-load-planning-forecasting- scheduling [19] EMD International A/S. Forecasting Solutions: LoadFor (Electricity Load Prediction). https://www.emd.dk/forecasting-solutions/loadfor/ [20] W. Wei, F. Liu, S. Mei, “Energy Pricing and Dispatch for Smart Grid Retailers Under Demand Response and Market Price Uncertainty,” IEEE Trans. Smart Grid, vol. 6, no. 3, pp. 1364-1374, May. 2015. [21] M. Hajati, H. Seifi, and M. K. Sheikh-El-Eslami, “Optimal retailer bidding in a DA market–a new method considering risk and demand elasticity,” Energy, vol. 36, no. 2, pp. 1332-1339, Feb. 2011. [22] R. Herranz, A. M. San Roque, J. Villar, and F. A. Campos, “Optimal demand-side bidding strategies in electricity spot markets,” IEEE Trans. power systems, vol. 27, no. 3, pp. 1204-1213, Aug, 2012. [23] M. Nazari, and A. A.Foroud, “Optimal strategy planning for a retailer considering medium and short-term decisions,” International journal of Electrical power & Energy systems, vol. 45, no. 1, pp. 107-116, Feb. 2013.

168

[24] A. Hatami, H. Seifi, and M. K. Sheikh-El-Eslami, “A stochastic-based decision-making framework for an electricity retailer: time-of-use pricing and electricity portfolio optimization,” IEEE Trans. Power Systems, vol. 26, no. 4, pp. 1808-1816, Nov.2011. [25] J. He, L. Cai, P. Cheng, and J. Fan, “Optimal Investment for Retail Company in Electricity Market,” IEEE Trans. Industrial Informatics, vol. 11, no. 5, pp. 1210-1219, Otc. 2015. [26] A. R. Hatami, H. Seifi, and M. K. Sheikh-El-Eslami, “Optimal selling price and energy procurement strategies for a retailer in an electricity market,” Electric Power Systems Research, vol. 79, no. 1, pp. 246-254, Jan. 2009. [27] J. Kettunen, A. Salo, and D. W. Bunn, “Optimization of electricity retailer's contract portfolio subject to risk preferences,” IEEE Trans. Power Systems, vol. 25, no. 1, pp. 117-128, Feb. 2010. [28] A. Ahmadi, M. Charwand, and J. Aghaei, “Risk-constrained optimal strategy for retailer forward contract portfolio,” International Journal of Electrical Power & Energy Systems, vol. 53, pp. 704-713, Dec. 2013. [29] M. Charwand, and Z. Moshavash, “Midterm decision-making framework for an electricity retailer based on Information Gap Decision Theory,” International Journal of Electrical Power & Energy Systems, vol. 63, pp. 185-195, Dec. 2014. [30] R. G. Karandikar, S. A. Khaparde, and S. V. Kulkarni, “Strategic evaluation of bilateral contract for electricity retailer in restructured power market,” International Journal of Electrical Power & Energy Systems, vol. 32, no. 5, pp. 457-463, Jun. 2010. [31] S. Gabriel, A. J. Conejo, M. A. Plazas, and S. Balakrishnan, “Optimal price and quantity determination for retail electric power contracts,” IEEE Trans. Power Systems, vol. 21, no. 1, pp. 180-187, Feb. 2006. [32] S. E. Fleten, and E. Pettersen, “Constructing bidding curves for a price-taking retailer in the Norwegian electricity market,” IEEE Trans. Power Systems, vol. 20, no. 2, pp. 701- 708, May 2005. [33] E. Celebi, and J. D. Fuller, “A model for efficient consumer pricing schemes in electricity markets,” IEEE Trans. Power Systems, vol. 22, no. 1, pp. 60-67, Feb. 2007. [34] M. Carrión, A. J. Conejo, and J. M. Arroyo, “Forward contracting and selling price determination for a retailer,” IEEE Trans. Power Systems, vol. 22, no. 4, pp. 2105-2114, Nov. 2007. [35] G. Dorini, P. Pinson, and H. Madsen, “Chance-constrained optimization of demand response to price signals,” IEEE Trans. Smart Grid, vol. 4, no. 4, pp. 2072-2080, Dec. 2013.

169

[36] O. Corradi, H. Ochsenfeld, H. Madsen, and P. Pinson, “Controlling electricity consumption by forecasting its response to varying prices,” IEEE Trans. Power Systems, vol. 28, no. 1, pp. 421-429, Feb. 2013. [37] R. Yu, W. Yang, and S. Rahardja, “A statistical demand-price model with its application in optimal real-time price,” IEEE Trans. Smart Grid, vol. 3, no. 4, pp. 1734- 1742, Dec. 2012. [38] Z. Xu, T. Deng, Z. Hu, Y. Song, and J. Wang, “Data-Driven Pricing Strategy for Demand-Side Resource Aggregators,” IEEE Trans. Smart Grid, vol.pp, no.99, pp. 1-1, Mar. 2016. [39] J. Saez-Gallego, J. M. Morales, M. Zugno, and H. Madsen, “A Data-Driven Bidding Model for a Cluster of Price-Responsive Consumers of Electricity,” IEEE Trans. Power Systems, vol.31, no.6, pp. 5001-5011, Nov. 2016. [40] R. K. Ahuja, and J. B. Orlin, “Inverse Optimization,” Operations Research, vol.49, no.5, pp.771-783, Jun. 2001. [41] P. Kotler, V. Wong, J. Saunders, and G. Armstrong, “Principles of Marketing (fourth European edition),” Prentice Hall, UK, 2005. http://s1.downloadmienphi.net/file/downloadfile8/148/1372857.pdf [42] A. Safdarian, M. Fotuhi-Firuzabad, and M. Lehtonen, “A Medium-Term Decision Model for DisCos: Forward Contracting and TOU Pricing,” IEEE Trans. Power Systems, vol.30, no.3, pp. 1143-1154, May 2015. [43] A. Safdarian, M. Fotuhi-Firuzabad, and M. Lehtonen, “Impacts of Time-Varying Electricity Rates on Forward Contract Scheduling of DisCos,” IEEE Trans. Power Delivery, vol.29, no.2, pp. 733-741, Apr. 2014. [44] F. Luo, Z. Y. Dong, Y. Chen, and J. Zhao, “Natural aggregation algorithm: a new efficient metaheuristic tool for power system optimizations,” Proceedings of Smart Grid Communications (SmartGridComm), 2016 IEEE International Conference on, Sydney, NSW, Australia, Nov. 6-9, 2016. [45] R. L. Haupt, and S. E. Haupt, “Practical Genetic Algorithms (the second edition),” Wiley, New York, NY, USA, 2004. [46] R. Storn and K. Price, “Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces,” Journal of Global Optimization, vol. 11, no. 4, pp 341–359, Dec.1997. [47] J. Kennedy ; and R. Eberhart, “Particle swarm optimization,” Proceedings of Neural Networks, 1995. Proceedings., IEEE International Conference on, Perth, WA, Australia, Nov.27 - Dec.1, 1995. [48] Dash optimization. “Dash's Xpress-MP Suite is the leading optimization software product for modeling and optimization.”

170

http://www.pse.ice.bas.bg/WWW_Systems_engineerig_laboratory/Links_systmeng/Das h%20Optimization%20-%20Leading%20Optimization%20Software.htm [49] GAMS. https://www.gams.com/ [50] I. Momber, S. Wogrin, and T. G. San Román, “Retail pricing: A bilevel program for PEV aggregator decisions using indirect load control,” IEEE Trans. Power Systems, vol. 31, no. 1, pp. 464-473, Jan. 2016. [51] A. Soroudi, and M. Ehsan, “IGDT based robust decision making tool for DNOs in load procurement under severe uncertainty,” IEEE Trans. Smart Grid, vol. 4, no. 2, pp. 886- 895, Jun. 2013. [52] S. Braithwait, D. Hansen, and M. O’Sheasy, “Retail electricity pricing and rate design in evolving markets,” Edison Electric Institute, pp. 1-57.Jul. 2007. [53] E. Celebi, and J. D. Fuller, “Time-of-use pricing in electricity markets under different market structures,” IEEE Trans. Power Systems, vol. 27, no. 3, pp. 1170-1181, Aug. 2012. [54] Y. Ding, S. Pineda, P. Nyeng, J. Østergaard, E. M. Larsen, and Q. Wu, “Real-time market concept architecture for EcoGrid EU-A prototype for European smart grids,” IEEE Trans. Smart Grid, vol. 4,no. 4, pp. 2006-2016, Dec. 2013 [55] T. Namerikawa, N. Okubo, R. Sato, Y. Okawa, and M. Ono, “Real-time pricing mechanism for electricity market with built-in incentive for participation,” IEEE Trans. Smart Grid, vol. 6, no. 6, pp. 2714-2724, Nov. 2015. [56] X. Liang, X. Li, R. Lu, X. Lin and X. Shen, “UDP: Usage-based dynamic pricing with privacy preservation for smart grid,” IEEE Trans. Smart Grid, vol. 4, no. 1, pp. 141- 150, Mar. 2013. [57] M. Roozbehani, M. A. Dahleh, and S. K. Mitter, “Volatility of power grids under real- time pricing,” IEEE Trans. Power Systems, vol. 27, no. 4, pp. 1926-1940, Nov. 2012. [58] S. A. Gabriel, A. J. Conejo, J. D. Fuller, B. F. Hobbs, and C. Ruiz, “Complementarity Modeling in Energy Market,” Springer, New York, 2013. [59] L. Jia, and L. Tong, “Dynamic Pricing and Distributed Energy Management for Demand Response,” IEEE Trans. Smart Grid, vol. 7, no. 2, pp. 1128-1136, Mar. 2016. [60] J. Shu, R. Guan, L. Wu, and B. Han, “A Bi-level Approach for Determining Optimal Dynamic Retail Electricity Pricing of Large Industrial Customers,” IEEE Trans. Smart Grid, vol. PP, no. 99, pp. 1-11, Jan. 2018. [61] J. Fortuny-Amat, and B. McCarl, “A representation and economic interpretation of a two-level programming problem,” The Journal of the Operational Research Society, vol.32, no.9, pp.783–792, Sept. 1981.

171

[62] S. Siddiqui, and S.A. Gabriel, “An SOS1-Based Approach for Solving MPECs with a Natural Gas Market Application,” Networks and Spatial Economics, vol.13, no.2, pp. 205-227, Jul. 2012. [63] N.Y. Soltani, S. J. Kim, and G. B. Giannakis, “Real-time load elasticity tracking and pricing for electric vehicle charging,” IEEE Trans. Smart Grid, vol. 6, no. 3, pp. 1303- 1313, May 2015.

[64] T. Hahn, Z. Tan, and W. Ko, “Design of Time-Varying Rate Considering CO2 Emission,” IEEE Trans. Smart Grid, vol. 4, no. 1, pp. 383-389, Mar. 2013. [65] C. Li, S. Tang, Y. Cao, Y. Xu, Y. Li, J. Li, and R. Zhang, “A new stepwise power tariff model and its application for residential consumers in regulated electricity markets,” IEEE Trans. Power Systems, vol. 28, no. 1, pp. 300-308, Feb. 2013. [66] R. García-Bertrand, “Sale prices setting tool for retailers,” IEEE Trans. Smart Grid, vol. 4, no. 4, pp. 2028-2035, Dec. 2013. [67] R. de Sá Ferreira, L.A. Barroso, P. R. Lino, M. M. Carvalho, and P. Valenzuela, “Time- of-use tariff design under uncertainty in price-elasticities of electricity demand: A stochastic optimization approach,” IEEE Trans. Smart Grid, vol. 4, no. 4, pp. 2285- 2295, Dec. 2013. [68] B. F. Hobbs, and U. Helman, “Complementarity-based equilibrium modeling for electric power markets" in Modeling Prices in Competitive Electricity Markets,” New York: Wiley, USA, 2004. [69] P. Yang, G. Tang, and A. Nehorai, “A game-theoretic approach for optimal time-of-use electricity pricing,” IEEE Trans. Power Systems, vol. 28, no. 2, pp. 884-892, May 2013. [70] R. Rashed Mohassel, A. Fung, F. Mohammadi, and K. Raahemifar, “A survey on Advanced Metering Infrastructure,” International Journal of Electrical Power & Energy Systems, vol. 63, pp. 473-484, Dec. 2014. [71] C Aswin Raj, E Aravind, B Ramya Sundaram, and K Vasudevan Shriram, “Smart Meter Based on Real Time Pricing,” Procedia Technology, pp. 120-124, vol. 21, Aug. 2015. [72] L. A. Martin, “Literature review for: Smart meters and real-time pricing in competitive retail electricity markets,” Jul. 16, 2013. http://cmd.org.au/wp- content/uploads/2015/08/Lit_Review_Smart_Meters_Leslie_Martin.pdf [73] H. Allcott, “The smart grid, entry, and imperfect competition in electricity markets,” National Bureau of Economic Research Working Paper 18071, Cambridge, May 2012. http://www.nber.org/papers/w18071 [74] European Commission. Country Report Finland 2017 Including an In-Depth Review on the prevention and correction of macroeconomic imbalances. Feb. 22, 2017.

172

https://ec.europa.eu/info/sites/info/files/2017-european-semester-country-report-finland- en.pdf [75] Energy Authority. National Report 2016 to the Agency for the Cooperation of Energy Regulators and to the European Commission. Jul. 8, 2016. https://www.energiavirasto.fi/documents/10191/0/National+Report+2016+Finland+151 8-601-2016.pdf/061a4522-d540-4870-a72c-80ce72a84b15 [76] Smart meters 101: France’s Linky electricity meters. Dec. 6, 2015. https://www.metering.com/features-analysis/smart-meters-101-frances-linky-electricity- meters/ [77] L. Bartelj, D. Paravan, A. F. Gubina, and R. Golob, “Valuating risk from sales contract offer maturity in electricity market,” International Journal of Electrical Power & Energy Systems, vol. 32, no. 2, pp. 147-155, Feb. 2010. [78] N. Mahmoudi-Kohan, M. P. Moghaddam, M. K. Sheikh-El-Eslami, “An annual framework for clustering-based pricing for an electricity retailer,” Electric Power Systems Research, vol. 80, no. 9, pp. 1042-1048, Sept. 2010. [79] M. J. Stevenson, L. F. Moreira do Amaral, M. Peat, “Risk management and the role of spot price predictions in the Australian retail electricity market,” Studies in Nonlinear Dynamics & Econometrics, vol. 10, no. 3, Article 4, Sept. 2006. [80] S. J. Deng, and S. S.Oren, “Electricity derivatives and risk management,” Energy, vol. 31, no. 6, pp. 940-953, Jun. 2006. [81] C. Wan, Z. Xu, Y. Wang, Z. Y. Dong, and K. P. Wong, “A Hybrid Approach for Probabilistic Forecasting of Electricity Price,” IEEE Trans. Smart Grid, vol.5, no.1, pp. 463-470, Jan. 2014. [82] X. Chen, Z. Y. Dong, K. Meng, Y. Xu, and K. P. Wong, and H. W. Ngan, “Electricity Price Forecasting With Extreme Learning Machine and Bootstrapping,” IEEE Trans. Power Systems, vol.27, no.4, pp. 2055-2062, Nov. 2012. [83] M. Rafiei, T. Niknam, and M. H. Khooban, “Probabilistic Forecasting of Hourly Electricity Price by Generalization of ELM for Usage in Improved Wavelet Neural Network,” IEEE Trans. Industrial Informatics, vol.pp, no.99, pp. 1-1, Jun. 2016. [84] C. Xiao, Z. Dong, Y. Xu, K. Meng, X. Zhou, and X. Zhang, “Rational and self-adaptive evolutionary extreme learning machine for electricity price forecast,” Memetic Computing, vol.8, no.3, pp. 223-233, Jun. 2016. [85] M. Hinterstocker, and S. Roon, “optimization of retail electricity pricing structures,” Proceedings of the International ETG Congress 2017, Nov. 28-29, 2017, Bonn, Germany.

173

[86] Y. Xu, and J. V. Milanović, “Artificial-Intelligence-Based Methodology for Load Disaggregation at Bulk Supply Point,” IEEE Trans. Power Systems, vol.30, no.2, Mar. 2015. [87] A. Reinhardt, P. Baumann, D. Burgstahler, M. Hollick, H. Chonov, M. Werner, and R. Steinmetz, “On the accuracy of appliance identification based on distributed load metering data,” Proceedings of the Second IFIP Conference on Sustainable Internet and ICT for Sustainability, Pisa, Italy, Oct. 4-5, 2012. [88] Y. X. Yu, B. Liu, and W. P. Luan, “Nonintrusive residential load monitoring and decomposition technology,” Southern Power System Technology, vol. 7, pp. 1-5, Aug. 2013. [89] M. Figueiredo, A. De Almeida, and B. Ribeiro, “Home electrical signal disaggregation for non-intrusive load monitoring (NILM) systems,” Neurocomputing, vol. 96, pp. 66- 73, Nov. 2012. [90] M. Dong, P. C. Meira, W. Xu, and C. Y. Chung, “Non-intrusive signature extraction for major residential loads,” IEEE Trans. on Smart Grid, vol.4, pp. 1421-1430, Sept. 2013. [91] D. Egarter, V. P. Bhuvana, and W. Elmenreich, “PALDi: Online load disaggregation via particle filtering,” IEEE Trans. on Instrumentation and Measurement, vol. 64, pp. 467-477, Feb. 2015. [92] S. B. Taieb, R. Huser, R. J. Hyndman, and M. G. Genton, “Forecasting Uncertainty in Electricity Smart Meter Data by Boosting Additive Quantile Regression,” IEEE Trans. Smart Grid, vol.7, no.5, pp. 2448-2455, Sept. 2016. [93] H. T. Nguyen, D.T. Nguyen, and L. B. Le, “Energy Management for Households with Solar Assisted Thermal Load Considering Renewable Energy and Price Uncertainty,” IEEE Trans. Smart Grid, vol.6, no.1, pp. 301-314, Jan. 2015. [94] S. Sun, M. Dong, and B. Liang, “Distributed Real-Time Power Balancing in Renewable-Integrated Power Grids With Storage and Flexible Loads,” IEEE Trans. Smart Grid, vol.7, no.5, pp. 2337-2349, Sept. 2016. [95] R. Couillet, S. M. Perlaza, H. Tembine, and M. Debbah, “Electrical Vehicles in the Smart Grid: A Mean Field Game Analysis,” IEEE Journal on Selected Areas in Communications, vol.30, no.6, pp. 1086-1096, Jul. 2012. [96] S. Li, W. Zhang, J. Lian, and K. Kalsi, “On reverse Stackelberg game and optimal mean field control for a large population of thermostatically controlled loads,” The 2016 American Control Conference, Boston, MA, USA, Jul. 6–8, 2016. [97] W. Su, A. Q. Huang, “A game theoretic framework for a next-generation retail electricity market with high penetration of distributed residential electricity suppliers,” Applied Energy, vol. 119, pp. 341-350, Apr. 2014.

174

[98] T. Hong, C. Chen, J. Huang, N. Lu, L. Xie, and H. Zareipour, “Guest Editorial Big Data Analytics for Grid Modernization,” IEEE Trans. Smart Grid, vol.7, no.5, pp. 2395-2396, Sept. 2016. [99] F. Luo, Z. Y. Dong, J. Zhao, X. Zhang, W. Kong, and Y. Chen, “Enabling the Big Data Analysis in the Smart Grid,” Proceedings of Power & Energy Society General Meeting, 2015 IEEE, Denver, CO, USA, Jul. 26-30, 2015. [100] F. Luo, Z. Y. Dong, Y. Chen, and Y. Xu, “Hybrid cloud computing platform: the next generation it backbone for smart grid,” Proceedings of the 2012 IEEE Power and Energy Society General Meeting, San Diego, California, Jul. 22-26, 2012. [101] F. Luo, J. Zhao, Z. Y. Dong, Y. Chen, Y. Xu, X. Zhang, and K. P. Wong, “Cloud- based information infrastructure for next-generation power grid: Conception, architecture, and applications,” IEEE Trans. Smart Grid, vol.7, no.4, pp. 1896 – 1912, Jul. 2016. [102] J. Dean, and S. Ghemawat, “Map Reduce: Simplified Data Processing on Large Clusters,” Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, vol.51, no.1, pp.107-113, San Francisco, California, Dec. 6- 8, 2004. [103] B. G. Kim, Y. Zhang, M. van der Schaar, and J. W. Lee, “Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning,” IEEE Trans. Smart Grid, vol.7, no.5, pp. 2187-2198, Sept. 2016. [104] D. He, W. Chen, L. Wang, and T. Y. Liu, “A Game-theoretic Machine Learning Approach for Revenue Maximization in Sponsored Search,” Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, Beijing, China, Aug.3-9, 2013. [105] X. Zhang, J. Li, H. Fu. UK retail electricity market reform and challenges. Automation of Electric Power Systems, vol. 40, no. 11, pp. 10-16, Oct. 2016. [106] D. Zhu, B. Chen, C. Gang. Institutional change of the electricity wholesale and retail markets in Singapore and its enlightenment. Price：Theory and Practice, no.10, pp. 56-58, 2014. [107] M. Zeng, J. Zhou, Y. Yu, et al. Experience from the reform of foreign electric power: a good reference for the funding of retail electricity market. Reformation & Strategy, vol. 25, no.4, pp. 179-182, 2009. [108] Y. Zhao, E. Yu. Study on retail electricity market (Part 1): competitive retail electricity market. Automation of Electric Power Systems, vol. 27, no.9, pp. 20-23, May 2003.

175

[109] M. J. Morey, L. D. Kirsch. Retail choice in electricity: what have we learned in 20 years? Electricity Markets Research Foundation, Feb. 11, 2016. https://www.hks.harvard.edu/hepg/Papers/2016/Retail%20Choice%20in%20Electricity %20for%20EMRF%20Final.pdf [110] Retail Market Review Committee. Power for the people. Report and recommendations for the Minister of Energy, Government of Alberta, Canada, Sept. 2012. http://www.energy.alberta.ca/Electricity/pdfs/RMRCreport.pdf [111] Council of European Energy Regulators. Electricity and gas retail market design with a focus on supplier switching and billing: guidelines of good practice, Jan. 24, 2012. https://www.ceer.eu/documents/104400/-/-/a13bb49a-f875-1bd0-785b-470d3291ef87 [112] ECME Consortium. The functioning of retail electricity markets for consumers in the European Union-final report. Nov. 2010. http://ec.europa.eu/consumers/archive/consumer_research/market_studies/docs/retail_el ectricity_full_study_en.pdf [113] Agency for the Cooperation of Energy Regulators. Ranking the competitiveness of retail electricity and gas markets at the national level in member states of the European Union and in Norway. Sept. 4, 2015. http://www.acer.europa.eu/en/electricity/market%20monitoring/documents_public/ipa %20final%20report.pdf [114] LIU Dongsheng. Australia electricity markets and retail companies [Online]. Energy Observer. [2017-01-19]. http://www.eothinker.com/eo/show.php?itemid=120 [115] ZHAO Junhua. How does the electricity retailer make money in Australia? Energy Observer [Online]. [2017-01-19]. http://www.eothinker.com/eo/show.php?itemid=40 [116] E. Lehto. Electricity prices in the Finnish retail market. Energy Policy, vol. 39, no.4, pp. 2179-2192, 2011. [117] Energy Authority. National Report 2015 to the Agency for the Cooperation of Energy Regulators and to the European Commission. Oct. 7, 2015. https://www.energiavirasto.fi/documents/10179/0/National+Report+2015+Finland+184 2-601-2015+20150710.pdf/7ebae177-6e7b-4ac0-b1ea-95628ca50742 [118] Fingrid. Electricity retail market business processes in Datahub. Apr. 29, 2016. http://www.fingrid.fi/fi/asiakkaat/asiakasliitteet/Datahub/ENG/Electricity%20retail%20 market%20business%20processes%20in%20Datahub%20version%201.0.pdf

176

[119] Q. Jiang, Y. Wang, H. Lin. Analysis on Relationship Between Electricity Wholesale Market and Retail Market Based on System Dynamics Method. Automation of Electric Power Systems, vol. 40, no. 24, pp. 77-84, Dec. 2016. [120] J. Liang, S. K. K. Ng, G. Kendall and J. W. Cheng, “Load signature study—Part I: Basic concept, structure, and methodology,” IEEE Trans. on Power Delivery, vol. 25, pp. 551-560, Apr. 2010. [121] F. Paradiso, F. Paganelli, A. Luchetta, D. Giuli and P. Castrogiovanni, “ANN-based appliance recognition from low-frequency energy monitoring data,” in Proc. 2013 IEEE 14th International Symposium and Workshops on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), June 4-7, 2013, Madrid, Spain, pp. 1-6. [122] M. Mittelsdorf, A. Hüwel, T. Klingenberg and M. Sonnenschein, “Submeter based Training of Multi-class Support Vector Machines for Appliance Recognition in Home Electricity Consumption Data,” in Proc. the 2nd International Conference on Smart Grids and Green IT Systems, May 9-10, 2013, Aachen, Germany, pp. 151-158. [123] A. G. Ruzzelli, C. Nicolas, A. Schoofs and G. M. O'Hare, “Real-time recognition and profiling of appliances through a single electricity sensor,” In Proc. 2010 7th Annual IEEE Communications Society Conference on Sensor Mesh and Ad Hoc Communications and Networks (SECON), June 21-25, 2010, Boston, MA, pp. 1-9. [124] F. Paganelli, F. Paradiso, S. Turchi, A. Luchetta, P. Castrogiovanni and D. Giuli, “Appliance Recognition in an OSGi-Based Home Energy Management Gateway,” International Journal of Distributed Sensor Networks, vol. 2015, 12 pages, Feb. 2015. [125] C. Belley, S. Gaboury, B. Bouchard and A. Bouzouane, “An efficient and inexpensive method for activity recognition within a smart home based on load signatures of appliances,” Pervasive and Mobile Computing, vol. 12, pp. 58-78, Jun. 2014. [126] F. Mcloughlin, A. Duffy, and M. Conlon. “Characterising domestic electricity consumption patterns by dwelling and occupant socio-economic variables: An Irish case study,” Energy and Buildings, vol. 48, pp. 240-248, May 2012. [127] D. J. Berndt and J. Clifford, “Using Dynamic Time Warping to Find Patterns in Time Series,” KDD workshop, vol. 10, pp. 359-370, Jul. 1994. [128] E. Keogh and C. A. Ratanamahatana, “Exact indexing of dynamic time warping,” Knowledge and information systems, vol. 7, pp. 358-386, Mar. 2005. [129] The R Project for Statistical Computing. https://www.r-project.org/. [130] T. Giorgino, “Computing and visualizing dynamic time warping alignments in R: the dtw package,” Journal of statistical Software, vol. 31, pp. 1-24, 2009.

177

[131] G. W. Corder, D. I. Foreman. Nonparametric statistics for non-statisticians: a step- by-step approach. John Wiley & Sons, 2009. [132] Gregory W. Corder and Dale I. Foreman, “Nonparametric Statistics for Non- Statisticians: A Step-by-Step Approach. Chapter 8-Tests for Nominal Scale Data: Chi- Square and Fisher Exact Test”, John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 2009. [133] IBM. Nonparametric Tests: Binomial Test. https://www.ibm.com/support/knowledgecenter/en/SSLVMB_24.0.0/spss/base/idh_ntbi .html [134] Gregory W. Corder and Dale I. Foreman, “Nonparametric Statistics for Non- Statisticians: A Step-by-Step Approach. Chapter 2-Testing Data for Normality”, John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 2009. [135] Gregory W. Corder and Dale I. Foreman, “Nonparametric Statistics for Non- Statisticians: A Step-by-Step Approach. Chapter 3-Comparing Two Related Samples: The Wilcoxon Signed Ranks Test”, John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 2009. [136] Gregory W. Corder and Dale I. Foreman, “Nonparametric Statistics for Non- Statisticians: A Step-by-Step Approach. Chapter 9-Test For Randomness: The Runs Test”, John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 2009. [137] P. Pikk and M. Viiding, “The dangers of marginal cost based electricity pricing,” Baltic Journal of Economics, vol. 13, no. 1, pp. 49-62, Jun. 2014. [138] W. Kong, Z. Y. Dong, G. Chen, Y. W. Jia, “A rule based domestic load profile generator for future smart grid,” in Proc. the 2014 IEEE Australasian Universities Power Engineering Conf. (AUPEC), Sept. 28-Oct. 1, 2014, Perth, WA, pp. 1-5. [139] S. Braithwait, D. Hansen, and M. O’Sheasy, “Retail electricity pricing and rate design in evolving markets,” Washington, D.C.: Edison Electric Institute, 2007. [140] J. Yang, J. Zhao, F. Luo, F. Wen, and Z.Y. Dong, “Decision-Making for Electricity Retailers: A Brief Survey,” IEEE Trans. Smart Grid, vol. PP, pp. 1-14, Jan. 2017. [141] G. Chicco, O. M. Ionel, and R. Porumb, “Electrical load pattern grouping based on centroid model with ant colony clustering,” IEEE Trans. Power Systems, vol. 28, pp. 1706-1715, May 2013. [142] G. Chicco, R. Napoli, F. Piglione, P. Postolache, M. Scutariu, and C. Toader, “Load pattern-based classification of electricity customers,” IEEE Trans. Power Systems, vol. 19, pp. 1232-1239, May 2004. [143] G. Chicco and I. S. Ilie, “Support vector clustering of electrical load pattern data,” IEEE Trans. Power Systems, vol. 24, pp. 1619-1628, Aug. 2009.

178

[144] G. J. Tsekouras, N. D. Hatziargyriou, and E. N. Dialynas, “Two-stage pattern recognition of load curves for classification of electricity customers,” IEEE Trans. Power Systems, vol. 22, pp. 1120-1128, Aug. 2007. [145] G. Chicco, R. Napoli, and F. Piglione, “Comparisons among clustering techniques for electricity customer classification,” IEEE Trans. Power Systems, vol. 21, pp. 933- 940, May 2006. [146] A. Albert and R.Rajagopal, “Cost-of-service segmentation of energy consumers,” IEEE Trans. Power Systems, vol. 29, pp. 2795-2803, Nov. 2014. [147] D. F. Rogers and G. G. Polak, “Optimal clustering of time periods for electricity demand-side management,” IEEE Trans. Power Systems, vol. 28, pp. 3842-3851, Nov. 2013. [148] S. Zhong and K. Tam, “A frequency domain approach to characterize and analyze load profiles,” IEEE Trans. Power Systems, vol. 27, pp. 857-865, May 2012. [149] D. M. Hanssens, L. J. Parsons, and R. L. Schultz. “Market response models: Econometric and time series analysis,” New York, USA: Springer, 2003. [150] J. J. Yang, J. Zhao, F. Wen, Z.Y. Dong, and Y.S. Xue, “Residential Appliance Identification and Load Modelling Based on Big Data Mining in Smart Grid Environment,” Electric Power Construction, vol. 37, no.12, pp. 11-23, Dec. 2016. [151] Y. Zhang, W. Chen, R. Xu, and J. Black, “A Cluster-Based Method for Calculating Baselines for Residential Loads,” IEEE Trans. Smart Grid, vol. 7, no. 5, pp. 2368-2377, Sept. 2016. [152] M. Koivisto, P. Heine, I. Mellin, and M. Lehtonen, “Clustering of Connection Points and Load Modeling in Distribution Systems,” IEEE Trans. Power Systems, vol. 28, no. 2, pp. 1255-1265, May 2013. [153] J. Kwac, J. Flora, and R. Rajagopal, “Household Energy Consumption Segmentation Using Hourly Data,” IEEE Trans. on Smart Grid, vol. 5, no. 1, pp. 420-430, Jan. 2014. [154] R. Li, C. Gu, F. Li, G. Shaddick, and M. Dale, “Development of Low Voltage Network Templates-Part I: Substation Clustering and Classification,” IEEE Trans. Power Systems, vol. 30, no. 6, pp. 3036-3044, Nov. 2015. [155] S. Haben, C. Singleton, and P. Grindrod, “Analysis and Clustering of Residential Customers Energy Behavioral Demand Using Smart Meter Data,” IEEE Trans. Smart Grid, vol. 7, no. 1, pp. 136-144, Jan. 2016. [156] T. Zhang, G. Zhang, J. Lu, X. Feng, and W. Yang, “A New Index and Classification Approach for Load Pattern Analysis of Large Electricity Customers,” IEEE Trans. Power Systems, vol. 27, no. 1, pp. 153-160, Feb. 2012.

179

[157] K. Mets, F. Depuydt, and C. Develder, “Two-Stage Load Pattern Clustering Using Fast Wavelet Transformation,” IEEE Trans. Smart Grid, vol. 7, no. 5, pp. 2250-2259, Sept. 2016. [158] M. Piao, H. S. Shon, J. Y. Lee, and K. H. Ryu, “Subspace projection method based clustering analysis in load profiling,” IEEE Trans. Power Syst., vol. 29, no. 6, pp. 2628– 2635, Nov. 2014. [159] Y. Wang, Q. Chen, C. Kang, and Q. Xia, “Clustering of Electricity Consumption Behavior Dynamics Toward Big Data Applications,” IEEE Trans. Smart Grid, vol. 7, no. 5, pp. 2437-2447, Sept. 2016. [160] R. Granell, C. J. Axon, and D. C. H. Wallom, “Impacts of Raw Data Temporal Resolution Using Selected Clustering Methods on Residential Electricity Load Profiles,” IEEE Trans. Power Systems, vol. 30, no. 6, pp. 3217-3224, Nov. 2015. [161] K. Zhou, S. Yang, and C. Shen, “A review of electric load classification in smart grid environment,” Renewable and Sustainable Energy Reviews, vol. 24, pp. 103-110, Aug. 2013. [162] G. Chicco, “Overview and performance assessment of the clustering methods for electrical load pattern grouping,” Energy, vol.42, no.1, pp. 68-80, Jun. 2012. [163] Australian Government Department of the Environment and Energy, “Smart Grid, Smart City,” Jul. 2014. [Online].Available:http://www.environment.gov.au/energy/programs/smartgridsmartcit y. [Accessed: 6- Dec- 2017]. [164] “Distribution test feeders,” IEEE Power Energy Society [Online]. Available: http://ewh.ieee.org/soc/pes/dsacom/testfeeders/index.html

180