Developing and Coordinating Autonomous Agents for Efficient Electricity Markets by Andrew Raymond Trueman Perrault A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Department of Computer Science University of Toronto c Copyright 2018 by Andrew Raymond Trueman Perrault Abstract Developing and Coordinating Autonomous Agents for Efficient Electricity Markets Andrew Raymond Trueman Perrault Doctor of Philosophy Department of Computer Science University of Toronto 2018 Whether for environmental, conservation, efficiency, or economic reasons, developing next generation electric power infrastructure is critical. Temporally relevant, granular data from smart meters provide new opportunities for data-driven management of the power grid. New developments|for example, electricity markets with multiple suppliers, the integration of renewable power sources into the system, and spikier demand patterns due to, say, electric vehicles|create new challenges for efficient grid op- eration. Computer science is uniquely positioned to assist with increasingly sophisticated techniques for handling and learning from large amounts of data. The methods of game theory and multi-agent systems provide a natural framework for modeling the competing incentives of electricity market par- ticipants. This thesis focuses on the use of learning, optimization, mechanism design, and preference elicitation methods to coordinate electricity demand and supply while respecting the incentives of market participants. Specifically, we propose an approach where an autonomous agent acts on behalf of each household, coordinating with inhabitants to relay information and make decisions on their behalf about electricity consumption. We focus on three problems that arise in developing such agents: (i) how to coordinate consumers' electricity use, (ii) how to share the costs of consumption among households (via their agents), and (iii) how to gather consumption preference data from consumers. Chapters 3 and 4 focus on different aspects of the first two problems. Both use a matching markets approach. In Chapter 3, we focus on the impact of demand smoothness and peaks on the supplier's cost, and in Chapter 4, on the impact of predictability. In both chapters, we develop new cost sharing schemes that are resilient to certain forms of strategic behavior on the part of the agents and that achieve strong performance in experiments. Chapter 5 studies the third problem. Motivated by control of heating and cooling systems, we present a new approach to preference elicitation, where the cost and accuracy of query responses is dependent on the user's familiarity with the conditions specified in the query. We show that despite the theoretical difficulty in this setting, we can build solvers that perform well in practice. ii Acknowledgements Thank you to my supervisor, Craig Boutilier, for inspiring my research and guiding an abstract vision into something concrete. To my committee, for their invaluable \on the ground" support at tough moments, as well as their outside perspective. To my parents, for endless hours of reading and editing. To my colleagues, collaborators and friends, especially Joanna Drummond, Elizabeth Greville, Marek Janicki, Aleksandr Kazachkov, Omer Lev, Nisarg Shah, Jake Snell, Tyrone Strangway, and Rory Tulk. iii Contents Acknowledgements iii Contents iv 1 Introduction 1 1.1 The Role of Artificial Autonomous Agents . 3 1.2 Experiential Elicitation . 3 1.3 Why Study Electricity in AI? . 4 1.4 Outline . 6 2 Background 8 2.1 Game Theory . 8 2.1.1 Non-Cooperative Game Theory . 8 2.1.2 Cooperative Game Theory . 11 2.1.3 Core Allocations . 12 2.1.4 The Shapley Value . 14 2.1.5 Convex Cooperative Games . 15 2.1.6 The Duality Between Cooperative Games and Markets . 16 2.2 Market Design . 17 2.2.1 Market Design in Economics . 18 2.2.2 Market Design in Computer Science . 19 2.3 Eliciting Preferences of Market Participants . 21 2.3.1 Mechanism Design . 21 2.3.2 The Vickrey-Clarke-Groves Mechanism . 24 2.3.3 Preference Elicitation and Query Types . 26 2.3.4 Query Strategies in Preference Elicitation . 28 2.4 The Smart Grid and the Role of Computer Science . 32 2.4.1 The Path of the Smart Grid . 33 2.4.2 The Technology Path to Deep Greenhouse Gas Emissions Cuts . 34 2.4.3 Artificial Intelligence and the Smart Grid . 36 3 Approximately Stable Pricing for Coordinated Purchasing of Electricity 37 3.1 Introduction . 37 3.2 Setting . 40 iv 3.3 Producer Price Functions (PPFs) . 43 3.3.1 Mixed Integer Program Encoding . 47 3.4 Cost Sharing and Stability Concepts . 50 3.4.1 Cost Sharing under the Marginal-Cost Defection Model . 50 3.4.2 Shapley-Like Payments . 53 3.4.3 Similarity-Based Envy-Free Payments . 54 3.5 Model of Consumer Demand . 56 3.6 Experiments . 58 3.6.1 Shapley-Like Payments . 58 3.6.2 Similarity-Based Envy-Free Payments . 60 3.7 Conclusion and Future Work . 61 4 Multiple-Profile Prediction-of-Use Games 64 4.1 Introduction . 64 4.2 Background . 66 4.2.1 Prediction-of-Use Games . 66 4.3 Multiple-Profile POU Games . 68 4.4 Properties of MPOU Games . 69 4.5 Incentives in MPOU Games . 71 4.6 Manipulation in MPOU Games . 74 4.7 Learning Utility Models . 75 4.8 Experiments . 78 4.8.1 Experimental Setup . 78 4.8.2 Results . 79 4.9 Conclusion and Future Work . 81 5 Experiential Preference Elicitation for Autonomous HVAC Systems 85 5.1 Introduction . 85 5.2 Background . 89 5.3 A Model of Experiential Elicitation . 91 5.4 EE with Relative Value Queries . 96 5.5 Query Response and Cost Model . 100 5.6 Experiments . 101 5.7 Conclusion and Future Work . 106 6 Conclusions and Future Work 109 6.1 Summary . 109 6.2 Future Work . 111 6.2.1 Integrating Chapters 3 and 4 . 111 6.2.2 Rationality of Players . 112 6.2.3 Incentives to Misreport . 113 Bibliography 116 v Chapter 1 Introduction Electricity markets and electricity distribution are the source of many intriguing technical problems for artificial intelligence (AI). Electricity is notable from an economic perspective because it fails to behave like an ideal commodity [Kirschen and Strbac, 2004]. It is delivered through a physical network governed by the laws of physics: the electrical grid. Demand and supply must be balanced continuously to keep the grid functioning (load balancing)|failure results in a long and expensive recovery process. Most load balancing is done outside of markets because of the speed of reaction required, on the order of seconds. Because supply is pooled through the grid, individuals cannot buy from a particular supplier directly. Storage is also very expensive. These elements combine to make electricity markets challenging to operate efficiently, in the economic sense. Our research and intuitions about how free markets lead to high-quality outcomes are difficult to apply in a setting that is sufficiently non-standard. The Nobel laureate Alvin Roth has written extensively on the challenge of defining good rules for in- teractions in markets [Roth, 2002]. He argues that no market operates completely removed from human design or conventions, and that it should be part of the role of economists in society to adjust the rules of markets to optimize outcomes. This area of study is called market design or design economics. While designing these rules is highly dependent on economic theory, there are often key computational chal- lenges in the operation of the market itself. Designed markets frequently do away with the spontaneity of organic ones where market participants choose their own partner with whom to interact. Instead, partic- ipants submit \bids" (which may be non-monetary) to a central authority, which \assigns" transactions to the market participants by running an algorithm. Such markets are referred to as matching markets, since they often match buyers to sellers to maximize economic efficiency or some related objective. More broadly, they coordinate the actions of the market participants. Electricity markets in particular are intensely designed. This design is driven by technical constraints, such as limitations in monitoring technology. For example, with traditional electricity meters, it is impossible to have a pricing scheme that depends on the time of consumption, as these meters do not record when electricity use occurs, only total usage. These markets are an interesting object of study, and Chapters 3 and 4 explore the problem of improving their efficiency from both the economic and computational perspectives. As we propose more complex market rules that increase efficiency, we have to be careful to not disrupt the desirable properties of existing, simple markets. Here two key issues arise. First, the more information an agent reports to the market, the more opportunity the agent has to manipulate the 1 Chapter 1. Introduction 2 outcome of the market by reporting information falsely, which we call misreporting. Second, what is good for the group of agents is not necessarily good for each individual agent|a rational, self-interested agent will take advantage of opportunities to improve its own outcome, potentially by making side deals with other agents. These actions may in turn reduce the welfare of the group. Economists call this the problem of stability. A stable allocation is one where no agent can benefit by making side deals or taking actions other than those prescribed by the market. Both of these issues are well understood in the abstract, but often require special treatment in any specific market. In Chapter 3, we apply a matching markets perspective to electricity exchange. Our use of matching markets is motivated by their application to supply chains (e.g., Chen and Roma [2010]), where they are used to increase economic efficiency while respecting global constraints on outcomes. Electricity gener- ator cost functions often have a complex structure, including features such as minimum and maximum production levels, multiple generation sources with varying costs, and ramp constraints that constrain production adjustments over time.