Energy Trust of Oregon Smart Thermostat Pilot Evaluation Prepared for Energy Trust of Oregon Prepared by Apex Analytics, LLC 1525 Spruce Street, Suite 200 Boulder, CO 80302 (303)590-9888 www.apexanalyticsllc.com March 1, 2016 Acknowledgements The Evaluation Team would like to acknowledge the review, support, and advice given by two third- party quality assurance reviewers, Scott Pigg (Seventh Wave) and Ken Agnew (DNV-GL). APEX ANALYTICS, LLC Energy Trust of Oregon Smart Thermostat Pilot Evaluation Table of Contents 1. Executive Summary ............................................................................................................................ 1-1 2. Introduction ....................................................................................................................................... 2-1 2.1 Evaluation Goals and Objectives ............................................................................................ 2-2 3. Background ........................................................................................................................................ 3-1 3.1 Participant Selection and Recruitment .................................................................................. 3-4 3.1.1 Participation Requirements ............................................................................................ 3-4 3.1.2 Recruitment .................................................................................................................... 3-7 3.1.3 Application ...................................................................................................................... 3-8 3.1.4 Quality Assurance ........................................................................................................... 3-8 4. Evaluation Methodology .................................................................................................................... 4-9 4.1 Staff Interviews ....................................................................................................................... 4-9 4.2 Participant Surveys ................................................................................................................. 4-9 4.3 Billing Analysis ...................................................................................................................... 4-10 4.3.1 Data Sources ................................................................................................................. 4-10 4.3.2 Gas Utility Data ............................................................................................................. 4-11 4.3.3 Study Groups ................................................................................................................ 4-11 4.3.4 Attrition 4-13 4.3.5 Billing Analysis Methodology ........................................................................................ 4-14 5. Findings .............................................................................................................................................. 5-1 5.1 Recruitment, Participation, and Implementation .................................................................. 5-1 5.1.1 Characteristics of Participants and their Homes ............................................................ 5-6 5.2 Participant Installation ........................................................................................................... 5-8 5.3 User Experience .................................................................................................................... 5-12 5.3.1 Features, Settings, and Participant Usage .................................................................... 5-12 5.3.2 Participant Behavior ..................................................................................................... 5-14 5.3.3 Customer Satisfaction ................................................................................................... 5-16 5.3.4 Comfort of Participant Homes ...................................................................................... 5-20 5.3.5 User experience related to implementation staff QA site visits .................................. 5-23 5.3.6 Participant commitment to energy savings.................................................................. 5-24 5.4 Energy Savings ...................................................................................................................... 5-27 5.4.1 Sample Characteristics ................................................................................................. 5-28 5.4.2 Energy Savings .............................................................................................................. 5-33 5.4.3 Sensitivity Analysis ........................................................................................................ 5-38 5.4.4 Subgroup Analysis......................................................................................................... 5-39 6. Conclusions and Recommendations .................................................................................................. 6-1 7. Appendices ......................................................................................................................................... 7-1 i Table of Contents Energy Trust of Oregon Smart Thermostat Pilot Evaluation List of Figures Figure 1. Satisfaction rating with smart thermostat 1-4 Figure 2. Nest and Lyric thermostats 3-1 Figure 3. Smart Thermostat Pilot recruitment process 3-6 Figure 4. Two-stage Sample Randomization Design 3-8 Figure 5. Pilot Two-Stage Randomization Results 4-13 Figure 6. Smart thermostat installation duration 5-8 Figure 7. Ease of smart thermostat installation 5-9 Figure 8. Satisfaction related to installation of the smart thermostat 5-10 Figure 9. Participants that experienced installation issues 5-11 Figure 10. Source of support for participants that experienced installation issues 5-11 Figure 11. Percentage of survey respondents finding specific features somewhat or very useful 5-13 Figure 12. Participants enabling occupancy-based settings (Auto-Away/Nest; Geofencing/Lyric) 5-14 Figure 13. Frequency of adjusting smart thermostat settings or using thermostat features 5-15 Figure 14. Additional non-installation issues with smart thermostat 5-16 Figure 15. Percent of all participants experiencing specific non-installation issues 5-17 Figure 16. Source of support for participants with smart thermostat non-installation issues 5-17 Figure 17. Satisfaction rating with smart thermostat 5-18 Figure 18. Satisfaction rating of participation in Smart Thermostat Pilot 5-19 Figure 19. Likelihood to recommend smart thermostat 5-19 Figure 20. If not involved in the Pilot, would you have returned the smart thermostat by now? 5-20 Figure 21. Comfort of home temperature compared to pre-smart thermostat period 5-21 Figure 22. Favorite aspect of the smart thermostats 5-22 Figure 23. Ease of smart thermostat operation – schedule and temperature adjustment 5-22 Figure 24. Ease of smart thermostat operation – overall user interface 5-23 Figure 25. Reasons for Smart Thermostat Pilot participation 5-25 Figure 26. Energy savings expectations 5-26 Figure 27. Does the $250 smart thermostat price tag make sense? 5-27 Figure 28. Distribution of pre-Pilot mean annual gas usage in therms by study group, for each treatment and comparison group pair 5-29 Figure 29. Box plots of monthly distribution of gas usage by study group, for each treatment and comparison group pair, January-April 2014 and 2015 5-32 Figure 30. Estimated pre- and post-installation mean annual gas usage for Nest recipients and randomized comparison homes 5-34 Figure 31. Estimated pre- and post-installation mean annual gas usage for Lyric recipients and randomized comparison homes 5-34 ii Table of Contents Energy Trust of Oregon Smart Thermostat Pilot Evaluation Figure 32. Estimated pre- and post-installation mean annual gas usage for Nest intention-to-treat and randomized comparison homes 5-36 Figure 33. Estimated pre- and post-installation mean annual gas usage for Lyric intention-to-treat and randomized comparison homes 5-36 Figure 34. Estimated pre- and post-installation mean annual gas usage for Nest recipients and matched comparison homes 5-37 Figure 35. Estimated pre- and post-installation mean annual gas usage for Lyric recipients and matched comparison homes 5-38 Figure 36. Comparison of average annual gas savings between the final participant sample, employees and contractors removed, homes with thermostats reportedly still installed, and homes with no major changes reported 5-41 Figure 37. Comparison of average annual gas savings by pre-Pilot annual gas use 5-42 Figure 38. Comparison of average annual gas savings by number of occupants 5-43 Figure 39. Comparison of average annual gas savings by geographic region 5-44 Figure 40. Comparison of average annual gas savings by furnace type 5-45 Figure 41. Comparison of average annual gas savings by secondary heating system 5-46 Figure 42. Comparison of average annual gas savings by prior thermostat type 5-47 Figure 43. Comparison of average annual gas savings by occupancy detection status 5-48 List of Tables Table 1. Comparison between Nest Heat Pump Control Pilot and Smart Thermostat Pilot ..................... 2-1 Table 2. Primary researchable questions and the associated tasks .......................................................... 2-3 Table 3. Features and naming conventions between the smart thermostats