Problem Methodology Empirical Summary

Aggregate Confusion: The Divergence of ESG Ratings

Florian Berg, Julian F. Koelbel, Roberto Rigobon

FEEN Webinar Series, October 7th, 2020

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Outline

1 Problem

2 Methodology Measurement, Scope, and Weights Implementation

3 Empirical Data Discrepancy Taxonomy Estimation Decomposition Rater Effect

4 Summary

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Outline

1 Problem

2 Methodology Measurement, Scope, and Weights Implementation

3 Empirical Data Discrepancy Taxonomy Estimation Decomposition Rater Effect

4 Summary

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Influence of ESG Ratings

ESG1 Rating Agencies are becoming influential institutions Investors with USD 80 trillions "integrate" ESG information Big players are buying ESG rating agencies Regulatory endorsement of ESG ratings possible

1ESG = Environmental, Social, Governance. A.k.a. Sustainability Ratings or ...... Corporate Social Responsibility (CSR) Ratings ...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Correlations

Investors: Which companies are "good"? Pricing implications? Companies: What is expected of us? Academics: How to work with ESG ratings empirically?

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Essential Elements of ESG Ratings

Scope: which attributes are included? Measurement: how are these attributes measured? Weights: how are indicators aggregated into one score?

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Why Sustainability Ratings diverge?

Three sources of divergence. Scope divergence: Ratings agencies rely on a different set of attributes. Weights divergence: Rating agencies take different views on the relative importance of attributes. Measurement divergence: Rating agencies measure the same attribute using different indicators. Results on Divergence Measurement explains 53% of the variation Scope 44% and Weights 3% Factors behind the measurement disagreement? Rater Effect accounts for one fifth of the variation

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Measurement, Scope, and Weights Empirical Implementation Summary Outline

1 Problem

2 Methodology Measurement, Scope, and Weights Implementation

3 Empirical Data Discrepancy Taxonomy Estimation Decomposition Rater Effect

4 Summary

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Atributes

An

A4

A3

A2

A1

...... Measurement Disagreement

I1,n An I2,n

A4

A3

I1,2 A2 I2,2

I1,1 A1 I2,1

Rating Agency 1 . Rating. . . Agency. . . . 2 ...... Scope Disagreement

I1,n An I2,n

A4 I2,3

I1,3 A3

I1,2 A2 I2,2

I1,1 A1 I2,1

Rating Agency 1 . Rating. . . Agency. . . . 2 ...... Weights Disagreement

I1,n An I2,n

A4 I2,3

R1 R2 I1,3 A3

I1,2 A2 I2,2

I1,1 A1 I2,1

Rating Agency 1 . Rating. . . Agency. . . . 2 ...... Problem Methodology Measurement, Scope, and Weights Empirical Implementation Summary Comparing raw data

Practically Impossible to find two rating agencies that measure the exact same attribute for the same firm (Only a dozen pairs). We need to match “raw data” otherwise Scope discrepancies would explain all the divergence. We created a Taxonomy for comparison purposes.

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Decomposition of Discrepancies: Without Categories

An Ifk,n

A4 Ifk,4

Rfk A3 Ifk,3

A2 Ifk,2

A1 Ifk,1

Attribute Indicator Rating

...... Decomposition of Discrepancies: With Categories

An Ifk,n

Cfk,m

A4 Ifk,4

Cfk,2 Rfk A3 Ifk,3

Cfk,1

A2 Ifk,2

A1 Ifk,1

Attribute Indicator Category Rating

...... Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Outline

1 Problem

2 Methodology Measurement, Scope, and Weights Implementation

3 Empirical Data Discrepancy Taxonomy Estimation Decomposition Rater Effect

4 Summary

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Data

Rating Agency Abbrv. Owner Indicators KLD KL MSCI 78 Sustainalytics SA Morningstar 163 Vigeo Eiris VI Moody’s 38 RobecoSAM RS S&P Global 80 Asset4 A4 Refinitiv 282 MSCI MS MSCI 68

ESG and indicator level data Baseline year 2014 (robustness check 2017) 924 common firms between raters

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Mean Absolute Deviation from the Median

Problem with Correlations Correlations at rating agency level hides firm heterogeneity Is the low correlation due to a small set of firms? Median Absolute Deviation For each firm compute ∑ MADf = |Rfk − median(Rf .)| k

where Rfk is the score of firm f and rating agency k. Group the top 100 and bottom 100 firms as measured by their mean median absolute deviations.

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Comparison of firms’ normalized scores.

Dexus Property Group Nokia Corporation Cisco Systems, Inc. Kingfisher plc Campbell Soup Company Accor S.A. Imperial Tobacco Group plc American Water Works Company, Inc. Kering SA Swedbank AB (publ) Land Securities Group plc Xcel Energy Inc. Telstra Corporation Limited Colgate−Palmolive Co. Newmont Mining Corporation Kesko Oyj International Business Machines Corporation Public Service Enterprise Group Inc. CA Technologies, Inc. Belgacom SA Ford Motor Co. Baker Hughes Incorporated Starbucks Corporation General Mills, Inc. BCE Inc. Bouygues SA Rolls Royce Holdings plc SGS SA Nikon Corporation Qantas Airways Limited Daikin Industries Ltd. Brambles Limited Swedish Match AB Astellas Pharma, Inc. IMI plc Morgan Stanley Nitto Denko Corporation Daiwa House Industry Co. Ltd. Sysmex Corp. Dassault Systemes SA Amgen Inc. Mitsubishi Heavy Industries Ltd. Svenska Handelsbanken AB (publ) Resona Holdings, Inc. Trelleborg AB Rating_Agency Sika AG Kirin Holdings Company, Limited Sustainalytics Chugai Pharmaceutical Co. Ltd. Takeda Pharmaceutical Company Limited RobecoSAM Philip Morris International, Inc. Kuraray Co. Ltd. Edenred SA Asset4 Banco Santander−Chile Eisai Co., Ltd. MSCI/KLD Apple Inc. Toyota Industries Corporation Vigeo Moody’s Corporation Nomura Research Institute, Ltd. SCANA Corp. Cigna Corp. Kobe Steel Ltd. HTC Corporation Mitsubishi Tanabe Pharma Corporation Daily Mail and General Trust plc Distribuidora Internacional de Alimentaci...... n, S.A. Luxottica Group SpA Challenger Limited Kerry Group plc Franklin Resources Inc. Kuehne + Nagel International AG Cardinal Health, Inc. Nippon Steel & Sumitomo Metal Corporation Ventas, Inc. Brenntag AG Shionogi & Co., Ltd. DENTSPLY International Inc. Health Care REIT, Inc. Kimco Realty Corporation AutoZone, Inc. CarMax Inc. Greek Organisation of Football Prognostics S.A. Fortune Brands Home & Security, Inc. Chipotle Mexican Grill, Inc. Robert Half International Inc. Commercial International Bank (Egypt) S.A.E. S.A.C.I. Falabella Helmerich & Payne, Inc. Sampo Oyj Sonic Healthcare Limited AmerisourceBergen Corporation Advance Auto Parts Inc. Roper Industries Inc. MediaTek Inc. Deutsche Wohnen AG Intuitive Surgical, Inc. Amphenol Corporation Turkiye Halk Bankasi A.S. Loews Corporation China Resources Land Ltd. Genuine Parts Company −2 −1 0 1 2 Normalized Rating

...... Comparison of firms’ normalized scores.

Dexus Property Group L’Oreal SA Nokia Corporation STMicroelectronics NV Cisco Systems, Inc. BT Group plc Kingfisher plc Henkel AG & Co. KGaA Campbell Soup Company Praxair, Inc. Accor S.A. Intel Corporation Imperial Tobacco Group plc Umicore S.A. American Water Works Company, Inc. GlaxoSmithKline plc Kering SA CRH plc Swedbank AB (publ) Sony Corporation Land Securities Group plc Applied Materials, Inc. Xcel Energy Inc. Banco Santander, S.A. Telstra Corporation Limited Vivendi Soci.....t..... Anonyme Colgate−Palmolive Co. Northern Trust Corporation Newmont Mining Corporation Verizon Communications Inc. Kesko Oyj NTT DOCOMO, Inc. International Business Machines Corporation International Paper Company Public Service Enterprise Group Inc. Outotec Oyj CA Technologies, Inc. Petrofac Limited Belgacom SA Exelon Corporation Ford Motor Co. Canadian National Railway Company Baker Hughes Incorporated Mahindra & Mahindra Ltd. Starbucks Corporation HSBC Holdings plc General Mills, Inc. Lite−On Technology Corp. BCE Inc. Amadeus IT Holding SA Bouygues SA Taiwan Semiconductor Manufacturing Company Limited Rolls Royce Holdings plc H & M Hennes & Mauritz AB (publ) SGS SA Kone Oyj Nikon Corporation ITV plc Qantas Airways Limited Nippon Telegraph and Telephone Corporation Daikin Industries Ltd. PostNL N.V. Brambles Limited CEMEX, S.A.B. de C.V. Swedish Match AB The Royal Bank of Scotland Group plc Astellas Pharma, Inc. Power Assets Holdings Limited IMI plc Entergy Corporation Morgan Stanley Advanced Micro Devices, Inc. Nitto Denko Corporation American Electric Power Co., Inc. Daiwa House Industry Co. Ltd. LG Electronics Inc. Sysmex Corp. Hitachi Chemical Co. Ltd. Dassault Systemes SA Oracle Corporation Amgen Inc. G4S plc Mitsubishi Heavy Industries Ltd. Samsung Electronics Co. Ltd. Svenska Handelsbanken AB (publ) Infineon Technologies AG Resona Holdings, Inc. Dominion Resources, Inc. Trelleborg AB Rating_Agency Hyundai Mobis Co.,Ltd. Rating_Agency Sika AG Honda Motor Co., Ltd. Kirin Holdings Company, Limited Sustainalytics Samsung Electro−Mechanics Co. Ltd. Sustainalytics Chugai Pharmaceutical Co. Ltd. Shaftesbury plc Takeda Pharmaceutical Company Limited RobecoSAM CTBC Financial Holding Co., Ltd. RobecoSAM Philip Morris International, Inc. CMS Energy Corp. Kuraray Co. Ltd. Alumina Ltd. Edenred SA Asset4 Iluka Resources Ltd. Asset4 Banco Santander−Chile Comcast Corporation Eisai Co., Ltd. MSCI/KLD Core Laboratories NV MSCI/KLD Apple Inc. Fomento Econ...... mico Mexicano, S.A.B de C.V Toyota Industries Corporation Vigeo Juniper Networks, Inc. Vigeo Moody’s Corporation Hannover R...... ck SE Nomura Research Institute, Ltd. Pfizer Inc. SCANA Corp. FUJIFILM Holdings Corporation Cigna Corp. Aisin Seiki Co., Ltd. Kobe Steel Ltd. Larsen & Toubro Limited HTC Corporation SSE plc Mitsubishi Tanabe Pharma Corporation Lam Research Corporation Daily Mail and General Trust plc Hyundai Engineering & Construction Co., Ltd. Distribuidora Internacional de Alimentaci...... n, S.A. Link Real Estate Investment Trust Luxottica Group SpA AU Optronics Corp. Challenger Limited Caterpillar Inc. Kerry Group plc Google Inc. Franklin Resources Inc. The Goodyear Tire & Rubber Company Kuehne + Nagel International AG The AES Corporation Cardinal Health, Inc. Swire Pacific Limited Nippon Steel & Sumitomo Metal Corporation Dr. Reddy’s Laboratories Ltd. Ventas, Inc. Itausa − Investimentos Itau S.A. Brenntag AG Western Digital Corporation Shionogi & Co., Ltd. Samsung Securities Co. Ltd. DENTSPLY International Inc. Honeywell International Inc. Health Care REIT, Inc. KT&G Corporation Kimco Realty Corporation William Hill plc AutoZone, Inc. Golden Agri−Resources Ltd CarMax Inc. Bridgestone Corp. Greek Organisation of Football Prognostics S.A. Bancolombia S.A. Fortune Brands Home & Security, Inc. Sumitomo Mitsui Financial Group, Inc. Chipotle Mexican Grill, Inc. Tyson Foods, Inc. Robert Half International Inc. Transocean Ltd. Commercial International Bank (Egypt) S.A.E. Toyo Seikan Group Holdings, Ltd. S.A.C.I. Falabella Grupo de Inversiones Suramericana S.A. Helmerich & Payne, Inc. West Japan Railway Company Sampo Oyj Porsche Automobil Holding SE Sonic Healthcare Limited Japan Exchange Group, Inc. AmerisourceBergen Corporation Hirose Electric Co. Ltd. Advance Auto Parts Inc. PICC Property and Casualty Co. Ltd. Roper Industries Inc. Tencent Holdings Ltd. MediaTek Inc. Kintetsu Corporation Deutsche Wohnen AG Toho Gas Co. Ltd. Intuitive Surgical, Inc. Marui Group Co., Ltd. Amphenol Corporation Tokyu Corporation Turkiye Halk Bankasi A.S. Fanuc Corporation Loews Corporation Odakyu Electric Railway Co., Ltd. China Resources Land Ltd. Cheng Shin Rubber Industries Co. Ltd. Genuine Parts Company Toho Co. Ltd. −2 −1 0 1 2 −3 −2 −1 0 1 2 3 Normalized Rating Normalized Rating

...... Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Mean Median Absolute Deviations

Very large heterogeneity at the firm level For the strong agreement set (100 smallest MAD’s) correlation of measures across rating agencies is 90% For the low agreement set (100 smallest MAD’s) correlation of measures across rating agencies is 32%

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Comparing Rankings

Financial Institutions care more about ranking than scores. i.e. Scores might be uncorrelated, but rankings might be tightly related. Common firms among the top and bottom 20 percent (164 firms)

Common among Top quantile Common among Bottom quantile Akzo Nobel NV Advance Auto Parts Inc. Australia & New Zealand Banking Group Limited Affiliated Managers Group Inc. Aviva plc America Movil S.A.B. de C.V. BMW AG Amphenol Corporation BNP Paribas SA Berkshire Hathaway Inc. Campbell Soup Company Cencosud S.A. Commonwealth Bank of Australia China Development Financial Holding Corporation Dexus Property Group China Resources Land Ltd. Diageo plc CP ALL Public Company Limited EDP-Energias de Portugal, S.A. Credit Saison Co. Ltd. Hewlett-Packard Company Deutsche Wohnen AG Imperial Tobacco Group plc Expedia Inc. Industria de Diseno Textil SA Genuine Parts Company Kingfisher plc Grupo Financiero Inbursa, S.A.B. de C.V. Koninklijke Philips N.V Hengan International Group Company Limited Mondi plc Intuitive Surgical, Inc. National Australia Bank Limited Japan Real Estate Investment Corporation Nokia Corporation Loews Corporation Renault SA MediaTek Inc. Schneider Electric S.A. MediPal Holdings Corporation Solvay SA Meiji Holdings Co., Ltd. STMicroelectronics NV Naver Corporation Swiss Re Ltd NCsoft Corporation Telecom Italia S.p.A. NEXON Co., Ltd. UPM-Kymmene Oyj Nippon Building Fund Inc. Wipro Ltd. Shimano Inc. Sumitomo Realty & Development Co. Ltd. Tokyo Electric Power Company, Incorporated

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Comparison of firms’ ranks.

Peugeot S.A. Koninklijke Philips N.V Bayerische Motoren Werke Aktiengesellschaft STMicroelectronics NV Akzo Nobel NV UPM−Kymmene Oyj Mondi plc EDP−Energias de Portugal, S.A. Swiss Re Ltd Schneider Electric S.A. Australia & New Zealand Banking Group Limited National Australia Bank Limited Dexus Property Group Aviva plc Wipro Ltd. BNP Paribas SA Cisco Systems, Inc. Industria de Diseno Textil SA Hewlett−Packard Company Telecom Italia S.p.A. Kingfisher plc Diageo plc Nokia Corporation Renault Soci.....t..... Anonym The Toronto−Dominion Bank Accor S.A. Campbell Soup Company Commonwealth Bank of Australia Solvay SA Imperial Tobacco Group plc Roche Holding AG 3M Company American Water Works Company, Inc. Kering SA Colgate−Palmolive Co. Swedbank AB (publ) Baker Hughes Incorporated Public Service Enterprise Group Inc. General Mills, Inc. Nikon Corporation SGS SA Ford Motor Co. Daikin Industries Ltd. IMI plc Banco Santander−Chile Rating_Agency Apple Inc. Franklin Resources Inc. Sustainalytics Moody’s Corporation Luxottica Group SpA RobecoSAM Nidec Corporation Brenntag AG Asset4 Kuehne + Nagel International AG Ventas, Inc. MSCI/KLD Health Care REIT, Inc. Shionogi & Co., Ltd. Vigeo DENTSPLY International Inc. CarMax Inc. Sonic Healthcare Limited AmerisourceBergen Corporation Robert Half International Inc. Helmerich & Payne, Inc. Advance Auto Parts Inc. S.A.C.I. Falabella JGC Corp. Sampo Oyj Hologic Inc. America Movil S.A.B. de C.V. Primary Health Care Limited Roper Industries Inc. MediaTek Inc. Deutsche Wohnen AG Naver Corporation Wynn Resorts Ltd. Amphenol Corporation CP ALL Public Company Limited FLIR Systems, Inc. Anhui Conch Cement Co. Ltd. Affiliated Managers Group Inc. Intuitive Surgical, Inc. Genuine Parts Company Harvey Norman Holdings Ltd. CITIC Limited Loews Corporation Berkshire Hathaway Inc. Meiji Holdings Co., Ltd. MediPal Holdings Corporation Expedia Inc. Cencosud S.A. China Development Financial Holding Corporation Credit Saison Co. Ltd. Shimano Inc. Hengan International Group Company Limited China Resources Land Ltd. Grupo Financiero Inbursa, S.A.B. de C.V. NEXON Co., Ltd. Sumitomo Realty & Development Co. Ltd. Tokyo Electric Power Company, Incorporated Japan Real Estate Investment Corporation NCsoft Corporation Nippon Building Fund Inc. 0.00 0.25 0.50 0.75 1.00 Rank

...... Comparison of firms’ ranks.

Peugeot S.A. Praxair, Inc. Koninklijke Philips N.V Intel Corporation Bayerische Motoren Werke Aktiengesellschaft Applied Materials, Inc. STMicroelectronics NV International Paper Company Akzo Nobel NV Sony Corporation UPM−Kymmene Oyj Outotec Oyj Mondi plc General Electric Company EDP−Energias de Portugal, S.A. Medtronic, Inc. Swiss Re Ltd Cognizant Technology Solutions Corporation Schneider Electric S.A. Toray Industries, Inc. Australia & New Zealand Banking Group Limited NTT DOCOMO, Inc. National Australia Bank Limited Verizon Communications Inc. Dexus Property Group Canadian National Railway Company Aviva plc H & M Hennes & Mauritz AB (publ) Wipro Ltd. Nippon Telegraph and Telephone Corporation BNP Paribas SA PostNL N.V. Cisco Systems, Inc. Entergy Corporation Industria de Diseno Textil SA HSBC Holdings plc Hewlett−Packard Company LG Electronics Inc. Telecom Italia S.p.A. Lite−On Technology Corp. Kingfisher plc Mahindra & Mahindra Ltd. Diageo plc Amadeus IT Holding SA Nokia Corporation Hasbro Inc. Renault Soci.....t..... Anonym Infineon Technologies AG The Toronto−Dominion Bank The Coca−Cola Company Accor S.A. Kone Oyj Campbell Soup Company ASICS Corp. Commonwealth Bank of Australia Power Assets Holdings Limited Solvay SA ITV plc Imperial Tobacco Group plc Samsung Electronics Co. Ltd. Roche Holding AG Dominion Resources, Inc. 3M Company ITOCHU Corporation American Water Works Company, Inc. CEMEX, S.A.B. de C.V. Kering SA The Royal Bank of Scotland Group plc Colgate−Palmolive Co. CTBC Financial Holding Co., Ltd. Swedbank AB (publ) Arkema S.A. Baker Hughes Incorporated Humana Inc. Public Service Enterprise Group Inc. Mitsubishi Chemical Holdings Corporation General Mills, Inc. Honda Motor Co., Ltd. Nikon Corporation Canadian Tire Corp. Ltd. SGS SA Hitachi Chemical Co. Ltd. Ford Motor Co. Alumina Ltd. Daikin Industries Ltd. Iluka Resources Ltd. IMI plc G4S plc Banco Santander−Chile Rating_Agency Halliburton Company Rating_Agency Apple Inc. Fomento Econ...... mico Mexicano, S.A.B de C.V Franklin Resources Inc. Sustainalytics Juniper Networks, Inc. Sustainalytics Moody’s Corporation Comcast Corporation Luxottica Group SpA RobecoSAM Core Laboratories NV RobecoSAM Nidec Corporation Hannover R...... ck SE Brenntag AG Asset4 Chr. Hansen Holding A/S Asset4 Kuehne + Nagel International AG Fluor Corporation Ventas, Inc. MSCI/KLD Newcrest Mining Limited MSCI/KLD Health Care REIT, Inc. CMS Energy Corp. Shionogi & Co., Ltd. Vigeo Hyundai Mobis Co.,Ltd. Vigeo DENTSPLY International Inc. Larsen & Toubro Limited CarMax Inc. NTT Data Corporation Sonic Healthcare Limited Link Real Estate Investment Trust AmerisourceBergen Corporation Caterpillar Inc. Robert Half International Inc. AU Optronics Corp. Helmerich & Payne, Inc. The Goodyear Tire & Rubber Company Advance Auto Parts Inc. Advanced Info Service Public Company Limited S.A.C.I. Falabella Tokyo Electron Limited JGC Corp. SSE plc Sampo Oyj Samsung Electro−Mechanics Co. Ltd. Hologic Inc. Lam Research Corporation America Movil S.A.B. de C.V. American Express Company Primary Health Care Limited Pfizer Inc. Roper Industries Inc. Grupo Financiero Banorte, S.A.B. de C.V. MediaTek Inc. Hyundai Engineering & Construction Co., Ltd. Deutsche Wohnen AG Man Group plc Naver Corporation DBS Group Holdings Ltd Wynn Resorts Ltd. The AES Corporation Amphenol Corporation FUJIFILM Holdings Corporation CP ALL Public Company Limited Microchip Technology Inc. FLIR Systems, Inc. Swire Pacific Limited Anhui Conch Cement Co. Ltd. Itausa − Investimentos Itau S.A. Affiliated Managers Group Inc. Dr. Reddy’s Laboratories Ltd. Intuitive Surgical, Inc. Aisin Seiki Co., Ltd. Genuine Parts Company William Hill plc Harvey Norman Holdings Ltd. Marsh & McLennan Companies, Inc. CITIC Limited Google Inc. Loews Corporation Western Digital Corporation Berkshire Hathaway Inc. KT&G Corporation Meiji Holdings Co., Ltd. Mitsubishi Estate Co., Ltd. MediPal Holdings Corporation Alliance Data Systems Corporation Expedia Inc. Bridgestone Corp. Cencosud S.A. Bancolombia S.A. China Development Financial Holding Corporation Samsung Securities Co. Ltd. Credit Saison Co. Ltd. Toyo Seikan Group Holdings, Ltd. Shimano Inc. Tabcorp Holdings Ltd. Hengan International Group Company Limited Mitsui Fudosan Co. Ltd. China Resources Land Ltd. Sumitomo Mitsui Financial Group, Inc. Grupo Financiero Inbursa, S.A.B. de C.V. Dentsu Inc. NEXON Co., Ltd. Grupo de Inversiones Suramericana S.A. Sumitomo Realty & Development Co. Ltd. SoftBank Corp. Tokyo Electric Power Company, Incorporated Porsche Automobil Holding SE Japan Real Estate Investment Corporation PICC Property and Casualty Co. Ltd. NCsoft Corporation Tencent Holdings Ltd. Nippon Building Fund Inc. Toho Gas Co. Ltd. 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Rank Rank

...... Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Taxonomy

Bottom-up taxonomy, resulting in 64 distinct criteria Two rules: Indicators can only be part of ONE category At least TWO matching indicators from different raters to establish a new category Example for the category "Water"

Rater Indicator Category Asset4 A4: 4.E.Emission Reduction/Discharge into Water System Water Asset4 A4: 4.E.Resource Reduction/Water Use Water Asset4 A4: 4.E.Resource Reduction/Water Recycling Water KLD KL: 3.ENV.CON.Water Management Water KLD KL: 3.ENV.STR.Water Stress Water RobecoSAM RS: 3.E.Water Operations Water RobecoSAM RS: 3.E.Water Related Risks Water Sustainalytics SA: 3.E.1.2.7 Water Intensity-Raw Score Water Sustainalytics SA: 3.E.1.3.4 Water Management Programmes-Raw Score Water Vigeo VI: 4.E.ENV2.1 Water Water MSCI MS: 5.Water Stress Mgmt Water

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Category Scores

KL KL KL KL KL SA SA SA SA VI VI VI RS RS A4 Average SA VI RS A4 MS VI RS A4 MS RS A4 MS A4 MS MS Access to Basic Services 0.08 0.13 0.85 0.49 0.15 0.16 0.31 Access to Healthcare 0.66 0.57 0.49 0.85 0.67 0.56 0.74 0.44 0.71 0.7 0.64 Anti-Competitive Practices -0.06 0.56 0.76 0 -0.05 0.56 0.30 Audit 0.57 0.66 0.62 0.62 Biodiversity 0.06 -0.08 0.06 0.66 0.61 0.41 0.47 0.47 0.01 0.2 0.29 Board 0.37 0.58 0.51 0.49 Board Diversity 0.8 0.80 Business Ethics 0.04 -0.11 0.4 0.6 0.33 0.03 0.01 -0.1 -0.15 0.38 0.14 Chairman Ceo Separation 0.59 0.59 Child Labor 0.49 0.49 Climate Risk Mgmt. 0.44 0.42 0.8 0.54 0.54 0.5 0.54 Clinical Trials 0.73 0.73 Collective Bargaining 0.59 -0.04 0 0.18 Community and Society -0.15 0.25 0.2 0.11 -0.1 -0.19 -0.13 0.51 0.5 0.56 0.16 Customer Relationship 0.38 -0.08 -0.09 0 -0.04 -0.13 -0.05 0.49 0.47 0.41 0.14 Diversity -0.06 -0.02 0.03 0.61 0.52 0.56 0.27 Employee Development 0.22 0.29 0.37 0.37 0.73 0.23 0.19 0.36 0.34 0.39 0.29 0.31 0.55 0.45 0.51 0.37 Environmental Policy 0.52 0.46 0.46 0.63 0.61 0.62 0.55 Environmental Reporting 0.52 0.25 0.36 0.38 GHG Emissions 0 -0.03 -0.06 0.28 0.31 0.5 0.17 GHG Policies 0.48 0.62 0.41 0.50 GMOs 0.38 0.43 0.25 0.35 Global Compact Member 0.92 0.92 Green Buildings 0.54 0.59 0.21 0.83 0.25 0.26 0.55 -0.02 0.66 0.28 0.42 Green Products 0.23 0.07 0.27 0.34 0.76 0.1 0.37 0.47 0.32 0.31 0.29 -0.05 0.53 0.44 0.53 0.33 Health and Safety 0.01 0.27 0.27 0.35 0.73 -0.1 -0.16 -0.16 -0.05 0.63 0.67 0.5 0.57 0.44 0.6 0.30 Human Rights 0 0.19 0.08 -0.01 -0.08 0.42 0.10 Indigenous Rights 0.26 -0.11 -0.46 -0.10 Labor Practices 0.21 -0.04 -0.14 0.07 0.1 0.2 0.14 0.32 0.27 0.54 0.45 0.43 0.35 0.34 0.37 0.24 Lobbying -0.28 -0.28 Product Safety -0.05 0.06 0.16 0 0.63 -0.14 -0.03 0.07 0.46 0.21 0.11 0.38 -0.03 0.1 0.14 Responsible Marketing -0.5 -0.06 -0.38 0.24 0.38 0.68 0 0.49 0.05 -0.1 0.08 Waste 0.34 0.23 0.33 0.30 Water 0.36 0.36 0.23 0.23 0.67 0.47 0.29 0.31 0.45 0.48 0.32 0.5 -0.02 0.24 0.44 0.36 Average 0.20 0.12 0.20 0.21 0.69 0.29 0.32 0.33 0.37 0.48 0.38 0.34 0.35 0.37 0.38 ...... Aggregation Rules

Sustainalytics RobecoSAM Asset4 Vigeo-Eiris MSCI KLD Access to Basic Services 0.019 - 0 - 0.138*** 0.065*** Access to Healthcare 0.051*** 0.004 0 - 0.079*** 0.051*** Animal Welfare 0.05*** - 0 - - - Anti-Competitive Practices - - 0.05*** 0.023*** 0 0.131*** Audit 0 - 0.026* 0.084*** - - Biodiversity 0 0 0 0.028*** 0.366*** 0.076*** Board 0.072*** - 0.196*** 0.113*** 0 - Board Diversity 0.043*** - 0 - - 0 Business Ethics 0.097*** 0.046*** 0.008 - 0 0.148*** Chairman Ceo Separation 0.039*** - 0.016 - - - Child Labor - - 0.008 0 - 0.046*** Climate Risk Mgmt. - 0.137*** 0.064*** - 0.069** 0.234*** Clinical Trials 0 - 0 - - - Collective Bargaining 0.051*** - 0.011* 0.072*** - - Community and Society 0.079*** 0.086*** 0.03* 0.001 - 0.14*** Corporate Governance - 0.048*** - - 0.198*** - Corruption 0.049*** - 0.022* 0.072*** 0.388*** 0.124*** Customer Relationship 0.127*** 0.097*** 0.086*** 0.027*** - 0.104*** Diversity 0.108*** - 0.066*** 0.159*** - 0.04*** ESG incentives 0.006 0 - - - - Electromagnetic Fields 0.021** 0 - - - - Employee Development 0.018* 0.221*** 0.116*** 0.067*** 0.406*** 0.149*** Employee Turnover 0.024* - 0 - - - Energy 0.032** 0.016*** 0.029** 0.103*** 0.194*** 0.046*** Environmental Fines 0 - 0 - - 0 Environmental Mgmt. System 0.199*** - 0.009 - - 0.205*** Environmental Policy 0.091*** 0.098*** 0.012 0.187*** - - Environmental Reporting 0.043** 0.039*** 0.007 - - - Financial Inclusion 0 - - - 0.089*** 0.061*** Forests 0.008 0.016* - - - - GHG Emissions 0.048*** - 0.002 0.033*** - 0.021** ...... Waste 0 0.005 0.035*** 0.009 - 0.186*** Water 0.03** 0.016*** 0.028** 0 0.035 0.175*** Unclassified Indicators Yes Yes Yes Yes Yes Yes R2 0.90 0.98 0.92 0.96 0.79 0.99 Firms 924 924 924 924 924 924 ...... Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Results of Estimation: Including Robustness

R2 from each specification OLS (signs) Random Forest Neural Network SASB (Categorization) Asset4 0.92 0.98 0.96 0.92 MSCI/KLD 0.99 0.93 0.98 0.98 RobecoSAM 0.98 0.99 0.99 0.98 Sustainalytics 0.90 0.95 0.93 0.90 Vigeo-Eiris 0.96 0.98 0.98 0.96

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Arithmetic Decomposition: Decomposition

The arithmetic decomposition is an exact decomposition:

∆a,b =∆(meas + ∆weights +)∆scope ( ) ∗ ∗ = C − C × wˆ + Rˆ − Rˆ − C − C × wˆ + Rˆ − Rˆ ajcom bjcom a,com b,com ajcom bjcom a,ex b,ex ˆ ˆ − ˆ − ˆ =Ra,com + Ra,ex Rb,com Rb,ex ˆ − ˆ =Ra Rb

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Arithmetic Decomposition: Results

Mean Absolute Deviations (over 924 firms)

Scope Measurement Weights Fitted True KLD 0.43 0.60 0.37 0.76 0.80 Sustainalytics 0.41 0.53 0.31 0.63 0.69 Vigeo 0.44 0.51 0.25 0.66 0.70 RobecoSAM 0.40 0.54 0.26 0.69 0.73 Asset4 0.39 0.53 0.33 0.66 0.75 MSCI 0.84 0.55 0.52 0.78 0.84 Average 0.48 0.54 0.34 0.69 0.75

Measurement is the most important source of divergence MSCI has a pronounced scope divergence

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Barrick Gold Corp.: Asset4 vs. KLD

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Rater Effect: Fixed Effect Results

Dummies R2 Firm 0.22 Firm + Firm-Rater 0.38 Firm + Firm-Category 0.47 Firm + Firm-Category + Firm-Rater 0.62

Rater effect adds between 14 and 16 percent!

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Rater Effect: LASSO

(a) KLD (b) RobecoSAM (c) Asset4

(d) Sustainalytics (e) Vigeo-Eiris (f) MSCI

...... Data Problem Discrepancy Methodology Taxonomy Empirical Estimation Summary Decomposition Rater Effect Robustness Checks

Alternative taxonomy based on 26 SASB criteria Statistical decomposition Repeat analysis for 2017 data Repeat analysis in rankings instead of normalized data

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Outline

1 Problem

2 Methodology Measurement, Scope, and Weights Implementation

3 Empirical Data Discrepancy Taxonomy Estimation Decomposition Rater Effect

4 Summary

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Key Findings

Measurement is the most important source of ESG rating divergence Measurement is in part driven by a rater effect

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence Problem Methodology Empirical Summary Implications

Academics Use average rating to represent the market’s consensus view Use of a single rating must be carefully justified Use more specific categories to test specific hypotheses Companies Our methodology can help to figure out why scores diverge Rating Agencies Differences between raters are not merely conceptual differences but also differences in measurement Rater effect has implications for the organizational structure of rating agencies Investors Either buy one rating that fits your purpose and trust the process Or buy raw data from different providers and construct your own

...... Florian Berg, Julian F. Koelbel, Roberto Rigobon Aggregate Confusion: Divergence