Replication / ML Reproducibility Challenge 2020 RESCIENCEC [Re] Replication Study of ’Generative causal explanations of black-box classifiers’ Ivo Verhoeven1, ID , Xinyi Chen1, ID , Qingzhi Hu1, ID , and Mario Holubar1, ID 1University of Amsterdam, Amsterdam, the Netherlands Edited by Koustuv Sinha Reviewed by Reproducibility Summary Anonymous Reviewers Received Scope of Reproducibility 29 January 2021 We verify the outcome of the methodology proposed in the article, which attempts to Published provide post-hoc causal explanations for black-box classifiers through causal reference. 27 May 2021 This is achieved by replicating the code step by step, according to the descriptions in the paper. All the claims in the paper have been examined, and we provide additional DOI 10.5281/zenodo.4835600 metric to evaluate the portability, expressive power, algorithmic complexity and the data fidelity of their framework. We have further extended their analyses to consider all benchmark datasets used, confirming results. Methodology We use the same architecture and (hyper)parameters for replication. However, the code has a different structure and we provide a more efficient implementation for themea- sure of information flow. In addition, Algorithm 1 in the original paper is not imple- mented in their repository, so we have also implemented Algorithm 1 ourselves and further extend their framework to another domain (text data), although unsuccessfully. Furthermore, we make a detailed table in the paper to show the time used to produce the results for different experiments reproduced. All models were trained on Nvidia GeForce GTX 1080 GPUs provided by Surfsaraʼs LISA cluster computing service at uni- versity of Amsterdam1. Results We reproduced the framework in the original paper and verified the main claims made by the authors in the original paper. However, the GCE model in extension study did not manage to separate causal factors and non-causal factors for a text classifier due to the complexity of fine-tuning the model. 1This is our course project for the master course Fairness, Accountability, Confidentiality and Transparency in AI at the University of Amsterdam. Lisa cluster: https://userinfo.surfsara.nl/systems/lisa. Copyright © 2021 I. Verhoeven et al., released under a Creative Commons Attribution 4.0 International license. Correspondence should be addressed to Ivo Verhoeven ([email protected]) The authors have declared that no competing interests exist. Code is available at https://github.com/shin-ee-chen/UvA_FACT_2021. – SWH swh:1:dir:9d960937f2b11bb67b53bbe4eaa940c14cadf9d1. Open peer review is available at https://openreview.net/forum?id=Vqtf4kZg2j. ReScience C 7.2 (#23) – Verhoeven et al. 2021 1 [Re] Replication Study of ’Generative causal explanations of black-box classifiers’ What was easy The original paper comes with extensive appendices, many of which contain crucial details for implementation and understanding of the intended function. The authors provide code for most of the experiments presented in the paper. Although at the begin- ning their code repository was not functional, we use it as a reference to re-implement our code. The author also updated their code two weeks after we start our own imple- mentation, which made it easy for us to verify the correctness of our re-implementation. What was difficult The codebase the authors provided was initially unusable, with missing or renamed imports, hardcoded filepaths and an all-around convoluted structure. Additionally, the description of Algorithm 1 is quite vague and no implementation of it was given. Beyond this, computational expense was a serious issue, given the need for inefficient training steps, and re-iterating training several times for hyperparameter search. Communication with original authors This reproducibility study is part of a course on fairness, accountability, confidentiality and transparency in AI. Since it is a course project where we interacted with other group in the forum, and another group also working with this paper has reached out to the authors about problems with the initial repository, we did not find necessary to do it again. 1 Introduction Machine learning is increasingly used in different applications. The wide-scale spread of these methods places more emphasis on transparent algorithmic decision making, which has the potential to mitigate the potential for disruptive social effects. Yet, despite reliable results of complex black boxes, their internal reasoning and inner workings are not necessarily apparent to end-users or even designers. As a result, not even trained experts can grasp the reasoning behind forecasts. Moreover, modern legislation have necessitated the opportunity challenging these systems, especially in heavily regulated domains, increasing the need for machine learning systems that are (post-hoc) inter- pretable [1]. Black-box artificial intelligence approaches like Deep Neural Networks have often proven to be able to capture complex dependencies within data, and allow for making accurate predictions. However, the actual internal logic used by systems dependent on such ap- proaches is often nebulous or totally unclear. In this paper, we will reproduce the paper which focuses on the explainability aspect of AI. Explainable Artificial Intelligence (XAI) refers to systems that seek to clarify how ablack- box AI model achieves its performance. Post-hoc XAI achieves the desired explainability be generating reasons for decisions after having trained a black-box classifier. This is often achieved by extracting correlations between input features and the eventual fore- casts [2]. This paper aims to reproduce such a post-hoc XAI algorithm, capable of pro- viding clear and interpretable explanations for complex black-box classifiers3 [ ]. The central contribution made by [3] is placing explanations in a causal-generative frame- work. By using a variational auto-encoder (VAE) [4] , low dimensional representations of the data can be achieved. By further incorporating a mutual information loss between the classifier and the latent variables, the latent space is decorrelated into factors that ReScience C 7.2 (#23) – Verhoeven et al. 2021 2 [Re] Replication Study of ’Generative causal explanations of black-box classifiers’ actively influence the classifierʼs prediction, and those that capture superfluous varia- tion in the underlying dataset. The causal-VAE , dubbed a generative causal explainer (GCE) by [3] can be studied and intervened with to provide a powerful framework for inducing explainability. There exists literature suggesting that some studies cannot be replicated [5]. This is valid also for the most respected journals and conferences in AI. Steps must be taken to ensure high trustworthiness of AI algorithms [6]. In the experiments presented here, we re-implement the framework used by [3], extend their analyses to all benchmark datasets discussed and provide an extension to a novel domain. When beginning our process, the officially provided code repository corresponding to the authorsʼ publication was not capable of running in all discussed scenarios, nor pro- ducing results similar to those presented 2. However, our re-implementation has ascer- tained all results, and been extended to incorporate all analyses, and can report their framework to be reproducible. All code has been made publicly available via a Github repository3. 2 Scope of Reproducibility The original paper establishes a causal inference framework. While intuitive, this intro- duces the additional challenge of balancing causal intervention with expressive latent variables. To overcome this hurdle, [3] integrate causal inference into the VAE learning objective. This system consists of two basic components: a way to describe the data dis- tribution, and a method of generating a consistent classification model. In order to get a clear objective of the reproduction, we consider the the main claim(s) of the original paper as: • Claim 1: creating a generative framework that describes a black-box classifier, and a process that achieves this while maintaining latent descriptions that result in high data fidelity and scalability of the overall system • Claim 2: their objective function helps the generative model disentangle the latent variables into causal and non-causal factors for the classifier. Their approach is sufficient for any black-box classifier that can provide class gradients andproba- bilities with respect to the classifier input • Claim 3: the resulting explanations show what is important to the accompanying classifier • Claim 4: the learned representation may also be used to explain counterfactual explanations as it incorporates both generative and causal modelling The standard of reproducibility demands that machine learning methods be routinely evaluated on the verifiability of their results. The following additional metrics (proper- ties) will be used as measurement of reproducibility: • Portability: the modelling domains and types of machine learning models that can benefit from their framework • Expressive Power: the explanation structure that can be produced by their frame- work • Algorithmic Complexity: the computational complexity of their algorithm 2Since, the authors have responded to suggestions and rewritten their code base substantially. 3https://github.com/shin-ee-chen/UvA_FACT_2021 ReScience C 7.2 (#23) – Verhoeven et al. 2021 3 [Re] Replication Study of ’Generative causal explanations of black-box classifiers’ • Data Fidelity: