Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor DeployedDeployed Validate and Deploy ModelModel Build an Event Driven Machine Learning Pipeline on Kubernetes Yasushi Osonoi Animesh Singh Developer Advocate IBM STSM, IBM kubeflow kfserving maintainer osonoi animeshsingh Center for Open Source Improving Enterprise AI lifecycle in Open Source Data and AI Technologies (CODAIT) Code – Build and improve practical frameworks to enable more developers to realize immediate value. Content – Showcase solutions for complex and real-world AI problems. Community – Bring developers and data scientists to engage with IBM • Team contributes to over 10 open source projects • 17 committers and many contributors in Apache projects • Over 1100 JIRAs and 66,000 lines of code committed to Apache Spark itself; over 65,000 LoC into SystemML • Over 25 product lines within IBM leveraging Apache Spark • Speakers at over 100 conferences, meetups, unconferences and more CODAIT codait.org 3 DEVELOPER ADVOCATE in TOKYO Tokyo Team is a part of Worldwide Developer Advocate Teams! Developer Advocate City Leader WW Developer Advocate WW Developer Advocate Client Developer Advocate AKIRA ONISHI NORIKO KATO KYOKO NISHITO YASUSHI OSONOI Program Manager WW Developer Advocate WW Developer Advocate Digital Developer Advocate TOSHIO YAMASHITA TAIJI HAGINO AYA TOKURA JUNKI SAGAWA @taiponrock https://developer.ibm.com/patterns/ https://developer.ibm.com/jp/ Please follow me @osonoi IBM’s history of actively fostering balanced communities 19 1998 “For more than 20 years, IBM and Red Hat have paved the way for open communities to power innovative IT solutions.” – Red Hat © 2019 IBM Corporation 8 AI and machine leaning IBM recently open sourced some key technologies for AI, including: •The AI Fairness 360 toolkit (AIF360), an open source software toolkit that can help detect and remove bias in machine learning models •The Adversarial Robustness Toolbox for rapid crafting and analysis of attack and defense methods for machine learning models •Fabric for Deep Learning (FfDL, pronounced fiddle), a deep learning platform offering TensorFlow, Caffe, PyTorch etc. as a Service on Kubernetes This space is as hot as they come, so look for more open source innovation from IBM in the months to come. Progress in Deep Learning IBM Watson IBM Deep Blue AlphaGo chess Jeopardy Apple’s Facebook’s Siri gets releases Siri face deep learning recognition 1997 AlexNet … 1997 2011 2012 2015 2016 2017 Introduced deep learning with GPUs 10 Progress in Deep Learning IBM Watson IBM Deep Blue AlphaGo chess Jeopardy Apple’s Facebook’s Siri gets releases Siri face deep learning recognition AlexNet 2011 … 1997 2011 2012 2015 2016 2017 Introduced deep learning with GPUs 11 Progress in Deep Learning IBM Watson IBM Deep Blue AlphaGo chess Jeopardy Apple’s Facebook’s Siri gets releases Siri face deep learning recognition AlexNet … 1997 2011 2012 2015 2016 20172017 Introduced deep learning with GPUs 12 IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation 13 Progress in Deep Learning IBM Watson IBM Deep Blue AlphaGo chess Jeopardy Apple’s Facebook’s Siri gets releases Siri face deep learning recognition AlexNet … 1997 2011 2012 2015 2016 2017 Introduced deep learning with GPUs IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation 14 Deep Learning = Training Artificial Neural Networks • 25 million “neurons” • 100 million connections (parameters) A human brain has: • 200 billion neurons • 32 trillion connections between them IBM Cloud / Watson and Cloud Platform / © 2018 IBM Corporation 15 Perception *Source: Hidden Technical Debt in Machine Learning Systems In reality … *Source: Hidden Technical Debt in Machine Learning Systems Neural Network Design Workflow domain data design Performance neural network HPO meets needs? • neural network structure • hyperparameters Start another experiment optimal hyperparameters NO Neural Network Design Workflow domain data design Performance neural network HPO meets needs? • neural network structure • hyperparameters Start another experiment optimal yes hyperparameters NO trained BAD Still model good! Cloud deploy evaluat e Let`s understand it from the context of an AI Lifecycle AI PLATFORM 20 We need a Cloud native AI Platform to build,AIOps train, deploy and monitor Models Assign Hyperparameters Initial Model and Train Create Model Prepared and AI Trained Analyzed Model Data PLATFORM Monitor Deployed Validate and Deploy Model Many tools available to build initial modelsAIOps Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor DeployedDeployed Validate and Deploy ModelModel Neural Network Modeller within Watson Studio An intuitive drag-and-drop, no-code interface for designing neural network structure https://developer.ibm.com/tutorials/create-and-experiment-with-dl-models-using-nn-modeler/ https://developer.ibm.com/patterns/validate-deep-learning-models/ March 30 2018 / © 2018 IBM Corporation 24 Many tools to train machine learning andAIOps deep learning models Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor DeployedDeployed Validate and Deploy ModelModel Training is accomplished. Model is readyAIOps – Can we trust it? Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor DeployedDeployed Validate and Deploy Can the model be trusted? ModelModel What does it take to trust a decision made by a machine? (Other than that it is 99% accurate)? #21, #32, #93 #21, #32, #93 Is it easy to Did anyone Is it fair? understand? tamper with it? Is it accountable? Our vision for Trusted AI Pillars of trust, woven into the lifecycle of an AI application FAIRNESS EXPLAINABILITY ROBUSTNESS ASSURANCE supported by an instrumented platform AIOpenScale So let`s start with vulnerability detectionAIOps of Models? Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor Is the model vulnerable to DeployedDeployed Validate and Deploy ModelModel adversarial attacKs? Enter: Adversarial Robustness Toolbox AIOps Assign Hyperparameters Initial Model and Train Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Monitor DeployedDeployed Validate and Deploy ModelModel ART The Adversarial Robustness Toolbox contains implementations of the following attacks: Deep Fool (Moosavi-Dezfooli et al., 2015) IBM Adversarial Robustness Fast Gradient Method (Goodfellow et al., 2014) Jacobian Saliency Map (Papernot et al., 2016) Universal Perturbation (Moosavi-Dezfooli et al., 2016) Virtual Adversarial Method (Moosavi-Dezfooli et al., Toolbox 2015) C&W Attack (Carlini and Wagner, 2016) NewtonFool (Jang et al., 2017) https://github.com/IBM/adversarial-robustness- toolbox The following defense methods are also supported: Feature squeezing (Xu et al., 2017) Spatial smoothing (Xu et al., 2017) Label smoothing (Warde-Farley and Goodfellow, 2016) ART Adversarial training (Szegedy et al., 2013) ART is a library dedicated to adversarial Virtual adversarial training (Miyato et al., 2017) machine learning. Its purpose is to allow rapid crafting and analysis of attack and defense methods for machine learning models. The Adversarial Robustness Toolbox provides an implementation for many state-of-the-art methods for attacking and defending classifiers. 31 Implementation for state-of-the-art methods for attacking and defending classifiers. Evasion attacks Evasion defenses Poisoning detection Robustness metrics • FGSM • Feature squeezing • Detection based on • CLEVER clustering activations • JSMA • Spatial smoothing • Empirical robustness • Proof of attack strategy • BIM • Label smoothing • Loss sensitivity • PGD • Adversarial training Evasion detection • Carlini & Wagner • Virtual adversarial Unified model API training • Detector based on • DeepFool inputs • Training • Thermometer encoding • NewtonFool • Detector based on • Prediction • Gaussian data activations • Universal perturbation augmentation • Access to loss and prediction gradients 32 ART Demo: https://art-demo.mybluemix.net/ 33 https://developer.ibm.com/patterns/integrate-adversarial-attacks-model-training-pipeline/ March 30 2018 / © 2018 IBM Corporation 34 https://developer.ibm.com/jp/patterns/integrate-adversarial-attacks-model-training-pipeline/ March 30 2018 / © 2018 IBM Corporation 35 Robustness check accomplished. How doAIOps we check for bias throughout lifecycle? Assign Hyperparameters Initial Model and Train Is the dataset biased? Create Model PreparedPrepared andand Trained AnalyzedAnalyzed Model DataData Are predictions Are model weights biased? Monitor biased? DeployedDeployed Validate and Deploy ModelModel Unwanted bias and algorithmic fairness Machine learning, by its very nature, is always a form of statistical discrimination Discrimination becomes objectionable when it places certain privileged groups at systematic advantage and certain unprivileged groups at systematic disadvantage Illegal in certain contexts Unwanted bias and algorithmic fairness Machine learning, by its very nature, is always a form of statistical discrimination Unwanted bias in training data yields models with unwanted bias that scale out Prejudice in labels Undersampling or oversampling 39 40 Supported bias mitigation algorithms Optimized Preprocessing