ApacheCon Europe Nov 2016, Seville, Spain

SMART MANUFACTURING With Streaming and Deep Leaning

Prajod Vettiyattil, Wipro @prajods https://in.linkedin.com/in/prajod 2

AGENDA

• Introduction • Smart Manufacturing • Applications • solution • CNN • Deeplearning4j (DL4J) • Apache Spark • Deploying

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 3

INTRODUCTION

• Industrialization • Automation • Scale • Commoditization • Accessibility

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 4

MEN AND MACHINES

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 5

KEY POINT

Machines can do what humans can’t

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 6

SMART MANUFACTURING

• Holistic view of entire process • Sourcing to product sale • Efficiency • Eco friendliness • Predict over react • Autonomous machines • Learning and AI • Use data and analytics

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 7

APPLICATIONS

• Detecting overheating • Equipment • Finished products • Glass defect detection

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 8

OVERHEATING

• Detect temperatures above defined boundaries

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 9

COOLING TOWER

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 10

ELECTRIC MOTOR

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 11

ELECTRIC MOTOR: OVERHEATING

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 12

OVERHEATING: LOWER RESOLUTION

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 13

ELECTRIC MOTOR: NORMAL VIEW

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 14

GLASS DEFECT

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 15

ZOOMING IN

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 16

UNDER A DIFFERENT LIGHT

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 17

AUTOMATING DETECTION

• Machine learning • Multivariate Analysis • Multiple categories of defects • Deep Learning

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 18

DEEP LEARNING: WHAT

• Neural Network based Machine Learning • Neural Network • 1 Input layer • 1 or more hidden layers • 1 Output layer • Basic unit of NN • Neuron or • Combine in many ways using multiple parameters • Model parameters • Hyperparameters Image from: ucalgary.ca

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 19

DEEP LEARNING: WHY

• Can discern complex input patterns • Higher accuracy than most other ML methods • for image analysis • Ideally suited for spatial data analysis • Higher time needed for training • Parallel execution • Higher level architectures for evolving needs • CNN, , LSTM

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata 20 @prajods

CONVOLUTIONAL NEURAL NETWORKS 21

CONVOLUTIONAL NEURAL NETWORK

• Convolutional Neural Network (CNN) • Convolve: mix two functions • multiplication and addition • Dot product • Pass many filters over an image • Image features • Images are 3D features • Discern the important features/signals • Filter = kernel

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 22

CNN: APPLICATION

• Self driving cars • Robotics • Drones • Industrial automation • Physical security • Medical labs • Wherever images or videos are used

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 23

IMAGE FOR CONVOLUTION

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 24

CONVOLUTION Filter matrix Image

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 25

CNN ARCHITECTURE

…..

)

)

etc

etc

tanh

tanh

probabiities

Pooling

Pooling

Convolution

Convolution

linear transform linear

linear transform linear

st

(RELU, (RELU,

(RELU, (RELU,

th

1

N

Output Output

Non

Non Fully connected Fullyconnected layers

Feature extraction + dimension reduction Classification

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 26

CNN PARAMETERS

• Model parameters • Hyperparameters • Filter values • Activation function • Weights • # layers • Biases • # nodes in each layer • Filter size and count • Regularization • Convolution layers • Maxpooling layers

Learned by NN Experience of designer

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 27

CNN: SAMPLE CODE

• MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .seed(seed).iterations(iterations).regularization(false).l2(0.003).activation(“relu") .learningRate(0.0007).weightInit(WeightInit.XAVIER) .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .updater(Updater.RMSPROP).momentum(0.9).list() .layer(0, convInit("cnn1", channels, 50 , new int[]{5, 5}, new int[]{1, 1}, new int[]{0, 0}, 0)) .layer(1, maxPool("maxpool1", new int[]{2,2})) …. .layer(4, new DenseLayer.Builder().nOut(500).build()) .layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD) .nOut(numLabels).activation("softmax").build()) .backprop(true).pretrain(false).cnnInputSize(height, width, channels).build();

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 28

DEEPLEARNING4J (DL4J)

• Deep learning • Open source • Apache 2.0 license • Java based • Distributed execution • Runs on Spark and Hadoop

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 29

DEEPLEARNING4J: FEATURES

• ND4J • N Dimensional Arrays/Tensors • Like numpy • Canova/DataVec • Data extraction, vectorization • Arbitrer • Evaluate and tune models • CPU or GPU • OpenBLAS, Intel MKL, CUDA

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 30

DEEPLEARNING4J: SUPPORT

• Community support • https://gitter.im/deeplearning4j/deeplearning4j • Commercial support • www.skymind.io

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 31

APACHE SPARK

• Incubated in 2009 at Berkeley University • 100s of contributors • Yahoo, Intel, UC Berkeley,…,50+ orgs • 10 to 100 times faster than Hadoop MR • https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 32

APACHE SPARK

• Complements Hadoop • Replaces Hadoop MR • Adds • In memory processing • Stream processing • Interactive queries • YARN or Mesos for clustering • Java, Scala, Python, R

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 33

SPARK: LOGICAL VIEW

Spark SQL Spark Streaming MLlib GraphX

Apache Spark Core

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 34

SPARK: DEPLOYMENT VIEW

Cache TaskTaskTask ExecutorExecutor Spark Driver Worker Node

Spark’s Cluster Cache Manager TaskTaskTask ExecutorExecutorExecutor Master Node Worker Node

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 35

PARALLEL TRAINING: SPARK + DL4J

Calculate model 1 parameters

Mini Averaged Data Split Data Batch Batch 1 parameters Spark Worker Node 1 from

workers Data

Spark Calculate

Mini Master Node model 2

Batch Batch 2 parameters Data Split Data Spark Worker Node 2

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata 36 @prajods

THE MACHINE LEARNING PROCESS 37

A LONG JOURNEY

Select problem Classify ML type Filter data Iterate data sets

Non ML solution ? Select 2 or 3 algo Convert data Evaluate perform

Analyze process Pick one Vectorize Validation set

Data available ? Gather data Select hyperparams Deploy

Define success KPI Analyze data Iterate hyperparams Continuous eval

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 38

SELECT PROBLEM

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 39

GATHER AND FORMAT DATA

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 40

PROGRAM

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 41

ITERATE WITH HYPERPARAMETERS AND DATA

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 42

CONTINUOUS EVALUATION

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 43

SUMMARY

• Automation and commoditization • Humans and machines • Beyond human capacity • Application to product quality inspection • Deep leaning and images: higher accuracy • CNN • DL4J + Spark • The Machine Learning process

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 44

REFERENCES

• https://en.wikipedia.org/wiki/Smart_manufacturing • http://karpathy.github.io/neuralnets/ • http://cs231n.github.io/convolutional-networks/ • https://deeplearning4j.org/convolutionalnets.html • https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ • https://en.wikipedia.org/wiki/Glass_production • http://www.qorpak.com/pages/lab-glass-defects-inspect-packaging • https://en.wikipedia.org/wiki/Automated_analyser

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 45

QUESTIONS

Prajod Vettiyattil, Wipro @prajods https://in.linkedin.com/in/prajod

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods