ApacheCon Big Data Europe Nov 2016, Seville, Spain
SMART MANUFACTURING With Apache Spark Streaming and Deep Leaning
Prajod Vettiyattil, Wipro @prajods https://in.linkedin.com/in/prajod 2
AGENDA
• Introduction • Smart Manufacturing • Applications • Deep learning solution • CNN • Deeplearning4j (DL4J) • Apache Spark • Deploying machine learning
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 3
INTRODUCTION
• Industrialization • Automation • Scale • Commoditization • Accessibility
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 4
MEN AND MACHINES
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 5
KEY POINT
Machines can do what humans can’t
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 6
SMART MANUFACTURING
• Holistic view of entire process • Sourcing to product sale • Efficiency • Eco friendliness • Predict over react • Autonomous machines • Learning and AI • Use data and analytics
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 7
APPLICATIONS
• Detecting overheating • Equipment • Finished products • Glass defect detection
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 8
OVERHEATING
• Detect temperatures above defined boundaries
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 9
COOLING TOWER
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 10
ELECTRIC MOTOR
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 11
ELECTRIC MOTOR: OVERHEATING
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 12
OVERHEATING: LOWER RESOLUTION
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 13
ELECTRIC MOTOR: NORMAL VIEW
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 14
GLASS DEFECT
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 15
ZOOMING IN
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 16
UNDER A DIFFERENT LIGHT
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 17
AUTOMATING DETECTION
• Machine learning • Multivariate Analysis • Multiple categories of defects • Deep Learning
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 18
DEEP LEARNING: WHAT
• Neural Network based Machine Learning • Neural Network • 1 Input layer • 1 or more hidden layers • 1 Output layer • Basic unit of NN • Neuron or Perceptron • Combine perceptrons in many ways using multiple parameters • Model parameters • Hyperparameters Image from: ucalgary.ca
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 19
DEEP LEARNING: WHY
• Can discern complex input patterns • Higher accuracy than most other ML methods • for image analysis • Ideally suited for spatial data analysis • Higher time needed for training • Parallel execution • Higher level architectures for evolving needs • CNN, RNN, LSTM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata 20 @prajods
CONVOLUTIONAL NEURAL NETWORKS 21
CONVOLUTIONAL NEURAL NETWORK
• Convolutional Neural Network (CNN) • Convolve: mix two functions • Matrix multiplication and addition • Dot product • Pass many filters over an image • Image features • Images are 3D features • Discern the important features/signals • Filter = kernel
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 22
CNN: APPLICATION
• Self driving cars • Robotics • Drones • Industrial automation • Physical security • Medical labs • Wherever images or videos are used
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 23
IMAGE FOR CONVOLUTION
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 24
CONVOLUTION Filter matrix Image
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 25
CNN ARCHITECTURE
…..
)
)
etc
etc
tanh
tanh
probabiities
Pooling
Pooling
Convolution
Convolution
linear transform linear
linear transform linear
st
(RELU, (RELU,
(RELU, (RELU,
th
1
N
Output Output
Non
Non Fully connected Fullyconnected layers
Feature extraction + dimension reduction Classification
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 26
CNN PARAMETERS
• Model parameters • Hyperparameters • Filter values • Activation function • Weights • # layers • Biases • # nodes in each layer • Filter size and count • Regularization • Convolution layers • Maxpooling layers
Learned by NN Experience of designer
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 27
CNN: SAMPLE CODE
• MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .seed(seed).iterations(iterations).regularization(false).l2(0.003).activation(“relu") .learningRate(0.0007).weightInit(WeightInit.XAVIER) .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .updater(Updater.RMSPROP).momentum(0.9).list() .layer(0, convInit("cnn1", channels, 50 , new int[]{5, 5}, new int[]{1, 1}, new int[]{0, 0}, 0)) .layer(1, maxPool("maxpool1", new int[]{2,2})) …. .layer(4, new DenseLayer.Builder().nOut(500).build()) .layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD) .nOut(numLabels).activation("softmax").build()) .backprop(true).pretrain(false).cnnInputSize(height, width, channels).build();
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 28
DEEPLEARNING4J (DL4J)
• Deep learning library • Open source • Apache 2.0 license • Java based • Distributed execution • Runs on Spark and Hadoop
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 29
DEEPLEARNING4J: FEATURES
• ND4J • N Dimensional Arrays/Tensors • Like numpy • Canova/DataVec • Data extraction, vectorization • Arbitrer • Evaluate and tune models • CPU or GPU • OpenBLAS, Intel MKL, Nvidia CUDA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 30
DEEPLEARNING4J: SUPPORT
• Community support • https://gitter.im/deeplearning4j/deeplearning4j • Commercial support • www.skymind.io
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 31
APACHE SPARK
• Incubated in 2009 at Berkeley University • 100s of contributors • Yahoo, Intel, UC Berkeley,…,50+ orgs • 10 to 100 times faster than Hadoop MR • https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 32
APACHE SPARK
• Complements Hadoop • Replaces Hadoop MR • Adds • In memory processing • Stream processing • Interactive queries • YARN or Mesos for clustering • Java, Scala, Python, R
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 33
SPARK: LOGICAL VIEW
Spark SQL Spark Streaming MLlib GraphX
Apache Spark Core
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 34
SPARK: DEPLOYMENT VIEW
Cache TaskTaskTask ExecutorExecutor Spark Driver Worker Node
Spark’s Cluster Cache Manager TaskTaskTask ExecutorExecutorExecutor Master Node Worker Node
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 35
PARALLEL TRAINING: SPARK + DL4J
Calculate model 1 parameters
Mini Averaged Data Split Data Batch Batch 1 parameters Spark Worker Node 1 from
workers Data
Spark Calculate
Mini Master Node model 2
Batch Batch 2 parameters Data Split Data Spark Worker Node 2
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata 36 @prajods
THE MACHINE LEARNING PROCESS 37
A LONG JOURNEY
Select problem Classify ML type Filter data Iterate data sets
Non ML solution ? Select 2 or 3 algo Convert data Evaluate perform
Analyze process Pick one Vectorize Validation set
Data available ? Gather data Select hyperparams Deploy
Define success KPI Analyze data Iterate hyperparams Continuous eval
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 38
SELECT PROBLEM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 39
GATHER AND FORMAT DATA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 40
PROGRAM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 41
ITERATE WITH HYPERPARAMETERS AND DATA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 42
CONTINUOUS EVALUATION
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 43
SUMMARY
• Automation and commoditization • Humans and machines • Beyond human capacity • Application to product quality inspection • Deep leaning and images: higher accuracy • CNN • DL4J + Spark • The Machine Learning process
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 44
REFERENCES
• https://en.wikipedia.org/wiki/Smart_manufacturing • http://karpathy.github.io/neuralnets/ • http://cs231n.github.io/convolutional-networks/ • https://deeplearning4j.org/convolutionalnets.html • https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ • https://en.wikipedia.org/wiki/Glass_production • http://www.qorpak.com/pages/lab-glass-defects-inspect-packaging • https://en.wikipedia.org/wiki/Automated_analyser
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods 45
QUESTIONS
Prajod Vettiyattil, Wipro @prajods https://in.linkedin.com/in/prajod
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods