Introduction Why to use (deep) neural networks? Types of deep neural networks Practical examples of deep neural networks
Deep Neural Networks Convolutional Neural Networks
René Pihlak
Department of Software Sciences School of Information Technologies Tallinn University of Technology
April 29, 2019
...... René Pihlak CNN Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Table of Contents
1 Introduction Types of training Self-introduction Types of structures Topics to cover Convolutional neural network 2 Why to use (deep) neural networks? 4 Practical examples of deep neural Description networks Comparision Road defect detection Popular frameworks YOLO3: darknet 3 Types of deep neural networks Estonian sign language
...... René Pihlak CNN 2nd year Master’s student
Current: Department of Software Sciences Past: Member of Board, Tiigrihüppe SA (now HITSA)
STUDIES:
WORK:
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
...... René Pihlak CNN Current: Department of Software Sciences Past: Member of Board, Tiigrihüppe SA (now HITSA)
2nd year Master’s student WORK:
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
STUDIES:
...... René Pihlak CNN Current: Department of Software Sciences Past: Member of Board, Tiigrihüppe SA (now HITSA)
WORK:
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
STUDIES: 2nd year Master’s student
...... René Pihlak CNN Current: Department of Software Sciences Past: Member of Board, Tiigrihüppe SA (now HITSA)
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
STUDIES: 2nd year Master’s student WORK:
...... René Pihlak CNN Past: Member of Board, Tiigrihüppe SA (now HITSA)
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
STUDIES: 2nd year Master’s student WORK: Current: Department of Software Sciences
...... René Pihlak CNN Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Background
STUDIES: 2nd year Master’s student WORK: Current: Department of Software Sciences Past: Member of Board, Tiigrihüppe SA (now HITSA)
...... René Pihlak CNN Why “deep neural networks”? What kind of deep neural networks? Practical uses of deep neural networks.
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Topics
...... René Pihlak CNN What kind of deep neural networks? Practical uses of deep neural networks.
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Topics
Why “deep neural networks”?
...... René Pihlak CNN Practical uses of deep neural networks.
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Topics
Why “deep neural networks”? What kind of deep neural networks?
...... René Pihlak CNN Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Topics
Why “deep neural networks”? What kind of deep neural networks? Practical uses of deep neural networks.
...... René Pihlak CNN What are today’s main topics? Are there in-class programming tasks today?
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Recap of Introduction
...... René Pihlak CNN Are there in-class programming tasks today?
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Recap of Introduction
What are today’s main topics?
...... René Pihlak CNN Are there in-class programming tasks today?
Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Recap of Introduction
What are today’s main topics?
...... René Pihlak CNN Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Recap of Introduction
What are today’s main topics? Are there in-class programming tasks today?
...... René Pihlak CNN Introduction Why to use (deep) neural networks? Self-introduction Types of deep neural networks Topics to cover Practical examples of deep neural networks Recap of Introduction
What are today’s main topics? Are there in-class programming tasks today?
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Table of Contents
1 Introduction Types of training Self-introduction Types of structures Topics to cover Convolutional neural network 2 Why to use (deep) neural networks? 4 Practical examples of deep neural Description networks Comparision Road defect detection Popular frameworks YOLO3: darknet 3 Types of deep neural networks Estonian sign language
...... René Pihlak CNN something to do with brain and nerves machine learning tool … applied statistics and probability theory
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer
h_1
in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN machine learning tool … applied statistics and probability theory
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer something to do with brain and nerves h_1
in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN tool … applied statistics and probability theory
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer something to do with brain and nerves h_1 machine learning
in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN … applied statistics and probability theory
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer something to do with brain and nerves h_1 machine learning tool in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN applied statistics and probability theory
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer something to do with brain and nerves h_1 machine learning tool in_1 h_2 … out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What are deep neural networks
input layer hidden layer output layer something to do with brain and nerves h_1 machine learning tool in_1 h_2 … out_1 applied statistics and probability theory in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN military and national security banks facebook/google smart phone camera applications speed cameras hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
...... René Pihlak CNN banks facebook/google smart phone camera applications speed cameras hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security
...... René Pihlak CNN facebook/google smart phone camera applications speed cameras hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks
...... René Pihlak CNN smart phone camera applications speed cameras hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks facebook/google
...... René Pihlak CNN speed cameras hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks facebook/google smart phone camera applications
...... René Pihlak CNN hospitals …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks facebook/google smart phone camera applications speed cameras
...... René Pihlak CNN …
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks facebook/google smart phone camera applications speed cameras hospitals
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Neural networks in everyday life
military and national security banks facebook/google smart phone camera applications speed cameras hospitals …
...... René Pihlak CNN reconnaissance car armored vehicle (background noise)
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Should we use (deep) neural networks?
...... René Pihlak CNN reconnaissance car armored vehicle (background noise)
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Should we use (deep) neural networks?
Identifying military vehicles:
...... René Pihlak CNN armored vehicle (background noise)
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Should we use (deep) neural networks?
Identifying military vehicles: reconnaissance car
...... René Pihlak CNN (background noise)
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Should we use (deep) neural networks?
Identifying military vehicles: reconnaissance car armored vehicle
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Should we use (deep) neural networks?
Identifying military vehicles: reconnaissance car armored vehicle (background noise)
...... René Pihlak CNN solution for a problem one of many tools
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What is the the purpose of using neural networks?
...... René Pihlak CNN one of many tools
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What is the the purpose of using neural networks?
solution for a problem
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks What is the the purpose of using neural networks?
solution for a problem one of many tools
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC Trained incrementally.
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC Trained incrementally. Lack of standards.
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC Trained incrementally. Lack of standards. Often the best solution currently available.
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC Trained incrementally. Lack of standards. Often the best solution currently available. Costly and/or time consuming.
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Pros & Cons
OK? TOPIC Trained incrementally. Lack of standards. Often the best solution currently available. Costly and/or time consuming. Buzzword.
...... René Pihlak CNN Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
...... René Pihlak CNN Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/
...... René Pihlak CNN Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/
...... René Pihlak CNN Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/
...... René Pihlak CNN OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/
...... René Pihlak CNN Eclipse Deeplearning4j: https://deeplearning4j.org/
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Some of the frameworks
Keras: https://keras.io/ Theano: http://www.deeplearning.net/software/theano/ Tensorflow: https://www.tensorflow.org/ Caffe: http://caffe.berkeleyvision.org/ OpenNN: http://www.opennn.net/ Eclipse Deeplearning4j: https://deeplearning4j.org/
...... René Pihlak CNN What is deep neural network? Is it the one and only tool for all problems? Does it have disadvantages? Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
...... René Pihlak CNN Is it the one and only tool for all problems? Does it have disadvantages? Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network?
...... René Pihlak CNN Is it the one and only tool for all problems? Does it have disadvantages? Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network?
...... René Pihlak CNN Does it have disadvantages? Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems?
...... René Pihlak CNN Does it have disadvantages? Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems?
...... René Pihlak CNN Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems? Does it have disadvantages?
...... René Pihlak CNN Which are the most popular frameworks?
Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems? Does it have disadvantages?
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems? Does it have disadvantages? Which are the most popular frameworks?
...... René Pihlak CNN Introduction Description Why to use (deep) neural networks? Comparision Types of deep neural networks Popular frameworks Practical examples of deep neural networks Recap of reasons to use neural networks
What is deep neural network? Is it the one and only tool for all problems? Does it have disadvantages? Which are the most popular frameworks?
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Table of Contents
1 Introduction Types of training Self-introduction Types of structures Topics to cover Convolutional neural network 2 Why to use (deep) neural networks? 4 Practical examples of deep neural Description networks Comparision Road defect detection Popular frameworks YOLO3: darknet 3 Types of deep neural networks Estonian sign language
...... René Pihlak CNN supervised learning unsupervised learning reinforcement learning
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Learning
...... René Pihlak CNN unsupervised learning reinforcement learning
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Learning
supervised learning
...... René Pihlak CNN reinforcement learning
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Learning
supervised learning unsupervised learning
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Learning
supervised learning unsupervised learning reinforcement learning
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Deep neural networks as a structure
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Deep neural networks as a structure
Traditional input layer hidden layer output layer
h_1
in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Deep neural networks as a structure
Traditional input layer hidden layer output layer
h_1 Autoencoder encoder decoder
in_1 out_1
in_1 h_2 en_1 de_1
out_1 in_2 h_1 out_2
in_2 h_3 en_2 de_2
out_2 in_3 h_2 out_3
en_3 in_3 h_4 de_3
in_4 out_4
h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Deep neural networks as a structure
Traditional input layer hidden layer output layer Computational graph
hi_1 h_1 Autoencoder encoder decoder in_1 out_1
in_1 out_1 hi_2
in_1 h_2 en_1 de_1 in_2 out_2
out_1 in_2 h_1 out_2 hi_3 h_1
in_2 h_3 en_2 de_2 hi_4 h_2 out_3 out_2 in_3 h_2 out_3 in_3
en_3 de_3 in_3 h_4 hi_5 h_3 out_4
in_4 out_4 in_4
hi_6 h_4 h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Deep neural networks as a structure
Traditional input layer hidden layer output layer Computational graph
hi_1 h_1 Autoencoder encoder decoder in_1 out_1
in_1 out_1 hi_2 in_1 h_2 en_1 de_1 in_2 out_2 Recurrent
out_1 in_2 h_1 out_2 hi_3 h_1
in_2 h_3 en_2 de_2 in_1 r_1 out_1 hi_4 h_2 out_3 out_2 in_3 h_2 out_3 in_3
en_3 de_3 in_3 h_4 hi_5 h_3 out_4
in_4 out_4 in_4
hi_6 h_4 h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network shapes
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network shapes
Traditional input layer hidden layer output layer
h_1
in_1 h_2
out_1
in_2 h_3
out_2
in_3 h_4
h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network shapes
Traditional input layer hidden layer output layer
h_1 Autoencoder encoder decoder
in_1 out_1
in_1 h_2 en_1 de_1
out_1 in_2 h_1 out_2
in_2 h_3 en_2 de_2
out_2 in_3 h_2 out_3
en_3 in_3 h_4 de_3
in_4 out_4
h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network shapes
Traditional input layer hidden layer output layer Computational graph
hi_1 h_1 Autoencoder encoder decoder in_1 out_1
in_1 out_1 hi_2
in_1 h_2 en_1 de_1 in_2 out_2
out_1 in_2 h_1 out_2 hi_3 h_1
in_2 h_3 en_2 de_2 hi_4 h_2 out_3 out_2 in_3 h_2 out_3 in_3
en_3 de_3 in_3 h_4 hi_5 h_3 out_4
in_4 out_4 in_4
hi_6 h_4 h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network shapes
Traditional input layer hidden layer output layer Computational graph
hi_1 h_1 Autoencoder encoder decoder in_1 out_1
in_1 out_1 hi_2 in_1 h_2 en_1 de_1 in_2 out_2 Recurrent
out_1 in_2 h_1 out_2 hi_3 h_1
in_2 h_3 en_2 de_2 in_1 r_1 out_1 hi_4 h_2 out_3 out_2 in_3 h_2 out_3 in_3
en_3 de_3 in_3 h_4 hi_5 h_3 out_4
in_4 out_4 in_4
hi_6 h_4 h_5
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN kernels: pooling:
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN pooling:
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
kernels: h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN pooling:
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
kernels: filters h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
kernels: filters pooling: h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Convolutional neural network as structure
kernels: filters pooling: reduction h_3 o_2
h_2 o_1
pooling h_1 convolution
input image conv kernels feature maps feature maps fully connected output (28 × 28 × 3) (7 × 7 × 3) (28 × 28 × 3 × 3) (14 × 14 × 3 × 3)
...... René Pihlak CNN Which are the three typical learning paradigms? Which are the four most common structures? What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
...... René Pihlak CNN Which are the four most common structures? What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms?
...... René Pihlak CNN Which are the four most common structures? What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms?
...... René Pihlak CNN What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures?
...... René Pihlak CNN What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures?
...... René Pihlak CNN What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures? What is the purpose of convolutional kernels?
...... René Pihlak CNN What effects are provided by pooling? Why is it important?
Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures? What is the purpose of convolutional kernels?
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures? What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
...... René Pihlak CNN Introduction Types of training Why to use (deep) neural networks? Types of structures Types of deep neural networks Convolutional neural network Practical examples of deep neural networks Recap of types of deep neural networks
Which are the three typical learning paradigms? Which are the four most common structures? What is the purpose of convolutional kernels? What effects are provided by pooling? Why is it important?
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Table of Contents
1 Introduction Types of training Self-introduction Types of structures Topics to cover Convolutional neural network 2 Why to use (deep) neural networks? 4 Practical examples of deep neural Description networks Comparision Road defect detection Popular frameworks YOLO3: darknet 3 Types of deep neural networks Estonian sign language
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Detecting defects
Deep Learning for Detection of Pavement Distress using Nonideal Photographic Images
Aleksei Tepljakov†, Andri Riid?, Rene Pihlak†, Kristina Vassiljeva†, Eduard Petlenkov†
†Department of Computer Systems, Tallinn University of Technology, Tallinn, Estonia ?Department of Software Science, Tallinn University of Technology, Tallinn, Estonia Email (corresponding author): aleksei.tepljakov at taltech.ee
Abstract—In this paper, a deep learning approach for detecting systems, location based solutions and cartography—developed pavement distress from nonideal photographic images of the a fast-speed mobile mapping system employing six high- road is investigated. Due to inconsistent data quality, part of resolution cameras for the recording the panoramic images of the associated machine learning challenge is to produce training and validation data that bears coherent information sufficient roads. The data (panoramic images and orthophotos assembled for the task of successfully training a deep convolutional neural from the panoramic images) collected by the system are further
network that provides required detection performance. In the visually inspected, detected deformations. . . . . are. . localized...... and...... paper, the proposed method for detecting pavement distress is digitized. All this is carried out. . manually. . . . . by. . experienced...... described. Work-in-progress experimental results are reported René Pihlak operators.CNN The resulting information is used by the Estonian and analyzed. Keywords—Pavement distress; defect detection; image recogni- Road Administration via a web application called EyeVi. tion; image processing; deep neural network. As a result of image acquisition, processing, and manual classification, the following data was made available for further I.INTRODUCTION analysis: two datasets containing more than a 1000 ortophotos Condition of roads is easily one of the more important signs with sufficiently high pixel resolution; each image contains of economical standard and general well-being in a given three companion files: a mask used to extract the road, and a country or region. Early detection and repair of pavement .vrt file containing geographical data about the image; manually defects avoids further degradation and brings down the overall added defect information in the form of .shp files. For road road maintenance cost. Efficient and timely road inspection is extraction, additional data in the form of camera shooting points therefore one of the key elements of a successful pavement is provided as well—that is, information about the location management system. Yet, periodical road surveys tend to be of the camera as it was taking the pictures. This is the initial rather costly and time-consuming if carried out in the traditional data which must be processed in an appropriate way such that way, i.e. by human visual inspection of the road surface. information about road distress is extracted from the original In recent years, automatic image-based road distress evalu- images. ation has become an option [1]. Although it is still an open B. Literature review research problem and subject to environmental conditions such as illumination level, shadows cast by nearby objects, etc., Over the last decade, a lot of research has addressed the great progress has been made in this area and various methods problem of automating the detection of pavement distress. ranging from filtering and thresholding to artificial neural Recently, computer vision and machine learning techniques, networks have been employed to carry out the task. as well as remote sensing devices with advanced pattern Public infrastructure undergoes aging as well as degradation recognition, are widely used not only in (open) academic due to weather conditions. The present research is motivated research projects, but also in commercial solutions based on by the fact that in Estonia the daily temperature can fluctuate proprietary software algorithms [2], [3]. In general, systems around 0◦ C for more than five months a year. Therefore, ice with good detection performance combine several approaches and snow melt during the day and freeze again by night. This for data acquisition and measurement. For example, LiDAR leads to expansion of cracks and other defects thus requiring technology allows to obtain a subsurface profile with all frequent road inspections. elevation information in addition to discovering changes in the properties of material [4]. Laser based systems provide A. Problem setting and initial data possibility of automatic analysis for surface characteristics such Toward the goal of efficient analysis of roads, Reach- as evenness and skid resistance. These otherwise excellent U Ltd.—a company specializing in geographic information solutions have one important drawback: most of such systems operate at low speed, i.e. under 10km/h. This low speed This research endeavor was partially supported by Archimedes Foundation decreases road traffic safety [5]. and Reach-U Ltd. in the scope of the smart specialization research and development project #LEP18032: “A methodology for computerised detection As a good example of such complex systems one can of pavement cracks and other road defects”. consider ARAN 9000 of University of Catania and mobile Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Detecting the road
• a large number of PNG image files. In the former case, additional data is saved with each image segment, such as presence and type of defects, geographic coordinates (thus allowing to locate the problematic segment on real world maps) and location on the entire image (so that the original image could be reconstructed later, if necessary). The partitioning algorithm extracts segments based on a simple grid and also makes use of oversampling to ensure all segments are captured, though some pixels may be represented in the resulting dataset more than once. There is also an option to ignore segments that are partially masked.
...... Fig. 1. Two instances of road area extraction. Green region — 500 pixel. . wide. . . . . D.. . . Classification...... performance evaluation line of joined camera points (initialRené Pihlak road areaCNN mask). Blue region — refined estimate of the road area obtained by backprojection (initial foreground for GrabCut). Violet region — initial background for GrabCut. Red line — borders There are four possible outcomes concerning the judgments of the road area delivered by GrabCut. Low color content of the lower image of road segments given by the classification system: in the figure implies that the lighting conditions were poor while shooting the panoramic images. • true negative (TN) — there is no defect and the system does not detect a defect; • true positive (TP ) — there is a defect and the system correctly detects it; 4) Application of GrabCut. GrabCut is one of the most • false negative (FN) — there is a defect, but the system powerful image segmentation techniques. The image is does not detect it; separated into initial foreground (using the refined road • false positive (FP ) — there is no defect, but the system mask) and background (far edges of the orthophoto) detects a defect. regions. Both the foreground and background regions Based on this, it is possible to impose accuracy criteria for are modeled as Gaussian Mixture Models. The GrabCut the system, where TN, TP , FN, and FP denote the total algorithm learns the color distributions by giving each counts of corresponding detection outcomes. In the case of this pixel a probability of belonging to the most feasible work, the requirements are recall which shows the percentage Gaussian component in one of the foreground or back- of how many actual defects were detected by the system and ground GMMs. A graph is built from the image and is defined as the final segmentation is performed using the iterative TP minimization algorithm of the graph cut to get a new Recall = (1) TP + FN classification of foreground and background pixels. The approach is sufficiently reliable, particularly on well- and precision which shows the percentage of how many of illuminated images where the road is visually distinguishable detected defects were actual defects and defined as from its shoulder and is not disturbed by occasional shadows TP Precision = . (2) (Fig. 1). TP + FP C. Image partitioning We also use the so called Matthews correlation coefficient Since the original images are of high resolution, they are (MCC) metric which is defined as partitioned into smaller fragments that we shall refer to as segments. The idea is to study the contents of each segment MCC = and to determine whether it depicts a pavement defect or not. TP · TN − FP · FN (3) The total of all of these segments forms the basis for training p(TP + FP )(TP + FN)(TN + FP )(TN + FN) the artificial neural network. Segments are extracted from each image automatically using because it provides a more reliable measure in case of the defect data made available by the company and are marked imbalanced data. with corresponding labels depending on the presence of a Finally, since convnets return probabilities and not discrete defect of a certain type or lack thereof. The resulting dataset values, one must use two threshold values: detection threshold may also be augmented as needed, i.e., the number of images Pdet and suspicion threshold Psus such that depicting the defects is artificially increased by applying various transformations to existing images such as translation and P (defect) > Pdet ⇒ defect is detected rotation (in theory, this should improve the efficiency of the convnet training). Finally, the segments are either exported into and • specific data placeholders—files of HDF5 format; P (defect) > Psus ⇒ defect is suspected. Introduction 2802 files containing various defects. In the latter case, more functionality in two separate PythonRoad defect libraries detection and also has Why to use (deep) neural networks? than half of the initially generated images were removed after graphical user interfaces developedYOLO3: using darknetPyQt5. The intended Types of deep neural networks manual inspection. end-user application entitled nnapplyEstonianincludes sign language a graphical user Practical examples of deep neural networks The final performance metrics of the best performing interface and allows to process arbitrary road images using the networks can be seen in Table I. These results are obtained by trained convnets showing the detected and suspected defects. feeding all the imagesDetecting from the dataset containing defects: 887 images resultsIt also generates a report in Microsoft Excel format. Examples into the partitioning algorithm and later to the each convnet for of both types of output from this application are depicted in validation. In this case, only segments taken from the pavement Figs. 4 and 5. part of the image are considered (this is obtained using the pavement extraction algorithm).
Table I. Validation: performance metrics of the best obtained convnets computed over the entire set of partitioned images.
Convnet Accuracy Precision Recall MCC
Trained from scratch? 0.96 0.17 0.43 0.25
Trained from scratch and 0.97 0.22 0.35 0.26 fine-tuned
VGG16 net fine-tuned v1 0.94 0.13 0.51 0.23
VGG16 net fine-tuned v2 0.97 0.23 0.39 0.29
VGG16 net 0.96 0.20 0.46 0.29 distortion-tuned Fig. 4. Example image with defect location suggestions generated by nnapply application ?training figures are 0.98 for precision and 0.90 for recall.
...... Several observations can be made. First, it is clear that the ...... accuracy parameter is not helpful in the assessment of the René Pihlak CNN obtained convnet’s performance. Indeed, rather high accuracy figures are not as relevant for real world applications as values of precision, recall, and MCC. Second, the performance figures are rather low, thus indicating the limited capacity of the convnets to successfully assess pavement distress in a reliable manner. The main reason for this is seen in the fact that the training data may not be sufficient and its quality may be inadequate for training of a general enough convnet. In addition, soft and hard shadows present in some of the images as well as foreign objects may also contribute toward the loss of performance [21]. Manual inspection of failed prediction cases also strongly points toward the inability of the neural Fig. 5. Example report concerning potential pavement defects generated by network to differentiate clean road edges from road edges with nnapply application defects. Next, comparing the fine-tuned VGG16 networks to the distortion-tuned one it is evident that it is possible to improve detection performance to some extent. This result hints at the IV. DISCUSSION fact that the initial image set contains only a limited set of images fit for either training or fine-tuning a coherent convnet. From the analysis of results, it is clear that a convnet suitable Therefore, further processing and augmentation of the dataset for reliable detection of pavement distress has not yet been is necessary. obtained. According to the validation results in Table I, the obtained convnets can be used for preliminary analysis of new B. Software solution images in automatic mode, however, are not yet suitable for As a result of this research and development project, a unsupervised standalone operation. fully-fledged Python software package was developed for the Further research and development is required to obtain qual- Reach-U Ltd that can be used to generate the data for training ifying convnets satisfying the required criteria of precision > and inspect it; train the deep learning convnets and apply 0.95 and recall > 0.9. Toward this end, several approaches them to arbitrary new images. The package comprises backend have been considered: Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Detecting defects: alternative approach
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Detecting defects: alternative approach
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Detecting defects: in numbers
Layer (type) Output Shape Params Layer (type) Output Shape Params decoder_in (None, 112) 0 Layer (type) Output Shape Params encoder_in (None, 128, 128, 3) 0 dense_1 (None, 65536) 7405568 conv2d_1 (None, 128, 128, 32) 2432 reshape_1 (None, 32, 32, 64) 0 encoder_in (None, 128, 128, 3) 0 conv2d_2 (None, 64, 64, 32) 50208 conv2d_tran (None, 64, 64, 64) 200768 encoder (None, 112) 7493200 conv2d_3 (None, 32, 32, 64) 100416 conv2d_tran (None, 128, 128, 32) 100384 decoder (None, 128, 128, 3) 7734755 flatten_1 (None, 65536) 0 conv2d_tran (None, 128, 128, 32) 25632 Total params: 15,227,955 latent_vec (None, 112) 7340144 conv2d_tran (None, 128, 128, 3) 2403 Trainable params: 15,227,955 decoder_out (None, 128, 128, 3) 0 Non-trainable params: 0 Total params: 7,493,200 Trainable params: 7,493,200 Total params: 7,734,755 Non-trainable params: 0 Trainable params: 7,734,755 Non-trainable params: 0
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks YOLO3
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks YOLO3
https://pjreddie.com/darknet/yolo/
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Estonian sign language recognition
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Estonian sign language recognition
...... René Pihlak CNN Does the results from a neural network depend on the quality of the input data? Do all neural networks work perfectly on all hardware configurations?
Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Recap of examples
...... René Pihlak CNN Do all neural networks work perfectly on all hardware configurations?
Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Recap of examples
Does the results from a neural network depend on the quality of the input data?
...... René Pihlak CNN Do all neural networks work perfectly on all hardware configurations?
Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Recap of examples
Does the results from a neural network depend on the quality of the input data?
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Recap of examples
Does the results from a neural network depend on the quality of the input data? Do all neural networks work perfectly on all hardware configurations?
...... René Pihlak CNN Introduction Road defect detection Why to use (deep) neural networks? YOLO3: darknet Types of deep neural networks Estonian sign language Practical examples of deep neural networks Recap of examples
Does the results from a neural network depend on the quality of the input data? Do all neural networks work perfectly on all hardware configurations?
...... René Pihlak CNN THANK YOU FOR YOUR ATTENTION!
...... Interesting projects
Generative Adversarial Networks (GAN): GauGAN GAN 2.0
...... René Pihlak CNN