3/10/2021 Agenda TODAY Frameworks for Deep Learning. Implementing simple networks with Keras. Implementing LeNet-5 with Keras and Caffe. TOMORROW Implementing Deep Neural Networks with TensorFlow 1.x. TensorBoard. The TensorFlow Eager API. 02/03/2021 Overview of the Apollo Framework for Autonomous Driving. 03/03/2021 The Perception and Prediction modules of Apollo. 2 Frameworks for DNNs DNNs are typically developed, trained, and inferred by means of specific frameworks (i.e., toolkits, libraries), because they provide a simple high-level interface (mostly in Python); simplify the creation of new DNNs using a large set of pre-implemented layers (e.g., convolutions); allow training and inferring DNNs on different devices (e.g., CPUs, GPUs, mobile) by changing a few lines of code; exploit frequent library updates with new features; In addition, a lot of examples and pre-trained networks are available online. 4 Frameworks for DNNs Frameworks for DNNs The deep learning landscape is constantly changing, hence many frameworks have been developed during the last years, most of them open source: The first widely adopted framework. Created in 2007 at the Montreal Institute for Learning Algorithms (MILA), headed by Yoshua Bengio, one of the pioneers of deep learning. Developed in Python. Can run DNNs on either CPU or GPU architectures. Since 2017, it is no longer maintained, due to the release of Different frameworks have different features, hence no one is many other frameworks, developed by large companies. the best for all the problems. Let’s consider some of them. 5 6 1 3/10/2021 Frameworks for DNNs Different design choices (2013) by UC Berkeley Configuration (2015) by Google typical usage file C++ (2015) by Amazon Model (2016) (Computational Network Toolkit) by specification Python Microsoft. Now called Microsoft Cognitive Toolkit. Programmatic (2017) by Facebook as an evolution of Caffe. R generation (2017) by Facebook. Caffe2 and Pytorch are going to be integrated into a single platform. Java (2017) (Deep Learning for Java) by Skymind as a deep learning library for Java and Scala. 7 8 High-level APIs High-level APIs Besides these frameworks, there are also high-level interfaces that are wrapped around one or multiple frameworks. Released in 2017 by Amazon and supported by Microsoft. High-level Python deep learning interface . Released in 2016 by François Chollet (Google). It wraps MXNet and soon it will also include CNTK. Python-based library for fast experimentation with DNN. Gluon is a direct competitor for Keras. It runs on top of TensorFlow, CNTK, Theano, or PlaidML. User-friendly, modular, and extensible. Quote: "API designed for human beings, not machines". It allows creating a DNN by stacking layers without . Released in 2017 by Google’s DeepMind. specifying math operations, but only layer types. It is built on top of TensorFlow. 9 10 High-level APIs Frameworks at a glance Open Neural Network Exchange Open source deep learning frameworks (2017) . In 2017, Microsoft, Amazon, Facebook, and others launched ONNX, an Open Neural Network Exchange format to represent deep learning models and port them between different frameworks. ONNX enables models to be trained in one framework and transferred to another for inference. ONNX supports Caffe2, CNTK, MXNet, PyTorch, and GoogleMicrosoft Amazon Microsoft Facebook TensorFlow. 11 12 2 3/10/2021 The complete stack Popularity GitHub users can “star” a repository to indicate that they like it. Appl. DNN So GitHub Stars can measure how popular a project is. High-level ONNX TensorFlow APIs Keras Gluon Mid-level Tensor APIs Theano Caffe Flow PyTorch CNTK MXNet Language Python C++ Java Caffe GitHubStars PyTorch OS Windows Linux Android iOS CNTK MXNet HW CPU DSP GPU TPU 13 14 Combined metrics Power score computed by mixing several criteria, as articles, TensorFlow is among the most used framework, together with books, citations, Github activities, Google search volume, etc. Caffe and PyTorch. Power score 2018 Released in 2015 by Google Main features Large community and support TensorBoard visualization tool Scalability to many platforms Good library mgmt and updates Not so intuitive interface Slower than other frameworks 15 16 The TensorFlow Stack Convolution Architecture For Feature Extraction High-level Released in 2013 by UC Berkeley TF APIs Estimators Keras TF learn TF-sim Mid-level Main features Datasets Layers Metrics Losses TF APIs Excellent for CNNs for image processing. Language Python Frontend C++ While in TF the network is created by programming, in Caffe layers are created by specifying a set of parameters. TF runtime TensorFlow Distributed Execution Engine Quite fast compared to other frameworks. Operating Command line, Python, C++, and MatLab interface. Systems Windows Linux Android iOS Not so easy to learn. Hardware CPU DSP GPU TPU Not so good with recurrent neural networks and sequence 17 models. 18 3 3/10/2021 Comparing frameworks Released in 2017 by Facebook Tutorial CNN RNN Easy-to-use Multiple GPU Keras Languages Speed Material modeling modeling API support compatible Main features Theano Python, C++ ++ ++ ++ + ++ NO YES Support for dynamic graphs. This is efficient when the input Tensor- Python, YES YES varies, as in text processing. In 2017, TensorFlow introduced Flow C++ +++ +++ ++ +++ ++ Eager Execution, to evaluate Python, Interactive debugging. Pytorch operations immediately, without C++ + +++ ++ ++ +++ YES NO building graphs. Easier to get started. Python, Caffe C++ + +++ NO + + YES NO Blend of high-level and low-level APIs. Python, R, MXNet Julia, Scala ++ ++ + ++ ++ YES YES Limited documentation. CNTK Python, C++ + + +++ + ++ YES YES No graphic visualization tools. DL4J Java, Scala +++ +++ +++ ++ ++ YES YES 19 20 NVIDIA framework NVIDIA also provides support for DNN development, but only on top of their GPU platforms: Appl. DNN Mid-level APIs Caffe PyTorch Theano Optimiz. Libraries cuDNN cuBLAS Language CUDA HW GPU 21 22 Data Set Finders Data sets There are a lot of data sets on the internet for training DNNs. Natural images . MNIST: handwritten digits (yann.lecun.com/exdb/mnist/). The following are general data set repositories that allow . CIFAR10 / CIFAR100: 32×32 image dataset with 10 / 100 categories searching for the one you need: (www.cs.utoronto.ca/~kriz/cifar.html). Kaggle: www.kaggle.com . COCO: large dataset for object detection, segmentation, and captioning (cocodataset.org). UCI Machine Learning Repository: mlr.cs.umass.edu/ml . ImageNet: large image database organized according to the WordNet hierarchy (www.image-net.org). VisualData: www.visualdata.io . Pascal VOC: dataset for image classification, object detection, and . CMU Library: guides.library.cmu.edu/machine-learning/datasets segmentation (https://pjreddie.com/projects/pascal-voc-dataset-mirror/). COIL 20: 128×128 images of 20 objects taken at different rotation angles Data sets are usually divided into categories. (www.cs.columbia.edu/CAVE/software/softlib/coil-20.php). COIL100: 128×128 images of 100 objects taken at different rotation angles (www1.cs.columbia.edu/CAVE/software/softlib/coil-100.php). 23 24 4 3/10/2021 Data sets Data sets Faces Speech . Labelled Faces in the Wild: 13,000 images of faces collected from the . TIMIT Speech Corpus: DARPA Acoustic-Phonetic Continuous Speech web, labelled with the person name (vis-www.cs.umass.edu/lfw). Corpus for phoneme classification (github.com/philipperemy/timit). Olivetti: images of several people faces at different angles . Aurora: Timit with noise and additional information (aurora.hsnr.de). (www.cs.nyu.edu/~roweis/data.html). Sheffield: 564 images of 20 individuals each shown in a range of poses Music (https://www.sheffield.ac.uk/eee/research/iel/research/face). Piano-midi.de: classical piano pieces (www.piano-midi.de). Text . Nottingham: over 1000 folk tunes (abc.sourceforge.net/NMD). 20 newsgroups: Classification task to map word occurrences into 20 . MuseData: a collection of classical music scores (musedata.stanford.edu). newsgroup IDs (qwone.com/~jason/20Newsgroups). JSB Chorales: a dataset of four-part harmonized chorales . Penn Treebank: used for next word prediction or next character prediction (https://github.com/czhuang/JSB-Chorales-dataset). (corochann.com/penn-tree-bank-ptb-dataset-introduction-1456.html). FMA: a dataset For Music Analysis (github.com/mdeff/fma). Broadcast News: large text dataset used for next word prediction (https://github.com/cyrta/broadcast-news-videos-dataset). 25 26 MNIST CIFAR-10 MNIST is the most popular datasets of handwritten digits. It CIFAR-10 consists of 60,000 32x32 color images organized in contains a training set of 60,000 examples and a test set of 10 classes (6,000 images per class). There are 50,000 training 10,000 examples. images and 10,000 test images: airplane automobile bird cat deer dog frog Size: 28 x 28 Greylevels: 256 horse (0 black, 1 white) train[i][0] or test[i][0]: i-th example ship Label: 0, 1, …, 9 train[i][1] or test[i][1]: i-th label 27 truck 28 COCO Olivetti Faces COCO is a large-scale and rich dataset for object detection, Olivetti Faces is a dataset containing 400 images of faces of segmentation, and captioning. several people at different angles. 330,000 images . 1.5 million object instances . 80 object categories . Number of images: 400 . 91 stuff categories . Image size: 64x64 . 5 captions per image . Color depth: 8 bit, grayscale [0-255] . 250,000 people with keypoints 29 30 5 3/10/2021 TensorFlow and Keras First of all, it must be clarified that: . Keras is an open-source machine learning library that generates code for TensorFlow and other frameworks; . Since 2017, the Keras API is integrated in TensorFlow and can be used as tf.keras (in a Python environment) . So neural networks can be developed either under Keras or TensorFlow, using the high levelAPI provided by tf.keras: 32 Installing TF using Pip Installing Anaconda TensorFlow and Keras may be installed by using Pip, the Another way to install TensorFlow and Keras is by leveraging package manager of Python.