Introduction to Deep Learning and Tensorflow

Introduction to Deep Learning and Tensorﬂow Day 1

MR ST LF FS SO IB

CINECA Roma - SCAI Department Marco Rorro Stefano Tagliaventi Luca Ferraro Francesco Salvadore Sergio Orlandini Isabella Baccarelli Rome, 7th-9th Nov 2018 Outline

Intro and ML Basics

Intro The Task The Performance The Experience

ML 1 Intro and ML Basics ingredients Data Model Objective f 2 Introduction Optimization

Tensorﬂow Graph and Session 3 A simple ML ingredients example

4 Tensorﬂow Outline

Intro and ML Basics

Intro The Task The Performance 1 The Experience Intro and ML Basics

ML ingredients Data 2 Introduction Model Objective f Optimization The Task

Tensorﬂow The Performance Measure Graph and Session The Experience A simple example

3 ML ingredients

4 Tensorﬂow The Big Picture

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example Machine learning basics

Intro and ML Basics

Intro The Task Machine learning is a form of applied statistic with increased The Performance The Experience emphasis on the use of computers to statistically estimate ML ingredients complicated functions and decreased emphasis on proving Data Model confidence intervals around these functions. Objective f Optimization • Machine learning is a subset of Artificial Intelligence that Tensorflow Graph and uses statistics to build models θ of data from data. Session A simple example • Machine learning systems use frequentist statistics, i.e estimate the true fixed models from uncertain observations. • Machine learning systems use Bayesian statistics, i.e find best model from families of models. • Machine learning is a matter of statistics Machine learning definition

Intro and ML Basics

Intro The Task The Performance The Experience ML • Arthur Samuel (1959). Machine Learning: Field of study that ingredients Data gives computers the ability to learn without being explicitly Model Objective f Optimization programmed. Tensorﬂow • Tom Mitchell (1998) Well-posed Learning Problem: A Graph and Session A simple computer program is said to learn from experience E with example respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E. Learning problems

Intro and ML Basics Playing checkers Intro The Task The Performance • The task T : playing checkers. The Experience

ML • The performance measure P : percent of games won against ingredients Data opponents. Model Objective f Optimization • The training experience E : playing practice games against Tensorﬂow itself. Graph and Session A simple example Recognize human faces in images • The task T : automatic human face recognition. • The performance measure P : percent of faces recognized. • The training experience E : learn from a database of face images. The Task

Intro and ML Basics Intro • Machine learning enable us to tackle tasks that are too The Task The Performance The Experience difﬁcult to solve with ﬁxed programs written and designed by

ML human beings. ingredients • Data fixed is referred to rule-based system where the decision Model Objective f process is defined at the design level of the algorithm. In such Optimization system output is deterministic with respect to the input. Tensorflow • Graph and probabilistic systems are in contrast to fixed programs. Session A simple • Machine learning deals with probabilistic systems. example • Time series forecasting • Computational linguistics • Game palying • Driving • The task shall not be confused with the learning phase, which instead is the procedure used to achieve the task. Tasks: Classification I

Intro and ML Basics • In this type of task, the computer program is asked to specify Intro The Task which of k categories some input belongs to. The Performance n The Experience f : R → {1, ..., k}. ML ingredients Map f could be a probability distribution over classes. Data Model A typical classification task is object recognition, where the Objective f Optimization input is an image. Tensorflow Graph and Session A simple example Tasks: Classification II

Intro and ML Basics

Intro The Task The Performance • The Experience Classiﬁcation with missing inputs: It is possible to

ML generalize the classiﬁcation task to take into account ingredients Data incomplete inputs or some missing variable. Instead of Model Objective f learning a single function, the system can learn a probability Optimization

Tensorflow distribution over all relevant variables, then solve the Graph and Session classification by marginalizing out the missing variables, i.e A simple example summing up over the unknown distributions to extract information on known ones. • A typical use case is related to medical diagnosis where some tests could be invasive or expensive. Tasks: Classification III

Intro and ML Basics

Intro The Task • Regression: In this type of task the program is asked to The Performance n The Experience predict a numerical value given some input. f : R → R Used ML ingredients for example in algorithmic trading. Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example Tasks: Structured output I

Intro and ML Basics • In structured output tasks output values are related to each Intro The Task The Performance other. One example is parsing a natural language sentence The Experience where the output is the tree that describes its grammatical ML ingredients structure as for example tagging verbs, nouns, adverbs and Data Model so on. Objective f Optimization • Transcription: In this type of task, the machine learning Tensorﬂow Graph and Session system is asked to observe a relatively unstructured A simple example representation of some kind of data and transcribe the information into discrete textual form. Examples are optical character recognition and speech recognition, where the computer program is asked to map a raw data (image pixels or waveform audio data ) to a sequence of characters. • Machine translation: Translate from symbols in some language to symbols in another language. TasksI

Intro and ML Basics • Anomaly detection: In this type of task, the computer Intro The Task program sifts through a set of events or objects and ﬂags The Performance The Experience some of them as being unusual or atypical. One example is ML ingredients credit card fraud detection, by modeling user purchasing Data Model habits. Objective f Optimization • Synthesis and sampling: In this type of task, the machine Tensorﬂow Graph and learning algorithm is asked to generate new examples that Session A simple example are similar to those into the training data. For example video games can use synthesis to automatically generate textures for large objects. Another example is the speech synthesis task, where given a written sentence an audio waveform is produced containing the spoken version with the requirements that there is no single correct output for each input. A large amount of variation in the output produce a more natural and realistic system. TasksII

Intro and • Denoising: Predict a clean output from a corrupted input. ML Basics • Intro Density estimation: In this kind of tasks the machine The Task The Performance learning algorithm is asked to learn a function The Experience n pmodel → where pmodel (x) can be interpreted as a ML R R ingredients probability density function(if x is continuous) or a probability Data Model Objective f mass function (if x is discrete) on the space that the Optimization examples where drawn from. In density estimation tasks the Tensorﬂow Graph and algorithm needs to learn the structure of the data. Session A simple example The Performance Measure,P

Intro and ML Basics Intro • To evaluate the abilities of a machine learning algorithm, we The Task The Performance The Experience must design a quantitative measure of its performance, ML usually related to the task T. ingredients Data • Model For classification tasks we can use the accuracy/error rate, Objective f Optimization i.e. the proportion of examples for which the model produces Tensorflow the correct/incorrect output. Graph and Session A simple • In machine learning field we are interested in evaluating the example performance on data that is not seen before. • We evaluate these performance measures using a test set of data that is separate from the data used for training the machine learning system. • It is often difficult to choose a performance measure corresponding well to the desired behavior of the system. The Experience, E I

Intro and ML Basics

Intro The Task The Performance The Experience ML Two main categories: ingredients Data Model • Unsupervised learning algorithms experience a dataset Objective f Optimization containing many features, then learn useful properties of the Tensorﬂow structure of this dataset. Graph and Session • In some contexts we wish to learn the entire probability A simple example distribution that generated a dataset • Implicitly : denoising, synthesis • Explicitly : density estimation The Experience, E II

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

• Supervised learning algorithm experience a dataset containing features, but each example is also associated with a label or target • Regression, classiﬁcation, structured output The Experience, E III

Intro and ML Basics

Intro The Task • Unsupervised and supervised are not formally defined terms, The Performance The Experience many ML technologies can be used to perform both. ML ingredients Data Model • Reinforced learning is another learning algorithm that Objective f Optimization interact with environment. Experience is not just made on a Tensorflow Graph and fixed dataset but there is a feedback between the algorithm Session A simple example and its experience. Outline

Intro and ML Basics

Intro The Task The Performance 1 Intro and ML Basics The Experience

ML ingredients Data 2 Introduction Model Objective f Optimization

Tensorﬂow 3 ML ingredients Graph and Session Data A simple example Model Objective function Optimization

4 Tensorﬂow ML ingredients

Intro and ML Basics

Intro The Task The Performance The Experience

ML • Data ingredients Data • The data input and output Model Objective f • Model Optimization

Tensorﬂow • The model used to generate output. Graph and Session • The objective function A simple example • The function used to train the model • Optimization Algorithm • The algorithm used to minimize loss function The dataI

Intro and ML Basics • One common way of describing an input dataset is with a Intro design matrix The Task The Performance • The Experience A matrix containing a different example in each row.

ML • Each columns of the matrix corresponds to a different feature. ingredients Data • For example, if we have 101 handwritten images 28x28 Model Objective f 101X784 Optimization representing digits, the design matrix is X ∈ R Tensorﬂow • A linear regression with 5 {y , x }, i = 1, 2, ..., 5 observation Graph and i i Session     A simple y1 1 x1 example y2 1 x2     β0 yi = β0 + β1xi y3 = 1 x3     β1 y4 1 x4 y5 1 x5 • In general if we have heterogeneous examples (not the same dimensions) an m row matrix {x(1), x(2), ..., x(m)} where each example vectors have different sizes. The Central ChallengeI

Intro and ML Basics • The system should perform well on new, previously unseen Intro The Task inputs not just those on which our model was trained. The Performance The Experience Generalization : the ability to perform well on previously ML ingredients unobserved inputs. Data Model • There are two important concepts used in machine learning: Objective f Optimization the training error and the test error that corresponds to a Tensorﬂow Graph and subdivision of the dataset. Session A simple example • Training error : compute some error measure on the training set. • Test error : the expected value of the error on a new input. Example: linear regression yˆ = wT x where w ∈ Rn • 1 ||X(train)w − y (train)||2 m(train) 2 • 1 ||X(test)w − y (test)||2 m(test) 2 The Central ChallengeII

Intro and ML Basics

Intro The Task The Performance The Experience

ML • The factors determining how well a machine learning ingredients Data algorithm will perform are its ability to: Model Objective f Optimization

Tensorflow 1 Make the training error small. Graph and Session A simple 2 Make the gap between training and test error small. These example two factors correspond to the two central challenges in machine learning: underfitting and overfitting . The Central Challenge III

Intro and ML Basics

Intro 1 Underfitting occurs when the model is not able to obtain a The Task The Performance sufficiently low error value on the training set. It has not The Experience enough capacity to represent data. ML ingredients Data 2 Overfitting occurs when the gap between the training error Model Objective f and test error is too large. Optimization

Tensorﬂow Graph and Session A simple example Hyperparameters

Intro and ML Basics

Intro The Task • Hyperparameters are settings of the machine learning The Performance The Experience algorithm that must be determined outside the learning ML algorithm itself. ingredients Data Model • For example in a polynomial regression the degree of the Objective f Optimization polynomial, is a hyperparameter wich act as a capacity (the Tensorﬂow Graph and ability to ﬁt a wide variety of functions) Session A simple example • To estimate hyperparameters a validation set of examples is used. • Typically the validation set is taken from the training set (two disjoint sets). • A typical choice is to use 20% of the training set as validation set. Regualrization

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f • Optimization Regularization is any modiﬁcation we make to a learning Tensorﬂow algorithm that is intended to reduce its generalization error Graph and Session but not its training error. A simple example The Model

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients • Parametric models such as linear regression, learn a Data Model function described by a parameter vector whose size is finite Objective f Optimization and fixed before any data is observed. Tensorflow Graph and • Non-parametric models have no such limitations. A Session A simple example practical example is the nearest neighbor regression.

y = f (x; θ) Artiﬁcial Neural NetworksI

Intro and ML Basics

Intro The Task The Performance The Experience ML • Biologically inspired model. ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

• a brief history: Artiﬁcial Neural NetworksII

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example • Inputs to neuron are wheighted then summed. • A term called bias independent of input is also added. • The output of the neuron is the application of an activation function to the sum previously computed. • In the perceptron model, the activation function is a threshold function. • Units(neuron) can be combined togheter to form a network Artiﬁcial Neural Networks III

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

• It is equivalent to function composition • Examples of activation functions: Artiﬁcial Neural NetworksIV

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorflow Graph and Session A simple example • Sigmoid and tanh used in classification tasks, slow to converge • ReLU most used, fast convergence, dead neuron issue. z • e j Softmax function σ(z)j = P zk is used in categorial k e distribution, it represents the probability over a finite number of categories The objective functionI

Intro and ML Basics • The objective function is a real values function that provides Intro a measure of how wrong the model is in terms of its ability to The Task The Performance estimate the relationship between inputs and outputs. The Experience

ML • It refers to an optimization problem, where the objectve ingredients Data (loss,cost) function is minimized or maximized respect to the Model Objective f parameters. Optimization

Tensorflow • An example is the mean square error function used in Graph and Session regression problems where: A simple example 1 Pm î i 2 J(θ) = MSEtrain(θ) = m i=1 ||y (θ) − y || is to be minimized with respect to the parameters θ • or the cross-entropy function used in classification problems: P H(p, q) = − x p(x) log q(x). • where p(x) is the true probability distribution. • It can be shown that these function can be derived using a general principle. The objective functionII

Intro and ML Basics • The Maximum Likelihood Estimation Intro m The Task θ = argmax ( ; θ) = argmax Q ( (i); θ) The Performance ML pmodel X i=1 pmodel x The Experience θ θ ML ingredients • (1) (m) Data where X = x , ...., x is a set of m examples drawn Model Objective f independently from the true but unknown data-generating Optimization distribution pdata(x) Tensorﬂow Graph and • pmodel (x; θ) is a parametric family of probability distribution Session A simple over the same space indexed by θ example • A more convenient but equivalent optimization problem is redeﬁning the ML taking the logarithm. Pm (i) θML = argmax i=1 log pmodel (x ; θ) θ • The ML method works if the true distribution p(data) lies within the model family pmodel (; θ). • The p(data) must correspond to exactly one value of θ. The optimization algorithm

Intro and ML Basics • Optimization refers to the task of either minimizing or Intro maximizing some function f (x) by altering x. The Task The Performance • The ﬁrst derivative of the cost function can be useful to The Experience minimize, or equivalently maximize , the cost function with ML ingredients respect to the parameters. Data 0 Model • For example we know that f (x − sign(f (x))) is less than f (x) Objective f Optimization for small Tensorﬂow • gradient descent: we can thus reduce f (x) by moving x in Graph and Session A simple the opposite sign of the derivative. example • In a multi dimensional fashion partial derivatives must be computed, so ∂ f (x) measures how f changes as only the ∂xi variable xi increases at point x. • The generalization of the gradient descent method is 0 x = x − ∇x f (x) • where , the learning rate, is a positive scalar determining the size of the step. The backpropagation algorithmI

Intro and ML Basics • The chain rule of calculus: Intro The Task • The Performance if y=g(x) and z=f(g(x))=f(y) The Experience dz = dz dy ML dx dy dx ingredients • if x ∈ m, y ∈ n and y = g(x): m → n and Data R R R R Model = ( ): n → Objective f z f y R R Optimization dz P dz dyj ∂y T = = ∇x z = ( ) ∇y z Tensorﬂow dxi j dyj dxi ∂x Graph and ∂y Session • is the nxm Jacobian matrix of g A simple ∂x example • The algorithm: 1 Forward propagate inputs example through the network. The backpropagation algorithmII

Intro and ML Basics 2 Compute the changes to the loss function backward to the Intro The Task input using the chain rule The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

• Use the derivative of loss function to update weights of the network: 0 x = x − ∇x f (x) Stochastic Gradient DescentI

Intro and ML Basics • An extension of the gradient descent algorithm is the Intro The Task stochastic gradient descent The Performance The Experience • Most used algorithm to train neural networks. ML ingredients • Data Given X the set of all inputs and Y the corresponding Model Objective f observed targets, P(Y |X, θ) is the probability to predict Y Optimization given X. Tensorﬂow Graph and Session • The conditional log-likelihood is: A simple example θML = argmax P(Y |X; θ) θ • If the examples in the dataset are independet and identically Pm (i) (i) distributed, θML = argmax i=1 logP(y |x : θ) θ • An example of cost function related to negative conditional 1 Pm (i) (i) log-likelihood is: J(θ) = m i=1 L(x , y , θ) where L(x(i), y (i), θ) = −logp(y|x : θ) Stochastic Gradient DescentII

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f • the gradient descent then become: Optimization m Tensorﬂow ∇ J(θ) = 1 P ∇ L(x(i), y (i), θ) Graph and θ m i=1 θ Session A simple example Stochastic Gradient Descent III

Intro and ML Basics

Intro The Task The Performance The Experience • It can be sampled using small set of samples drawn uniformly ML 0 ingredients from the training set x(1), ..., x(m ) Data Model Objective f 1 Pm (i) (i) Optimization g = m0 ∇θ i=1 L(x , y , θ) Tensorﬂow Graph and Session θ ← θ − g A simple example 0 • the sample x(1), ..., x(m ) is called minibatch • Minibatch size is typically chosen small and ﬁxed. AdaGrad

Intro and • ML Basics Adapts the learning rates of all model parameters by scaling

Intro them inversely proprortional to the square root of the sum of The Task The Performance all the historical squared values of the gradient. The Experience • The parameters with the largest partial derivative of the loss ML ingredients have a correspondingly rapid decrease in their learning rate. Data Model • Objective f The net effect is grater progress in mthe more gently slope Optimization directions of parameter space. Tensorﬂow Graph and Session A simple example

• Suitable for not all deep learning models. • Rapidly is designed to converge when applied to a convex function. RMSProp

Intro and ML Basics Intro • Better that AdaGrad in the nonconvex setting The Task The Performance The Experience • Exponentially weighted moving average. ML • ingredients Decaying discard history from exterme past so that it can Data Model converge rapidly after ﬁnding a convex bowl. Objective f Optimization

Tensorﬂow Graph and Session A simple example Adam

Intro and • ML Basics Adaptive moments

Intro • Variant of RMSProp and momentum The Task The Performance • As a general rule momentum appplied to gradient descent, The Experience reduce oscillating contribution. ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example Outline

Intro and ML Basics

Intro The Task The Performance The Experience 1 Intro and ML Basics ML ingredients Data Model 2 Introduction Objective f Optimization

Tensorﬂow Graph and 3 ML ingredients Session A simple example 4 Tensorﬂow Graph and Session A simple example Deep learning frameworks

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

and others https://skymind.ai/wiki/comparison-frameworks-dl4j-tensorﬂow-pytorch Framework desired features

Intro and ML Basics

Intro The Task The Performance The Experience We are interested in: ML ingredients • classical machine learning algorithms Data Model Objective f • deep learning approach (especially convolutional neural Optimization network) Tensorﬂow Graph and Session • high level language (Python) A simple example • little/no programming effort • integration with existing pipelines • multi-node CPU and GPU support TensorFlow

Intro and ML Basics Intro • Google Brain’s second generation machine learning system The Task The Performance The Experience • computations are expressed as stateful data-flow graphs ML • ingredients automatic differentiation capabilities Data Model • optimization algorithms: gradient and proximal gradient Objective f Optimization based Tensorflow Graph and • Session code portability (CPUs, GPUs, on desktop, server, or mobile A simple example computing platforms) • Python interface is the preferred one (Java, C++ and Go also exist) • installation through: virtualenv, pip, Docker, Anaconda, from sources • Apache 2.0 open-source license Tensorflow

Intro and ML Basics • Tensorﬂow is a computational framework for building Intro The Task machine learning models The Performance The Experience • High-level, object-oriented API (tf.estimator) ML • ingredients Libraries for common model components Data (tf.layers/tf.losses/tf.metrics) Model Objective f • Optimization Lower-level APIs (TensorFlow)

Tensorﬂow Graph and Session A simple example Tensorﬂow

Intro and ML Basics • The foundation of computation in TensorFlow is the Graph Intro object. This holds a network of nodes, each representing one The Task The Performance operation, connected to each other as inputs and outputs The Experience

ML • a graph protocol buffer ingredients • Data a runtime that executes the (distributed) graph Model Objective f Optimization • What are protocol buffers? Tensorflow Graph and • Session Protocol buffers are Google’s language-neutral, A simple example platform-neutral, extensible mechanism for serializing structured data - think XML • define how you want your data to be structured • then you can use special generated source code to easily write and read your structured data • All of TensorFlow’s file formats are based on Protocol Buffers • define data structures in text files, and the protobuf tools generate classes in C, Python, and other languages that can load, save, and access the data Data Flow Graphs

Intro and ML Basics

Intro • TensorFlow separates deﬁnition of computations from their The Task The Performance execution The Experience • Phase 1: assemble a graph ML ingredients • Phase 2: use a session to execute operations in the graph. Data Model Objective f • not true in eager mode Optimization

Tensorﬂow Graph and Session A simple example Tensor

Intro and ML Basics

Intro The Task The Performance The Experience

ML ingredients Data • An n-dimensional array Model Objective f Optimization • 0-d tensor: scalar (number) Tensorﬂow Graph and • 1-d tensor: vector Session A simple example • 2-d tensor: matrix • and so on TensorFlow: basic usage

Intro and ML Basics

Intro import tensorflow as tf The Task The Performance # define graph nodes(on default graph) The Experience i n p u t 1= tf. constant([1.0, 1.0, 1.0, 1.0]) ML i n p u t 2= tf. constant([2.0, 2.0, 2.0, 2.0]) ingredients output= tf.add(input1, input2) Data Model # launcha session Objective f s e s s= tf. Session() Optimization # run graph on session and retrieve results Tensorﬂow Graph and r e s u l t= sess.run(output) Session A simple # close the session example s e s s.close()

• graph-related lines • session-related lines • evaluation-related lines TensorFlow Graph Class

Intro and ML Basics Intro An instance of tf.Graph class: The Task The Performance The Experience • represents, as a directed graph, the dataflow of the ML ingredients computation we want to perform; Data Model • graph nodes are called operations (instances of tf.Operation Objective f Optimization class); Tensorflow • not necessarily a mathematical operation, but also a Graph and Session variable/constant definition, ... A simple example • each operation involves zero or more instances of tf.Tensor class, it produces zero or more instances of same class; • a tf.Tensor is a multi-dimensional array; • in these examples we are using the default graph (accessible through tf.get_default_graph() function). Session Class

Intro and ML Basics

Intro The Task The Performance The Experience An instance of tf.Session class: ML ingredients Data • is used for running graph nodes; Model Objective f • Optimization may own resources, so use Session.close() method when

Tensorﬂow you are done with it; Graph and Session • A simple it can be customizable example • you chan choose hardware where evaluation is performed; • it can provide results of node evaluations through Session.run() method. Tensors: variables, constants, placeholders

Intro and ML Basics Intro An instance of tf.Variable class: The Task The Performance The Experience • implements the mathematical concept of variable; ML ingredients • maintains it’s value accross different Session.run() calls; Data Model • Objective f can be assigned a new value through the function tf.assign() Optimization (and others of type tf.assign...()) Tensorﬂow Graph and Session • it requires an initialization step. A simple example

Other Tensors: • use tf.constant() function to initialize constant tensors; • use tf.placeholder() for a tensor that will be fed later during the evaluation. approx. minimization of a univariate functional (I)

Intro and import tensorflow as tf ML Basics import numpy as np Intro # define variable and placeholders The Task The Performance a_op= tf. placeholder( tf. float32, shape=(1) ) The Experience b_op= tf. placeholder( tf. float32, shape=(1) ) x_op= tf. Variable( [ 2.0 ], dtype=tf. float32) ML ingredients y_op= a_op* x_op**2 + b_op* x_op Data Model Objective f o p t i m i z e r= tf.train. AdamOptimizer(0.1) Optimization min_step= optimizer. minimize( y_op)

Tensorﬂow i n i t _ o p= tf. global_variables_initializer() Graph and Session A simple example

Deﬁne tensors: placeholders for a and b parameters, variable for x. Overloaded operators will call pointwise math operations: what we are actually calling is:

y_op= tf.add( tf. multiply( tf.pow( x_op, 2 ), a_op), t f. multiply(x_op, b_op))

Deﬁne optimization method, pass the function to be optimized. Deﬁne an operation for variables initialization. approx. minimization of a univariate functional (II)

Intro and # launcha session ML Basics s e s s= tf. Session()

Intro # initialize variables The Task s e s s.run(init_op) The Performance The Experience a= [ 1.0] ML b = [-2.0] ingredients y= [] Data f o r ii in range( 100 ): Model _, y_p= sess.run([ min_step, y_op], Objective f Optimization f e e d _ d i c t={ a_op:a, b_op:b}) y. append( y_p) Tensorﬂow Graph and x= sess.run(x_op) Session A simple example s e s s.close()

Deﬁne a session and initialize all variables of the graph. Deﬁne some Python variables: function parameters a and b and a vector y to hold function value at each iteration. Implement main loop: note the Python dictionary feed_dict and the return values of sess.run() method. Retrieve the estimated value of the minimum point. approx. minimization of a univariate functional (III)

Intro and ML Basics import matplotlib. pyplot as plt Intro The Task p r i n t(’solution:%f,(true:%f)’ The Performance The Experience %(x,- np.divide(b, np. multiply([a], 2.0) )))

ML p l t.plot(y) ingredients p l t.xlabel(’iterations’) Data p l t.ylabel(’function value’) Model p l t.show() Objective f Optimization

Tensorﬂow Graph and Session A simple example

Estimated value is 1.002937, real is 1.0. Tensorboard

Intro and • ML Basics Tensorboard visualizes TensorFlow graph, quantitative

Intro metrics and other data (images) The Task The Performance # before run.sees() The Experience w r i t e r= tf. summary. FileWriter(’./graphs’, tf. get_default_graph()) ML ingredients w r i t e r.close()# close the writer when you’re done using it Data Model tensorboard--logdir="./graphs" Objective f Optimization

Tensorﬂow Graph and Session A simple example Linear regression (I)

Intro and ML Basics import tensorflow as tf import numpy as np Intro import matplotlib. pyplot as plt The Task The Performance # Set up the data witha noisy linear relationship betweenX andY. The Experience num_examples= 50 ML X= np.array([np. linspace(-2,4, num_examples), ingredients np. linspace(-6,6, num_examples)]) Data X += np. random. randn(2, num_examples) Model x,y=X Objective f x_with_bias= np.array([(1.,a) fora inx]). astype(np. float32) Optimization

Tensorﬂow l o s s e s=[] Graph and training_steps =50 Session learning_rate = 0.002 A simple example

1 generate noisy input data 2 set slack variables and ﬁx algorithm −

− parameters

−

− − − − Linear regression (II)

Intro and ML Basics # Start of graph description # Set up all the tensors, variables, and operations. Intro A= tf. constant(x_with_bias) The Task The Performance t a r g e t= tf. constant(np. transpose([y]). astype(np. float32)) The Experience w e i g h t s= tf. Variable(tf. random_normal([2, 1], 0, 0.1))

ML yhat= tf. matmul(A, weights) ingredients y e r r o r= tf. subtract(yhat, target) Data l o s s= tf.nn. l2_loss(yerror) Model Objective f Optimization update_weights= t f.train. GradientDescentOptimizer(learning_rate). minimize(loss) Tensorﬂow Graph and s e s s= tf. Session() Session A simple s e s s.run( tf. global_variables_initializer()) example f o r_ in range(training_steps): # Repeatedly run the operations, updating variables s e s s.run( update_weights) l o s s e s. append( sess.run( loss))

3 define tensorflow constants and variables 4 define nodes 5 start evaluation Linear regression (III)

Intro and ML Basics

Intro The Task The Performance # Training is done, get the final values The Experience b e t a s= sess.run( weights) yhat= sess.run( yhat) ML ingredients Data Model Objective f Optimization

Tensorﬂow Graph and Session A simple example

− −

−

− − − − Advantages and disadvantages

Intro and TensorFlow in general: ML Basics

Intro High-level, object-oriented API (tf .estimator) The Task The Performance Libraries for common model components The Experience

ML (tf .layers/tf .losses/tf .metrics) ingredients Data Lower-level APIs (TensorFlow) Model Objective f Optimization very simple installation/setup on supported platforms Tensorﬂow Graph and Session not too easy to install and setup on unsupported platform A simple example plenty of tutorials, exhaustive documentation tools for exporting (partially) trained graphs (see MetaGraph) debugger, graph ﬂow visualization as a Python user: “Python API is the most complete and the easiest to use” numpy interoperability Keras interoperability Moving forward

Intro and ML Basics

Intro The Task The Performance The Experience • tensorflow.org ML ingredients • TensorFlow Dev Summit 2018 Data Model • Objective f TensorFlow Dev Summit 2017 Optimization • Tensorflow Stanford CS 20: Tensorflow for Deep Learning Research Graph and Session A simple example

Useful library: • Keras • Horovod