Deep Learning with Java, Part 2 PU Machine Learning Mit Java, Weka Und UIMA

Deep Learning with Java, part 2 PU Machine learning mit Java, Weka und UIMA

Nils Reiter, [email protected]

February 2, 2021 Winter term 2020/21

Reiter Deep Learning with Java, part 2 2021-02-02 1 / 15 Looking Back

Section 1

Looking Back

Reiter Deep Learning with Java, part 2 2021-02-02 2 / 15 Looking Back Exercise 11

https://github.com/idh-cologne-machine-learning-mit-java/exercise-11

Reiter Deep Learning with Java, part 2 2021-02-02 3 / 15 ▶ Similar to Python ▶ Plan ▶ Today / Exercise 11: Data preparation ▶ Next week / Exercise 12: Training an actual network

Looking Back Libraries

▶ Data representation: ND4J ▶ ETL library: Datavec ▶ Stores n-dimensional arrays outside of the Java heap ▶ “API mimics the semantics of Numpy, Matlab and scikit-learn” ▶ libnd4j: C++ part ▶ Neural networks: Deeplearning4J (DL4J) ▶ Can run on CUDA (GPU), partially written in C++ ▶ Complexity sources ▶ Data sets that don’t fit in memory (unlike Weka) ▶ Eﬀiciency for productive use

Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 ▶ Plan ▶ Today / Exercise 11: Data preparation ▶ Next week / Exercise 12: Training an actual network

Looking Back Libraries

Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 Looking Back Libraries

Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 Deep Learning with Java

Section 2

Deep Learning with Java

Reiter Deep Learning with Java, part 2 2021-02-02 5 / 15 ▶ Two steps ▶ Define layout of artificial neural network and training parameters ▶ Launch training, supervise progress

Deep Learning with Java DL4J

▶ Open source, integrated with Hadoop and Spark → Learning can be distributed over compute clusters

Reiter Deep Learning with Java, part 2 2021-02-02 6 / 15 Deep Learning with Java DL4J

▶ Open source, integrated with Hadoop and Spark → Learning can be distributed over compute clusters ▶ Two steps ▶ Define layout of artificial neural network and training parameters ▶ Launch training, supervise progress

Reiter Deep Learning with Java, part 2 2021-02-02 6 / 15 1 MultiLayerConfiguration conf 2 = new NeuralNetConfiguration.Builder() 3 .seed(seed).list() 4 .layer(new DenseLayer.Builder().nIn(5) 5 .nOut(10).activation(Activation.TANH) 6 .weightInit(WeightInit.ZERO).build()) 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) ▶ Layout 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) ▶ Number of layers ▶ 12 .activation(Activation.SOFTMAX).nIn(3) Size of each layer 13 .nOut(outputNum).build()) ▶ Type of layers 14 .build();

Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern

Deep Learning with Java Designing Artificial Neural Networks

▶ Parameters ▶ Activation function ▶ Regularization ▶ Weight updater ▶ Weight initialization ▶ Loss function ▶ Seed values

Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 1 MultiLayerConfiguration conf 2 = new NeuralNetConfiguration.Builder() 3 .seed(seed).list() 4 .layer(new DenseLayer.Builder().nIn(5) 5 .nOut(10).activation(Activation.TANH) 6 .weightInit(WeightInit.ZERO).build()) 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) 12 .activation(Activation.SOFTMAX).nIn(3) 13 .nOut(outputNum).build()) 14 .build();

Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern

Deep Learning with Java Designing Artificial Neural Networks

▶ Parameters ▶ Activation function ▶ Regularization ▶ Weight updater ▶ Weight initialization ▶ Loss function ▶ Seed values ▶ Layout ▶ Number of layers ▶ Size of each layer ▶ Type of layers

Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern

Deep Learning with Java Designing Artificial Neural Networks

▶ Parameters 1 MultiLayerConfiguration conf ▶ Activation function 2 = new NeuralNetConfiguration.Builder() ▶ Regularization 3 .seed(seed).list() ▶ 4 .layer(new DenseLayer.Builder().nIn(5) Weight updater 5 .nOut(10).activation(Activation.TANH) ▶ Weight initialization 6 .weightInit(WeightInit.ZERO).build()) ▶ Loss function 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) ▶ Seed values 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) ▶ Layout 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) ▶ Number of layers ▶ 12 .activation(Activation.SOFTMAX).nIn(3) Size of each layer 13 .nOut(outputNum).build()) ▶ Type of layers 14 .build();

Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 Deep Learning with Java Designing Artificial Neural Networks

Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern

Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 ▶ Epoch: One time training on the entire training data set ▶ More epochs make the result better, but take more time ▶ Argument types to fit(...) ▶ fit(DataSet) / fit(DataSetIterator) ▶ fit(INDArray, INDArray) ▶ fit(INDArray, int[]) ▶ fit(MultiDataSet) / fit(MultiDataSetIterator iterator)

Deep Learning with Java Training Artificial Neural Networks

1 // Create a network from 2 // a configuration 3 MultiLayerNetwork network 4 = new MultiLayerNetwork(conf); 5 6 // Initialize it 7 network.init(); 8 9 // Train one epoch 10 network.fit(dataset); 11 12 // Train n epochs 13 network.fit(dataset, n);

Reiter Deep Learning with Java, part 2 2021-02-02 8 / 15 Deep Learning with Java Training Artificial Neural Networks

1 // Create a network from ▶ Epoch: One time training on the entire 2 // a configuration training data set 3 MultiLayerNetwork network 4 = new MultiLayerNetwork(conf); ▶ More epochs make the result better, but 5 take more time 6 // Initialize it ▶ 7 network.init(); Argument types to fit(...) 8 ▶ fit(DataSet) / fit(DataSetIterator) 9 // Train one epoch ▶ fit(INDArray, INDArray) 10 network.fit(dataset); ▶ 11 fit(INDArray, int[]) 12 // Train n epochs ▶ fit(MultiDataSet) / 13 network.fit(dataset, n); fit(MultiDataSetIterator iterator)

Reiter Deep Learning with Java, part 2 2021-02-02 8 / 15 Deep Learning with Java Non-Linear Network Architectures

▶ MultiLayerNetwork: Stacked layers ▶ ComputationGraph: Arbitrary directed acyclic graphs ▶ Multiple inputs and outputs ▶ Complex structures ▶ Joint learning of multiple tasks

Reiter Deep Learning with Java, part 2 2021-02-02 9 / 15 demo Deep Learning with Java The Problem of Teaching Deep Learning

▶ Deep learning is best taught using examples ▶ Examples need to be brief and easily graspable → Standard data sets with pre-defined reading routines ▶ Reality: Your data is always a little different from standard data sets ⇒ What is taught is not reflective of reality

Reiter Deep Learning with Java, part 2 2021-02-02 11 / 15 Deep Learning with Java A Real Dataset

▶ Amazon reviews, used for sentiment classification ▶ It’s large: Test data: 400 000 reviews, Training data: 3 600 000 ▶ Preprocessing ▶ Establish vocabulary (count words, preserve words with frequency > 15) ▶ Lowercase everything ▶ Replace punctuation and digits by space ▶ Count words in vocabulary, replace string column by NDArray ▶ Convert labels to category ▶ Convert category to integer

Reiter Deep Learning with Java, part 2 2021-02-02 12 / 15 Deep Learning with Java

demo

Reiter Deep Learning with Java, part 2 2021-02-02 13 / 15 Next Exercise

Section 3

Next Exercise

Reiter Deep Learning with Java, part 2 2021-02-02 14 / 15 Next Exercise Exercise 12 – the last one!

https://github.com/idh-cologne-machine-learning-mit-java/exercise-12

Reiter Deep Learning with Java, part 2 2021-02-02 15 / 15