Deep Learning with Java, part 2 PU Machine learning mit Java, Weka und UIMA Nils Reiter, [email protected] February 2, 2021 Winter term 2020/21 Reiter Deep Learning with Java, part 2 2021-02-02 1 / 15 Looking Back Section 1 Looking Back Reiter Deep Learning with Java, part 2 2021-02-02 2 / 15 Looking Back Exercise 11 https://github.com/idh-cologne-machine-learning-mit-java/exercise-11 Reiter Deep Learning with Java, part 2 2021-02-02 3 / 15 I Similar to Python I Plan I Today / Exercise 11: Data preparation I Next week / Exercise 12: Training an actual network Looking Back Libraries I Data representation: ND4J I ETL library: Datavec I Stores n-dimensional arrays outside of the Java heap I “API mimics the semantics of Numpy, Matlab and scikit-learn” I libnd4j: C++ part I Neural networks: Deeplearning4J (DL4J) I Can run on CUDA (GPU), partially written in C++ I Complexity sources I Data sets that don’t fit in memory (unlike Weka) I Efficiency for productive use Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 I Plan I Today / Exercise 11: Data preparation I Next week / Exercise 12: Training an actual network Looking Back Libraries I Data representation: ND4J I ETL library: Datavec I Stores n-dimensional arrays outside of the Java heap I “API mimics the semantics of Numpy, Matlab and scikit-learn” I libnd4j: C++ part I Neural networks: Deeplearning4J (DL4J) I Can run on CUDA (GPU), partially written in C++ I Complexity sources I Data sets that don’t fit in memory (unlike Weka) I Efficiency for productive use I Similar to Python Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 Looking Back Libraries I Data representation: ND4J I ETL library: Datavec I Stores n-dimensional arrays outside of the Java heap I “API mimics the semantics of Numpy, Matlab and scikit-learn” I libnd4j: C++ part I Neural networks: Deeplearning4J (DL4J) I Can run on CUDA (GPU), partially written in C++ I Complexity sources I Data sets that don’t fit in memory (unlike Weka) I Efficiency for productive use I Similar to Python I Plan I Today / Exercise 11: Data preparation I Next week / Exercise 12: Training an actual network Reiter Deep Learning with Java, part 2 2021-02-02 4 / 15 Deep Learning with Java Section 2 Deep Learning with Java Reiter Deep Learning with Java, part 2 2021-02-02 5 / 15 I Two steps I Define layout of artificial neural network and training parameters I Launch training, supervise progress Deep Learning with Java DL4J I Open source, integrated with Hadoop and Spark ! Learning can be distributed over compute clusters Reiter Deep Learning with Java, part 2 2021-02-02 6 / 15 Deep Learning with Java DL4J I Open source, integrated with Hadoop and Spark ! Learning can be distributed over compute clusters I Two steps I Define layout of artificial neural network and training parameters I Launch training, supervise progress Reiter Deep Learning with Java, part 2 2021-02-02 6 / 15 1 MultiLayerConfiguration conf 2 = new NeuralNetConfiguration.Builder() 3 .seed(seed).list() 4 .layer(new DenseLayer.Builder().nIn(5) 5 .nOut(10).activation(Activation.TANH) 6 .weightInit(WeightInit.ZERO).build()) 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) I Layout 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) I Number of layers I 12 .activation(Activation.SOFTMAX).nIn(3) Size of each layer 13 .nOut(outputNum).build()) I Type of layers 14 .build(); Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern Deep Learning with Java Designing Artificial Neural Networks I Parameters I Activation function I Regularization I Weight updater I Weight initialization I Loss function I Seed values Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 1 MultiLayerConfiguration conf 2 = new NeuralNetConfiguration.Builder() 3 .seed(seed).list() 4 .layer(new DenseLayer.Builder().nIn(5) 5 .nOut(10).activation(Activation.TANH) 6 .weightInit(WeightInit.ZERO).build()) 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) 12 .activation(Activation.SOFTMAX).nIn(3) 13 .nOut(outputNum).build()) 14 .build(); Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern Deep Learning with Java Designing Artificial Neural Networks I Parameters I Activation function I Regularization I Weight updater I Weight initialization I Loss function I Seed values I Layout I Number of layers I Size of each layer I Type of layers Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern Deep Learning with Java Designing Artificial Neural Networks I Parameters 1 MultiLayerConfiguration conf I Activation function 2 = new NeuralNetConfiguration.Builder() I Regularization 3 .seed(seed).list() I 4 .layer(new DenseLayer.Builder().nIn(5) Weight updater 5 .nOut(10).activation(Activation.TANH) I Weight initialization 6 .weightInit(WeightInit.ZERO).build()) I Loss function 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) I Seed values 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) I Layout 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) I Number of layers I 12 .activation(Activation.SOFTMAX).nIn(3) Size of each layer 13 .nOut(outputNum).build()) I Type of layers 14 .build(); Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 Deep Learning with Java Designing Artificial Neural Networks I Parameters 1 MultiLayerConfiguration conf I Activation function 2 = new NeuralNetConfiguration.Builder() I Regularization 3 .seed(seed).list() I 4 .layer(new DenseLayer.Builder().nIn(5) Weight updater 5 .nOut(10).activation(Activation.TANH) I Weight initialization 6 .weightInit(WeightInit.ZERO).build()) I Loss function 7 .layer(new DenseLayer.Builder().nIn(10).nOut(3) I Seed values 8 .activation(Activation.SIGMOID) 9 .weightInit(WeightInit.UNIFORM).build()) I Layout 10 .layer(new OutputLayer.Builder( 11 LossFunctions.LossFunction.HINGE) I Number of layers I 12 .activation(Activation.SOFTMAX).nIn(3) Size of each layer 13 .nOut(outputNum).build()) I Type of layers 14 .build(); Builder design patterns: https://en.wikipedia.org/wiki/Builder_pattern Reiter Deep Learning with Java, part 2 2021-02-02 7 / 15 I Epoch: One time training on the entire training data set I More epochs make the result better, but take more time I Argument types to fit(...) I fit(DataSet) / fit(DataSetIterator) I fit(INDArray, INDArray) I fit(INDArray, int[]) I fit(MultiDataSet) / fit(MultiDataSetIterator iterator) Deep Learning with Java Training Artificial Neural Networks 1 // Create a network from 2 // a configuration 3 MultiLayerNetwork network 4 = new MultiLayerNetwork(conf); 5 6 // Initialize it 7 network.init(); 8 9 // Train one epoch 10 network.fit(dataset); 11 12 // Train n epochs 13 network.fit(dataset, n); Reiter Deep Learning with Java, part 2 2021-02-02 8 / 15 Deep Learning with Java Training Artificial Neural Networks 1 // Create a network from I Epoch: One time training on the entire 2 // a configuration training data set 3 MultiLayerNetwork network 4 = new MultiLayerNetwork(conf); I More epochs make the result better, but 5 take more time 6 // Initialize it I 7 network.init(); Argument types to fit(...) 8 I fit(DataSet) / fit(DataSetIterator) 9 // Train one epoch I fit(INDArray, INDArray) 10 network.fit(dataset); I 11 fit(INDArray, int[]) 12 // Train n epochs I fit(MultiDataSet) / 13 network.fit(dataset, n); fit(MultiDataSetIterator iterator) Reiter Deep Learning with Java, part 2 2021-02-02 8 / 15 Deep Learning with Java Non-Linear Network Architectures I MultiLayerNetwork: Stacked layers I ComputationGraph: Arbitrary directed acyclic graphs I Multiple inputs and outputs I Complex structures I Joint learning of multiple tasks Reiter Deep Learning with Java, part 2 2021-02-02 9 / 15 demo Deep Learning with Java The Problem of Teaching Deep Learning I Deep learning is best taught using examples I Examples need to be brief and easily graspable ! Standard data sets with pre-defined reading routines I Reality: Your data is always a little different from standard data sets ) What is taught is not reflective of reality Reiter Deep Learning with Java, part 2 2021-02-02 11 / 15 Deep Learning with Java A Real Dataset I Amazon reviews, used for sentiment classification I It’s large: Test data: 400 000 reviews, Training data: 3 600 000 I Preprocessing I Establish vocabulary (count words, preserve words with frequency > 15) I Lowercase everything I Replace punctuation and digits by space I Count words in vocabulary, replace string column by NDArray I Convert labels to category I Convert category to integer Reiter Deep Learning with Java, part 2 2021-02-02 12 / 15 Deep Learning with Java demo Reiter Deep Learning with Java, part 2 2021-02-02 13 / 15 Next Exercise Section 3 Next Exercise Reiter Deep Learning with Java, part 2 2021-02-02 14 / 15 Next Exercise Exercise 12 – the last one! https://github.com/idh-cologne-machine-learning-mit-java/exercise-12 Reiter Deep Learning with Java, part 2 2021-02-02 15 / 15.