DOCSLIB.ORG

Machine Learning Tutorial

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Machine Learning Tutorial Photos placed in horizontal position with even amount of white space between photos and header Machine Learning Tutorial Danny Dunlavy, 01461 Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2018-7925 TR Goals for this Tutorial § Introduction to main concepts in Machine Learning § Preparation for participation in MLDL Workshop Caveats § Awareness stressed over education § Neural Networks/Deep Learning mostly avoided § Deep Learning Tutorial: Thursday, July 19, 2018 7/18/18 ML Tutorial 2 Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. --Tom Mitchell, Machine Learning, 1997 7/18/18 ML Tutorial 3 Example: Handwriting Recognition § Task (T): § recognizing and classifying handwritten numbers within images § Performance measure (P): § percent of numbers correctly classified § Experience (E): § a database of handwritten numbers with given classifications Example adapted from Tom Mitchell, Machine Learning, 1997 Data from MNIST database, http://yann.lecun.com/exdb/mnist/ 7/18/18 ML Tutorial 4 Example ML Workflow Data Features Model Solution Evaluation Instance Label Label Correct 5 0 87% 0 1 96% 4 2 84% . 1 3 82% . 3 0 4 1 7/18/18 ML Tutorial 5 Feature Engineering § Feature engineering is the process of using domain knowledge to create feature § Often manual, time-consuming process § Many machine learning algorithms take vectors as inputs § Raw data often is not in vector format § For many data types, there are existing conventions for creating feature vectors Pedro Domingos. 2012. A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. 7/18/18 ML Tutorial 6 Feature Vectors: Images Pixel Values (vectorized) Image Processing Features (Feature Detectors) § Edge, corner, blob, ridge detection § Histogram of Oriented Gradients (HoG) § Hu’s Invariant Moments § Local binary patterns (LBP) § Hough transform . Example Software: Python: scikit-image Matlab: Image Processing Toolbox Julia: JuliaImages (ImageFeatures) R: https://github.com/bnosac/image Reference: Image Feature Detectors and Descriptors. Eds. Awad Image from Matlab 2018a demo: street1.jpg and Hassaballah, Springer, 2016. 7/18/18 ML Tutorial 7 Works generally with Feature Vectors: Text counts of observations on discrete domains § Vector Space Model § Variations (Bag of Words Model) § Stop words (high frequency) § Document 1 § the, a , and The quick brown fox jumped over the lazy dog. § Stemming § Document 2 § jumps, jumped -> jump The brown dog jumped over the dog fence. § N-grams Doc 1 Doc 2 § quicK brown quick 1 0 § brown fox brown 1 1 § fox jump fox 1 0 § jump 1 1 Weighting § over 1 1 TF-IDF (term frequency- Document Matrix Document - inverse doc frequency) dog 1 2 Term fence 0 1 Salton, et al., 1975. A vector space model for automatic Manning and Schutze, Foundations of Statistical indexing. Communications of the ACM, 18(11), 613-620. Natural Language Processing. MIT Press. 1999. 7/18/18 ML Tutorial 8 ! Sequential Data ! § Natural Language Processing (NLP) Part of speech at np-tl nn-tl jj-tl nn-tl vbd nr at Word The Fulton County Grand Jury said Friday an !! !! W. Nelson Francis and Henry Kucera, 1979. The Brown Corpus: A Standard Corpus of Present-Day Edited American English. YG! MZOMC!@MH3;N*CMBJN3B* § Computer NetworkYG! MZOMC!@MH3;N*CMBJN3B* Traffic Analysis YGF! M[9#6&1#$%*B#%09* YGF! M[9#6&1#$%*B#%09* g)*!*9;)S.'8').!89383,!49!*9;)*797!HHC!A'79)!39]&9.;93!;).8#'.'.S!='A9!$'.73!)=!0&2#.!#;8').3W!4#<$!Z84)!$'.73[,! *&.!Z84)!$'.73[!#.7!3$'K!K9*=)*297!5%!GGg)*!*9;)S.'8').!89383,!49!*9;)*797!HHC!A'79)!39]&9.;93!;).8!&.'A9*3'8%!38&79.83!#8!#S93!)=!BREQQN!#'.'.S!='A9!$'.73!)=!0&2#.!#;8').3W!4#<$!Z84)!$'.73[,!?9!8))$!9#;0!#;8').!3;9.9!e!8'293!ZG!=)*! *&.!Z84)!$'.73[!#.7!3$'K!K9*=)*297!5%!GG7#8#5#39!#.7!B!=)*!*9;)S.'8').[N!!?9!&397!#!38#8';!;)<)*!!&.'A9*3'8%!38&79.83!#8!#S93!)=!BREQQN!PPM!A'79)!;#29*#!4'80!GC=K3!=*#29!*#89,!40';0!4#3!K)3'8').97!?9!8))$!9#;0!#;8').!3;9.9!e!8'293!ZG!=)*! 8)!#!J!2989*3!<).S!4#<$!4#%!8)!8#$9!=&<<!5)7%!2)A929.8!7#8#5#39!#.7!B!=)*!*9;)S.'8').[N!!?9!&397!#!38#8';!;)<)*!PPM!A'79)!;#29*#!4'80!GC=K3!=*#29!*#89,!40';0!4#3!K)3'8').97!)=!J!389K3N!V#;0!A'79)!=*#29!4#3!7)4.!3#2K<97!=*)2!IQC\eRC! K'\9<3!8)!GHCmQeC!K'\9<3N!"09!A'79)!=*#29!*#89!4#3!#<3)!7)4.E3#2K<97!8)!.)*2#<'a9!809!2)8').!3K997!).!9#;0!2)8').N!8)!#!J!2989*3!<).S!4#<$!4#%!8)!8#$9!=&<<!5)7%!2)A929.8!)=!J!389K3N!V#;0!A'79)!=*#29!4#3!7)4.!3#2K<97!=*)2!IQC\eRC! K'\9<3!8)!GHCmQeC!K'\9<3N!"09!A'79)!=*#29!*#89!4#3!#<3)!7)4.E3#2K<97!8)!.)*2#<'a9!809!2)8').!3K997!).!9#;0!2)8').N!!34546G >.!80'3!9\K9*'29.8,!49!79='.9!#!7#8#!398!)!40';0!8#$93!809!=)*2!)=!#!./ 34546 !89.3)*,!409*9!(kGG!Z809!.&259*!)=!K9)K<9[,! >.!80'3!9\K9*'29.8,!49!79='.9!#!7#8#!398!Wylie,!40';0!8#$93!809!=)*2!)=!#! et al., Using NoSQL Databases for Streaming! G !89.3)*,!409*9!(kGG!Z809!.&259*!)=!K9)K<9[,!Network Analysis, LDAV, 2012. VkJ!Z809!.&259*!)=!#;8').!;<#3393[,!#.7!)6kGC!Z809!.&259*!)=!'2#S9!=9#8&*93[N!./ VkJ!Z809!.&259*!)=!#;8').!;<#3393[,!#.7!§ Video Analysis 6kGC!Z809!.&259*!)=!'2#S9!=9#8&*93[N! ?#<$B! ?#<$Q! `&.B! `&.Q! :$'K! ?#<$B! ?#<$Q! `&.B! `&.Q! :$'K! Takayuki Hori, Jun Ohya and Jun Kurumisawa, Computational Imaging, 2010. 7/18/18 ML Tutorial 9 ?#<$! ?#<$! `&.! `&.! :$'K! ?'80!#*23!?#<$! ?'80)&8!#*23!?#<$! h)4!3K997!`&.! ('S0!3K997!`&.! :$'K! ?'80!#*23!4#A'.S! ?'80)&8!#*23!4#A'.S! h)4!3K997! ('S0!3K997! 4#A'.S! 4#A'.S! g'S&*9!IW!g'A9!1'.73!)=!(&2#.!T;8').3!#.7!V\K<#'.!)=!809!2)8').N! g'S&*9!IW!g'A9!1'.73!)=!(&2#.!T;8').3!#.7!V\K<#'.!)=!809!2)8').N! YGK! O#645$*!"#$%&'&8+%&5$*C#40-%4* YGK! O#645$*!"#$%&'&8+%&5$*C#40-%4* g)*!*9;)S.'8').!89383,!49!&397!BHJ!3#2K<93!ZGG!K9)K<9!+!J!#;8').3[N!"09!*9;)S.'8').!*93&<83!#*9!30)4.!'.!g'SN!R,!'.! 40';0!809!*9;)S.'8').!#;;&*#;'93!=)*!809!J!#;8').3!#*9!30)4.Ng)*!*9;)S.'8').!89383,!49!&397!BHJ!3#2K<93!ZGG!K9)K<9!+!J!!")!30)4!809!A#<'7'8%!)=!)&*!K*)K)397!2980)7,!)&*!2980)7!#;8').3[N!"09!*9;)S.'8').!*93&<83!#*9!30)4.!'.!g'SN!R,!'.! '3!;)2K#*97!4'80!D9#*938!D9'S05)*!*&<9!ZDD[!#.7!O*'.;'K#<!P)2K).9.8!T.#<%3'3!ZOPT[!5#397!2980)73N!"09!DDE5#397!40';0!809!*9;)S.'8').!#;;&*#;'93!=)*!809!J!#;8').3!#*9!30)4.N!")!30)4!809!A#<'7'8%!)=!)&*!K*)K)397!2980)7,!)&*!2980)7! '3!;)2K#*97!4'80!D9#*938!D9'S05)*!*&<9!ZDD[!#.7!O*'.;'K#<!P)2K).9.8!T.#<%3'3!ZOPT[!5#397!2980)73N!"09!DDE5#397! 2980)7!;#<;&<#893!809!7'==9*9.;9!598499.!809!'.K&8!7#8#!#.7!9#;0!7#8#!38)*97!'.!809!89.3)*#),!#.7!809!#;8').!80#8!S'A93!809! 2980)7!;#<;&<#893!809!7'==9*9.;9!598499.!809!'.K&8!7#8#!#.7!9#;0!7#8#!38)*97!'.!809!89.3)* ,!#.7!809!#;8').!80#8!S'A93!809! 2'.'2#<!7'==9*9.;9!'3!809!*93&<8!)=!809!*9;)S.'8').N!"09!OPTE5#397!*9;)S.'8').!2980)7!&393!V'S9.!A#<&93!)58#'.97!=*)2!#) 809!A#*'#.;9E;)A#*'#.;9!2#8*'\!'.!809!7#8#5#39,!#.7!39#*;093!2'.'2#<!7'==9*9.;9!'3!809!*93&<8!)=!809!*9;)S.'8').N!"09!OPTE5#397!*9;)S.'8').!2980)7!&393!V'S9.!A#<&93!)58#'.97!=*)2!=)*!809!#;8').!80#8!S'A93!809!2)38!2'.'2#<!7'==9*9.;9!)=!809! V'S9.!A#<&93N!!809!A#*'#.;9E;)A#*'#.;9!2#8*'\!'.!809!7#8#5#39,!#.7!39#*;093!=)*!809!#;8').!80#8!S'A93!809!2)38!2'.'2#<!7'==9*9.;9!)=!809! V'S9.!A#<&93N!! g'S&*9!RW!`9;)S.'8').!`93&<83!=)*!9#;0!#;8').N! ! g'S&*9!RW!`9;)S.'8').!`93&<83!=)*!9#;0!#;8').N! ! @:,AB,@CDQ+."*8+EFGG++EFGGHIBT @:,AB,@CDQ+."*8+EFGG++EFGGHIBT Downloaded from SPIE Digital Library on 30 May 2012 to 198.102.153.2. Terms of Use: http://spiedl.org/terms Downloaded from SPIE Digital Library on 30 May 2012 to 198.102.153.2. Terms of Use: http://spiedl.org/terms ! ! Major Types of Machine Learning § Unsupervised Learning § Supervised Learning § Semi-supervised Learning § Reinforcement Learning 7/18/18 ML Tutorial 10 Unsupervised Learning § Tasks § Clustering (grouping) § Dimensionality reduction § Anomaly detection § Association § Generative modeling § Experience (data) § Instances are unlabeled § Performance measures § Challenging due to lack of labels/known solutions § Validation often leverages labeled data sets (labels only used in testing) Fisher, 1936.
Load more

Recommended publications
  • Least Squares Regression Principal Component Analysis a Supervised Dimensionality Reduction Method for Machine Learning in Scientific Applications
    Universitat Politecnica` de Catalunya Bachelor's Degree in Engineering Physics Bachelor's Thesis Least Squares Regression Principal Component Analysis A supervised dimensionality reduction method for machine learning in scientific applications Supervisor: Author: Dr. Xin Yee H´ector Pascual Herrero Dr. Joan Torras June 29, 2020 Abstract Dimension reduction is an important technique in surrogate modeling and machine learning. In this thesis, we present three existing dimension reduction methods in de- tail and then we propose a novel supervised dimension reduction method, `Least Squares Regression Principal Component Analysis" (LSR-PCA), applicable to both classifica- tion and regression dimension reduction tasks. To show the efficacy of this method, we present different examples in visualization, classification and regression problems, com- paring it to state-of-the-art dimension reduction methods. Furthermore, we present the kernel version of LSR-PCA for problems where the input are correlated non-linearly. The examples demonstrated that LSR-PCA can be a competitive dimension reduction method. 1 Acknowledgements I would like to express my gratitude to my thesis supervisor, Professor Xin Yee. I would like to thank her for giving me this wonderful opportunity and for her guidance and support during all the passing of this semester, putting herself at my disposal during the difficult times of the COVID-19 situation. Without her, the making of this thesis would not have been possible. I would like to extend my thanks to Mr. Pere Balsells, for allowing students like me to conduct their thesis abroad, as well as to the Balsells Foundation for its help and support throughout the whole stay.
    [Show full text]
  • Nonparametric Regression
    Nonparametric Regression Statistical Machine Learning, Spring 2015 Ryan Tibshirani (with Larry Wasserman) 1 Introduction, and k-nearest-neighbors 1.1 Basic setup, random inputs Given a random pair (X; Y ) Rd R, recall that the function • 2 × f0(x) = E(Y X = x) j is called the regression function (of Y on X). The basic goal in nonparametric regression is ^ d to construct an estimate f of f0, from i.i.d. samples (x1; y1);::: (xn; yn) R R that have the same joint distribution as (X; Y ). We often call X the input, predictor,2 feature,× etc., and Y the output, outcome, response, etc. Importantly, in nonparametric regression we do not assume a certain parametric form for f0 Note for i.i.d. samples (x ; y );::: (x ; y ), we can always write • 1 1 n n yi = f0(xi) + i; i = 1; : : : n; where 1; : : : n are i.i.d. random errors, with mean zero. Therefore we can think about the sampling distribution as follows: (x1; 1);::: (xn; n) are i.i.d. draws from some common joint distribution, where E(i) = 0, and then y1; : : : yn are generated from the above model In addition, we will assume that each i is independent of xi. As discussed before, this is • actually quite a strong assumption, and you should think about it skeptically. We make this assumption for the sake of simplicity, and it should be noted that a good portion of theory that we cover (or at least, similar theory) also holds without the assumption of independence between the errors and the inputs 1.2 Basic setup, fixed inputs Another common setup in nonparametric regression is to directly assume a model • yi = f0(xi) + i; i = 1; : : : n; where now x1; : : : xn are fixed inputs, and 1; : : : are still i.i.d.
    [Show full text]
  • Enabling Deeper Learning on Big Data for Materials Informatics Applications
    www.nature.com/scientificreports OPEN Enabling deeper learning on big data for materials informatics applications Dipendra Jha1, Vishu Gupta1, Logan Ward2,3, Zijiang Yang1, Christopher Wolverton4, Ian Foster2,3, Wei‑keng Liao1, Alok Choudhary1 & Ankit Agrawal1* The application of machine learning (ML) techniques in materials science has attracted signifcant attention in recent years, due to their impressive ability to efciently extract data‑driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector‑ based materials representation as input to build accurate property prediction models. We fnd that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to signifcantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.
    [Show full text]
  • COVID-19 and Machine Learning: Investigation and Prediction Team #5
    COVID-19 and Machine Learning: Investigation and Prediction Team #5: Kiran Brar Olivia Alexander Kimberly Segura 1 Abstract The COVID-19 virus has affected over four million people in the world. In the U.S. alone, the number of positive cases have exceeded one million, making it the most affected country. There is clear urgency to predict and ultimately decrease the spread of this infectious disease. Therefore, this project was motivated to test and determine various machine learning models that can accurately predict the number of confirmed COVID-19 cases in the U.S. using available time-series data. COVID-19 data was coupled with state demographic data to investigate the distribution of cases and potential correlations between demographic features. Concerning the four machine learning models tested, it was hypothesized that LSTM and XGBoost would result in the lowest errors due to the complexity and power of these models, followed by SVR and linear regression. However, linear regression and SVR had the best performance in this study which demonstrates the importance of testing simpler models and only adding complexity if the data requires it. We note that LSTM’s low performance was most likely due to the size of the training dataset available at the time of this research as deep learning requires a vast amount of data. Additionally, each model’s accuracy improved after implementing time-series preprocessing techniques of power transformations, normalization, and the overall restructuring of the time-series problem to a supervised machine learning problem using lagged values. This research can be furthered by predicting the number of deaths and recoveries as well as extending the models by integrating healthcare capacity and social restrictions in order to increase accuracy or to forecast infection, death, and recovery rates for future dates.
    [Show full text]
  • Kernel Smoothing Methods
    Kernel Smoothing Methods Hanchen Wang Ph.D Candidate in Information Engineering, University of Cambridge September 29, 2019 Hanchen Wang ([email protected]) Kernel Smoothing Methods September 29, 2019 1 / 18 Overview 1 6.0 what is kernel smoothing? 2 6.1 one-dimensional kernel smoothers 3 6.2 selecting the width λ of the kernel p 4 6.3 local regression in R p 5 6.4 structured local regression models in R 6 6.5 local likelihood and other models 7 6.6 kernel density estimation and classification 8 6.7 radial basis functions and kernels 9 6.8 mixture models for density estimation and classifications 10 6.9 computation considerations 11 Q & A: relationship between kernel smoothing methods and kernel methods 12 one more thing: solution manual to these textbooks Hanchen Wang ([email protected]) Kernel Smoothing Methods September 29, 2019 2 / 18 6.0 what is kernel smoothing method? a class of regression techniques that achieve flexibility in esti- p mating function f (X ) over the domain R by fitting a different but simple model separately at each query point x0. p resulting estimated function f (X ) is smooth in R fitting gets done at evaluation time, memory-based methods require in principle little or no training, similar as kNN lazy learning require hyperparameter setting such as metric window size λ kernels are mostly used as a device for localization rather than high-dimensional (implicit) feature extractor in kernel methods Hanchen Wang ([email protected]) Kernel Smoothing Methods September 29, 2019 3 / 18 6.1 one-dimensional kernel smoothers, overview
    [Show full text]
  • Input Output Kernel Regression: Supervised and Semi-Supervised Structured Output Prediction with Operator-Valued Kernels
    Journal of Machine Learning Research 17 (2016) 1-48 Submitted 11/15; Revised 7/16; Published 9/16 Input Output Kernel Regression: Supervised and Semi-Supervised Structured Output Prediction with Operator-Valued Kernels C´elineBrouard [email protected] Helsinki Institute for Information Technology HIIT Department of Computer Science, Aalto University 02150 Espoo, Finland IBISC, Universit´ed'Evry´ Val d'Essonne 91037 Evry´ cedex, France Marie Szafranski [email protected] ENSIIE & LaMME, Universit´ed'Evry´ Val d'Essonne, CNRS, INRA 91037 Evry´ cedex, France IBISC, Universit´ed'Evry´ Val d'Essonne 91037 Evry´ cedex, France Florence d'Alch´e-Buc [email protected] LTCI, CNRS, T´el´ecom ParisTech Universit´eParis-Saclay 46, rue Barrault 75013 Paris, France IBISC, Universit´ed'Evry´ Val d'Essonne 91037 Evry´ cedex, France Editor: Koji Tsuda Abstract In this paper, we introduce a novel approach, called Input Output Kernel Regression (IOKR), for learning mappings between structured inputs and structured outputs. The approach belongs to the family of Output Kernel Regression methods devoted to regres- sion in feature space endowed with some output kernel. In order to take into account structure in input data and benefit from kernels in the input space as well, we use the Re- producing Kernel Hilbert Space theory for vector-valued functions. We first recall the ridge solution for supervised learning and then study the regularized hinge loss-based solution used in Maximum Margin Regression. Both models are also developed in the context of semi-supervised setting. In addition we derive an extension of Generalized Cross Validation for model selection in the case of the least-square model.
    [Show full text]
  • Kernel Smoothing Toolbox ∗ for MATLAB
    Kernel Smoothing Toolbox ∗ for MATLAB Contents 1 Kernel regression 2 1.1 Startmenu .................................. 2 1.2 Basicmenuandsettingofparameters . ... 4 1.3 Estimationofoptimalparameters . ... 6 1.4 “Eye-control”method ............................ 7 1.5 Comparing of methods for bandwidth selection . ...... 8 1.6 The final estimation of the regression function . ....... 9 2 Kernel quality indexes 11 2.1 Startmenu .................................. 11 2.2 Basicmenu .................................. 14 3 Two-dimensional density estimation 16 3.1 Startmenu .................................. 16 3.2 Basicmenuandsettingofparameters . ... 19 3.3 Finalestimationofthedensity. ... 22 ∗Jan Kol´aˇcek, Dept. of Mathematics and Statistics, Masaryk University, Kotl´aˇrsk´a2, Brno, Czech Republic, [email protected], available on http : //math.muni.cz/ ∼ kolacek/docs/kerns.zip 1 1 Kernel regression 1.1 Start menu The Start menu (Figure 1) for kernel regression is called up by command >> ksregress 1♠ ❙ ❙❙✇ 2♠ ✑✰✑ 3♠ 4♠ ✑✰✑ Figure 1: Start menu You can skip this menu by typing input data as an argument >> ksregress(x,y); where vectors x and y should have the same length n and they mark x and y axes of measurements. If we know also the right regression function f (for example for sim- ulated data), we can set it as the next argument. For more see >> help ksregress. After execution of this command directly the window on Figure 3 is called up. In the Start menu, you have several possibilities how to define input data. You can load it from a file (button 1♠) or simulate data (button 2♠). In fields 3♠you can list your variables in current workspace to define input data. If your workspace is empty, these fields are non-active.
    [Show full text]
  • Krls: a Stata Package for Kernel-Based Regularized Least
    JSS Journal of Statistical Software , Volume , Issue . http://www.jstatsoft.org/ krls:A Stata Package for Kernel-Based Regularized Least Squares Jeremy Ferwerda Jens Hainmueller Massachusetts Institute of Technology Stanford University Chad J. Hazlett University of California Los Angeles Abstract The Stata package krls implements kernel-based regularized least squares (KRLS), a machine learning method described in Hainmueller and Hazlett(2014) that allows users to tackle regression and classification problems without strong functional form assumptions or a specification search. The flexible KRLS estimator learns the functional form from the data, thereby protecting inferences against misspecification bias. Yet it nevertheless allows for interpretability and inference in ways similar to ordinary regression models. In particular, KRLS provides closed-form estimates for the predicted values, variances, and the pointwise partial derivatives that characterize the marginal effects of each independent variable at each data point in the covariate space. The method is thus a convenient and powerful alternative to OLS and other GLMs for regression-based analyses. We also provide a companion package and replication code that implements the method in R. Keywords: machine learning, regression, classification, prediction, Stata, R. 1. Overview GLMs remain the workhorse modeling technology for most regression and classification prob- lems in social science research. GLMs are relatively easy to use and interpret, and allow a variety of outcome variable types with different assumed conditional distributions. However, by using the data in a linear way within the appropriate link function, all GLMs impose strin- gent functional form assumptions that are often potentially inaccurate for social science data. For example, linear regression typically requires that the marginal effect of each covariate is constant across the covariate space.
    [Show full text]
  • Iprior: an R Package for Regression Modelling Using I-Priors
    iprior: An R Package for Regression Modelling using I-priors Haziq Jamil Wicher Bergsma Universiti Brunei Darussalam London School of Economics Abstract This is an overview of the R package iprior, which implements a unified methodology for fitting parametric and nonparametric regression models, including additive models, multilevel models, and models with one or more functional covariates. Based on the principle of maximum entropy, an I-prior is an objective Gaussian process prior for the regression function with covariance kernel equal to its Fisher information. The regres- sion function is estimated by its posterior mean under the I-prior, and hyperparameters are estimated via maximum marginal likelihood. Estimation of I-prior models is simple and inference straightforward, while small and large sample predictive performances are comparative, and often better, to similar leading state-of-the-art models. We illustrate the use of the iprior package by analysing a simulated toy data set as well as three real- data examples, in particular, a multilevel data set, a longitudinal data set, and a dataset involving a functional covariate. Keywords: Gaussian, process, regression, objective, prior, empirical, Bayes, RKHS, kernel, EM, algorithm, Nystr¨om, random, effects, multilevel, longitudinal, functional, I-prior, R. 1. Introduction For subject i 2 f1; : : : ; ng, assume a real-valued response yi has been observed, as well as a row vector of p covariates xi = (xi1; : : : ; xip), where each xik belongs to some set Xk, k = 1; : : : ; p. To describe the dependence of the yi on the xi, we consider the regression model yi = α + f(xi) + i (1) iid −1 i ∼ N(0; ) where f is some regression function to be estimated, and α is an intercept.
    [Show full text]
  • Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
    Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains Matthew Tancik1∗ Pratul P. Srinivasan1;2∗ Ben Mildenhall1∗ Sara Fridovich-Keil1 Nithin Raghavan1 Utkarsh Singhal1 Ravi Ramamoorthi3 Jonathan T. Barron2 Ren Ng1 1University of California, Berkeley 2Google Research 3University of California, San Diego Abstract We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low- dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities. 1 Introduction A recent line of research in computer vision and graphics replaces traditional discrete representations of objects, scene geometry, and appearance (e.g. meshes and voxel grids) with continuous functions parameterized by deep fully-connected networks (also called multilayer perceptrons or MLPs). These MLPs, which we will call “coordinate-based” MLPs, take low-dimensional coordinates as inputs (typically points in R3) and are trained to output a representation of shape, density, and/or color at each input location (see Figure 1).
    [Show full text]
  • Kernel Regularized Least Squares: Reducing Misspecification Bias with a Flexible and Interpretable Machine Learning Approach
    Advance Access publication October 10, 2013 Political Analysis (2014) 22:143–168 doi:10.1093/pan/mpt019 Kernel Regularized Least Squares: Reducing Misspecification Bias with a Flexible and Interpretable Machine Learning Approach Jens Hainmueller Department of Political Science, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139 e-mail: [email protected] (corresponding author) Chad Hazlett Downloaded from Department of Political Science, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139 e-mail: [email protected] Edited by R. Michael Alvarez http://pan.oxfordjournals.org/ We propose the use of Kernel Regularized Least Squares (KRLS) for social science modeling and inference problems. KRLS borrows from machine learning methods designed to solve regression and classification problems without relying on linearity or additivity assumptions. The method constructs a flexible hypothesis space that uses kernels as radial basis functions and finds the best-fitting surface in this space by minimizing a complexity-penalized least squares problem. We argue that the method is well-suited for social science inquiry because it avoids strong parametric assumptions, yet allows interpretation in ways at Stanford University on December 4, 2015 analogous to generalized linear models while also permitting more complex interpretation to examine nonlinearities, interactions, and heterogeneous effects. We also extend the method in several directions to make it more effective for social inquiry, by (1) deriving estimators for the pointwise marginal effects and their variances, (2) establishing unbiasedness, consistency, and asymptotic normality of the KRLS estimator under fairly general conditions, (3) proposing a simple automated rule for choosing the kernel bandwidth, and (4) providing companion software.
    [Show full text]
  • Deconvoluting Kernel Density Estimation and Regression for Locally Diferentially Private Data Farhad Farokhi
    www.nature.com/scientificreports OPEN Deconvoluting kernel density estimation and regression for locally diferentially private data Farhad Farokhi Local diferential privacy has become the gold-standard of privacy literature for gathering or releasing sensitive individual data points in a privacy-preserving manner. However, locally diferential data can twist the probability density of the data because of the additive noise used to ensure privacy. In fact, the density of privacy-preserving data (no matter how many samples we gather) is always fatter in comparison with the density function of the original data points due to convolution with privacy- preserving noise density function. The efect is especially more pronounced when using slow-decaying privacy-preserving noises, such as the Laplace noise. This can result in under/over-estimation of the heavy-hitters. This is an important challenge facing social scientists due to the use of diferential privacy in the 2020 Census in the United States. In this paper, we develop density estimation methods using smoothing kernels. We use the framework of deconvoluting kernel density estimators to remove the efect of privacy-preserving noise. This approach also allows us to adapt the results from non-parametric regression with errors-in-variables to develop regression models based on locally diferentially private data. We demonstrate the performance of the developed methods on fnancial and demographic datasets. Government regulations, such as the roll-out of the General Data Protection Regulation in the European Union (EU) (https ://gdpr-info.eu), the California Consumer Privacy Act (https ://oag.ca.gov/priva cy/ccpa), and the development of the Data Sharing and Release Bill in Australia (https ://www.pmc.gov.au/publi c-data/data-shari ng-and-relea se-refor ms.) increasingly prohibit sharing customer’s data without explicit consent 1.
    [Show full text]