Predictive Models in Health Research Department of Statistics Department of Biomedical Data Science

41st Meeting of the International Society for Clinical Biostatistics ISCB, Krakow, Poland, August 23-27, 2020

1 / 31 Some Take Home Messages

This talk is about supervised learning: building models from data that predict an outcome using a collection of input features. • There are some powerful and exciting tools for making predictions from data. • They are not magic! You should be skeptical. They require good data and proper internal validation. • Human judgement and ingenuity are essential for their success. • They may be overkill — if simpler and more traditional methods do as well, they are preferable (Occam’s razor).

2 / 31 Some Definitions

Machine Learning constructs algorithms that can learn from data. Statistical Learning is a branch of applied statistics that emerged in response to , emphasizing statistical models and assessment of uncertainty. Data Science is the extraction of knowledge from data, using ideas from mathematics, statistics, machine learning, computer science, engineering, ... All of these are very similar — with different emphases.

3 / 31 Simple? Yes Done properly in practice? Rarely

In God we trust. All others bring data.∗

∗ Statistical “proverb” sometimes attributed to W. Edwards Deming.

Internal Model Validation

• IMPORTANT! Don’t trust me or anyone who says they have a wonderful machine learning algorithm, unless you see the results of a careful internal validation. • Eg: divide data into two parts A and B. Run algorithm on part A and then test it on part B. Algorithm must not have seen any of the data in part B. • If it works in part B, you have (some) confidence in it

4 / 31 Done properly in practice? Rarely

In God we trust. All others bring data.∗

∗ Statistical “proverb” sometimes attributed to W. Edwards Deming.

Internal Model Validation

• IMPORTANT! Don’t trust me or anyone who says they have a wonderful machine learning algorithm, unless you see the results of a careful internal validation. • Eg: divide data into two parts A and B. Run algorithm on part A and then test it on part B. Algorithm must not have seen any of the data in part B. • If it works in part B, you have (some) confidence in it Simple? Yes

4 / 31 In God we trust. All others bring data.∗

∗ Statistical “proverb” sometimes attributed to W. Edwards Deming.

Internal Model Validation

• IMPORTANT! Don’t trust me or anyone who says they have a wonderful machine learning algorithm, unless you see the results of a careful internal validation. • Eg: divide data into two parts A and B. Run algorithm on part A and then test it on part B. Algorithm must not have seen any of the data in part B. • If it works in part B, you have (some) confidence in it Simple? Yes Done properly in practice? Rarely

4 / 31 ∗

∗ Statistical “proverb” sometimes attributed to W. Edwards Deming.

Internal Model Validation

• IMPORTANT! Don’t trust me or anyone who says they have a wonderful machine learning algorithm, unless you see the results of a careful internal validation. • Eg: divide data into two parts A and B. Run algorithm on part A and then test it on part B. Algorithm must not have seen any of the data in part B. • If it works in part B, you have (some) confidence in it Simple? Yes Done properly in practice? Rarely

In God we trust. All others bring data.

4 / 31 Internal Model Validation

• IMPORTANT! Don’t trust me or anyone who says they have a wonderful machine learning algorithm, unless you see the results of a careful internal validation. • Eg: divide data into two parts A and B. Run algorithm on part A and then test it on part B. Algorithm must not have seen any of the data in part B. • If it works in part B, you have (some) confidence in it Simple? Yes Done properly in practice? Rarely

In God we trust. All others bring data.∗

∗ Statistical “proverb” sometimes attributed to W. Edwards Deming.

4 / 31 vary in shape. These call for different approaches.

Wide Data Thousands / Millions of Variables

Hundreds of Samples Screening and fdr, Lasso, SVM, Stepwise

Tall Data We have too many variables; prone to overfitting. Need to remove variables, or regularize, or both.

Tens / Hundreds of Variables

Thousands / Millions of Samples GLM, Random Forests, Boosting, Deep Learning

Sometimes simple models (linear) don’t suffice. We have enough samples to fit nonlinear models with many interactions, and not too many variables. Good automatic methods for doing this.

5 / 31 Big data vary in shape. These call for different approaches.

Tall and Wide Data Thousands / Millions of Variables

Millions to Billions of Samples

Tricks of the Trade

Exploit sparsity Random projections / hashing Variable screening Subsample rows Divide and recombine Case/ control sampling MapReduce ADMM (divide and conquer) . . .

6 / 31 Outline of remainder of talk

• Large scale lasso models using glmnet (tall and wide) • Random Forest used as ensemble learner (tall) • Deep learning used for image classification (wide) • Machine learning overkill (very wide)

7 / 31 glmnet Fit regularization paths for a variety of GLMs with lasso and elastic net penalties; e.g. logistic regression p Pr(Y = 1 | X = x) X log = β + x β Pr(Y = 0 | X = x) 0 j j j=1 • Lasso penalty [Tibshirani, 1996] induces sparsity in Pp coefficients: j=1 |βj| ≤ s. It shrinks them toward zero, and sets many to zero. • Fit efficiently using coordinate descent. Handles sparse X naturally, and exploits sparsity of solutions, warms starts, variable screening, and includes methods for model selection using cross-validation. glmnet team: TH, Jerome Friedman, Rob Tibshirani, Noah Simon, Junyang Qian, Balasubramanian Narasimhan, Ken Tay.

8 / 31 9 / 31 UK Biobank Genomic Data 15 36 91 245 599 1413 3209 6888 13395 22803 34754 47673

● Lasso (train) 0.85 ● ● Lasso (val) ● ● ● ● ● ● ● ● ReLasso (train) ● Build phenotype ● ● ● ● ● ReLasso (val) ● ● ● ● ●

0.80 ● ● ● prediction models ● ● ● ● ● ● ● ● ● ● ● using genotypes at ● ● ● ● 0.75 ● ● ● ● ● ● 800K loci (SNPs) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.70 ● ● for 500K ● ● ● ● ● ● ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● R ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● individuals. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.65 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Here we predict ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Standing Height ● ● ● ● ● 0.60 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● using lasso model ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.55 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● via , and ● ● ● glmnet ● achieve R2 = 70%. 0.50

0 20 40 60 80

Lambda Index

10 / 31 bioRxiv preprint first posted online May. 7, 2019; doi: http://dx.doi.org/10.1101/630079. The copyright holder for this preprint (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.

Algorithmic details in pictures

Iteration 1 Iteration 2 0.4 0.4 0.2 0.2 0.0 0.0 Coefficients Coefficients −0.2 −0.2 −0.4 −0.4

0 10 20 30 40 0 10 20 30 40

Lambda Index completed fit Lambda Index new valid fit new invalid fit Iteration 3 Iteration 4 0.4 0.4 0.2 0.2 0.0 0.0 Coefficients Coefficients −0.2 −0.2 −0.4 −0.4

0 10 20 30 40 0 10 20 30 40

Lambda Index Lambda Index 11 / 31 Figure 1: The lasso coefficient profile that shows the progression of the BASIL algorithm. The previously finished part of the path is colored grey, the newly completed and verified is in green, and the part that is newly computed but failed the verification is colored red.

of expensive disk read operations. At each iteration, we roll out the solution path progressively, which is illustrated in Figure 1 and will be detailed in the next section. In addition, we propose optimization specific for the SNP data in the UK Biobank studies to speed up the procedure.

1.4 Outline of the paper The rest of the paper is organized as follows. Section 2 describes the proposed batch screening iterative lasso (BASIL) algorithm for the • Gaussian family in detail and its extension to other problems such as logistic regression. Section 3 discusses related methods and packages for solving large-scale lasso problems. •

4 UK Biobank — Asthma

Build polygenic risk scores for disease phenotypes

Odds ratio plot - Polygenic Risk Score percentiles

odds ratio 4 Cases Controls 45k 3.5 40k

3 35k

2.5 30k

25k 2 Asthma odds ratio 20k Number of individuals 1.5 15k

1 10k

5k 0.5 0 0 20 40 60 80 100 Asthma PRS Percentile

12 / 31 UK Biobank — Asthma

13 / 31 UK Biobank — Asthma

Age of first event - Asthma Top 1% 0.5

0.4

Top 5%

0.3 Top 10%

Proportion Asthma event 0.2

0.1 40-60%

Bottom 10%

0 10 20 30 40 50 60 70 80 Age

14 / 31 UK Biobank — Asthma

Variable Level N Events Hazard Ratio P value

Polygenic risk score 40-60% 62139 5017 Reference

Bottom 10% 31036 896 0.36 (0.34, 0.39) <0.001

Top 1% 3085 1494 7.31 (6.90, 7.74) <0.001

Top 10% 15480 3636 3.06 (2.93, 3.20) <0.001

Top 5% 12403 3894 4.27 (4.09, 4.45) <0.001

Eosinophil threshold below .18 73655 6392 Reference

above .18 50488 8545 1.61 (1.56, 1.67) <0.001

Sex Female 66292 8330 Reference

Male 57851 6607 0.87 (0.85, 0.90) <0.001

Body Mass Index Below 25 40969 4414 Reference

25-30 53273 6220 1.04 (1.00, 1.08) 0.06

30-35 21600 2880 1.12 (1.07, 1.17) <0.001

35-40 6010 954 1.26 (1.17, 1.35) <0.001

Above 40 2291 469 1.59 (1.44, 1.75) <0.001

15 / 31 More on glmnet

Current version 4.02. Check out the website at glmnet.stanford.edu, and find the four vignettes there. • All GLM family() choices are accommodated, along with the original hardwired families. e.g. family=binomial(link="probit"), family=negative.binomial(theta=5) • New relax=TRUE argument which allows for refitting the active set without regularization. • C index for survival models • New functions for assessing fits • Convenient functions for fitting big GLMs • Progress bar via trace=TRUE

16 / 31 Predicting the Pathogenicity of Missense Variants

Goal: prioritize list of candidate genes for prostate cancer

Joint work with Epidemiology colleagues Weiva Sieh, Nilah Monnier Ioannidis, Joe Rothstein, Alice Whittemore, ···

REVEL — rare exome variant ensemble learner

Ioannidis, N., ··· , Sieh, W. (Oct 2016) Amer. J. Human Genetics

17 / 31 Approach

• A number of existing scores for disease status do not always agree (e.g SIFT, MutPred). • Idea is to use a Random Forest algorithm to integrate these scores into a single consensus score for predicting disease. • We will use existing functional prediction scores, conservation scores, etc as features — 12 features in all. • Data acquired through Human Gene Mutation Database, SwissVar and ClinVar. Neutral Disease Train 123,706 6,182 Test 2,406 1,953

18 / 31 Correlation of Features A B 1 0.20 phyloP (primate) phastCons (primate) phastCons (placental) SiPhy 0.8 GERP++ RS 0.15 phyloP (placental) MutationTaster LRT 0.6 phyloP (vertebrate) 0.10 phastCons (vertebrate) FATHMM 0.4 MutPred

SIFT Relative importance MutationAssessor 0.05 PROVEAN 0.2 VEST Polyphen2 HDIV Polyphen2 HVAR 0 0 LRT LRT SIFT SIFT VEST SiPhy VEST SiPhy MutPred MutPred FATHMM FATHMM PROVEAN PROVEAN GERP++ RS GERP++ RS MutationTaster MutationTaseter phyloP (primate) phyloP (primate) Polyphen2 HDIV Polyphen2 HDIV Polyphen2 HVAR Polyphen2 HVAR MutationAssessor MutationAssessor phyloP (placental) phyloP (placental) phyloP (vertebrate) phyloP (vertebrate) phastCons (primate) phastCons (primate) phastCons (placental) phastCons (placental) phastCons (vertebrate) phastCons (vertebrate)

19 / 31 Decision Trees

SIFT > .9 x

SLR > .8 LRT > .8 x x

GERP >.2 Pr(D)=.65 Pr(D)=.60 SIFT > .5 x x

Pr(D)=.95 Pr(D)=.75 Pr(D)=.55 Pr(D)=.25

Pr(D)=.75 Trees use the features to create subgroups in the data to refine the estimate of disease. Shallow trees are too coarse/inaccurate. 20 / 31 Random Forests

Leo Breiman (1928–2005)

• Deep trees (fine subgroups) are more accurate, but very noisy. • Idea: fit many (1000s) different and very-deep trees, and average their predictions to reduce the noise. • How to get different trees? - Grow trees to bootstrap subsampled versions of the data. - Randomly ignore variables as candidates for splits. Random Forests are very effective and give accurate predictions. They are automatic, and give good CV estimates of prediction error (for free!). R package RandomForest.

21 / 31 Results for REVEL A B 1.0 1.00

0.95 0.8 0.90

0.6 0.85

AUC 0.80 0.4 REVEL (0.957) REVEL MetaLR (0.917) MetaLR True positive rate MetaSVM (0.933) 0.75 MetaSVM KGGSeq (0.893) KGGSeq 0.2 CONDEL (0.898) CONDEL Eigen (0.910) 0.70 Eigen CADD (0.902) CADD DANN (0.833) DANN 0 0.65 0 0.2 0.4 0.6 0.8 1.0 All (>0) <0.1 0.1-0.5 0.5-1.0 1.0-3.0 >3.0 False positive rate Neutral variant AF range [%] Performance evaluated on independent test set, and REVEL compared with 7 other ensemble competitors.

22 / 31 A B Feature Importance 1 0.20 phyloP (primate) phastCons (primate) phastCons (placental) SiPhy 0.8 GERP++ RS 0.15 phyloP (placental) MutationTaster LRT 0.6 phyloP (vertebrate) 0.10 phastCons (vertebrate) FATHMM 0.4 MutPred

SIFT Relative importance MutationAssessor 0.05 PROVEAN 0.2 VEST Polyphen2 HDIV Polyphen2 HVAR 0 0 LRT LRT SIFT SIFT VEST SiPhy VEST SiPhy MutPred MutPred FATHMM FATHMM PROVEAN PROVEAN GERP++ RS GERP++ RS MutationTaster MutationTaseter phyloP (primate) phyloP (primate) Polyphen2 HDIV Polyphen2 HDIV Polyphen2 HVAR Polyphen2 HVAR MutationAssessor MutationAssessor phyloP (placental) phyloP (placental) phyloP (vertebrate) phyloP (vertebrate) phastCons (primate) phastCons (primate) phastCons (placental) phastCons (placental) phastCons (vertebrate) phastCons (vertebrate)

23 / 31 Deep Learning

8 16 32 8 4 32 16 100 500 32 2 pool pool convolve pool convolve flatten convolve Convolutional neural network (CNN) for image modeling.

• Big successes with image classification. Revolutionary! • Many digital images in healthcare: mammograms, MRI scans, X-rays, optical scans, skin photographs, ... • Modern software tools accessible to all (e.g. keras and tensorflow in R).

24 / 31 Diabetic Retinopathy

25 / 31 Two recent articles (2019-20)

26 / 31 Validation Set Performance for Referable Diabetic Retinopathy

[A] 8788 images, 8 experts [B] 1745 images, 7 experts

27 / 31 Image Segmentation

Deep Learning CNN on MRI scans

Identify deep cervical spine extensors and measure muscle fat in- filtration after whiplash injury

Adapt pretrained net- work

Compare with expert raters

28 / 31 GT = Ground Truth (average of three raters) CNN = Convolutional Neural Network 29 / 31 Overkill - article in Nature Medicine 2001

Simple linear method does as well, and does gene selection Elements of Statistical Learning, Section 18.2

30 / 31 ··· and now for some cheap marketing ... 9781107149892 Efron & Hastie JKT C M Y K Hastie • Tibshirani • Friedman

STS Efron & hasti Springer Series in Statistics “HowSpringer and why isSeries computational in Statistics statistics taking over the world? In this serious Springer Texts in Statistics James · Witten · Hastie Tibshirani Springer Texts in Statistics is Max H. Stein Professor, The twenty-first century has seen a work of synthesis that is also fun to read, Efron and Hastie give their take on the Professor of Statistics, and Professor of breathtaking expansion of statistical Gareth James · · Trevor Hastie · unreasonable effectiveness of statistics and machine learning in the context of a Biomedical Data Science at Stanford University. methodology, both in scope and in An Introduction to Statistical Learning series of clear, historically informed examples.” He has held visiting faculty appointments at influence. “Big data,” “data science,” with Applications in R — Andrew Gelman, Columbia University Harvard, UC Berkeley, and Imperial College and “machine learning” have become

London. Efron has worked extensively on familiar terms in the news, as statistical Gareth James “Computer Age Statistical Inference is written especially for those who want to hear An Introduction to Statistical Learning provides an accessible overview of the fi eld theories of statistical inference, and is the the big ideas, and see them instantiated through the essential mathematics that methods are brought to bear upon the of statistical learning, an essential toolset for making sense of the vast and complex inventor of the bootstrap sampling technique. defines statistical analysis. It makes a great supplement to the traditional curricula enormous data sets of modern science data sets that have emerged in fi elds ranging from biology to fi nance to marketing to Daniela Witten Trevor Hastie He received the National Medal of Science in for beginning graduate students.” and commerce. How did we get here? astrophysics in the past twenty years. Th is book presents some of the most important Trevor Hastie • Robert Tibshirani • Jerome Friedman 2005 and the Guy Medal in Gold of the Royal e And where are we going? modeling and prediction techniques, along with relevant applications. Topics include Trevor Hastie — Rob Kass, Carnegie Mellon University Statistical Society in 2014. Robert Tibshirani linear regression, classifi cation, resampling methods, shrinkage approaches, tree-based The Elements of Statictical Learning This book takes us on an exhilarating methods, support vector machines, clustering, and more. Color graphics and real-world “This is a terrific book. It gives a clear, accessible, and entertaining account of the Robert Tibshirani Trevor Hastie journey through the revolution in data

is John A. Overdeck Professor, S Comput examples are used to illustrate the methods presented. Since the goal of this textbook Jerome Friedmaninterplay between theory and methodological development that has driven statistics analysis following the introduction of

Professor of Statistics, and Professor of in the computer age. The authors succeed brilliantly in locating contemporary tatisti is to facilitate the use of these statistical learning techniques by practitioners in sci- electronic computation in the 1950s. ence, industry, and other fi elds, each chapter contains a tutorial on implementing the Biomedical Data Science at Stanford University. algorithmic methodologies for analysis of ‘big data’ within the framework of Beginning with classical inferential analyses and methods presented in R, an extremely popular open source statistical He is coauthor of Elements of Statistical established statistical theory.” During the past decade there has been an explosion in computation and information tech- theories – Bayesian, frequentist, Fisherian soft ware platform. Learning, a key text in the field of modern — Alastair Young, Imperial College London nology. With it have come vast amounts of data in a variety of fields such as medicine, biolo- – individual chapters take up a series Two of the authors co-wrote Th e Elements of Statistical Learning (Hastie, Tibshirani 1 gy, finance, and marketing. The challenge of understanding these data has led to the devel- Bradley Efron data analysis. He is also known for his work An Introduction of influential topics: survival analysis, and Friedman, 2nd edition 2009), a popular reference book for statistics and machine opment of new tools in the field of statistics, and spawned new areas such as data mining, on generalized additive models andThe principal Elements“This is a guided tourof of modern statistics that emphasizes the conceptual and logistic regression, empirical Bayes, the learning researchers. An Introduction to Statistical Learning covers many of the same machine learning, and bioinformatics. Many of these tools have common underpinnings but computational advances of the last century. Authored by two masters of the field, it A e r curves, and for his contributions to the R c Trevor Hastie topics, but at a level accessible to a much broader audience. Th is book is targeted at are often expressed with different terminology. This book describes the important ideas in Learning The Elements of Statistical offers just the right mix of mathematical analysis and insightful commentary.” jackknife and bootstrap, random forests, An Introduction to Statistical Learning computing environment. Hastie was awarded I al statisticians and non-statisticians alike who wish to use cutting-edge statistical learn- these areas in a common conceptual framework. While the approach is statistical, the neural networks, Markov chain Monte the Emmanuel and Carol Parzen prize for — Hal Varian, Google ing techniques to analyze their data. Th e text assumes only a previous course in linear to Statisticalemphasis is on concepts rather than mathematics. Many examples are given, with a liberal Carlo, inference after model selection,

regression and no knowledge of matrix algebra. use of color graphics. It should be a valuable resource for statisticians and anyone interested Statistical Innovation in 2014. Statistical Learning g and dozens more. The distinctly modern in data mining in science or industry. The book’s coverage is broad, from supervised learning “Efron and Hastie guide us through the maze of breakthrough statistical Gareth James nf

is a professor of statistics at University of Southern California. He has e (prediction) to unsupervised learning. The many topics include neural networks, support Institute of Mathematical Statistics methodologies following the computing evolution: why they were developed, their approach integrates methodology and

published an extensive body of methodological work in the domain of statistical learn- vector machines, classification trees and boosting—the first comprehensive treatment of this properties, and how they are used. Highlighting their origins, the book helps us algorithms with statistical inference. The

Monographs ing with particular emphasis on high-dimensional and functional data. Th e conceptual Learningtopic in any book. Data Mining,Inference,andunderstand each method’s Prediction roles in inference and/or prediction.” e r n book ends with speculation on the future framework for this book grew out of his MBA elective courses in this area. Editorial Board: — Galit Shmueli, National Tsing Hua University Computer Age direction of statistics and data science. Daniela Witten is an assistant professor of biostatistics at . Her This major new edition features many topics not covered in the original, including graphical D. R. Cox () research focuses largely on high-dimensional statistical machine learning. She has models, random forests, ensemble methods, least angle regression & path algorithms for the B. Hambly (University of Oxford) “A masterful guide to how the inferential bases of classical statistics can provide a contributed to the translation of statistical learning techniques to the fi eld of genomics, with Applicationslasso, non-negative in R matrix factorization, and spectral clustering. There is also a chapter on S. Holmes (Stanford University) principled disciplinary frame for the data science of the twenty-first century.” ce through collaborations and as a member of the Institute of Medicine committee that methods for “wide” data (p bigger than n), including multiple testing and false discovery rates. led to the report Evolution of Translational Omics. J. Wellner (University of Washington) — Stephen Stigler, University of Chicago, author of Seven Pillars of Statistical Wisdom Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Statistical Trevor Hastie and Robert Tibshirani are professors of statistics at Stanford University, and Stanford University. They are prominent researchers in this area: Hastie and Tibshirani Second Edition “A refreshing view of modern statistics. Algorithmics are put on equal footing with are co-authors of the successful textbook Elements of Statistical Learning. Hastie and developed generalized additive models and wrote a popular book of that title. Hastie co- intuition, properties, and the abstract arguments behind them. The methods covered Tibshirani developed generalized additive models and wrote a popular book of that developed much of the statistical modeling software and environment in R/S-PLUS and title. Hastie co-developed much of the statistical modeling soft ware and environment are indispensable to practicing statistical analysts in today’s big data and big invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the computing landscape.” in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data- and is co-author of the very successful An Introduction to the Bootstrap. mining tools including CART, MARS, projection pursuit and gradient boosting. — Robert Gramacy, The University of Chicago Booth School of Business Inference Statistics STATISTICS ISBN 978-1-4614-7137-0  ---- Algorithms, Evidence, and Data Science

9 781461 471370 Cover illustration: Pacific Ocean wave, North Shore, Oahu, Hawaii. © Brian Sytnyk / Getty Images. Cover designed by Zoe Naylor. › springer.com Printed in the United Kingdom

Thank you!

Summary

• We saw examples of Lasso, Random Forest, and Deep Learning. • We saw an example where a Neural Networks was overkill, and a simpler model was sufficient.

31 / 31 9781107149892 Efron & Hastie JKT C M Y K Hastie • Tibshirani • Friedman

STS Efron & hasti Springer Series in Statistics “HowSpringer and why isSeries computational in Statistics statistics taking over the world? In this serious Springer Texts in Statistics James · Witten · Hastie Tibshirani Springer Texts in Statistics Bradley Efron is Max H. Stein Professor, The twenty-first century has seen a work of synthesis that is also fun to read, Efron and Hastie give their take on the Professor of Statistics, and Professor of breathtaking expansion of statistical Gareth James · Daniela Witten · Trevor Hastie · Robert Tibshirani unreasonable effectiveness of statistics and machine learning in the context of a Biomedical Data Science at Stanford University. methodology, both in scope and in An Introduction to Statistical Learning series of clear, historically informed examples.” He has held visiting faculty appointments at influence. “Big data,” “data science,” with Applications in R — Andrew Gelman, Columbia University Harvard, UC Berkeley, and Imperial College and “machine learning” have become

London. Efron has worked extensively on familiar terms in the news, as statistical Gareth James “Computer Age Statistical Inference is written especially for those who want to hear An Introduction to Statistical Learning provides an accessible overview of the fi eld theories of statistical inference, and is the the big ideas, and see them instantiated through the essential mathematics that methods are brought to bear upon the of statistical learning, an essential toolset for making sense of the vast and complex inventor of the bootstrap sampling technique. defines statistical analysis. It makes a great supplement to the traditional curricula enormous data sets of modern science data sets that have emerged in fi elds ranging from biology to fi nance to marketing to Daniela Witten Trevor Hastie He received the National Medal of Science in for beginning graduate students.” and commerce. How did we get here? astrophysics in the past twenty years. Th is book presents some of the most important Trevor Hastie • Robert Tibshirani • Jerome Friedman 2005 and the Guy Medal in Gold of the Royal e And where are we going? modeling and prediction techniques, along with relevant applications. Topics include Trevor Hastie — Rob Kass, Carnegie Mellon University Statistical Society in 2014. Robert Tibshirani linear regression, classifi cation, resampling methods, shrinkage approaches, tree-based The Elements of Statictical Learning This book takes us on an exhilarating methods, support vector machines, clustering, and more. Color graphics and real-world “This is a terrific book. It gives a clear, accessible, and entertaining account of the Robert Tibshirani Trevor Hastie journey through the revolution in data

is John A. Overdeck Professor, S Comput examples are used to illustrate the methods presented. Since the goal of this textbook Jerome Friedmaninterplay between theory and methodological development that has driven statistics analysis following the introduction of

Professor of Statistics, and Professor of in the computer age. The authors succeed brilliantly in locating contemporary tatisti is to facilitate the use of these statistical learning techniques by practitioners in sci- electronic computation in the 1950s. ence, industry, and other fi elds, each chapter contains a tutorial on implementing the Biomedical Data Science at Stanford University. algorithmic methodologies for analysis of ‘big data’ within the framework of Beginning with classical inferential analyses and methods presented in R, an extremely popular open source statistical He is coauthor of Elements of Statistical established statistical theory.” During the past decade there has been an explosion in computation and information tech- theories – Bayesian, frequentist, Fisherian soft ware platform. Learning, a key text in the field of modern — Alastair Young, Imperial College London nology. With it have come vast amounts of data in a variety of fields such as medicine, biolo- – individual chapters take up a series Two of the authors co-wrote Th e Elements of Statistical Learning (Hastie, Tibshirani 1 gy, finance, and marketing. The challenge of understanding these data has led to the devel- Bradley Efron data analysis. He is also known for his work An Introduction of influential topics: survival analysis, and Friedman, 2nd edition 2009), a popular reference book for statistics and machine opment of new tools in the field of statistics, and spawned new areas such as data mining, on generalized additive models andThe principal Elements“This is a guided tourof of modern statistics that emphasizes the conceptual and logistic regression, empirical Bayes, the learning researchers. An Introduction to Statistical Learning covers many of the same machine learning, and bioinformatics. Many of these tools have common underpinnings but computational advances of the last century. Authored by two masters of the field, it A e r curves, and for his contributions to the R c Trevor Hastie topics, but at a level accessible to a much broader audience. Th is book is targeted at are often expressed with different terminology. This book describes the important ideas in Learning The Elements of Statistical offers just the right mix of mathematical analysis and insightful commentary.” jackknife and bootstrap, random forests, An Introduction to Statistical Learning computing environment. Hastie was awarded I al statisticians and non-statisticians alike who wish to use cutting-edge statistical learn- these areas in a common conceptual framework. While the approach is statistical, the neural networks, Markov chain Monte the Emmanuel and Carol Parzen prize for — Hal Varian, Google ing techniques to analyze their data. Th e text assumes only a previous course in linear to Statisticalemphasis is on concepts rather than mathematics. Many examples are given, with a liberal Carlo, inference after model selection,

regression and no knowledge of matrix algebra. use of color graphics. It should be a valuable resource for statisticians and anyone interested Statistical Innovation in 2014. Statistical Learning g and dozens more. The distinctly modern in data mining in science or industry. The book’s coverage is broad, from supervised learning “Efron and Hastie guide us through the maze of breakthrough statistical Gareth James nf

is a professor of statistics at University of Southern California. He has e (prediction) to unsupervised learning. The many topics include neural networks, support Institute of Mathematical Statistics methodologies following the computing evolution: why they were developed, their approach integrates methodology and

published an extensive body of methodological work in the domain of statistical learn- vector machines, classification trees and boosting—the first comprehensive treatment of this properties, and how they are used. Highlighting their origins, the book helps us algorithms with statistical inference. The

Monographs ing with particular emphasis on high-dimensional and functional data. Th e conceptual Learningtopic in any book. Data Mining,Inference,andunderstand each method’s Prediction roles in inference and/or prediction.” e r n book ends with speculation on the future framework for this book grew out of his MBA elective courses in this area. Editorial Board: — Galit Shmueli, National Tsing Hua University Computer Age direction of statistics and data science. Daniela Witten is an assistant professor of biostatistics at University of Washington. Her This major new edition features many topics not covered in the original, including graphical D. R. Cox (University of Oxford) research focuses largely on high-dimensional statistical machine learning. She has models, random forests, ensemble methods, least angle regression & path algorithms for the B. Hambly (University of Oxford) “A masterful guide to how the inferential bases of classical statistics can provide a contributed to the translation of statistical learning techniques to the fi eld of genomics, with Applicationslasso, non-negative in R matrix factorization, and spectral clustering. There is also a chapter on S. Holmes (Stanford University) principled disciplinary frame for the data science of the twenty-first century.” ce through collaborations and as a member of the Institute of Medicine committee that methods for “wide” data (p bigger than n), including multiple testing and false discovery rates. led to the report Evolution of Translational Omics. J. Wellner (University of Washington) — Stephen Stigler, University of Chicago, author of Seven Pillars of Statistical Wisdom Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Statistical Trevor Hastie and Robert Tibshirani are professors of statistics at Stanford University, and Stanford University. They are prominent researchers in this area: Hastie and Tibshirani Second Edition “A refreshing view of modern statistics. Algorithmics are put on equal footing with are co-authors of the successful textbook Elements of Statistical Learning. Hastie and developed generalized additive models and wrote a popular book of that title. Hastie co- intuition, properties, and the abstract arguments behind them. The methods covered Tibshirani developed generalized additive models and wrote a popular book of that developed much of the statistical modeling software and environment in R/S-PLUS and title. Hastie co-developed much of the statistical modeling soft ware and environment are indispensable to practicing statistical analysts in today’s big data and big invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the computing landscape.” in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data- and is co-author of the very successful An Introduction to the Bootstrap. mining tools including CART, MARS, projection pursuit and gradient boosting. — Robert Gramacy, The University of Chicago Booth School of Business Inference Statistics STATISTICS ISBN 978-1-4614-7137-0  ---- Algorithms, Evidence, and Data Science

9 781461 471370 Cover illustration: Pacific Ocean wave, North Shore, Oahu, Hawaii. © Brian Sytnyk / Getty Images. Cover designed by Zoe Naylor. › springer.com Printed in the United Kingdom

Thank you!

Summary

• We saw examples of Lasso, Random Forest, and Deep Learning. • We saw an example where a Neural Networks was overkill, and a simpler model was sufficient.

··· and now for some cheap marketing ...

31 / 31 Thank you!

Summary

• We saw examples of Lasso, Random Forest, and Deep Learning. • We saw an example where a Neural Networks was overkill, and a simpler model was sufficient.

··· and now for some cheap marketing ... 9781107149892 Efron & Hastie JKT C M Y K Hastie • Tibshirani • Friedman

STS Efron & hasti Springer Series in Statistics “HowSpringer and why isSeries computational in Statistics statistics taking over the world? In this serious Springer Texts in Statistics James · Witten · Hastie Tibshirani Springer Texts in Statistics Bradley Efron is Max H. Stein Professor, The twenty-first century has seen a work of synthesis that is also fun to read, Efron and Hastie give their take on the Professor of Statistics, and Professor of breathtaking expansion of statistical Gareth James · Daniela Witten · Trevor Hastie · Robert Tibshirani unreasonable effectiveness of statistics and machine learning in the context of a Biomedical Data Science at Stanford University. methodology, both in scope and in An Introduction to Statistical Learning series of clear, historically informed examples.” He has held visiting faculty appointments at influence. “Big data,” “data science,” with Applications in R — Andrew Gelman, Columbia University Harvard, UC Berkeley, and Imperial College and “machine learning” have become

London. Efron has worked extensively on familiar terms in the news, as statistical Gareth James “Computer Age Statistical Inference is written especially for those who want to hear An Introduction to Statistical Learning provides an accessible overview of the fi eld theories of statistical inference, and is the the big ideas, and see them instantiated through the essential mathematics that methods are brought to bear upon the of statistical learning, an essential toolset for making sense of the vast and complex inventor of the bootstrap sampling technique. defines statistical analysis. It makes a great supplement to the traditional curricula enormous data sets of modern science data sets that have emerged in fi elds ranging from biology to fi nance to marketing to Daniela Witten Trevor Hastie He received the National Medal of Science in for beginning graduate students.” and commerce. How did we get here? astrophysics in the past twenty years. Th is book presents some of the most important Trevor Hastie • Robert Tibshirani • Jerome Friedman 2005 and the Guy Medal in Gold of the Royal e And where are we going? modeling and prediction techniques, along with relevant applications. Topics include Trevor Hastie — Rob Kass, Carnegie Mellon University Statistical Society in 2014. Robert Tibshirani linear regression, classifi cation, resampling methods, shrinkage approaches, tree-based The Elements of Statictical Learning This book takes us on an exhilarating methods, support vector machines, clustering, and more. Color graphics and real-world “This is a terrific book. It gives a clear, accessible, and entertaining account of the Robert Tibshirani Trevor Hastie journey through the revolution in data

is John A. Overdeck Professor, S Comput examples are used to illustrate the methods presented. Since the goal of this textbook Jerome Friedmaninterplay between theory and methodological development that has driven statistics analysis following the introduction of

Professor of Statistics, and Professor of in the computer age. The authors succeed brilliantly in locating contemporary tatisti is to facilitate the use of these statistical learning techniques by practitioners in sci- electronic computation in the 1950s. ence, industry, and other fi elds, each chapter contains a tutorial on implementing the Biomedical Data Science at Stanford University. algorithmic methodologies for analysis of ‘big data’ within the framework of Beginning with classical inferential analyses and methods presented in R, an extremely popular open source statistical He is coauthor of Elements of Statistical established statistical theory.” During the past decade there has been an explosion in computation and information tech- theories – Bayesian, frequentist, Fisherian soft ware platform. Learning, a key text in the field of modern — Alastair Young, Imperial College London nology. With it have come vast amounts of data in a variety of fields such as medicine, biolo- – individual chapters take up a series Two of the authors co-wrote Th e Elements of Statistical Learning (Hastie, Tibshirani 1 gy, finance, and marketing. The challenge of understanding these data has led to the devel- Bradley Efron data analysis. He is also known for his work An Introduction of influential topics: survival analysis, and Friedman, 2nd edition 2009), a popular reference book for statistics and machine opment of new tools in the field of statistics, and spawned new areas such as data mining, on generalized additive models andThe principal Elements“This is a guided tourof of modern statistics that emphasizes the conceptual and logistic regression, empirical Bayes, the learning researchers. An Introduction to Statistical Learning covers many of the same machine learning, and bioinformatics. Many of these tools have common underpinnings but computational advances of the last century. Authored by two masters of the field, it A e r curves, and for his contributions to the R c Trevor Hastie topics, but at a level accessible to a much broader audience. Th is book is targeted at are often expressed with different terminology. This book describes the important ideas in Learning The Elements of Statistical offers just the right mix of mathematical analysis and insightful commentary.” jackknife and bootstrap, random forests, An Introduction to Statistical Learning computing environment. Hastie was awarded I al statisticians and non-statisticians alike who wish to use cutting-edge statistical learn- these areas in a common conceptual framework. While the approach is statistical, the neural networks, Markov chain Monte the Emmanuel and Carol Parzen prize for — Hal Varian, Google ing techniques to analyze their data. Th e text assumes only a previous course in linear to Statisticalemphasis is on concepts rather than mathematics. Many examples are given, with a liberal Carlo, inference after model selection,

regression and no knowledge of matrix algebra. use of color graphics. It should be a valuable resource for statisticians and anyone interested Statistical Innovation in 2014. Statistical Learning g and dozens more. The distinctly modern in data mining in science or industry. The book’s coverage is broad, from supervised learning “Efron and Hastie guide us through the maze of breakthrough statistical Gareth James nf

is a professor of statistics at University of Southern California. He has e (prediction) to unsupervised learning. The many topics include neural networks, support Institute of Mathematical Statistics methodologies following the computing evolution: why they were developed, their approach integrates methodology and

published an extensive body of methodological work in the domain of statistical learn- vector machines, classification trees and boosting—the first comprehensive treatment of this properties, and how they are used. Highlighting their origins, the book helps us algorithms with statistical inference. The

Monographs ing with particular emphasis on high-dimensional and functional data. Th e conceptual Learningtopic in any book. Data Mining,Inference,andunderstand each method’s Prediction roles in inference and/or prediction.” e r n book ends with speculation on the future framework for this book grew out of his MBA elective courses in this area. Editorial Board: — Galit Shmueli, National Tsing Hua University Computer Age direction of statistics and data science. Daniela Witten is an assistant professor of biostatistics at University of Washington. Her This major new edition features many topics not covered in the original, including graphical D. R. Cox (University of Oxford) research focuses largely on high-dimensional statistical machine learning. She has models, random forests, ensemble methods, least angle regression & path algorithms for the B. Hambly (University of Oxford) “A masterful guide to how the inferential bases of classical statistics can provide a contributed to the translation of statistical learning techniques to the fi eld of genomics, with Applicationslasso, non-negative in R matrix factorization, and spectral clustering. There is also a chapter on S. Holmes (Stanford University) principled disciplinary frame for the data science of the twenty-first century.” ce through collaborations and as a member of the Institute of Medicine committee that methods for “wide” data (p bigger than n), including multiple testing and false discovery rates. led to the report Evolution of Translational Omics. J. Wellner (University of Washington) — Stephen Stigler, University of Chicago, author of Seven Pillars of Statistical Wisdom Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Statistical Trevor Hastie and Robert Tibshirani are professors of statistics at Stanford University, and Stanford University. They are prominent researchers in this area: Hastie and Tibshirani Second Edition “A refreshing view of modern statistics. Algorithmics are put on equal footing with are co-authors of the successful textbook Elements of Statistical Learning. Hastie and developed generalized additive models and wrote a popular book of that title. Hastie co- intuition, properties, and the abstract arguments behind them. The methods covered Tibshirani developed generalized additive models and wrote a popular book of that developed much of the statistical modeling software and environment in R/S-PLUS and title. Hastie co-developed much of the statistical modeling soft ware and environment are indispensable to practicing statistical analysts in today’s big data and big invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the computing landscape.” in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data- and is co-author of the very successful An Introduction to the Bootstrap. mining tools including CART, MARS, projection pursuit and gradient boosting. — Robert Gramacy, The University of Chicago Booth School of Business Inference Statistics STATISTICS ISBN 978-1-4614-7137-0  ---- Algorithms, Evidence, and Data Science

9 781461 471370 Cover illustration: Pacific Ocean wave, North Shore, Oahu, Hawaii. © Brian Sytnyk / Getty Images. Cover designed by Zoe Naylor. › springer.com Printed in the United Kingdom

31 / 31 Summary

• We saw examples of Lasso, Random Forest, and Deep Learning. • We saw an example where a Neural Networks was overkill, and a simpler model was sufficient.

··· and now for some cheap marketing ... 9781107149892 Efron & Hastie JKT C M Y K Hastie • Tibshirani • Friedman

STS Efron & hasti Springer Series in Statistics “HowSpringer and why isSeries computational in Statistics statistics taking over the world? In this serious Springer Texts in Statistics James · Witten · Hastie Tibshirani Springer Texts in Statistics Bradley Efron is Max H. Stein Professor, The twenty-first century has seen a work of synthesis that is also fun to read, Efron and Hastie give their take on the Professor of Statistics, and Professor of breathtaking expansion of statistical Gareth James · Daniela Witten · Trevor Hastie · Robert Tibshirani unreasonable effectiveness of statistics and machine learning in the context of a Biomedical Data Science at Stanford University. methodology, both in scope and in An Introduction to Statistical Learning series of clear, historically informed examples.” He has held visiting faculty appointments at influence. “Big data,” “data science,” with Applications in R — Andrew Gelman, Columbia University Harvard, UC Berkeley, and Imperial College and “machine learning” have become

London. Efron has worked extensively on familiar terms in the news, as statistical Gareth James “Computer Age Statistical Inference is written especially for those who want to hear An Introduction to Statistical Learning provides an accessible overview of the fi eld theories of statistical inference, and is the the big ideas, and see them instantiated through the essential mathematics that methods are brought to bear upon the of statistical learning, an essential toolset for making sense of the vast and complex inventor of the bootstrap sampling technique. defines statistical analysis. It makes a great supplement to the traditional curricula enormous data sets of modern science data sets that have emerged in fi elds ranging from biology to fi nance to marketing to Daniela Witten Trevor Hastie He received the National Medal of Science in for beginning graduate students.” and commerce. How did we get here? astrophysics in the past twenty years. Th is book presents some of the most important Trevor Hastie • Robert Tibshirani • Jerome Friedman 2005 and the Guy Medal in Gold of the Royal e And where are we going? modeling and prediction techniques, along with relevant applications. Topics include Trevor Hastie — Rob Kass, Carnegie Mellon University Statistical Society in 2014. Robert Tibshirani linear regression, classifi cation, resampling methods, shrinkage approaches, tree-based The Elements of Statictical Learning This book takes us on an exhilarating methods, support vector machines, clustering, and more. Color graphics and real-world “This is a terrific book. It gives a clear, accessible, and entertaining account of the Robert Tibshirani Trevor Hastie journey through the revolution in data

is John A. Overdeck Professor, S Comput examples are used to illustrate the methods presented. Since the goal of this textbook Jerome Friedmaninterplay between theory and methodological development that has driven statistics analysis following the introduction of

Professor of Statistics, and Professor of in the computer age. The authors succeed brilliantly in locating contemporary tatisti is to facilitate the use of these statistical learning techniques by practitioners in sci- electronic computation in the 1950s. ence, industry, and other fi elds, each chapter contains a tutorial on implementing the Biomedical Data Science at Stanford University. algorithmic methodologies for analysis of ‘big data’ within the framework of Beginning with classical inferential analyses and methods presented in R, an extremely popular open source statistical He is coauthor of Elements of Statistical established statistical theory.” During the past decade there has been an explosion in computation and information tech- theories – Bayesian, frequentist, Fisherian soft ware platform. Learning, a key text in the field of modern — Alastair Young, Imperial College London nology. With it have come vast amounts of data in a variety of fields such as medicine, biolo- – individual chapters take up a series Two of the authors co-wrote Th e Elements of Statistical Learning (Hastie, Tibshirani 1 gy, finance, and marketing. The challenge of understanding these data has led to the devel- Bradley Efron data analysis. He is also known for his work An Introduction of influential topics: survival analysis, and Friedman, 2nd edition 2009), a popular reference book for statistics and machine opment of new tools in the field of statistics, and spawned new areas such as data mining, on generalized additive models andThe principal Elements“This is a guided tourof of modern statistics that emphasizes the conceptual and logistic regression, empirical Bayes, the learning researchers. An Introduction to Statistical Learning covers many of the same machine learning, and bioinformatics. Many of these tools have common underpinnings but computational advances of the last century. Authored by two masters of the field, it A e r curves, and for his contributions to the R c Trevor Hastie topics, but at a level accessible to a much broader audience. Th is book is targeted at are often expressed with different terminology. This book describes the important ideas in Learning The Elements of Statistical offers just the right mix of mathematical analysis and insightful commentary.” jackknife and bootstrap, random forests, An Introduction to Statistical Learning computing environment. Hastie was awarded I al statisticians and non-statisticians alike who wish to use cutting-edge statistical learn- these areas in a common conceptual framework. While the approach is statistical, the neural networks, Markov chain Monte the Emmanuel and Carol Parzen prize for — Hal Varian, Google ing techniques to analyze their data. Th e text assumes only a previous course in linear to Statisticalemphasis is on concepts rather than mathematics. Many examples are given, with a liberal Carlo, inference after model selection,

regression and no knowledge of matrix algebra. use of color graphics. It should be a valuable resource for statisticians and anyone interested Statistical Innovation in 2014. Statistical Learning g and dozens more. The distinctly modern in data mining in science or industry. The book’s coverage is broad, from supervised learning “Efron and Hastie guide us through the maze of breakthrough statistical Gareth James nf

is a professor of statistics at University of Southern California. He has e (prediction) to unsupervised learning. The many topics include neural networks, support Institute of Mathematical Statistics methodologies following the computing evolution: why they were developed, their approach integrates methodology and

published an extensive body of methodological work in the domain of statistical learn- vector machines, classification trees and boosting—the first comprehensive treatment of this properties, and how they are used. Highlighting their origins, the book helps us algorithms with statistical inference. The

Monographs ing with particular emphasis on high-dimensional and functional data. Th e conceptual Learningtopic in any book. Data Mining,Inference,andunderstand each method’s Prediction roles in inference and/or prediction.” e r n book ends with speculation on the future framework for this book grew out of his MBA elective courses in this area. Editorial Board: — Galit Shmueli, National Tsing Hua University Computer Age direction of statistics and data science. Daniela Witten is an assistant professor of biostatistics at University of Washington. Her This major new edition features many topics not covered in the original, including graphical D. R. Cox (University of Oxford) research focuses largely on high-dimensional statistical machine learning. She has models, random forests, ensemble methods, least angle regression & path algorithms for the B. Hambly (University of Oxford) “A masterful guide to how the inferential bases of classical statistics can provide a contributed to the translation of statistical learning techniques to the fi eld of genomics, with Applicationslasso, non-negative in R matrix factorization, and spectral clustering. There is also a chapter on S. Holmes (Stanford University) principled disciplinary frame for the data science of the twenty-first century.” ce through collaborations and as a member of the Institute of Medicine committee that methods for “wide” data (p bigger than n), including multiple testing and false discovery rates. led to the report Evolution of Translational Omics. J. Wellner (University of Washington) — Stephen Stigler, University of Chicago, author of Seven Pillars of Statistical Wisdom Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Statistical Trevor Hastie and Robert Tibshirani are professors of statistics at Stanford University, and Stanford University. They are prominent researchers in this area: Hastie and Tibshirani Second Edition “A refreshing view of modern statistics. Algorithmics are put on equal footing with are co-authors of the successful textbook Elements of Statistical Learning. Hastie and developed generalized additive models and wrote a popular book of that title. Hastie co- intuition, properties, and the abstract arguments behind them. The methods covered Tibshirani developed generalized additive models and wrote a popular book of that developed much of the statistical modeling software and environment in R/S-PLUS and title. Hastie co-developed much of the statistical modeling soft ware and environment are indispensable to practicing statistical analysts in today’s big data and big invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the computing landscape.” in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data- and is co-author of the very successful An Introduction to the Bootstrap. mining tools including CART, MARS, projection pursuit and gradient boosting. — Robert Gramacy, The University of Chicago Booth School of Business Inference Statistics STATISTICS ISBN 978-1-4614-7137-0  ---- Algorithms, Evidence, and Data Science

9 781461 471370 Cover illustration: Pacific Ocean wave, North Shore, Oahu, Hawaii. © Brian Sytnyk / Getty Images. Cover designed by Zoe Naylor. › springer.com Printed in the United Kingdom

Thank you!

31 / 31