The analysis of RNA-Seq experiments using approximate likelihood Daniel C. Jones A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2020 Reading Committee: Walter L. Ruzzo, Chair Sreeram Kannan Georg Seelig Program Authorized to Offer Degree: Paul G. Allen School of Computer Science & Engineering ©Copyright 2020 Daniel C. Jones University of Washington Abstract The analysis of RNA-Seq experiments using approximate likelihood Daniel C. Jones Chair of the Supervisory Committee: Walter L. Ruzzo Paul G. Allen School of Computer Science & Engineering The analysis of mRNA transcript abundance with RNA-Seq is a central tool in molecular biology research, but often analyses fail to account for the uncertainty in these estimates, which can be significant, especially when trying to disentangle isoforms or duplicated genes. Preserving uncertainty necessitates a full probabilistic model of the all the sequencing reads, which quickly becomes intractable, as experiments can consist of billions of reads. To overcome these limitations, we propose a new method of approximat- ing the likelihood function of a sparse mixture model, using a technique we call the Pólya tree transformation. We demonstrate that substituting this approximation for the real thing achieves most of the benefits with a fraction of the computational costs, leading to more accurate detection of differential transcript expression. 3 4 For Maddie. Let’s have some more normal ones. Contents 1 the analysis of rna-seq experiments 11 1.1 Introduction 11 1.2 The RNA-Seq quantification problem 12 1.2.1 Methods that avoid deconvolution 13 1.2.2 Methods that embrace deconvolution 15 1.2.3 Approximations 31 1.3 The two-step style of RNA-Seq analysis 33 1.3.1 Dealing with estimation uncertainty 34 2 approximatingtherna-seqlikelihoodfunction 37 2.1 Approximating likelihood with variational inference 37 2.1.1 Practical benefits to likelihood approximation 39 2.1.2 Minimizing KL divergence 39 2.2 Designing an approximation 41 2.2.1 Common distributions on the simplex 42 2.2.2 Transformations onto the simplex 43 2.2.3 Hierarchical stick-breaking 52 2.2.4 Some mathematical properties of the Pólya tree trans- formation 57 2.2.5 Heuristics for tree-based transformations 68 2.2.6 Base distributions 70 2.3 Evaluating approximations 71 2.3.1 A dataset for evaluating goodness of fit 73 2.3.2 Testing procedures 73 2.3.3 The Pólya tree transform improves the goodness of fit to likelihood marginals 75 2.3.4 Approximate likelihood introduces little error into fold-change lower bound estimates 81 2.3.5 Estimating and sampling from approximated likeli- hood can be faster than bootstrap sampling 83 2.4 Notes on implementation 84 2.4.1 Approximating likelihood 85 2.4.2 Bias correction 85 2.4.3 Inference 87 2.5 Conclusion 88 3 applications of approximate likelihood 89 3.1 Tissue classification using random forests 89 3.2 Pairwise correlation between transcripts 90 3.3 Models of transcript differential expression 93 3.3.1 Prior on regression coefficients 94 3.3.2 Modeling variance 94 3.3.3 Scale penalization 94 5 6 c o n t e n t s 3.3.4 Calling differential expression using minimum ef- fect size 95 3.3.5 Modeling gene expression and isoform usage 96 3.4 Approximate likelihood models outperform other models in identifying differentially expressed transcripts 96 3.5 Differential expression calls made with approximate likeli- hood are more internally consistent 97 4 scalingupapproximatelikelihoodanalysis 101 4.1 Fixed topologies with sequence based heuristics 101 4.1.1 Tractable hierarchical clustering of transcripts by 푘-mer sets 102 4.1.2 The 푘-mer heuristic produces an equally accurate approximation 103 4.2 Efficient sampling of transcript expression 103 4.2.1 Exploiting Dirichlet quasi-conjugacy 104 4.2.2 Beta distributions and the Pólya tree transforma- tion 109 5 conclusion 111 a somenotesontheeffectivelengthtransforma- t i o n 113 bibliography 117 Index of mathematical notation Likelihood Notation ℝ+ Positive real numbers 푛 푛+1 푛+1 Δ 푛-dimensional unit simplex, i.e., {푥 ∈ ℝ+ | ∑푖=1 푥푖 = 1} 푟 Set of reads (representation is left unspecified) 푟(푖) Set of reads belonging to the 푖th sample 푥 Relative expression vectors 푥(푖) Relative expression vector (in Δ푛−1) for the 푖th sample. 푧푖푗 Latent indicator variables that is 1 iff read 푖 was generated from transcript 푗 휃 Unspecified model parameters 푚 Number of RNA-seq reads in a particular sample. 푛 Number of annotation transcripts. 푝푗(⋅) Transcript 푗’s probability distribution over reads 풫 Normalized RNA-Seq likelihood function (equivalent to a posterior over expression under uniform prior) ℓ푖 Length of fragment 푖 퐿푗 Length of transcript 푗 핃푗 Effective length of transcript 푗 Tree Notation left(푖) Index of the left child of the node at index 푖 right(푖) Index of the left child of the node at index 푖 leaves(푖) Set of indexes of leaf nodes in the subtree rooted at 푖 internal(푖) Set of indexes of internal (i.e., non-leaf) nodes in the subtree rooted at 푖 ℒ Vector giving the number of internal nodes in each internal node’s left subtree ℛ Vector giving the number of internal nodes in each internal node’s right subtree 7 Overview This dissertation is organized into four chapters. 1. The analysis of RNA-Seq experiments. We give a brief overview of methodologies that have been brought to bear to analyze gene and transcript expression in RNA-Seq experiments over the last twelve years. The dominant probabilistic view of RNA-Seq treats it as a sparse mixture model, with fixed components, where relative mRNA abundance is represented by the mixture coefficients. 2. Approximating the RNA-seq likelihood function. Here we develop a new approximation to the RNA-Seq likelihood function. Though the focus will be on RNA-Seq, this approach is generalizable to other sparse mixture models. The approximation is based on a fully gen- eralized family of stick breaking transformations we call Pólya tree transformations. We prove some useful properties of this class of transformations, and then go on to show the approximation is an ex- tremely accurate despite using only a constant number of parame- ters per transcript. 3. Applications of approximate likelihood. Several applications of ap- proximate likelihood are presented. Most importantly, we build a probabilistic model of differential expression, and show it improves on existing state of the art methods. We also examine simple pair- wise correlation and classification tasks where approximate likeli- hood shows improvements over using point estimates or bootstrap sampling. 4. Scaling up approximate likelihood analysis. Finally, with the ulti- mate goal of scaling up inference to large numbers of cells in isoform level single cell datasets, we describe two possible improvements. First, we show an alternative heuristic for building Pólya tree trans- formations that allows for one size fits all transformations. Second, we explore the idea of quasi-conjugacy that allows efficient sampling from the posterior in models using a Dirichlet prior. Some of the material preseted here previously appeared in the preprint: Jones, Daniel C., and Walter L. Ruzzo. ”Polee: RNA-Seq analy- sis using approximate likelihood.” bioRxiv (2020). 9 The analysis of RNA-seq experiments 1 1.1 Introduction The increasing ability over the past two decades to perform large scale measurement of mRNA abundance has transformed how molecular biol- ogy research is done. Prior to the introduction of complementary DNA microarrays in the mid-1990s (Schena et al., 1995), studying the dynamics of gene expression was limited to overall RNA quantity or to a few specific RNA at a time. With this revolutionary expansion in the scope, microar- rays presented new statistical challenges. Multiple-testing, noise, normal- ization, cross-hybridization, and reconciling multiple vendor’s arrays were all issues tackled in the ensuing years by new software and statistical meth- ods. A similar technological leap was made a decade later with the advent of high-throughput RNA sequencing, or “RNA-seq” (Wang, Gerstein, and Sny- der, 2009). The new technique, enabled in particular by new Illumina’s new sequencers capable of sequencing millions of short (initially, 30-35nt) fragments at a time, promised more sensitivity, accuracy, and flexibility than microarrays. RNA-seq offers expression information at the nucleotide level without being chained to a particular set of probes or annotated genes. When new information becomes available (a better genome assembly or more accurate gene annotations, for example), RNA-seq data can be triv- ially reanalyzed. As was the case with microarrays, the promise of higher fidelity tran- scriptomics came with a set of new computational and statistical challenges. Most of these challenges stem from one limitation: RNA-seq reads are short. Current sequencing technologies are capable of generating long reads with limited throughput (e.g., nanopore sequencing), or short reads with high throughput (e.g., sequencing by synthesis). When the goal is accurate quan- tification, as is usually the case in RNA-seq experiments, the latter is prefer- able. Though the reads have grown longer in the ensuing decade (from 30-35nt to 100-200nt on Illumina’s platform), RNA-seq for the most part does not directly quantify full RNA molecules. The median human RNA transcript length is 794nt (Ensembl, 2018), much longer than the typical read. The overcome this limitation, RNA molecules are fragmented (usually randomly, using sonication), and the ends of these fragments are sequenced with short reads. This approach complicates analysis because it introduces ambiguity. Typically transcripts have unique sequences, but they may dif- fer from a paralog or another isoform of the same gene by only a few bases.