Bittremieux et al. 2017 • Computational quality control tools for 1/12

Computational quality control tools for mass spectrometry proteomics

Wout Bittremieux1,2, Dirk Valkenborg3,4,5, Lennart Martens6,7,8, Kris Laukens1,2

1Department of Mathematics and Computer Science, University of Antwerp, 2020 Antwerp, Belgium; 2Biomedical Informatics Research Center Antwerp (biomina), University of Antwerp/Antwerp University Hospital, 2650 Edegem, Belgium; 3Flemish Institute for Technological Research (VITO), 2400 Mol, Belgium; 4CFP, University of Antwerp, 2020 Antwerp, Belgium; 5I-BioStat, Hasselt University, 3590 Diepenbeek, Belgium; 6Medical Biotechnology Center, VIB, 9000 Ghent, Belgium; 7Department of Biochemistry, Ghent University, 9000 Ghent, Belgium; 7Bioinformatics Institute Ghent, Ghent University, 9052 Zwijnaarde, Belgium Corresponding author: [email protected], [email protected]

Abstract

As mass spectrometry-based proteomics has matured during the past decade a growing emphasis has been placed on quality control. For this purpose multiple computational quality control tools have been introduced. These tools generate a set of metrics that can be used to assess the quality of a mass spectrometry experiment. Here we review which dierent types of quality control metrics can be generated, and how they can be used to monitor both intra- and inter-experiment performance. We discuss the principal computational tools for quality control and list their main characteristics and applicability. As most of these tools have specic use cases it is not straightforward to compare their performance. For this survey we used dierent sets of quality control metrics derived from information at various stages in a mass spectrometry process and evaluated their eectiveness at capturing qualitative information about an experiment using a supervised learning approach. Furthermore, we discuss currently available algorithmic solutions that enable the usage of these quality control metrics for decision-making.

1 Introduction of quality control is then to leverage the experimental set- up to comprehend how well an instrument performs and In the past decade mass spectrometry (MS)-based pro- how condent the results from the experiments are. teomics has evolved into an extremely powerful analytical technique to identify and quantify in complex bio- Related to the experimental design and based on the type logical samples. This high-throughput approach can yield a of performance we want to monitor there are multiple ap- considerable volume of complex data for each experiment. proaches to quality control. A typical example consists of As it has matured, over the last few years a growing em- the use ofQC samples with a simple sample content in- phasis has been placed on quality assurance (QA). This terleaved between the biological samples. The interesting attention on quality assurance is of the utmost importance aspect of suchQC samples is that they have a controlled, to safeguard condence in the acquired results: in cases limited, and known sample content. They are typically mea- where this has been lacking mass spectrometry proteomics sured on a frequent basis, which allows to extract periodic has sometimes suered from exaggerated claims [4, 20]. information on the performance of the mass spectrome- To anticipate this evolution, a shift to “quality by design” ter. Of course, to understand this performance expressive is now taking place [51]. This means that the “designing QC metrics that provide information indicative of the qual- and developing formulations and manufacturing processes ity of the experimental results need to be derived. Some ensure a predened product quality.” As such, quality assur- straightforward and commonly usedQC metrics include ance consists of multiple aspects of which quality control the number of identications or the sequence coverage. Al- (QC) is an essential component, but other elements such as though these metrics give a global view of the performance, a careful experimental design [12, 25, 35] are equally vital. they do not allow us to pinpoint specic elements of the Whereas the experimental design has to be established workow where a failure might have arisen. Instead, more prior to the initiation of an experiment, quality control granularQC metrics providing information on the chro- takes place while or after the experimental results are ob- matography, the ion signal, the spectrum acquisition, etc., tained. Nonetheless, quality control and experimental de- might be used. sign should not be discussed in isolation, as they are inter- woven. For example, aQC sample can consist of a single peptide, a single digest, or a complex lysate, and Over the years dozens ofQC metrics have been proposed, this decision inuences the type ofQC metric(s) that can be generated by a range of bioinformatics tools. In this paper investigated [5, 30, 41]. Furthermore, one has to decide how we will list the mainQC tools and explain their use cases manyQC runs to include in the experiment and to what and capabilities. Furthermore, we will provide an empirical extent and in which order theseQC runs are interleaved assessment of which type ofQC metrics is most adequate with the biological samples under consideration. The goal in detecting low-quality experiments. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 2/12

QC metric Value

RT-TIC-Q1 0.344

RT-TIC-Q2 0.182

RT-TIC-Q3 0.198

RT-TIC-Q4 0.276 … … …

QC metric Value

RT-TIC-Q1 0.393

RT-TIC-Q2 0.169

RT-TIC-Q3 0.168

RT-TIC-Q4 0.271

experimental results intra-experiment metrics inter-experiment metrics

Figure 1: Intra-experiment metrics evaluate the quality of a single experiment, whereas inter-experiment metrics can be used to compare the quality of multiple experiments.

1.1 ality control metrics types of tools, we will further make a distinction between tools that generate metrics for individual experiments, tools We can primarily distinguishQC metrics based on whether that compare a limited group of experiments and do not they represent information about a single experiment, or necessarily require a complex back-end for data storage, about multiple experiments, as illustrated in gure1. and tools for longitudinal tracking that storeQC data for a Intra-experiment metrics give information about a single large number of experiments. experiment and are computed at the level of individual scans or identications. These metrics show the evolution of a A second distinction between various metrics can be specic measure over the experiment run time, such as, made based on from which stage in a mass spectrome- for example a chromatogram of the total ion current (TIC) try workow they represent the quality of the system. As over the retention time (RT), or the mass accuracy of the shown in gure2, we can distinguish between instrument identied spectra. metrics, identication (ID)-free metrics, andID-based met- Inter-experiment metrics, on the other hand, assess a spe- rics. cic part of the quality of an experiment using a single ID-free metrics andID-based metrics are similar in the measurement for the whole experiment. These values can sense that they are both computed from the spectral results. subsequently be compared for multiple experiments, for ID-free metrics are derived solely from the spectral results, example through a longitudinal analysis to evaluate the per- i.e. from the raw spectral data directly generated by the formance over time. Often an intra-experiment metric can mass spectrometer. These metrics aim to capture infor- be converted to an inter-experiment metric through sum- mation over the whole mass spectrometry workow and marization. This is illustrated in gure1, where a TIC chro- include for example the shape of the peaks or the course of matogram enables the assessment of the chromatographic TIC detailing the chromatography, the number ofMS1 and performance by visualizing the intensity distribution over MS2 scans or the scan rate detailing the spectrum acquisi- the retention time. Using summary statistics this continu- tion, or the charge state distribution detailing the ionization. ous information can be converted to inter-experiment met- The advantage ofID-free metrics is that they are generated rics detailing the fraction of the total retention time that directly from the raw spectral data, which makes it possi- was required to accumulate a certain amount of the TIC, ble to instantly generate these metrics as soon as a mass which gives a high-level assessment at the experiment level spectrometry run has been completed. of the chromatographic stability. ID-based metrics are derived from the spectral results To compare inter-experiment metrics multiple observa- as well, but they combine these data with subsequently tions for dierent experiments are required. Therefore,QC obtained identication results. Examples include aforemen- tools that analyze these metrics usually include a database tioned metrics such as the number of identications in terms back-end for the persistent storage of historical data. On the of peptide-spectrum matches (PSMs), peptides, or proteins; other hand, intra-experiment metrics can be computed from or the sequence coverage for a known sample. Other de- only a single experiment and there is no comparison with tailed metrics can be computed as well, for example by external data. As a result,QC tools that exclusively gener- comparing the dierence inRT for similar identications ate intra-experiment metrics are generally easier to set up, to assess the chromatographic stability, the number of spec- as no external data storage needs to be provided. Because tra identied as the same peptide to measure the dynamic the use cases and requirements dier between these two sampling, or by linking information similar to theID-free Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 3/12

• OpenMS • OpenMS • iMonDB • QuaMeter • proteoQC • SimpatiQCo • PTXQC • SProCoP • QuaMeter • SimpatiQCo instrument ID-free ID-based

sample mass spectrometer spectra bioinformatics

Figure 2: QC tools can capture qualitative information at dierent stages of a mass spectrometry experiment. For each type ofQC metrics the representative tools are listed. metrics with the identication results. Compared toID-free 2 QC tools metrics, the computation ofID-based metrics is somewhat more involved because it additionally requires the identi- In recent years, quality control has become a key focus of cations results. Furthermore, the computation ofID-based attention in academic, industrial, and governmental pro- metrics can be negatively inuenced by suboptimal iden- teomics laboratories. This trend is exemplied (and possibly tication settings. However, in general the inclusion of driven) by the numerousQC tools that have been devel- identications can provide a more detailed qualitative as- oped over the past few years. Initial work by Rudnick et al. sessment of the experimental results. [45] described for the rst time how computationalQC met- rics can be used to objectively assess the quality of a mass Finally, instrument metrics do not look at the spectral spectrometry proteomics experiment. Whereas previously data but derive information directly from instrument read- quality control was mostly performed manually by moni- outs. These are typically very sensitive, low-level metrics, toring a few key measurements, this work showed how a such as the status of the ion source, the vacuum, or a turbo comprehensive set ofQC metrics can be used to thoroughly pump, depending on the type of instrument. An advantage investigate the system performance. A set of 46 mainlyID- of instrument metrics is that they directly indicate which based metrics was dened and implemented in a pipeline part of the instrument is outside its normal range of oper- of Perl programs by researchers at the National Institute ation. This facilitates troubleshooting and can be a driver of Standards and Technology (NIST), called NIST MSQC. for maintenance scheduling. On the other hand, these met- This set of metrics has since then been reimplemented in rics cannot be directly related to the experimental results, several lab-specic data processing pipelines. Support for instead they provide a secondary source ofQC informa- NIST MSQC itself has been discontinued in early 2016 and tion. Furthermore, instrument metrics are instrument- and the original implementation is no longer available, but sev- vendor-specic, and are typically not included in open le eral of the reimplementations remain under active develop- formats such as the mzML format [38]. ment. It has been demonstrated that computationalQC metrics Each distinct type of metric can give a dierent view provide objective criteria that can accurately capture the on the quality of the data. However, not all metrics are quality of a mass spectrometry experiment, and there has always applicable; often metrics are especially relevant for been a proliferation of tools that can compute such metrics. a particular type of sample. For example, monitoring the Here, we will detail the primary tools, their characteris- sequence coverage is mostly applicable when using samples tics, and their usage. Table1 provides an overview of the that contain a single protein digest, whereas the number of discussed tools. protein identications is applicable to samples that consist of a complex lysate. Additionally, the type of experiment 2.1 Tools evaluating individual also plays an important role. For example, the number of identications is very relevant for a discovery experiment, experiments but less so for a targeted experiment. In contrast, instrument 2.1.1 aMeter metrics are largely agnostic to the type of experiment and the sample content, but they can signicantly vary between QuaMeter was initially developed as a user-friendly and dierent instrument models and vendors. open-source alternative to NIST MSQC. NIST MSQC con- Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 4/12

Operating Experiment Instru- ID- ID- Tool Interface Website system type ment free based QuaMeter command- Windows, discovery http://proteowizard.sourceforge. × [33, 59] line Linux DDA XX net/ OpenMS cross- discovery KNIME × http://www.openms.de/ [58] platform DDA XX proteoQC cross- discovery http://bioconductor.org/packages/ R × × [60] platform DDA X proteoQC discovery Windows, PTXQC & quan- R cross- × × https://github.com/cbielow/PTXQC [8] tication X platform DDA discovery, http://proteome.gs.washington. SProCoP targeted Skyline Windows × × edu/software/skyline/tools/ [6] SRM& X sprocop.html PRM SimpatiQCo discovery http://ms.imp.ac.at/?goto= web Windows × [43] DDA XX simpatiqco iMonDB https://bitbucket.org/ GUI Windows any × × [10] X proteinspector/imondb/

Table 1: An overview of the discussedQC tools and their main characteristics. sisted of a graphical user interface (GUI) wrapper around command-line functionality the computation can easily be multiple individual tools and scripts with various inter- automated. This makes QuaMeter a powerful tool that com- dependencies, which resulted in a complex pipeline. Ad- putes an extensive set of inter-experimentQC metrics. ditionally, some elements of this pipeline could only be modied to a limited extent. NIST MSQC could exclusively 2.1.2 OpenMS compute metrics from Thermo Scientic raw les, and only supported three search engines to provide identications: OpenMS is a comprehensive open-source software library the NIST MSPepSearch or the SpectraST [31] spectral li- that oers a wide range of algorithms and tools for mass brary search engines, or the OMSSA [24] sequence database spectrometry-based proteomics and metabolomics [49]. It search engine. These limitations restricted the applicability consists of various small processing tools that can be used to of NIST MSQC. Instead, QuaMeter consists of a single multi- construct complex analysis workows [2, 27]. These work- platform command-line application that is able to compute ows can be designed visually using the KNIME workow QC metrics from raw les originating from instruments pro- engine [7], where each tool functions as an individual node duced by multiple vendors. Using the ProteoWizard [13] in the workow. library it is able to read spectral data stored in a wide variety The various OpenMS nodes can be used to build com- of vendor-specic raw les (restricted to the Windows plat- plexQC pipelines [58]. The providedQC nodes can com- form) and open standard le formats, such as mzML [38]. pute a set of intra-experiment metrics, consisting of both Further, it can utilize identication results produced by any ID-free andID-based metrics. OpenMS supports a range search engine in the standard mzIdentML [26] or pepXML of search engines to generate identications for theID- format through external processing using IDPicker [32]. based metrics, for which there exist specic nodes, includ- ing Mascot, MS-GF+ [28], Myrimatch [52], OMSSA [24], The initial QuaMeter version [33] computed a set of 42 and X!Tandem [16]. ExampleQC metrics include the num- ID-basedQC metrics equivalent to those dened by Rudnick ber of spectra (identied or otherwise), peptides, and pro- et al. [45]. In a subsequent version QuaMeter improved teins; mass accuracy statistics; and the mass over charge upon this by also including functionality to compute a set and retention time acquisition ranges. These metrics are of 45 ID-freeQC metrics [59]. Both sets of metrics are complemented by various plots that provide further details, inter-experiment summary metrics, although the output such as a TIC chromatogram, a histogram of the mass accu- is exported to simple tab-delimited text les, so the visual- racy of the identied peptides, or a histogram of the charge ization and analysis thereof has to be done using external distribution of the detected ion features. OpenMS exports software or code scripts. Without advanced visualization this information to an Extensible Markup Language (XML) or analysis functionality QuaMeter focuses solely on com- qcML le [58], which can be visualized in a web browser putingQC metrics. Especially the set ofID-free metrics, through an embedded stylesheet, or to a portable document which requires only the spectral data, can very easily be format (PDF) report. computed. For the set ofID-based metrics some prior pro- Due to the wealth of algorithms and tools that are avail- cessing of the identication results by IDPicker is required, able in the OpenMS software library, the providedQC work- which can make this process slightly more cumbersome. ows can potentially be easily extended to compute addi- Only a limited conguration is required, and through the tional metrics. Furthermore, there is no need to be restricted Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 5/12 to algorithms natively provided by OpenMS, as the available label-free quantication. After initial processing of the spec- functionality can easily be extended through custom nodes, tral data by MaxQuant, PTXQC uses the MaxQuant output for example by using the built-in support for the R statistical results to compute variousQC metrics. PTXQC requires programming language [44]. This makes it possible to build as input the custom text les generated by MaxQuant and granular workows and achieve a very ne-grained control, the MaxQuant conguration settings, and hence cannot be although expert knowledge of the OpenMS ecosystem and used to process any other type of data. As PTXQC is written the KNIME environment is recommended to do so. The in the R programming language, it is fully cross-platform. constructed workows can subsequently be exported and Additionally, easy drag-and-drop functionality to execute shared. Both OpenMS and KNIME are cross-platform tools, the QC analyses is provided for the Windows operating ensuring the universal applicability of these workows. system. PTXQC produces an extensive report that contains a set 2.2 Tools comparing groups of experiments of 24 intra- and inter-experiment metrics. These metrics are divided into four categories corresponding to the specic 2.2.1 proteoQC MaxQuant output source the metrics are derived from: “Pro- The proteoQC package [60] for the R programming lan- teinGroups”, “Evidence”, “Msms”, and “MsmsScans”. The guage [23, 44] can be used to generate a Hyper Text Markup metrics cover a wide range of information, including the Language (HTML) report detailing the experimental qual- intensity of the detected features and peptides, the poten- ity. Prior to executing proteoQC the experimental design tial presence of contaminants, the mass accuracy of the has to be specied by conguring each spectral data le identied peptides and fragments, the number of missed representing a sample as belonging to a specic fraction, cleavages detailing the enzyme specicity, and the num- technical replicate, and biological replicate. The generated ber of identied peptides and proteins. Other metrics are QC report contains intra-experiment metrics for each indi- specically related to the MaxQuant “match-between-runs vidual sample, as well as aggregated information to compare (MBR)”[14] functionality. MBR aligns the retention times samples at the level of their fractions, technical replicates, of multiple runs and transfers their identications across and biological replicates. features that have the same accurate mass and a similar To generate a set of intra-experimentID-based metrics retention time, providing more data for the downstream for each sample, proteoQC uses the rTANDEM package [22] quantication of proteins. PTXQC assesses the MBR per- to interface the X!Tandem [16] sequence database search en- formance by evaluating the retention time alignment and gine in R to provide identication results. For each sample by checking whether the identication transfer seems cor- some individual metrics andQC plots are generated, such as rect. All of these metrics are then visualized and compared a breakdown of the precursor ion charge states, the mass ac- between the dierent raw les that constitute the consid- curacy, information on the number of spectra and peptides ered MaxQuant project using detailed gures. Furthermore, that were used to identify distinct proteins during protein each of the metrics is converted to an individual score for inference, etc. Furthermore, when identifying the data pro- each experiment using automated scoring functions. Most teoQC automatically adds the common Repository of Adven- of these scores are absolute scores generated by comparing titious Proteins (cRAP)( http://www.thegpm.org/crap/) the observation to a threshold, for example such as whether database to the user-provided protein database. The cRAP the number of detected contaminants is too excessive, or database contains contaminants such as common laboratory generated by evaluating a specic characteristic of the ob- proteins, like trypsin, or contaminants transfered through servation, for example such as the extent to which the mass dust or contact, like keratin, and proteoQC reports which of deviations are centered around zero. Other scores are com- these contaminants were detected in the samples. Addition- puted for a single raw le using the other raw les as a ally, proteoQC reports on the reproducibility of the results reference, for example by comparing the number of missed by comparing the number of identied spectra, peptides, cleavages in each individual raw le to the average number and proteins per fraction, technical replicate, and biological of missed cleavages. Finally, some other scores are evalu- replicate, and their overlap between the replicates. ated relative to settings extracted from MaxQuant, such as By incorporating the experimental design proteoQC can the mass accuracy compared to the width of the precursor make informed comparisons between individual samples, mass window. All these scoring functions generate inter- which providesQC information on an additional level. Fur- experiment metrics that are used to compare the quality thermore, proteoQC is fully cross-platform within the pop- of the dierent experiments. Usefully, PTXQC provides a ular R programming language. However, as theQC pipeline heatmap overview of the inter-experiment metrics, which has to be congured programmatically, some R experience yields an assessment of the quality at a glance and facilitates is recommended to utilize proteoQC. pinpointing the low-performing experiments. Although PTXQC can exclusively be used to analyze MaxQuant results, through this tight integration it is able to 2.2.2 PTXQC compute some highly relevant and specializedQC metrics. Proteomics Quality Control (PTXQC)[8] is an R-based qual- These metrics do not only assess the quality of the spec- ity control pipeline for MaxQuant [15], a highly popular tral data, but also provide information on the subsequent software suite for . Like MaxQuant, bioinformatics processing by MaxQuant. Furthermore, the PTXQC supports a wide range of quantitative proteomics addition of a high-level heatmap at the start of the report workows, including stable isotope labeling with amino is very useful to get a quick overview of the quality, after acids in cell culture (SILAC), tandem mass tags (TMT), and which the more detailed visualizations can be employed to Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 6/12 further investigate potential problems. the protein sequence coverage. For eachQC metric the range of acceptable values is learned based on the historical 2.2.3 SProCoP observations using robust statistical measures to take outly- ing values into account. This information is then displayed Statistical Process Control in Proteomics (SProCoP)[6] is a in the metric plots using a color-coded background band to QC script written in R [44] that can be used as a plugin [11] highlight deviating system performance. Further, external for the popular Skyline [34] tool for targeted proteomics. messages can be entered manually, for example pertaining SProCoP applies well-established statistical process con- to instrument maintenance. These messages will be super- trol techniques such as the Shewhart control chart and the imposed on the metric plots to relate the external events to Pareto chart. The purpose of a Shewhart control chart is the evolution of the metrics. to track performance over time and identify outliers that SimpatiQCo consists of a number of dierent components, deviate excessively from the expected behavior. Further, such as the database, the web server, and various processing the Pareto chart is a combination of a bar and line graph, tools. These components need to be installed individually, which displays the number of deviating measurements for and although a step-by-step installation guide is available each metric along with its cumulative percentage, and pro- online, this complicated process is not recommended for vides feedback on which metrics are more variable and may novice users. Furthermore, not all of the conguration can require attention. be done through the graphical web-based client. For exam- Using these statistical process control techniques SPro- ple, to process raw les these must be able to be linked to CoP monitors the performance of ve inter-experiment a specic instrument. Unfortunately, an instrument deni- QC metrics based on targeted peptides present inQC sam- tion can only be created by manually adding a record in the ples with a known sample content or spiked into real sam- corresponding table of the PostgreSQL database. ples: signal intensity, mass measurement accuracy, reten- SimpatiQCo is a powerful tool to track system perfor- tion time reproducibility, peak full width at half maximum mance over time, albeit with some technical limitations. (FWHM), and peak symmetry. Measurement thresholds are Namely, SimpatiQCo is only able to process raw les gen- dened empirically based on a reference set of samples with erated on a limited number of instrument models and only a known good quality, after which the performance of other supports the commercial Mascot search engine for peptide samples in the Skyline project can be investigated. identications. Through its integration with Skyline SProCoP is vendor- independent and can be used for a wide range of tar- geted and discovery workows. Additionally these statisti- 2.3.2 iMonDB cal process control techniques are available online (http: Unlike the previous tools the Instrument MONitoring //www.qcmylcms.com/) and have been implemented in the DataBase (iMonDB)[10] does not compute metrics from Panorama [47] repository for targeted proteomics from Sky- the spectral results, but extracts instrument metrics from line. Panorama AutoQC is a utility application that monitors the raw les. The iMonDB uses a MySQL database to store for new data les and automatically invokes Skyline to pro- its information. This database acts as a server, with two sep- cess the data. TheQC metrics are stored in Panorama and arate standalone GUI applications that can connect to the the statistical process control charts similar to SProCoP can database as clients, each with a specic task: the iMonDB be visualized through the Panorama web application. Collector processes raw les and stores the instrument met- rics in the database, whereas the iMonDB Viewer retrieves 2.3 Tools for longitudinal tracking the information from the database and visualizes it. The iMonDB supports a wide range of instruments man- 2.3.1 SimpatiQCo ufactured by Thermo Scientic, although it does not sup- SIMPle AuTomatIc Quality COntrol (SimpatiQCo)[43] not port other instrument vendors. Prior to extracting instru- only computes variousQC metrics, it also stores and visual- ment metrics from a raw le, a corresponding instrument izes these metrics for longitudinal monitoring of the system denition has to be created. This can be done through performance. It uses a PostgreSQL database as back-end, the iMonDB Collector, which allows the full congura- and an Apache webserver to provide a web-based front-end tion through its GUI. Further, extraction of the instrument for conguration and visualization. metrics can be done manually through the GUI, or can be SimpatiQCo can computeQC metrics from a limited se- done through command-line functionality provided by the lection of Thermo Scientic and SCIEX instruments. Raw iMonDB Collector. This command-line functionality can les from these instruments can be uploaded to the web be used to automatically run the iMonDB Collector using server manually, or can be added automatically through a an external scheduling tool, such as the native operating “hot folder” that is monitored continuously for new raw les. system scheduler. These raw les are then submitted to a linked Mascot server The behavior over time of the metrics for each instru- for peptide identications. Next, SimpatiQCo calculates a ment can be viewed using the iMonDB Viewer. Similar to range ofID-free andID-basedQC metrics such as the num- functionality provided by SimpatiQCo it is possible to add ber ofMS1 andMS2 scans, the number of identied PSMs additional information pertaining to external events and and proteins, the TIC, and information on lock masses (if show this on the metric plots to link this to the evolution applicable). Further, specic peptides and proteins can be of the metrics. It is also possible to export a PDF le of the investigated in detail using metrics such as the peak area external events for reporting purposes. and width and the elution time of peptides of interest, and A unique aspect of the iMonDB is that this is the only Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 7/12 tool that is able to systematically analyze instrument met- cic to certain experimental workows and sample types. rics. The advantage of these instrument metrics, which Meanwhile most tools also represent some of their QC in- provide information at the lowest level, is their high sensi- formation through visualizations. Although these quickly tivity, which makes it possible to detect emerging defects provide useful insights for human users, this data is not in a timely fashion. However, because these metrics are suitable for an objective, automatic comparison. instrument-dependent they are usually not retained during To compare dierent types of metrics we used the set conversion to open formats, such as mzML [38]. Due to this of instrument metrics computed by the iMonDB[10], the limitation the iMonDB needs to work with vendor-specic set ofID-free metrics computed by QuaMeter [59], and the raw les directly, which is currently limited to Thermo set ofID-based metrics as identied by Rudnick et al. [45]. Scientic raw les. Furthermore, there is a multitude of These sets of metrics are very comprehensive and all of instrument metrics that are extracted, which makes it hard these inter-experiment metrics can readily be used to com- to comprehend which metrics are most useful to monitor pare experiments to each other. To be able to determine systematically, even for expert users. Nevertheless, these whether or not these metrics can capture qualitative infor- instrument metrics can be very useful to detect malfunc- mation about an experiment, we used a public dataset for tioning instrument elements before these have a deleterious which the quality of the experiments is known. The dataset eect on the experimental results, preventing potential loss consists of a number of complex quality controlLC-MS runs of valuable sample content. performed on several dierent instruments at the Pacic Northwest National Laboratory (PNNL)[3]. Each sample 2.4 Other tools had an identical content (whole cell lysate of Shewanella oneidensis), and the quality of the various runs has been As mentioned previously, NIST MSQC [45] was the rst manually annotated by expert instrument operators as be- tool that generated computationalQC metrics, although it ing either “good”, “ok”, or “poor”. We split up the various was recently retired in early 2016. runs depending on the instrument type, being either “Exac- Metriculator [54] is a web-based tool for storing and vi- tive”, “LTQ IonTrap”, “LTQ Orbitrap”, or “Velos Orbitrap”, sualizingQC metrics longitudinally. However, Metriculator with each of these instrument groups consisting of multiple does not computeQC metrics directly but critically depends individual instruments. We refer to the original publica- upon an embedded version of NIST MSQC. Unfortunately, tion by Amidan et al. [3] for further information on the the installation process for Metriculator is not very straight- experimental procedures and the dataset details. forward; it has many Ruby dependencies whose installation This public dataset already contains the precomputed set might fail, and which are presently outdated or even no ofID-free metrics by QuaMeter and the set ofID-based longer supported. metrics by SMAQC (the PNNL in-house reimplementa- LogViewer [50] is a simple visualization tool that presents tion of the NIST MSQC metrics dened by Rudnick et al. a set of 11 instrument metrics, such asMS1 andMS2 ions [45]; https://github.com/PNNL-Comp-Mass-Spec/SMAQC). injection times, andID-free metrics, such as the charge We further used the iMonDB to compute the set of instru- state and mass distributions. As input it uses log les from ment metrics. To this end all experimental raw les, pre- Thermo instruments exported by RawXtract [39], which computedQC metrics, and the expert quality annotations has been deprecated presently. were retrieved from the PRoteomics IDEntications (PRIDE) A dierent approach is used by SprayQc [46]. Whereas database [57]. the other discussed tools computeQC metrics post- To quantify the expressiveness of these three sets of met- acquisition, SprayQc directly interfaces with periph- rics, each capturing a dierent type ofQC information, we eral equipment to continuously monitor its performance. employed a binary classier. As the quality of the experi- SprayQc is able to automatically track the stability of the ments was manually assessed by expert instrument opera- electrospray through computer vision, the status of the liq- tors, this labeling can be used as the ground truth to train uid chromatography (LC) pumps, the temperature of the the classier. We used the acceptable experiments, with column oven, and the continuity of the data acquisition. In their quality designated as either “good” or “ok”, as the pos- case a malfunctioning is detected SprayQc can automati- itive class, and the inferior experiments, with their quality cally take corrective actions and warn the instrument op- designated as “poor”, as the negative class. When given an erator. This is a valuable approach to minimize the loss experiment represented by itsQC metrics, the classication of precious sample content and provide early notications, task consists of correctly predicting the experiment’s qual- and it can complement the otherQC tools that provide a ity. Prior to training the classier we removed redundant post-acquisition quality assessment. features that have a very low variance and we rescaled the features robust to outliers by centering by the median and scaling by the interquartile range. Next, for each separate 3 Metrics evaluation instrument type we trained a random forest classier, for which we split the data into 65%–35% training and testing We compared various sets of metrics to assess their eec- subsets that are equally stratied according to their quality tiveness in expressing the quality of a mass spectrometry labels. This classier has been coded in Python and uses proteomics experiment. Typically this is not a straightfor- the random forest implementation from scikit-learn [42], ward task because, as we have reviewed in the previous along with functionality provided by NumPy [56] and pan- sections, eachQC tool has its own characteristics and re- das [40]. The code is available as open source at https: quirements, and use cases can vary as some tools are spe- //bitbucket.org/proteinspector/qc-evaluation/. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 8/12

decision-making based on these metrics. However, inter-

1.0 preting these metrics is not trivial. First, considerable do- main knowledge is required to understand what each metric

0.8 signies. Second, the metrics form a high-dimensional data space, which complicates their analysis. Dierent elements

0.6 in a mass spectrometry workow do not function in iso- lation but instead inuence each other, which has to be

0.4 taken into account while analyzing metrics representing True Positive Rate information about these elements. Therefore, univariate

0.2 approaches are generally insucient; instead multivariate

ID-free (AUC = 0.96) approaches that can deal with the high-dimensional data ID-based (AUC = 0.99) 0.0 Instrument (AUC = 0.90) space should be preferred, while also taking the curse of 0.0 0.2 0.4 0.6 0.8 1.0 dimensionality into account [1]. False Positive Rate To this end Wang et al. [59] have developed a robust mul- Figure 3: ROC curve showing the classication perfor- tivariate statistical toolkit to interpretQC metrics. They mance for the Velos Orbitrap instrument type. ROC curves have used a principal component analysis (PCA) transforma- for the other instrument types indicate similar results (data tion to reduce the data to a low-dimensional approximation, not shown). in which they were able to successfully detect outlier low- quality experiments based on pairwise dissimilarities. Fur- thermore, they developed an analysis of variance (ANOVA) As illustrated by the receiver operator characteristic model which enabled them to identify whether the observed (ROC) curve in gure3 all three types ofQC metrics are variability was attributable to lab-dependent factors, batch adept at discriminating high-quality experiments from low- eects, or biological variability. Such work driving the un- quality experiments. This shows that all of the dierent derstanding ofQC metrics is highly valuable, and these tools can give us valuable insights into the quality of an analyses have been applied to great eect for multiple stud- experiment, and that information captured at various dif- ies. For example, it was used to assess the quality of the ferent stages of the mass spectrometry process should be experimental results for various studies conducted by the investigated.ID-based metrics slightly outperformID-free National Cancer Institute (NCI) Clinical Proteomic Tumor metrics, most likely because theID-based metrics can em- Analysis Consortium (CPTAC)[48, 53, 61]. ploy additional information provided by the identications. Similar work was done by Bittremieux et al. [9], who ap- This dierence is minimal however, which is perhaps not plied unsupervised outlier detection to identify low-quality surprising as both types of metrics take similar properties of experiments. Subsequently they used a specialized outlier the spectra into account. This reinforces previous research interpretation technique to determine whichQC metrics which showed that ID-based metrics are not signicantly mostly contributed to the decrease in quality. The advan- inuenced by slight dierences in the identications, such tage of this approach is that allQC metrics are used to as when using an alternative search engine [33]. This also identify low-quality experiments, unlike when using a di- shows the excellent ecacy ofID-free metrics in objectively mensionality reduction, such as PCA, which discards some evaluating the quality based solely on spectral information. of the information. Meanwhile, the advanced outlier in- BecauseID-based metrics require additional computational terpretation pinpointing the most relevantQC metrics can steps to obtain the identications, whereasID-free metrics yield actionable information for domain experts to optimize can be directly computed from the spectral results,ID-free their experimental set-up. metrics might be preferred if a speedy quality assessment is required. In contrast, instrument metrics perform a lit- Whereas these previous analyses used unsupervised tech- tle worse at correctly identifying low-quality experiments. niques, Amidan et al. [3] trained a supervised classier to This is likely because they are only secondary results that discriminate low-quality experiments from high-quality ex- are not always directly related to the data quality. Never- periments. A supervised approach will generally perform theless, these metrics still have merit as they do not depend better than an unsupervised approach but will require initial on a specic type of experiment or sample content, but are training. Furthermore, a supervised classier might have applicable on all occasions. Furthermore, by combining to be retrained to adapt it to data generated by a dierent the individual classiers for the various types of metrics in instrument or in a dierent laboratory. Amidan et al. [3] an ensemble classier a further performance gain can be have expended signicant eort in manually annotating the achieved because the dierent types of metrics each provide quality of over a thousand experiments to generate train- a complementary view on the quality. ing data, which allowed them to build a highly performant logistic regression classier. These analyses are extremely valuable, as they allow us 4 Using QC metrics for to achieve a deeper understanding of the mass spectrometry decision-making processes and the properties of what makes a high-quality experiment. These algorithmic approaches provide a thor- As tools for computational quality control have prolifer- ough quality assessment of the spectral data, which enforces ated in recent years, the challenge in this eld is now shift- informed decision-making, and which has the potential to ing from the computation ofQC metrics toward informed automatically drive the spectral acquisition in the future. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 9/12

5 Conclusion [4] Baggerly, K. A., Morris, J. S., Edmonson, S. R., and Coombes, K. R. “Signal in Noise: Evaluating Reported We have given an overview of the available computational Reproducibility of Serum Proteomic Tests for Ovar- tools to generateQC metrics for mass-spectrometry-based ian Cancer.” In: JNCI Journal of the National Can- proteomics. These tools enable assessing the performance cer Institute 97.4 (Feb. 16, 2005), pp. 307–309. doi: of the experimental set-up and detecting unreliable results. 10.1093/jnci/dji008. These are essential requirements to inspire condence in the [5] Bereman, M. S. “Tools for Monitoring System Suit- experimental results, which will prove to be a crucial step in ability in LC MS/MS Centric Proteomic Experiments.” the maturation of proteomics technologies, and which will In: PROTEOMICS 15 (5-6 Mar. 2015), pp. 891–902. allow us to for example routinely apply these technologies doi: 10.1002/pmic.201400373. into a clinical setting [36, 51]. Another potential applica- tion where an accurate assessment of the data quality is [6] Bereman, M. S., Johnson, R., Bollinger, J., Boss, Y., et paramount, is in the reuse of public data [19, 21, 29, 37]. As al. “Implementation of Statistical Process Control for public data repositories keep expanding and the potential Proteomic Experiments via LC MS/MS.” In: Journal for data reuse grows, we envision that data submissions to of The American Society for Mass Spectrometry 25.4 public repositories will soon have to be accompanied by (Apr. 2014), pp. 581–587. doi: 10.1007/s13361-013- QC parameters at the time of submission, or will have a 0824-5. standard set ofQC metrics calculated automatically after [7] Berthold, M. R., Cebron, N., Dill, F., Gabriel, T. R., submission [37]. et al. “KNIME - the Konstanz Information Miner: Finally, most currentQC tools are limited to the typical Version 2.0 and Beyond.” In: ACM SIGKDD Explo- use case of bottom-up data-dependent acquisition (DDA) rations Newsletter 11.1 (June 2009), pp. 26–31. doi: discovery experiments, and theirQC metrics often cannot 10.1145/1656274.1656280. be directly translated to other types of experiments. Less [8] Bielow, C., Mastrobuoni, G., and Kempa, S. “Pro- research has been done onQC for other types of workows, teomics Quality Control: Quality Control Software such as data-independent acquisition (DIA)[18] or top- for MaxQuant Results.” In: Journal of Proteome Re- down proteomics [55], or even related mass spectrometry- search 15.3 (Mar. 4, 2016), pp. 777–787. doi: 10.1021/ based domains, such as metabolomics [17]. In the next few acs.jproteome.5b00780. years we will likely see the eorts onQC expanded to these types of workows as well, which will further bolster the [9] Bittremieux, W., Meysman, P., Martens, L., Valken- diverse and powerful mass spectrometry ecosystem. borg, D., et al. “Unsupervised Quality Assessment of Mass Spectrometry Proteomics Experiments by Multivariate Quality Control Metrics.” In: Journal of Acknowledgments Proteome Research 15.4 (Apr. 1, 2016), pp. 1300–1307. doi: 10.1021/acs.jproteome.6b00028. W.B., D.V., L.M., and K.L. acknowledge the support of the VLAIO grant “InSPECtor” (IWT project 120025). [10] Bittremieux, W., Willems, H., Kelchtermans, P., Martens, L., et al. “iMonDB: Mass Spectrometry Qual- ity Control through Instrument Monitoring.” In: Jour- References nal of Proteome Research 14.5 (May 1, 2015), pp. 2360– 2366. doi: 10.1021/acs.jproteome.5b00127. [1] Aggarwal, C. C., Hinneburg, A., and Keim, D. A. “On [11] Broudy, D., Killeen, T., Choi, M., Shulman, N., et al. “A the Surprising Behavior of Distance Metrics in High Framework for Installable External Tools in Skyline.” Dimensional Space.” In: Proceedings of the 8th In- In: Bioinformatics 30.17 (Sept. 1, 2014), pp. 2521–2523. ternational Conference on Database Theory - ICDT doi: 10.1093/bioinformatics/btu148. ’01. Vol. 1973. Lecture Notes in Computer Science. London, England: Springer Berlin Heidelberg, 2001, [12] Cairns, D. A. “Statistical Issues in Quality Control pp. 420–434. doi: 10.1007/3-540-44503-X_27. of Proteomic Analyses: Good Experimental Design and Planning.” In: PROTEOMICS 11.6 (Mar. 2011), [2] Aiche, S., Sachsenberg, T., Kenar, E., Walzer, M., et pp. 1037–1048. doi: 10.1002/pmic.201000579. al. “Workows for Automated Downstream Data Analysis and Visualization in Large-Scale Compu- [13] Chambers, M. C., Maclean, B., Burke, R., Amodei, D., tational Mass Spectrometry.” In: PROTEOMICS 15.8 et al. “A Cross-Platform Toolkit for Mass Spectrome- (Apr. 2015), pp. 1443–1447. doi: 10 . 1002 / pmic . try and Proteomics.” In: Nature Biotechnology 30.10 201400391. (Oct. 10, 2012), pp. 918–920. doi: 10.1038/nbt.2377. [3] Amidan, B. G., Orton, D. J., LaMarche, B. L., Monroe, [14] Cox, J., Hein, M. Y., Luber, C. A., Paron, I., et al. “Ac- M. E., et al. “Signatures for Mass Spectrometry Data curate Proteome-Wide Label-Free Quantication by Quality.” In: Journal of Proteome Research 13.4 (Apr. 4, Delayed Normalization and Maximal Peptide Ratio 2014), pp. 2215–2222. doi: 10.1021/pr401143e. Extraction, Termed MaxLFQ.” In: Molecular & Cel- lular Proteomics 13.9 (Sept. 1, 2014), pp. 2513–2526. doi: 10.1074/mcp.M113.031591. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 10/12

[15] Cox, J. and Mann, M. “MaxQuant Enables High Pep- [26] Jones, A. R., Eisenacher, M., Mayer, G., Kohlbacher, tide Identication Rates, Individualized p.p.b.-Range O., et al. “The mzIdentML Data Standard for Mass Accuracies and Proteome-Wide Protein Quan- Mass Spectrometry-Based Proteomics Results.” In: tication.” In: Nature Biotechnology 26.12 (Nov. 30, Molecular & Cellular Proteomics 11.7 (July 1, 2012), 2008), pp. 1367–1372. doi: 10.1038/nbt.1511. pp. M111.014381–M111.014381. doi: 10.1074/mcp. [16] Craig, R. and Beavis, R. C. “TANDEM: Matching Pro- M111.014381. teins with Tandem Mass Spectra.” In: Bioinformatics [27] Junker, J., Bielow, C., Bertsch, A., Sturm, M., et al. 20.9 (Feb. 19, 2004), pp. 1466–1467. doi: 10.1093/ “TOPPAS: A Graphical Workow Editor for the Anal- bioinformatics/bth092. ysis of High-Throughput Proteomics Data.” In: Jour- [17] Dunn, W. B., Wilson, I. D., Nicholls, A. W., and Broad- nal of Proteome Research 11.7 (July 6, 2012), pp. 3914– hurst, D. “The Importance of Experimental Design 3920. doi: 10.1021/pr300187f. and QC Samples in Large-Scale and MS-Driven Un- [28] Kim, S. and Pevzner, P. A. “MS-GF+ Makes Progress targeted Metabolomic Studies of Humans.” In: Bio- towards a Universal Database Search Tool for Pro- analysis 4.18 (Sept. 2012), pp. 2249–2264. doi: 10. teomics.” In: Nature Communications 5 (Oct. 31, 2014), 4155/bio.12.204. p. 5277. doi: 10.1038/ncomms6277. [18] Egertson, J. D., MacLean, B., Johnson, R., Xuan, Y., [29] Kinsinger, C. R., Apel, J., Baker, M., Bian, X., et et al. “Multiplexed Peptide Analysis Using Data- al. “Recommendations for Mass Spectrometry Data Independent Acquisition and Skyline.” In: Nature Quality Metrics for Open Access Data (Corollary Protocols 10.6 (May 21, 2015), pp. 887–903. doi: 10. to the Amsterdam Principles).” In: Molecular & Cel- 1038/nprot.2015.055. lular Proteomics 10.12 (Feb. 3, 2012), O111.015446– [19] Eisenacher, M., Schnabel, A., and Stephan, C. “Qual- O111.015446. doi: 10.1074/mcp.O111.015446. ity Meets Quantity - Quality Control, Data Standards [30] Köcher, T., Pichler, P., Swart, R., and Mechtler, K. and Repositories.” In: PROTEOMICS 11.6 (Mar. 2011), “Quality Control in LC-MS/MS.” In: PROTEOMICS pp. 1031–1036. doi: 10.1002/pmic.201000441. 11.6 (Mar. 6, 2011), pp. 1026–1030. doi: 10.1002/ [20] Ezkurdia, I., Calvo, E., Del Pozo, A., Vázquez, J., et al. pmic.201000578. “The Potential Clinical Impact of the Release of Two [31] Lam, H., Deutsch, E. W., Eddes, J. S., Eng, J. K., et al. Drafts of the Human Proteome.” In: Expert Review “Development and Validation of a Spectral Library of Proteomics 12.6 (Nov. 2, 2015), pp. 579–593. doi: Searching Method for Peptide Identication from 10.1586/14789450.2015.1103186. MS/MS.” In: PROTEOMICS 7.5 (Mar. 5, 2007), pp. 655– [21] Foster, J. M., Degroeve, S., Gatto, L., Visser, M., et al. 667. doi: 10.1002/pmic.200600625. “A Posteriori Quality Control for the Curation and [32] Ma, Z.-Q., Dasari, S., Chambers, M. C., Litton, M. D., Reuse of Public Proteomics Data.” In: PROTEOMICS et al. “IDPicker 2.0: Improved Protein Assembly with 11.11 (June 11, 2011), pp. 2182–2194. doi: 10.1002/ High Discrimination Peptide Identication Filtering.” pmic.201000602. In: Journal of Proteome Research 8.8 (Aug. 7, 2009), [22] Fournier, F., Joly Beauparlant, C., Paradis, R., and pp. 3872–3881. doi: 10.1021/pr900360j. Droit, A. “rTANDEM, an R/Bioconductor Package [33] Ma, Z.-Q., Polzin, K. O., Dasari, S., Chambers, M. C., for MS/MS Protein Identication.” In: Bioinformatics et al. “QuaMeter: Multivendor Performance Metrics 30.15 (Aug. 1, 2014), pp. 2233–2234. doi: 10.1093/ for LC-MS/MS Proteomics Instrumentation.” In: Ana- bioinformatics/btu178. lytical Chemistry 84.14 (July 17, 2012), pp. 5845–5850. [23] Gatto, L. and Christoforou, A. “Using R and Biocon- doi: 10.1021/ac300629p. ductor for Proteomics Data Analysis.” In: Biochimica [34] MacLean, B., Tomazela, D. M., Shulman, N., Cham- et Biophysica Acta (BBA) - Proteins and Proteomics bers, M., et al. “Skyline: An Open Source Docu- 1844.1 (Jan. 2014), pp. 42–51. doi: 10 . 1016 / j . ment Editor for Creating and Analyzing Targeted Pro- bbapap.2013.04.032. teomics Experiments.” In: Bioinformatics 26.7 (Apr. 1, [24] Geer, L. Y., Markey, S. P., Kowalak, J. A., Wagner, L., 2010), pp. 966–968. doi: 10.1093/bioinformatics/ et al. “Open Mass Spectrometry Search Algorithm.” btq054. In: Journal of Proteome Research 3.5 (Oct. 11, 2004), [35] Maes, E., Kelchtermans, P., Bittremieux, W., De pp. 958–964. doi: 10.1021/pr0499491. Grave, K., et al. “Designing Biomedical Proteomics [25] Hu, J., Coombes, K. R., Morris, J. S., and Baggerly, Experiments: State-of-the-Art and Future Perspec- K. A. “The Importance of Experimental Design in Pro- tives.” In: Expert Review of Proteomics 13.5 (Apr. 25, teomic Mass Spectrometry Experiments: Some Cau- 2016), pp. 495–511. doi: 10.1586/14789450.2016. tionary Tales.” In: Briengs in Functional Genomics 1172967. and Proteomics 3.4 (Jan. 1, 2005), pp. 322–331. doi: [36] Martens, L. “Bringing Proteomics into the Clinic: The 10.1093/bfgp/3.4.322. Need for the Field to Finally Take Itself Seriously.” In: PROTEOMICS - Clinical Applications 7 (5-6 June 2013), pp. 388–391. doi: 10.1002/prca.201300020. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 11/12

[37] Martens, L. “Public Proteomics Data: How the Field [48] Slebos, R. J. C., Wang, X., Wang, X., Zhang, B., et Has Evolved from Sceptical Inquiry to the Promise al. “Proteomic Analysis of Colon and Rectal Carci- of in Silico Proteomics.” In: EuPA Open Proteomics noma Using Standard and Customized Databases.” 11 (June 2016), pp. 42–44. doi: 10.1016/j.euprot. In: Scientic Data 2 (June 23, 2015), p. 150022. doi: 2016.02.005. 10.1038/sdata.2015.22. [38] Martens, L., Chambers, M., Sturm, M., Kessner, D., [49] Sturm, M., Bertsch, A., Gröpl, C., Hildebrandt, A., et et al. “mzML—a Community Standard for Mass Spec- al. “OpenMS – An Open-Source Software Framework trometry Data.” In: Molecular & Cellular Proteomics for Mass Spectrometry.” In: BMC Bioinformatics 9.1 10.1 (Jan. 1, 2011), R110.000133–R110.000133. doi: (Mar. 26, 2008), p. 163. doi: 10.1186/1471-2105-9- 10.1074/mcp.R110.000133. 163. [39] McDonald, W. H., Tabb, D. L., Sadygov, R. G., Mac- [50] Sweredoski, M. J., Smith, G. T., Kalli, A., Graham, Coss, M. J., et al. “MS1, MS2, and SQT—three Uni- R. L. J., et al. “LogViewer: A Software Tool to Visu- ed, Compact, and Easily Parsed File Formats for alize Quality Control Parameters to Optimize Pro- the Storage of Shotgun Proteomic Spectra and Iden- teomics Experiments Using Orbitrap and LTQ-FT tications.” In: Rapid Communications in Mass Spec- Mass Spectrometers.” In: Journal of Biomolecular trometry 18.18 (Sept. 30, 2004), pp. 2162–2168. doi: Techniques 22.4 (Dec. 2011), pp. 122–126. PMID: 10.1002/rcm.1603. 22131886. [40] McKinney, W. “Data Structures for Statistical Com- [51] Tabb, D. L. “Quality Assessment for Clinical Pro- puting in Python.” In: Proceedings of the 9th Python teomics.” In: Clinical Biochemistry 46.6 (Apr. 2013), in Science Conference. Ed. by van der Walt, S. and pp. 411–420. doi: 10.1016/j.clinbiochem.2012. Millman, J. Austin, Texas, USA, 2010, pp. 51–56. 12.003. [41] Paulovich, A. G., Billheimer, D., Ham, A.-J. L., Vega- [52] Tabb, D. L., Fernando, C. G., and Chambers, M. C. Montoto, L., et al. “Interlaboratory Study Characteriz- “MyriMatch: Highly Accurate Tandem Mass Spectral ing a Yeast Performance Standard for Benchmarking Peptide Identication by Multivariate Hypergeomet- LC-MS Platform Performance.” In: Molecular & Cel- ric Analysis.” In: Journal of Proteome Research 6.2 lular Proteomics 9.2 (Feb. 1, 2010), pp. 242–254. doi: (Feb. 2, 2007), pp. 654–661. doi: 10.1021/pr0604054. 10.1074/mcp.M900222-MCP200. [53] Tabb, D. L., Wang, X., Carr, S. A., Clauser, K. R., et al. [42] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., “Reproducibility of Dierential Proteomic Technolo- et al. “Scikit-Learn: Machine Learning in Python.” In: gies in CPTAC Fractionated Xenografts.” In: Journal Journal of Machine Learning Research 12 (Oct 2011), of Proteome Research 15.3 (Mar. 4, 2016), pp. 691–706. pp. 2825–2830. doi: 10.1021/acs.jproteome.5b00859. [43] Pichler, P., Mazanek, M., Dusberger, F., Weilnböck, L., [54] Taylor, R. M., Dance, J., Taylor, R. J., and Prince, et al. “SIMPATIQCO: A Server-Based Software Suite J. T. “Metriculator: Quality Assessment for Mass Which Facilitates Monitoring the Time Course of LC- Spectrometry-Based Proteomics.” In: Bioinformatics MS Performance Metrics on Orbitrap Instruments.” 29.22 (Nov. 15, 2013), pp. 2948–2949. doi: 10.1093/ In: Journal of Proteome Research 11.11 (Nov. 2, 2012), bioinformatics/btt510. pp. 5540–5547. doi: 10.1021/pr300163u. [55] Toby, T. K., Fornelli, L., and Kelleher, N. L. “Progress [44] R Core Team. R: A Language and Environment for in Top-down Proteomics and the Analysis of Proteo- Statistical Computing. R Foundation for Statistical forms.” In: Annual Review of 9.1 Computing. Vienna, Austria, 2016. (June 12, 2016), pp. 499–519. doi: 10.1146/annurev- [45] Rudnick, P. A., Clauser, K. R., Kilpatrick, L. E., anchem-071015-041550. Tchekhovskoi, D. V., et al. “Performance Metrics for [56] Van der Walt, S., Colbert, S. C., and Varoquaux, G. Liquid Chromatography-Tandem Mass Spectrome- “The NumPy Array: A Structure for Ecient Nu- try Systems in Proteomics Analyses.” In: Molecular merical Computation.” In: Computing in Science & & Cellular Proteomics 9.2 (Feb. 1, 2010), pp. 225–241. Engineering 13.2 (Mar. 2011), pp. 22–30. doi: 10. doi: 10.1074/mcp.M900223-MCP200. 1109/MCSE.2011.37. [46] Scheltema, R. A. and Mann, M. “SprayQc: A Real- [57] Vizcaíno, J. A., Csordas, A., del-Toro, N., Dianes, J. A., Time LC-MS/MS Quality Monitoring System to Max- et al. “2016 Update of the PRIDE Database and Its imize Uptime Using o the Shelf Components.” In: Related Tools.”In: Nucleic Acids Research 44 (D1 Jan. 4, Journal of Proteome Research 11.6 (June 1, 2012), 2016), pp. D447–D456. doi: 10.1093/nar/gkv1145. pp. 3458–3466. doi: 10.1021/pr201219e. [58] Walzer, M., Pernas, L. E., Nasso, S., Bittremieux, W., et [47] Sharma, V., Eckels, J., Taylor, G. K., Shulman, N. J., al. “qcML: An Exchange Format for Quality Control et al. “Panorama: A Targeted Proteomics Knowledge Metrics from Mass Spectrometry Experiments.” In: Base.” In: Journal of Proteome Research 13.9 (Sept. 5, Molecular & Cellular Proteomics 13.8 (Aug. 1, 2014), 2014), pp. 4205–4210. doi: 10.1021/pr5006636. pp. 1905–1913. doi: 10.1074/mcp.M113.035907. Bittremieux et al. 2017 • Computational quality control tools for mass spectrometry proteomics 12/12

[59] Wang, X., Chambers, M. C., Vega-Montoto, L. J., Bunk, D. M., et al. “QC Metrics from CPTAC Raw LC- MS/MS Data Interpreted through Multivariate Statis- tics.” In: Analytical Chemistry 86.5 (Mar. 4, 2014), pp. 2497–2509. doi: 10.1021/ac4034455. [60] Wen, B. and Gatto, L. proteoQC: An R package for proteomics data quality control. R package version 1.6.0. 2016. [61] Zhang, B., Wang, J., Wang, X., Zhu, J., et al. “Pro- teogenomic Characterization of Human Colon and Rectal Cancer.” In: Nature 513.7518 (July 20, 2014), pp. 382–387. doi: 10.1038/nature13438.