Protein structure validation by generalized linear model root-mean-square deviation prediction Anurag Bagaria,1 Victor Jaravine,1 Yuanpeng J. Huang,2 Gaetano T. Montelione,2 and Peter Gu¨ ntert1,3* 1Institute of Biophysical Chemistry, Center for Biomolecular Magnetic Resonance, and Frankfurt Institute of Advanced Studies, Goethe University Frankfurt, 60438 Frankfurt am Main, Germany 2Department of Molecular Biology and Biochemistry, Center for Advanced Biotechnology and Medicine, and Northeast Structural Genomics Consortium, Rutgers University, and Robert Wood Johnson Medical School, Piscataway, New Jersey 08854 3Graduate School of Science and Technology, Tokyo Metropolitan University, Hachioji, Tokyo 192-0397, Japan Received 19 October 2011; Revised 17 November 2011; Accepted 19 November 2011 DOI: 10.1002/pro.2007 Published online 23 November 2011 proteinscience.org Abstract: Large-scale initiatives for obtaining spatial protein structures by experimental or computational means have accentuated the need for the critical assessment of protein structure determination and prediction methods. These include blind test projects such as the critical assessment of protein structure prediction (CASP) and the critical assessment of protein structure determination by nuclear magnetic resonance (CASD-NMR). An important aim is to establish structure validation criteria that can reliably assess the accuracy of a new protein structure. Various quality measures derived from the coordinates have been proposed. A universal structural quality assessment method should combine multiple individual scores in a meaningful way, which is challenging because of their different measurement units. Here, we present a method based on a generalized linear model (GLM) that combines diverse protein structure quality scores into a single quantity with intuitive meaning, namely the predicted coordinate root-mean-square deviation (RMSD) value between the present structure and the (unavailable) ‘‘true’’ structure (GLM-RMSD). For two sets of structural models from the CASD-NMR and CASP projects, this GLM-RMSD value was compared with the actual accuracy given by the RMSD value to the corresponding, experimentally determined reference structure from the Protein Data Bank (PDB). The correlation coefficients between actual (model vs. reference from PDB) and predicted (model vs. ‘‘true’’) heavy-atom RMSDs were 0.69 and 0.76, for the two datasets from CASD-NMR and CASP, respectively, which is considerably higher than those for the individual scores (20.24 to 0.68). The GLM-RMSD can thus predict the accuracy of protein structures more reliably than individual coordinate-based quality scores. Keywords: NMR; protein structure validation; CASD-NMR; CASP; structure quality; Gaussian network model; RMSD; GLM-RMSD Abbreviations: CASD-NMR, critical assessment of protein structure determination by NMR; CASP, critical assessment of protein structure prediction; GNM, Gaussian network model; GLM, generalized linear model; MLR, multiple linear regression; NMR, nuclear magnetic resonance; PDB, Protein Data Bank; RMSD, root-mean-square deviation. Additional Supporting Information may be found in the online version of this article. Grant sponsors: Volkswagen Foundation; Deutsche Forschungsgemeinschaft; Grant number: DFG grant JA1952/1-1 (to V.J. and P.G.); Grant sponsors: e-NMR and WeNMR projects of the European Commission and Japan Society for the Promotion of Science (JSPS); National Institutes of Health Protein Structure Initiative; Grant number: U54 GM094597 (to G.T.M.). *Correspondence to: Peter Gu¨ ntert, Institute of Biophysical Chemistry, Goethe University Frankfurt, Max-von-Laue-Str. 9, 60438 Frankfurt am Main, Germany. E-mail: [email protected] Published by Wiley-Blackwell. VC 2011 The Protein Society PROTEIN SCIENCE 2012 VOL 21:229—238 229 Introduction Results Obtaining three-dimensional structures of proteins Here, we present the generalized linear model-RMSD is a priority for the biological research community (GLM-RMSD) method to evaluate the quality of pro- since several decades. Accurate protein structures tein models by predicting their RMSDs relative to the are crucial, for example, for drug design or screening native structure. Originally a simple multiple linear applications. Protein structures can be obtained by regression (MLR) method was tested, using several 2,3,5,6,19 7 experimental techniques such as X-ray crystallogra- quality scores provided by the PSVS server. phy and nuclear magnetic resonance (NMR) spec- After performing a statistical model selection and sig- troscopy, or predicted by computational methods nificance study (as described in ‘‘Model Selection sec- such as homology modeling, threading, fold recogni- tion’’) on the initially included scores, a GLM-RMSD tion, molecular dynamics simulation, and so forth.1 using only four out of the initial eight scores was Generally, the accuracy of a structural model deter- found to perform better. We applied the GLM-RMSD mines the range of its potential applications. For method to two sets of protein structure models from example, it would be nearly meaningless to use a the CASD-NMR and CASP8 projects, respectively. protein structure for structure-based drug design if The resulting GLM-RMSD values were correlated we are unsure about the quality of the target protein with the actual accuracy of the protein structure mod- model. Many measures have been proposed for the els in terms of the RMSD of the coordinates between assessment of structural quality,2 including torsion the model and the corresponding reference structure angle distributions, steric clashes, three-dimensional from the Protein Data Bank (PDB). profiles, residue environments, fold prediction, eval- uations of experimental restraints, and so forth, MLR predicted RMSD which have been implemented in popular structure MLR is a multivariate statistical technique for validation software packages including Procheck,3 examining the linear correlations between two or Molprobity,4 WHAT IF,5 Verify3D,6 ProsaII,7 Protein more independent variables and a single dependent Structure Validation Software suite (PSVS),8 and so variable. Here, we consider a linear model by which LM forth. There have been discussions as to why an the predicted RMSD values yi (LM-RMSD) individual assessment score should directly correlate between the i-th model structure from the training with a coordinate accuracy metric,9 as well as set and the corresponding native protein structure attempts to obtain composite quality assessment depend linearly on m (normalized) validation scores 10–12 scores. These methods can address global and/or (xi1, ..., xim) local accuracy13 and can express these in terms of root-mean-square deviation (RMSD) values.14 Alter- Xm yLM ¼ b x þ a natively, a model-specific scoring function can be i j ij j¼1 constructed by a support vector machine.15 Structural genomics projects aim at experimen- for proteins i ¼ 1, ..., n. The LM-RMSD thus repre- tally determining three-dimensional structures of sents an approximation of the (known) accuracy of representative members of as many different fold the model structure. The value of the j-th validation families as possible, whereas comparative modeling score for the model structure i is denoted by xij. tries to complement this by providing structural Each score describes a particular aspect of structure models of homologues of the representative proteins. quality. The constants a and bj are determined by a However, for successful predictions it is essential that linear least-squares fit to the actual RMSDi values the quality of the experimentally determined struc- from a training set of n known structures that mini- tures is adequate. Evaluating NMR protein structure mizes the v2 value determination methods is the aim of the critical assessment project critical assessment of protein ! Xn Xm 2 2 structure determination by nuclear magnetic reso- v ¼ RMSDi À bjxij þ a 16 nance (CASD-NMR) for NMR solution structures. A i¼1 j¼1 similar, large-scale project for the critical assessment of structure prediction is CASP.17 Evaluating the Generalized linear model quality of NMR structures has always been a A GLM20 is a generalization of the above model, GLM challenging task, and several erroneous structures where the values yi are assumed to belong to a were found.18,19 These investigations have shown the distribution from the exponential family with a l > GLM ¼ l difficulty to evaluate reliably the accuracy of NMR mean 0 and E(yi ) . In our case the gamma structures using only traditional quality indicators distribution is chosen from the family as it is non- based on coordinates or conformational restraint data negative (like the RMSD values) and showed the and highlighted the need for a tight integration of closest similarity in shape to the histogram of the more sophisticated structure validation tools in NMR RMSDi values. The validity of this assumption was structure determination projects. evaluated by fitting a gamma distribution to the 230 PROTEINSCIENCE.ORG Protein Structure Validation by GLM-RMSD Table I. Mean Values (Standard Deviation) of the (Unnormalized) Quality Scores Sequences/ Dataset structures DP Verify3D ProsaII P-u/w P-All MolProb GNM Size CASD-NMR 16/65 0.75 (0.09) À1.42 (2.07) À2.58 (3.06) À1.84 (1.18) À0.62 (0.97) À3.55 (8.18) 0.81 (0.03) 106.15 (29.71) CASP 85/85 0.39 (0.09) 0.74 (1.33) 0.50 (1.27) À2.19 (1.47) À1.11 (1.12) À18.77 (5.43) 0.51 (0.06) 133.75 (56.77) Combined 101/150 0.55 (0.20) À0.20 (2.00) À0.83 (2.70) À2.04 (1.37) À0.90 (1.08) À12.17 (10.13) 0.64 (0.16) 121.79 (48.95) histograms of RMSD values from the CASD-NMR, coarse-grained approach to estimate the average CASP, and Combined datasets by maximum likeli- coordinate fluctuation. GNM scores were reported to hood fitting using the function fitdist() from the R be correlated to protein stability (DG) and to the software environment for statistical computing RMSD itself.25,26 (viii) The protein size given by the (http://www.r-project.org/) package ‘‘fitdistrplus’’ number of residues (Size), which was included (Supporting Information Fig.