PROTEINS: Structure, Function, and Bioinformatics 57:702–710 (2004) Scoring Function for Automated Assessment of Protein Structure Template Quality Yang Zhang and Jeffrey Skolnick* Center of Excellence in Bioinformatics, University at Buffalo, Buffalo, New York ABSTRACT We have developed a new scoring aligned, the resulting full-length model might be of poorer function, the template modeling score (TM-score), to quality. The template assessment problem becomes particu- assess the quality of protein structure templates larly relevant during the development of efficient fold and predicted full-length models by extending the recognition algorithms, since different sequence–structure approaches used in Global Distance Test (GDT)1 and alignment schemes or parameters can result in various MaxSub.2 First, a protein size-dependent scale is levels of alignment confidence with an associated loss or exploited to eliminate the inherent protein size gain of alignment coverage.3–6 Therefore, a single assess- dependence of the previous scores and appropri- ment score that has an appropriate balance of alignment ately account for random protein structure pairs. accuracy and coverage and that is strongly related to the Second, rather than setting specific distance cutoffs quality of the final full-length model is essential. Equally and calculating only the fractions with errors below important, it must differentiate between a random and a the cutoff, all residue pairs in alignment/modeling statistically significant prediction. are evaluated in the proposed score. For compari- Highly related to the above problems, several interest- son of various scoring functions, we have con- ing scoring functions have been developed for the purpose structed a large-scale benchmark set of structure of sequence-dependent comparison of two structures of templates for 1489 small to medium size proteins different lengths (in contrast to sequence-independent using the threading program PROSPECTOR_3 and structure alignment algorithms).1,2,13–15 For example, with built the full-length models using MODELLER and MaxSub,2 Siew and coworkers tried to identify the maxi- TASSER. The TM-score of the initial threading align- mum substructure in which the distances between equiva- ments, compared to the GDT and MaxSub scoring lent residues of two structures after superposition are functions, shows a much stronger correlation to the below some threshold value, such as 3.5 Å. Since the quality of the final full-length models. The TM-score MaxSub scoring function only counts those residues in- is further exploited as an assessment of all ‘new fold’ cluded in the substructure, the spatial information of the targets in the recent CASP5 experiment and shows a templates outside the substructure is omitted. For ex- close coincidence with the results of human-expert ample, Figure 1(a) shows the MaxSub superposition of the visual assessment. These data suggest that the TM- native structure of 2sas_ and the template alignment (94% score is a useful complement to the fully automated coverage, which is the ratio of the number of aligned assessment of protein structure predictions. The residues to the number of target residues) obtained from executable program of TM-score is freely download- the threading program PROSPECTOR_3,6 where residue able at http://bioinformatics.buffalo.edu/TM-score. pairs of distance Ͻ3.5 Å are highlighted in red (50% Proteins 2004;57:702–710. © 2004 Wiley-Liss, Inc. coverage), with the remainder of the aligned residues in INTRODUCTION yellow. The templates in Figure 1(b) (original alignment having 94% coverage) and Figure 1(c) (the ‘well-aligned’ The canonical comparative modeling/threading-based part with 50% coverage) therefore have the same MaxSub- protein structure prediction procedure consists of two score, which is only associated with the set of red residues. steps: (i) finding a solved structure related to the target However, the facility of the templates for the final full- 3–6 sequence (i.e. template) and (ii) building a full-length length structure modeling can be significantly different. 7–10 model based on the template. The quality of the Using the structure building program MODELLER,7,10 for resulting full-length model is usually assessed by the root example, the template in Figure 1(b) results in a full- 11,12 mean square deviation (RMSD) between equivalent length model with a RMSD from native of 4.4 Å, while the atoms in the model and native structures. However, template in Figure 1(c) results in a full-length model RMSD alone is not sufficient to estimate the quality of the initial templates because the alignment coverage can be very different in different approaches.3–6 Obviously, a *Correspondence to: Jeffrey Skolnick, Center of Excellence in template witha2ÅRMSD to native having 50% align- Bioinformatics, University at Buffalo, 901 Washington St., Buffalo, NY 14203. E-mail: [email protected]. ment coverage is not necessarily better for structure Received 30 December 2003; Revised 16 March 2004, 25 May 2004; modeling than one with a RMSD of 3 Å but having 80% Accepted 9 June 2004 alignment coverage. While the template aligned regions Published online 8 October 2004 in Wiley InterScience are better in the former because fewer residues are (www.interscience.wiley.com). DOI: 10.1002/prot.20264 © 2004 WILEY-LISS, INC. SCORING FUNCTION FOR TEMPLATE QUALITY 703 Fig. 1. (a) Superposition between the template and the native structure of 2sas from MaxSub. The native structure is shown as thin backbone in white; the template is shown as thick backbone in yellow, which is from Chain-A of 2scp in the PDB library17 hit by PROSPECTOR_3.6 The residues with pairwise distances lower than 3.5 Å are highlighted in red for both template and native. (b) The whole template alignment has 94% coverage. (c) The substructure of the template alignment with a distance to native of less than 3.5 Å (‘well-aligned’ part) has 50% coverage. The templates in (b) and (c) have the same MaxSub score of 0.434. However, using MODELLER,7,10 the template in (b) results in a full-length model of RMSD 4.4 Å, while the template in (c) results in a full-length model of RMSD 12.5 Å. having a RMSD from native of 12.5 Å. By way of further illustration, in a large-scale benchmark set of PROSPEC- TOR_3 alignments (see below), there are 81 cases having MaxSub-scores between 0.4 and 0.45. The RMSD values of the final full-length models built from MODELLER vary from 3.5 to 35.7 Å with a standard deviation of 4.8 Å. Thus, there is no apparent correlation between the Maxsub-score and the quality of the resulting full-length model. In their GDT_TS scoring function, Zemla and cowork- ers1,13 further identify multiple maximum substructures associated with several different threshold cutoffs (e.g. 1, 2,4,and8Åasused in the recent CASP5 experiment16). The GDT_TS-score is defined as the average coverage of the target sequence of the substructures with the four different distance thresholds. Since the GDT_TS-score focuses only on the size of the substructures, the detailed match information of templates/models and native struc- tures is partially missed (e.g. residues with deviations ranging from 4.1–8 Å from native have identical contribu- tions to the scoring function). Zemla13 further addressed Fig. 2. The average MaxSub-scores (triangles) and GDT-scores (circles) as a function of protein size. The data are calculated from all pairs this problem by introducing more distance thresholds. of 3656 proteins taken from PDB library17 that have Ͻ30% pairwise Another problem associated with these score functions is sequence identity. The statistical error bars are smaller than the size of the dependence of the score magnitudes on the evaluated the points. The solid and dashed lines are the nonlinear least square Marquardt–Levenberg fit of the MaxSub and GDT data to a power-law proteins’ size. In other words, one must address the issue equation of f(L) ϭ aLb, where L is the length of the smaller protein of the of what the corresponding score value of a pair of randomly corresponding structure pair. The fit parameters a and b are as indicated. related structures would be. In Figure 2, we plot the average MaxSub- and GDT-scores as a function of the length of protein for random structure pairs in the Protein been observed by many authors when structure similarity Data Bank (PDB)17 that have pairwise sequence identity is measured by structure alignment18,19 or RMSD calcula- of less than 30%. These scores show a power-law depen- tion.20,21 To eliminate the dependence on protein size, dence on the protein size. Obviously, a given absolute Levitt and Gerstein18 and Ortiz and coworkers19 con- score, such as GDT ϭ 0.4 or MaxSub ϭ 0.3, can reflect a verted their structure alignment score into a statistical significant alignment for a target of 400 residues, but it is significance score, called the P-value, on the basis of the close to a random selection in the PDB for a target of 40 statistics of their random structure database. For their residues. This significant size dependence renders the relative RMSD, Betancourt and Skolnick20 normalized the absolute magnitude of these scoring functions meaning- RMSD by the average RMSD from random structure pairs less. with similar size and radii of gyration. In the RMSD-100 The significant protein size dependence of structure score,21 Carugo and Pongor divided the RMSD by a factor similarity for randomly related structure pairs has also of 1 ϩ͌N/100, with N representing the protein length. 704 Y. ZHANG AND J. SKOLNICK In this article, we extend the above approaches and develop a new scoring function for the assessment of threading templates, which we call the template modeling (TM) score.