Vol. 19 no. 1 2003, pages i1–i7 BIOINFORMATICS DOI: 10.1093/bioinformatics/btg1029
APDB: a novel measure for benchmarking sequence alignment methods without reference alignments Orla O’Sullivan 1, Mark Zehnder 3, Des Higgins 1, Philipp Bucher 3, Aurelien Grosdidier 3 and Cedric´ Notredame 2, 3,∗
1Department of Biochemistry, University College, Cork, Ireland, 2Information Gen´ etique´ et Structurale, CNRS UMR-1889, 31, Chemin Joseph Aiguier, 13402 Marseille, France and 3Swiss Institute of Bioinformatics, Chemin des Boveresse, 155, 1066 Epalinges, Switzerland
Received on January 6, 2000; revised on Month xx, 2000; accepted on February 20, 2000
ABSTRACT techniques used in bioinformatics require the assembly Author please Motivation: We describe APDB, a novel measure for eval- of a multiple sequence alignment at some point. These check use of A and B uating the quality of a protein sequence alignment, given include phylogenetic tree reconstruction, detection of heads is correct two or more PDB structures. This evaluation does not re- remote homologues through the use of profiles or HMMs, quire a reference alignment or a structure superposition. secondary and tertiary structure prediction and more re- APDB is designed to efficiently and objectively benchmark cently the identification of the nsSNPs (non synonymous multiple sequence alignment methods. Single Nucleotide Polymorphisms) that are most likely to Results: Using existing collections of reference multiple alter a protein function. All of these important applica- sequence alignments and existing alignment methods, tions demonstrate the need to improve existing multiple we show that APDB gives results that are consistent sequence alignment methods and to establish their true with those obtained using conventional evaluations. We limits and potential. Doing so is complicated, however, also show that APDB is suitable for evaluating sequence because most multiple sequence alignment methods alignments that are structurally equivalent. We conclude rely on a complicated combination of greedy heuristic that APDB provides an alternative to more conventional algorithms meant to optimize an objective function. methods used for benchmarking sequence alignment This objective function is an attempt to quantify the packages. biological quality of an alignment. Almost every multiple Availability: APDB is implemented in C, its source code alignment package uses a different empirical objective and its documentation are available for free on request function of unknown biological relevance. In practice, from the authors. most of these algorithms are known to perform well on Contact: [email protected] some protein families and less well on others, but it is difficult to predict this in advance. It can also be very INTRODUCTION hard to establish the biological relevance of a multiple We introduce APDB (Analyze alignments with PDB), a alignment of poorly characterized protein families. See new method for benchmarking and improving multiple se- Duret and Abdeddaim (2000) and Notredame (2002) for quence alignment packages with minimal human interven- two recent reviews of the wide variety of techniques that tion. We show how it is possible to avoid the use of ref- have been used to make multiple alignments. erence alignments when PDB structures are available for Given such a wide variety of methods and such poor at least two homologous sequences in a test alignment. theoretical justification for most of them, the main option Using this method it should become possible to system- for a rational comparison is systematic benchmarking. atically benchmark or train multiple sequence alignment This is usually accomplished by comparing the alignments methods using all known structures, in a completely auto- produced by various methods with ‘reference’ alignments matic manner. of the same sequences assembled by specialists with There are strong justifications for improving multiple the help of structural information. Barton and Sternberg sequence alignment methods. Many sequence analysis (1987) made an early systematic attempt to validate a mul- tiple sequence alignment method using structure based ∗ To whom correspondence should be addressed. alignments of globins and immunoglobulins. Later on,
Bioinformatics 19(1) c Oxford University Press 2003; all rights reserved. 1 O.O’Sullivan et al.
Notredame and Higgins (1996) used another collection alignment procedures break this tie in an arbitrary fashion, of such alignments assembled by Pascarella and Argos leading to a reference alignment that represents only one (1992). More recently, it has become common practice possible arrangement of aligned residues. to use BAliBASE (Thompson et al., 1999); a collection This problem becomes most serious when the sequences of multiple sequence alignments assembled by specialists one is considering are distantly related (less than 30% and designed to systematically address the different types identity). Unfortunately, this is also the most interesting of problems that alignment programs encounter, such as level of similarity where most sequence alignment meth- the alignment of a distant homologue or long insertions ods make errors and where it is important to accurately and deletions. In this work, we examined two such refer- benchmark existing algorithms. The APDB method that ence collections: BaliBase and Homstrad (Mizuguchi et we describe in this work has been designed to specifically al., 1998), a collection of high quality multiple structural address this problem and remove, almost entirely, the alignments. need for arbitrary decisions when using structures to There are two simple ways to use a reference alignment evaluate the quality of a multiple sequence alignment. for the purpose of benchmarking Karplus and Hu (2001). In APDB, a target alignment is not evaluated against a One may count the number of pairs of aligned residues reference alignment. Rather, we measure the quality of the in the target alignment that also occur in the reference structural superposition induced by the target alignment alignment and divide this number by the total number of given any structures available for the sequences it con- pairs of residues in the reference. This is the Sum of Pairs tains. By treating the alignment as the result of some sort Score (SPS). The main drawback is that it is not very of structure superposition, we simply measure the fraction discriminating and tends to even out differences between of aligned residues whose structural neighborhoods are methods. The more popular alternative is the Column similar. This makes it possible to avoid the most expen- Score (CS) where one measures the percentage of columns sive and controversial element of the MSA benchmark- in the target alignment that also occur in the reference ing methods: the reference multiple sequence alignment. alignment. This is widely used and is considered to be APDB requires just three parameters. This is tiny if we a stringent measure of alignment performance. In order compare it with any reference alignment where each pair to avoid the problem of unalignable sections of protein of aligned residue can arguably be considered as a free pa- sequences (i.e. segments that cannot be superimposed), it rameter. is common practice to annotate the most reliable regions In this work we show how the APDB measure was of a multiple structural alignment and to only consider designed and characterized on a few carefully selected these core regions for the evaluation. In BaliBase, the core pairs of structures. Among other things we explored its regions make up slightly less than 50% of the total number sensitivity to parameter settings and various sequence of alignment columns. and structure properties, such as similarity, length, or Such use of multiple sequence alignment collections for alignment quality. Finally, APDB was used to benchmark benchmarking is very convenient because of its simplicity. known methods using two popular data sets: BaliBase and However, a major problem is the heavy reliance on the Homstrad. These were either used as standard reference correctness of the reference alignment. This is serious alignments or as collections of structures suitable for because, by nature, these reference alignments are at least APDB. partially arbitrary. Although structural information can be It should be noted that there are several methods for handled more objectively than sequence information, the evaluating the quality of structure models and predictions assembly of a multiple structural alignment remains a very using known structures. The development of these has complex problem for which no exact solution is known. been driven by the need to evaluate entries in the CASP As a consequence, any reference multiple alignment based protein structure prediction competition and have been on structure will necessarily reflect some bias from the reviewed by Cristobal et al. (2001). These all depend methods and the specialist who made the assembly. The on generating structure superpositions between the model second drawback is that given a set of structures there and the target and evaluating the quality of the match can be more than one correct alignment. This plurality using, for example, RMSD between the two or using some results from the fact that a structural superposition does measure of the number of alpha carbons that superimpose not necessarily translate unambiguously into one sequence well (e.g. MaxSub by Siew et al. (2000)). In principle, alignment. For instance, if we consider that the residues this could also be used to benchmark alignment methods. to be aligned correspond to the residues whose alpha However, one serious disadvantage is the requirement for carbons are the closest in the 3-D superposition, it is a superposition, which is itself a difficult problem. A easy to imagine that sometimes an alpha carbon can second disadvantage is the way RMSD measures behave be equally close to the alpha carbons of two potential with different degrees of sequence divergence and their homologous residues. Most structure based sequence sensitivity to local or global alignment differences. We i2 APDB
have carefully designed APDB so that on the one hand correctX:Y (Z : W) is equal to 1 if d(X, Z) and d(Y,W) it remains very simple but on the other hand it is able to are sufficiently similar as set by T 1. measure the similarity of the structural environments in a aligned(X : Y) is equal to 1 if most pairs Z : W in the manner that lends itself to measuring alignment quality. X : Y bubble are correct as set by T 2. ( : ) = SYSTEM AND METHODS consideredX:Y Z W 1 ( , )< ( , )< The APDB scoring function if d X Z Brad and d Y W Brad (1) ( : ) = APDB is a measure designed to evaluate how consistent correctX:Y Z W 1 an alignment is with the structure superposition this if d(X, Z)
i3 O.O’Sullivan et al.
A second method for generating sub-optimal alignments was based on the PROSUP package (Lackner et al., 2000). PROSUP takes two structures, makes a rigid body superposition and generates all the sequence alignments that are consistent with this superposition, thus producing alternative alignments that are equivalent from a structural point of view. Typically PROSUP yields 5 to 25 alternative alignments within a very narrow range of RMSDs.
Comparison of apdb with other standard measures In order to compare the APDB measure with more con- ventional measures, we used the Column Score (CS) mea- sure as provided by the aln compare package (Notredame et al., 2000). CS measures the percentage of columns in a test alignment that also occur in the reference alignment. In BAliBASE this measure is restricted to those columns annotated as core region in the reference. Although alter- native measures have recently been introduced (Karplus and Hu, 2001), CS has the advantage of being one of the most widely used and the simplest method available today.
Fig. 1. Tuning of Brad, the bubble radius using sub-optimal RESULTS AND DISCUSSION alignments of two sequences from HOM 9 Each graph represents Fine tuning of apdb the correlation between CS and APDB for 4 different Bubble Radius values (Brad of 6, 8, 10 and 12A).û In each graph, each dot represents Three parameters control the behaviour of APDB: Brad a sub-optimal alignment from HOM 9, sampled from the genetic (the bubble radius), T1 (the difference of distance thresh- algorithm. old) and T2 (the fraction of the bubble neighbourhood that must support the alignment of two residues). We exhaus- tively studied the tuning effect of each of these param- Generation of multiple alignments eters using HOM 9 and parameterised APDB so that its We compared the performance of APDB on two different behaviour is as consistent as possible with the behaviour multiple alignment methods. We tested the widely used of CS on HOM 9. ClustalW version 1.80 (Thompson et al., 1994). We also In Figure 1 we show the relationship between CS tested the more recent T-Coffee version 1.37 (Notredame and APDB for 250 sub-optimal alignments generated by et al., 2000) using default parameters. genetic algorithm for one of the 9 test cases from HOM 9 over 4 different settings of Brad, the Bubble Radius. While Generation of suboptimal alignments the two scoring schemes are in broad agreement, the In order to evaluate the sensitivity of APDB to the correlation improves dramatically as Brad increases. This quality of an alignment, we used an improved version trend can be summarised using the correlation coefficient of the genetic algorithm SAGA (Notredame and Higgins, measured on each of the graphs similar to those shown in 1996) in order to generate populations of sub-optimal Figure 1. The overall results for all nine HOM 9 test cases alignments. In each case a pair of sequences was chosen are shown in Figure 2. These results clearly show that the in HOM 9 and 50 random alignments were generated behaviour of APDB is best for values of Brad of 10 Aorû and allowed to evolve within SAGA so that their quality above. With these values the level of correlation between gradually improved (as measured by their similarity with CS and APDB increases and so does the agreement across the HOMSTRAD reference alignment). Ten alignments all 9 test cases. We chose 10 Aû as the default value in were sampled at each generation in order to build a order to ensure a proper behaviour while retaining as much collection of alternative alignments with varying degrees as possible the local character of the measure. Given the of quality. This algorithm was stopped when optimality default value of 10 Aû for Brad, we examined T1 and T2 in was reached, thus typically yielding collections of a few a similar fashion and found the most appropriate values as hundred alignments. 1 Aû for T1 and 70% for T2. i4 APDB
Fig. 2. Correlation between the Column Score measure (CS) and APDB on HOM 9 Each HOM 9 test set is labelled according to its average percent sequence identity as measured on the HOMSTRAD reference. The horizontal axis indicates the value of Brad. The vertical axis indicates the correlation coefficient between CS and APDB as measured on a population of sub-optimal alignments similar to the ones in Figure 1. Each dot indicates a correlation coefficient measured on one HOM 9 test set, using the indicated value of Brad. Each HOM 9 test set is an alignment between two sequences whose PDB names are as follows: 17: 2gar versus 1fmt, 18: ljfl versus lb74, 33: 1isi versus 11be, 43: 2cev versus 1d3v, 52: 1aq0 versus 1ghs, 63: 2gnk versus 2pii, 71: 1hcz versus 1cfm, 82: 1dvg versus 1qq8, 89: 1k25 versus 1qme.
Sensitivity of apdb to sequence and structure same and clearly indicates that APDB gives consistent similarity results regardless of the structural similarity between the It is important to verify that the behaviour of APDB structures being considered. remains consistent across a wide range of sequence similarity levels. It is especially important to make sure Suitability of apdb for analysing sub-optimal that when two different alignments of the same sequences alignments are evaluated, the best one (as judged by comparison with Collections of sub-optimal alignments for each of the the HOMSTRAD reference) always gets the best APDB nine HOM 9 test sets were generated using SAGA and score. In order to check for this, we used the genetic evaluated for their CS scores and APDB scores. These algorithm to generate sub-optimal alignments for each test results were pooled and are displayed on the graph case in HOM 9. In each case, we gathered a collection shown on Figure 4. This Figure indicates good agreement of 250 sub-optimal alignments with CS scores of 0Ð40%, between the CS and the APDB score regardless of the 41Ð60%, 61Ð80% and 81Ð100%. The CS score measures level of optimality within the alignment being considered. the agreement between an alignments and its reference It suggests that APDB is informative over the complete in HOMSTRAD. We then measured the average APDB range of CS values. It also confirms that APDB is not ‘too score in each of these collections. Each of these measures generous’ with sub-optimal alignments corresponds to a dot in Figure 3 where vertically aligned We also checked whether sequence alignments that series of dots correspond to different measures made on are structurally equivalent obtain similar APDB scores the same HOM 9 test set. even if they are different at the sequence level. For this Figure 3 clearly shows that regardless of the percent purpose, we used PROSUP (Lackner et al., 2000). Given identity within the HOM 9 test set being considered, a pair of structures, PROSUP generates several alignments alignments with better CS scores always correspond to a that are equally good from a structure point of view better APDB score (this results in the lines never crossing (similar RMSD), but can be very different at the sequence one another on Fig. 3). We did a similar analysis using level (different Column Score). We manually identified the RMSD as measured on the HOMSTRAD alignment two such test sets in HOMSTRAD and the results are in place of sequence identity. The behaviour was the summarized in Table 1. For each of these two test sets, we
i5 O.O’Sullivan et al.
Fig. 3. Estimation of the sensitivity of APDB to sequence identity On this graph, each set of vertically aligned dots corresponds to a single HOM 9 test set. The 9 HOM 9 test sets are arranged according to their average identity (17Ð89%, see Figure 2). Each dot represents the average APDB score of a population of 250 sub-optimal alignments (generated by genetic algorithm) with a similar CS score (binned in four groups representing CS of <40%, 41Ð60%, 61Ð80% and 81Ð100%) generated for one of the 9 HOM 9 test sets.
Table 1. Evaluating PROSUP suboptimal alignments with APDB
Set St1 St2 ALN RMSD CS APDB
1 1e96B 1a17 aln1 1.45Aû 100.0 80.2 1e96B 1a17 aln2 1.50A˚ 65.6 80.7 2 1cd8 1qfpa aln1 2.95Aû 100.0 18.7 1cd8 1qfpa aln2 2.95A˚ 55.1 17.9
Set indicates the test set index, St1 and St2 indicate the two structures being aligned by PROSUP, ALN indicates the alignment being considered, RMSD shows the RMSD associated with this alignment, CS indicates its CS score, with the CS score of aln1 alignments being set to 100 because they are used as references. APDB indicates the APDB score.
In both test sets, using aln1 as a reference for the CS measure leads to the conclusion that aln2 is mostly incorrect (cf. CS column of Table 1). This is not true since these alignments are structurally equivalent as indicated by their RMSDs. In such a situation, APDB behaves much more appropriately and gives to each couple aln1/aln2 Fig. 4. Correlation between CS and APDB on the complete scores that are nicely consistent with their RMSD, thus HOM 9 test set Each dot corresponds to a sub-optimal alignment indicating that APDB can equally well reward two sub- of one of the HOM 9 test cases, generated by genetic algorithm. For each alignment the graph plots the APDB score against its CS optimal alignments when these are equivalent from a counterpart. structural point of view. Using apdb to benchmark alignment methods Table 2 shows the average CS and APDB scores for the selected in the output of PROSUP two alignments (aln1 test sets in each of the four Bali 91 categories being and aln2) to which PROSUP assigns similar RMSDs. aln1 considered here and in HOM 43. The highest scores in is used as a reference and therefore gets a CS score of all cases, for both measures, come from the reference 100 while the CS score of the second alignment (aln2)is column (the last column). This is desirable providing computed by direct comparison with its aln1 counterpart. the reference alignments really are consistent with the i6 APDB
Table 2. Correlation between APDB and CS on BaliBase and Homstrad local evaluation and the absence of a reference alignment, the only possible effect of non-superposable regions is Set N ClustalW T-Coffee Reference to decrease the proportion of residues found aligned in CS APDB CS APDB CS APDB a structurally optimal sequence alignment, thus yielding scores lower than 100 in the case of distantly related B91 R1 35 70.1 59.9 67.7 58.3 100 64.7 structures. B91 R2 23 32.7 26.6 33.9 47.1 100 55.2 A key advantage of APDB is its simplicity. It only re- B91 R3 22 46.4 38.5 48.6 46.9 100 53.2 quires three parameters and a few PDB files. Most impor- B91 R4 11 52.0 59.5 52.5 64.5 100 65.7 tantly, APDB does not require any arbitrary manual in- H43 43 35.4 60.2 38.9 61.6 100 72.9 tervention such as the assembly of a reference alignment. In the short term, all the existing collections of reference Test Set: indicates the test set being considered, either one of the alignment could easily be integrated and extended with BaliBase 91 references (B91R#) or HOM 43(H43), a subset of HOMSTRAD. N indicates the number of test alignments in this category. APDB. In the longer term, APDB could also be used to ClustalW indicates a set of measures made on alignments generated with evaluate and compare existing collections of alignments ClustalW. T-Coffee indicates similar measures made on T-Coffee generated such as profiles, when structures are available. alignments. Reference indicates measures made on the reference alignments as provided in BaliBase or in Homstrad. CS columns are the Column Score measures while APDB indicates similar measures made using APDB. REFERENCES Barton,G.J. and Sternberg,M.J.E. (1987) A strategy for the rapid underlying structures. If we now compare the columns multiple alignment of protein sequences: confidence levels from twobytwo,wefind that every variation on CS from one tertiary structure comparisons. J. Mol. Biol., 198, 327Ð337. column to another agrees with the corresponding variation Cristobal,S., Zemla,A., Fischer,D., Rychlewski,L. and Elofsson,A. of APDB. For instance in row 1 (Bali 91 Ref1), when (2001) A study of quality measures for protein threading models. T-Coffee/CS is lower than ClustalW/CS, T-Coffee/APDB BMC Bioinformatics, 2,5. is also lower. This observation is true for the whole Duret,L. and Abdeddaim,S. (2000) Multiple alignment for struc- table, regardless of the pair of results being considered. tural, functional, or phylogenetic analyses of homologous se- When considering the 134 alignments one by one, this quences. Bioinformatics, Sequence, Structure and Databanks. Higgins,D. and Taylor,W. (eds), Oxford University Press, Ox- observation remains true in more than 70 % of the cases. ford. Karplus,K. and Hu,B. (2001) Evaluation of protein multiple align- CONCLUSION ments by SAM-T99 using the BAliBASE multiple alignment test This work introduces APDB, a novel method that makes set. Bioinformatics, 17, 713Ð720. it possible to evaluate the quality of a sequence alignment Lackner,P., Koppensteiner,W.A., Sippl,M.J. and Domingues,F.S. when two or more tertiary structures of the sequences it (2000) ProSup: a refined tool for protein structure alignment. contains are available. This method does not require a ref- Protein Eng., 13, 745Ð752. Mizuguchi,K., Deane,C.M., Blundell,T.L. and Overington,J.P. erence alignment and it does not depend on any complex (1998) HOMSTRAD: a database of protein structure alignments procedure such as structure superposition or sequence for homologous families. Protein Sci., 7, 2469Ð2471. alignment. We show here that APDB sensitivity is com- Notredame,C. (2002) Recent progress in multiple sequence align- parable with that of CS, a well-established measure that ments. Pharmacogenomics, 3, 131Ð144. compares a target alignment with a reference alignment. Notredame,C. and Higgins,D.G. (1996) SAGA: sequence alignment Our results also indicate that APDB can discriminate by genetic algorithm. Nucleic Acids Res., 24, 1515Ð1524. better than CS between structurally correct sub-optimal Notredame,C., Higgins,D.G. and Heringa,J. (2000) T-Coffee: a sequence alignments and structurally incorrect sequence novel algorithm for multiple sequence alignment. J. Mol. Biol., alignments, even when the structures being considered are 302, 205Ð217. distantly related. Pascarella,S. and Argos,P. (1992) A data bank merging related Apart from the cost associated with their assembly, a protein structures and sequences. Protein Eng., 5, 121Ð137. Siew,N., Elofsson,A., Rychlewski,L. and Fischer,D. (2000) Max- serious problem with reference alignments is that they Sub: an automated measure for the assessment of protein struc- need to be annotated to remove from the evaluation ture prediction quality. Bioinformatics, 16, 776Ð785. regions that correspond to non-superposable portions of Thompson,J., Higgins,D. and Gibson,T. (1994) CLUSTAL W: im- the structures. This is necessary because otherwise these proving the sensitivity of progressive multiple sequence align- regions (whose alignment cannot be trusted) will bias a ment through sequence weighting, position-specific gap penalties CS evaluation toward rewarding the arbitrary alignment and weight matrix choice. Nucleic Acids Res., 22, 4673Ð4690. conformation displayed in the reference. Table 2 illustrates Thompson,J., Plewniak,F. and Poch,O. (1999) BaliBase: a bench- well the fact that such an annotation is not necessary mark alignment database for the evaluation of multiple sequence in APDB. In our measure, thanks to the combination of alignment programs. Bioinformatics, 15(1),87Ð88.
i7