TRACKING ERROR: an Essential Tool in Evaluating Index Funds

JOURNAL OF INVESTMENT CONSULTING TRACKING ERROR: An Essential Tool in Evaluating Index Funds

By Chris Tobe, CFA

As the dollars being indexed grow and portfolio management becon1es more sophisticated, consultants should demand tighter tracking to the index in question from their index n1anagers. In fact, the key to assessiI1g index fund managers is determining if their performance has tracked their benchmarks within reasonable tolerances. The alltl10r reviews what is known about this aspect of performance evaluation. He also sl1ares 11is experie11ce in detern1ining what factors caused a difference in performance for a $3 billion S&P 500 index fund.

This article is based on a study written by the author and should be no attempt to outperform the index or Dr. Ken Miller for Kentucky State Auditor Edward B. bencl1mark. William F Sharpe holds that a passive Hatchett Jr. The study was reported in the July 27, 1998, strategy requires the manager issue of Pensions and Investments. Earlier versions of this ... to hold every security from the market, with each article were presented at two conferences on indexing. represented in the same manner as in the n1arket. Thus, if security X represents 3 percent of the value ver $1 trillion is now being managed of the securities in the market, a passive investor's within index funds. Lipper lists the portfolio will have 3 percent of its value invested in Vanguard 500 Index fund as the second X. Equivalently; a passive manager will hold the same percentage of the total outstanding amount of largest stock fllnd in the world with a each security in the market. 1 value of over $91 billio11 as of August 31. Frank Vanguard's website Cwww.vanguard.com) glos Russell Company recently reported that its top sary defilles index investing as "an investment strat institutional clients had over 46 perce11t of their egy involving the creation of a portfolio tailored as U.s. equity portfolios indexed, with 29 percent of closely as possible to match the perforn1ance of a their non-U.S. stock portfolios and 9 percent of specific stock or bond market index... and thus their fixed income portfolios n1anaged passively expected to provide a highly predictable return rel Russell attributes tllis growth ill dollars indexed to ative to the specific benchmarks...." the demand for lower asset management fees and This brings us to the issue of modifying an transaction costs, increased market efficiency, as indexed portfolio. Some indexing managers will well as the development of more benchmark index drop issues because it isn't practical to buy all the es. Witl1 many new entrants into index manage securities in the index. This practice is acceptable if ment and many plan sponsors themselves indexing, tlle mallager is using optimization or sampling tech new standards of performance evaluation need to niques that blind the manager to any security analy be established. sis factors. Other managers, however, use this selec tion process to enhance the overall return on the What Is an Index Fund? portfolio. The question for consultants is "to what imply put, index funds are paSSively man degree does this practice change the management of aged portfolios. This strategy seeks only to tl1e portfolio to an active style?" At the 1998 AIMR match the composition and investment annual conference, Vanguard's ]ol1n Bogle said that return of a specific stock, bond, or other tlle inte11t of the manager is the important distinc market index or performance benchmark. There tion. Modification could be acceptable, he

8 THE]OURNAL OF INVESTMENT CONSULTING © 1999 Investment Management Consultants Association Inc. Reprint with permission only. JOURNAL OF INVESTMENT CONSULTING explained, if the intent is to mirror the index. But ties) should have higher tracking error over once a manager taints the intent with any active short periods, but this should revert to zero over decision other th.an n1irroring, then the fund is no longer periods. longer passively managed. According to Bogle, a Since non-mean reverting tracking error is manager who tries to track a 1000 stock index by largely associated with fees and other costs, all of sampling with 300 stocks is still indexil1g. Anot11er the higher fee retail fU11ds tend to have negative manager who holds 497 stocks of the S&P 500 tracking error. Trading costs within the funds would index and substitutes the other thTee with tobacco lead to non-mean reverting tracking error. Always stocks in the hope that they will increase in value is maintaining a cash position creates long-term nega actively il1vesting, not indexing. tive tracking error, but the error could be positive over shorter periods. What Is Tracking Error? A11drew England discussed the causes of track lthough tracking error is not a new con ing error in an article on a United Kingdom website: cept, it is still not widely understood by One-third of index tracker funds fail to deliver ade many buyers of index funds. "Trackil1g quate performance according to Reuters error" can be defined as the percentage Hindsights performance differe11ce il1 total return between an index fund n1easurement service. This AmOUGH a11d the benchmarking index the fund was designed is based on the consisten to replicate. Even if a self-styled index manager out cy and persistency of pas TRACKING ERROR performs the deSignated benchmark, the difference sive funds in producing in performance is tracking error. returns that match their In the literature and among investment profes benchmarks. Factors that IS NOTA NEW sionals, the concept is sometimes addressed using can create performance differe11t terms. At the IMN-Fabozzi Superbowl of inefficiencies in trackers CONCEPT, IT IS Indexing conference in December 1998, Charles include the methods used Galligan, who writes the Bear Stearns I11dex by managers, dividends, STILL NOT WIDELY Forecast, referred more precisely to "index tracking taxes, and the size of funds. The approach used error." At the April 1999 IRR Global Indexing UNDERSTOOD BY by a passive fund (whether Summit in Bermuda, John Goodwin of Northern sampling, full replication, MANY OF Trust Quantitative Advisors broke tl1e concept or derivative style) and the BUYERS down to mean reverting and non-mean reverting changing composition of tracking error. Mean reverting can be positive or indices can also affect per INDEX FUNDS. negative, usually reverting back to zero over longer formance. The 111easure periods. Non-mean reverting is always negative, is ment of a tracker is based never recovered, and is best exemplified by fees and on the certainty of performance returns rather than transaction costs. the probability of performance. With an actively In this article, tracking error is defined as the managed one, you attempt to distinguish whether perce11tage difference in total return. between an the performance is due to skill or luck. With an index fund and its benchmark index. index tracker, researchers are trying to judge if a tracking error is due to the investment process or to random factors. There are two elements to this. The The Causes of Tracking Error first being whether the actual returns are in line with oodwin's mean reverting/non-mean benchmarks. The second is calculating the amount reverting categories help us examine the of tracking error, and any relative volatility is lon.g-term effects of tracking error. Mean deemed as negative. 2 reverting tracking errors are primarily Vanguard's website also discusses this issue the result of not holding the exact weighting of each under Myths on IndeXing: security in the index. It is assumed that iI1 the long Some fund sponsors are offering index funds with run all securities in an index will have performances cut-rate expense ratios in an effort to market the similar to the index and revert to the mean. A port "lowest cost index funds." Be wary of this practice for two reasons: folio sampling strategy (not holding all the securi

First, such funds are likely to raise their expense Galligan suggested that there could be 5 to 13 basis ratios-without warning-after a period of time. points inherent in managing an index fund: 3 to 5 Second, the performance of an index fund is from commissions, 2 to 3 from dividends, up to 3 affected not only by its expense ratio, but by from cash flows, and up to 2 basis points from factors such as brokerage costs and fund manage changes in the benchmarked index. ment practices. John Rerlkthaler of Mornin.gstar has years of Thus, those funds without adept management experience researching index funds. He has sug may have a higher tracking error relative to their tar gested some rough rules of thun1b for evaluation: get benchmark. S&P 500, ±50 basis points (including expenses); REIT or MidCap, 100 to 150 basis points; and Different Standards for Different Funds Russell 2000 or another sn1all cap index, 200 he tolerance for tracking error depends on basis points. many factors, but the most important is The reasons for these different standards are the index to be tracked and the size of the easily explained. How expensive is it to purchase fund. The S&P 500 has by far the most the securities contained in the benchmarked index? funds indexed to it. Most investment professionals, What about a manager's experience? Th.ere is a lot including Bogle, agree that funds indexed to it more historical experience in. rUl1.ning an S&P 500 should be held to higher standards with respect index fund than in running a smaller cap or to tracking error. Gus Sauter of Vanguard has this REIT fund. to say: Formulating an index involves a l1umber of fac Large cap indexes such as the S&P 500 contain very tors, including the number of different issues in the liquid securities and are almost always replicated by index and how they should be weighted. commercially offered funds. In these funds tracking Some indexes only can be replicated through error is closely scrutinized and almost every basis optimization or other sampling techniques. What point must be explained. It is cost effective to repli about the size of the fund? Oth.er considerations cate a new fund with $5 million. As the fund grows, include how incon1e is reinvested in the index though, the sampling process can be replaced by and how often securities comprising the index 3 complete replication if appropriate. are ch.anged. JOhIl Loftus points Ollt th.at "most successful William Bernstein cited the most well-known managers of large index funds have exhibited his cases of positive tracking error with respect to small torical tracking error of 20 basis points or lower." cap index funds: He thinks future performance could achieve a zero Gus Sauter and Rex Sinquefield have created funds tracking error. "1 which are beating their benchmarks. Even after During his Superbowl of Indexing talk, expenses. Like clockwork. Spookier still they are

Table 1 RETURNS BY QUARTER IN 0/0 3rd qtr 4th qtr 1st qtr 2nd qtr Total TrkEr 96 96 97 97

S&P 3.10 8.30 2.69 17.50 34.727 Mgr 1 3.04 8.36 2.72 17.45 34.705 -0.022 Mgr 2 3.08 8.37 2.88 17.51 34.786 -0.069 Mgr 3 3.06 8.40 2.62 17.46 34.661 -0.066 Mgr4 3.10 8.34 2.68 17.47 34.729 -0.002 Mgr 5 3.07 8.37 2.67 17.48 34.725 -0.001 Mgr 6 3.20 8.38 2.59 17.44 34.750 -0.030 Source: Pensions & Investments PIPER database

doing it with index funds! This is attributed to a As Bogle would ask, "what was the intent of "patient buyer" discount block-trading strategy This these exclusions?" The exclusion of 175 stocks was strategy has resulted in negative total trading costs, the result of the impiemelltation of an optimization despite the poor liquidity of small company stocks. 5 program. S&P has long been aware that some index An informed observer might say that these funds invariably use optin1ization or other sampling managers could be providing the liquidity: Who are techniques. S&P has reminded the industry that the market makers anyway? Who is getting the such strategies have the pote11tial to introduce trading profit? tracking error. The Kentucky fund was large enough for full replication and it was recommended that Higher Standards for the Biggest it do so. S&P 500 Funds Problems in performance measurement can also lead to tracking error. Olle conclusion is that using hould S&P 500 index funds with assets over a cash rather than an accrual method of measure $1 billion be lleld to lower tracking error ment can lead to tracking error. (See the author's spreads (higher performance standards) article in the Winter 1999 issue of The Journal of tllan other index funds? An inforn1al sllrvey Portfolio Measurement [vol. 3, no. 20] for a discus of large public plan managers revealed a consensus sion of these issues.) standard for S&P 500 tracking error of less than 10 basis points. Is this an unreasonable standard? Conclusion Table 1, wllich was constructed from Pensions and Investments 1997 PIPER database of major insti he tracking error for a fund such as the tlltional index managers, shows tracking errors Kentucky $3 billioll S&P sao index fund ra11ging from zero to seve11 basis points. Dictating a should have been no more than five basis tracking error of less than 10 basis points for large POilltS per annum. Evell large institutions S&P 500 index funds does not seem to be an unrea and plans shouldn't manage an S&P 500 fund inter sonable standard. nally unless their managers can meet the four-point target. External funds should understand that reten tion of a state account requires this same standard Finding the Causes of of indexing performance. Tracking Error As the dollars under index management grow ne large public plan fired an S&P 500 and institutional investors gain more experience index manager because of an allnual and sophistication, sponsors will expect and tracking error of 13 basis points. The demand tighter trackillg to any index, not just the plan then switched to an internally large cap benchmarks. Consultants need to stay managed S&P 500 index fund with the objective of abreast of this particular aspect of the passive man lower costs and subsequent lower tracking error. agement debate. Such an objective may not be easily met. A review Endnotes of one $3 billion inter11ally n1anaged S&P sao 1. William F Sharp, "The Arithmetic of Active index fUlld in the Kentucky state retirement system Management," Financial Analysts Journal 47, no. 1 revealed a 30 basis point tracking error for the (1991): 7-9. entire 1997 fiscal year and a 70 basis POillt tracking 2. Andrew England, "Trackers Fail to Deliver," error in the fourth quarter of 1996. Online Investment Week, www.invweek.co.uklinvweekl In determining the cause of the tracking error 3. Gus Sauter, "Medium and Small Capitalization in tllis $3 billion internal fund, it was fOllnd that the Indexing," Professional Perspectives on IndeXing, ed. Frank]. Fabozzi (New Hope, Pa.: Fabozzi Associates, fund had excluded internationally based stocks, 1997). illcluding Royal Dutch Sllell, Unilever, Seagram 4. John Loftus, "Enhanced EqUity Indexing," Corporation, and Northern Telecom. These four Professional Perspectives on Indexing, ed. Frank J. Fabozzi stocks represented over $87 million in allocated (New Hope, Pa.: Fabozzi Associates, 1997). nlarket cap. Also excluded were stocks with low S. William Bernstein, "How to Beat the Value Line safety ratings. TIle fund oWlled orily 325 Benchmarking Indexing's Ultin1ate Irony;" www.effi issues, not 500. cientfrontier.comlef/998/indexfun