The Ratings Game Evaluating the three groups that rate the charities By Stephanie Lowell, Brian Trelstad, & Bill Meehan Stanford Social Innovation Review Summer 2005 Copyright © 2005 by Leland Stanford Jr. University All Rights Reserved DO NOT COPY Stanford Social Innovation Review 518 Memorial Way, Stanford, CA 94305-5015 Ph: 650-725-5399. Fax: 650-723-0516 Email: [email protected], www.ssireview.com 38 STANFORD SOCIAL INNOVATION REVIEW by STEPHANIE LOWELL, BRIAN TRELSTAD, & BILL MEEHAN the ratings game The tsunami that struck South Asia in ited scale of its operations in the affected December 2004 will be remembered not areas. Another group, Direct Relief Inter- only for the scale of the human misery it national, assured donors it was depositing caused, killing hundreds of thousands and its flood of donations into a separate bank displacing millions, but also for the account, and that the salaries of its unprecedented global outpouring of char- employees would not be ity it evoked. Within a few weeks of the paid out of these donations, disaster, over $400 million (on the way to as part of its effort to maxi- Evaluating an estimated total of $1 billion) had been mize the amount that would raised by U.S. aid organizations alone; fur- reach the victims. the thermore, a large proportion of those If all of this heralds a new donations was made via the Internet. age in philanthropy, where three groups “The response has been unprece- the Internet will be a domi- dented,” says Mike Kiernan of Save the nant force in charity, bringing that rate Children USA, “greater than any other dis- a new sense of accountabil- aster or crisis in (our) more than 70 years ity and transparency to the the of operation.” By April, roughly 20 percent process, there are three of the $63 million Save the Children USA online services already in charities had collected for tsunami victims had come place that stand to benefit. in through its Web site – a 100-fold increase These three have built Web- from pre-tsunami levels. Other groups based charity rating services, available to reported a similar shift in giving patterns. give prospective donors information and In response, some of the charities ben- guidance about the groups they wish to efiting from this surge in donations started consider for support. They are the BBB behaving in new ways, too. For example, Wise Giving Alliance, which uses a set of the U.S. branch of Doctors Without Bor- 20 standards to monitor the operations ders announced a week after the disaster and financial stability of national chari- that it had already raised as much money ties, and uses a “pass-fail” system of grad- ILLUSTRATION BY JOYCE HESSELBERTH/IMAGES.COM ILLUSTRATION as it could responsibly use, given the lim- ing; Charity Navigator, which rates non- www.ssireview.com ~ FOR PERSONAL USE ONLY ~ DO NOT DISTRIBUTE ~ STANFORD SOCIAL INNOVATION REVIEW39 profits on a set of organizational efficiency and organizational capac- ity metrics with a “star-based” sys- tem of one to four stars, using algo- rithms derived from publicly reported financial data; and the American Institute of Philanthropy (AIP), which rates nonprofits with grades of “A+” through “F” using financial ratios and analysis of char- ities’ financial statements, including their 990s and audited financials.”1 Not surprisingly, all three of these services saw their user traffic grow exponentially in the wake of the tsunami. At Charity Navigator, for example, traffic grew tenfold, from an average of 5,000 unique vis- itors a day to over 50,000 during the week following the tragedy. Over the past few years, each of these ratings sites has sought to establish itself as the authority for Sri Lanka’s eastern coast was devastated by last December’s tsunami. The disaster led to an donors seeking information to unprecedented outpouring of charity all over the world. guide their giving decisions. And their influence is starting to be felt, as many nonprofits now gram effectiveness; and they generally do a poor job of conducting proudly tout their high ratings from these organizations on their analysis in important qualitative areas such as management Web sites (“Save the Children awarded 4-star rating from strength, governance quality, or organizational transparency. Charity Navigator”), and portals such as Earthlink direct users To be fair, these are early days for the ratings business; all to the “top-rated charities” identified by the ratings agencies. of the sites are less than six years old2 and each is still working We conducted a detailed study of the agencies to determine on improving its methodology, growing its user base, and devel- how useful a service they provide. The results were sobering: oping a sustainable business model for its services. Our review of their methodologies indicates that these sites indi- But as traffic to these rating sites grows, and donors make vidually and collectively fall well short of providing meaning- important decisions using potentially misleading data and analy- ful guidance for donors who want to support more efficient and sis, the agencies’ potential to do harm may outweigh their abil- effective nonprofits. ity to inform. In this article, we review some of the strengths Based on our study, the major weaknesses of the ratings agen- and weaknesses of the ratings agencies and consider how to build cies are threefold: They rely too heavily on simple analysis and a more effective and transparent system for nonprofit ratings ratios derived from poor-quality financial data; they overem- and evaluation. phasize financial efficiency while ignoring the question of pro- Rating the Raters PHOTOGRAPH BY BRUNO STEVENS / COSMOS AURORA STEPHANIE LOWELL is an independent nonprofit consultant and an To assess how well the ratings agencies do their job, we put them alumnus of McKinsey & Company’s Boston office. She can be reached at to a two-part test. First, we put ourselves in the position of a [email protected]. hypothetical donor for tsunami relief. How helpful would these BRIAN TRELSTAD is the CFO of Acumen Fund and an alumnus of agencies have been? Second, we conducted a more thorough McKinsey’s New Jersey office. He can be reached at btrelstad@acumen- review of each of the rating agencies’ services, tried to under- fund.org. stand their evaluation methodology, and interviewed their lead- WILLIAM F. MEEHAN III is a director of McKinsey & Company, ers in an effort to understand their potential for guiding effec- senior lecturer in strategic management and Class of 1978 Lecturer (2004- tive donor decision making. 2005) at Stanford University Graduate School of Business. He is also the First, we tested how each rating agency would rank seven chairman of Philanthropic Research Inc., the parent of GuideStar. He of the 10 largest recipients of tsunami aid (see chart, p. 41).3 All can be reached at [email protected]. seven got a “pass” from Wise Giving, three or four stars from 40STANFORD SOCIAL INNOVATION REVIEW ~ FOR PERSONAL USE ONLY ~ DO NOT DISTRIBUTE ~ www.ssireview.com RATINGS FOR TOP RECIPIENTS OF TSUNAMI AID IN THE UNITED STATES Wise Giving Alliance Charity Charity Navigator Forbes* Watch American Red Cross Meets Standards A- 4 stars (64.11) 1) 91% 2) 80% 3) 166% ($236m) US Fund for UNICEF Meets Standards C-** 4 stars (61.29) 1) 89% 2) 91% 3) 103% ($68m) Save the Children USA Meets Standards A 4 stars (61.59) 1) 91% 2) 87% 3) 92% ($41m) World Vision USA Meets Standards B+ 4 stars (61.02) 1) 81% 2) 87% 3) 92% ($29m) CARE USA Meets Standards A- 4 stars (66.73) 1) 92% 2) 89% 3) 85% ($25m) Doctors Without Meets Standards A 3 stars (54.80) 1) 86% 2) 89% 3) 92% Borders ($20m) AmeriCares Meets Standards A 4 stars (62.40) 1) 99% 2) 99% 3) 94% ($20m) *Forbes annually ranks charities using three ratios: 1) Charitable services as percent of total expenses; 2) Percent of private support remaining after fundraising expenses; and 3) Percent of private support remaining after surplus. **Charity Watch discounts in-kind revenues in its calculations. SOURCE: WEB SITES Charity Navigator, and with the exception of the US Fund for bers review nonprofits in the areas of financial efficiency and UNICEF, a B+ or higher from AIP. With the exception of the stability, governance and oversight, performance measurement, US Fund for UNICEF, whose poor grade isn’t explained on the and the quality and accuracy of the organization’s fundraising AIP Web site, does the analysis really help you make an affir- and informational material. “Finances alone are only a piece of mative decision? Is there enough information to distinguish the picture, and in fact can give you a ‘false positive’ on the health one of these charities from the rest? While it is certainly reas- of the organization,” notes Bennett Weiner, Wise Giving’s chief suring to know that contributing to any one of these seems to operating officer. be a reasonable choice (with the debatable exception of Wise Giving’s 20 evaluation standards were developed over UNICEF), the ratings sites collectively fail the test to actually three years with input from hundreds of donors, nonprofit inform positive donor choice about allocating scarce capital leaders, government regulators, and academics. The agency among competing options. interacts with the nonprofits they review both by contacting For our more in-depth review, we analyzed each agency’s them to discuss issues or concerns, and then follows up by finances and their Web sites. We then interviewed the senior lead- posting an implementation guide on its Web site to ensure ership at each organization to understand their methodology.