Unpacking Charity Navigator's Rating System

Home , Charity (practice), Charity Navigator, Intelligent Giving, Nonprofit organization

White Paper #3 Evaluating the Evaluator: Unpacking

Charity Navigator’s Rating System

Summary Charity Navigator (CN) operates a system for evaluating U.S. public charities, with the purpose of helping donors make informed decisions. In this paper we summarize the organization’s rating process and dissect its evaluation methodology. We find that while CN offers a needed standard for nonprofit accountability and has potential to fulfill its mission in the future, its current system has substantial deficiencies that can misinform donors and harm nonprofits. Major problems include the composition of the Financial Health Rating, which is not a valid gauge of nonprofits’ financial health. We offer recommendations to donors on alternative research methods, to nonprofits and management consultants on presenting informative data to the public, and to CN on constructive improvements to its system.

CN issued revisions to its methodology in June, 2016, subsequent to the first edition of this paper. We address these changes in an addendum on page 14 here. The revisions do not substantively alter this paper’s findings or recommendations.

pimgconsulting.com | (206) 282-7464

Introduction

How can we determine if a public charity is effective and responsible in carrying out its work?

This is an important question. There are over one million public charities in the U.S.1 These organizations are authorized by the federal government, serve a wide range of functions in our society and account for the bulk of the nonprofit sector, which constitutes over 5% of the U.S. economy.2 Taxpayers subsidize charitable organizations in two ways: public charities are generally exempt from federal taxation and many receive charitable contributions, which are often tax-deductible for donors. The public has a clear interest in ensuring that nonprofits

fulfill their purposes.

Charity Navigator (CN) is a nonprofit organization that describes itself as “America’s leading independent charity evaluator.”3 Founded in 2001, the organization’s mission statement is: “Charity Navigator works to guide intelligent giving. By guiding intelligent giving, we aim to advance a more efficient and responsive philanthropic marketplace, in which givers and the charities they support work in tandem to overcome our nation’s and the world’s most persis- tent challenges.”4 CN uses a set of calculations to rate nonprofits on several criteria, culmina-

ting with familiar “star” ratings on aspects it has identified as keys to donor understanding.

To the extent that individuals and institutions use CN’s ratings to influence their charitable 6 donation decisions, the organization’s methods can influence resource distribution. As donors get increasingly savvy about available research tools, more nonprofit leaders are concerned about their CN ratings and the perceptions these ratings may fuel among potential supporters. It is therefore worthwhile to examine CN’s methodology and inquire about its validity in addressing its objectives.

In this paper, we evaluate the evaluator. We examine CN’s rating system and assess its strengths, weaknesses and overall relevance to the cause of promoting an effective nonprofit sector.

On pages 2-3, we establish context for CN’s work with brief groundwork on performance measurement. On pages 4-6 we summarize CN’s evaluation system. We then present detailed analysis of the system’s strengths and weaknesses, on pages 7-11. Finally, on 2016 © GROUP MANAGEMENT INTEREST

pages 12-13 we offer recommendations on actions CN and its users can take to improve

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC data and communication in the nonprofit sector, looking forward. PUBLIC

1 National Center for Charitable Statistics, http://nccs.urban.org/statistics/quickfacts.cfm 1 2 Independent Sector, http://www.independentsector.org/economic_role, 2014 3 We use “CN” for brevity; Charity Navigator does not generally use this acronym in its communications. 4 Charity Navigator, http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=17

Performance Metrics from a Donor’s Perspective

In recent decades, the use of “measurable outcomes” has become a pronounced trend in the nonprofit sector, driven largely by funders interested in seeing the results of their support. To understand where to best direct funds, a donor may logically want to know both whether a nonprofit is producing good outcomes and how well it is run.

Once an individual or institutional donor has established interest in a nonprofit’s work, there are three basic questions they should answer affirmatively before writing a check. Building on

our opening query on page one,

1. Is the organization effective in conducting its work? 2. Is it managed with integrity and competence? 3. Should I direct money to this particular entity, given other alternatives?

Outcome measurement in the nonprofit sector has traditionally focused on program evaluation, assessment of organizations’ mission-related work. Program evaluation methods have varied widely, without clear standards in many fields of nonprofit work. The difficulty of measuring mission impact is one complicating factor for many nonprofits. The availability of funding for evaluation of short- and long-term outcomes can be another obstacle. Thus, answering

Question 1 is not always straightforward.

Management practices of nonprofits can also vary widely from one organization to another. ngCharity Navigator's Rating System Donors support specific organizations, not causes or social needs in the abstract, so understanding management performance is crucial. This endeavor brings its own challenges. In contrast to businesses, which are typically judged on their profitability relative to competitors, nonprofits exist to deliver societal value, and face a higher level of public accountability. While financial results are important, profitability is inadequate as a catch-all metric for nonprofit management strength, and answering Question 2 is also a nuanced process.

Addressing the first two questions requires standardized information on organizational performance and clear methodology to produce this data. Question 3 is comparative, and may

valuatingthe Evaluator:Unpacki Statisticians and researchers have tackled many similarly challenging tasks.

E PUBLIC PUBLIC Consider, for example, the medical problem of determining, “Is this person healthy?” To investigate, health professionals will take various measurements. Some metrics, such as age, height and weight, are descriptive, and give context to an evaluator. Many metrics, such as cholesterol or insulin levels in a patient’s blood, are tests for specific problems. Others, such as blood pressure, heart rate and temperature, are general indicators of either the presence or absence of any of a wide range of potential concerns. In statistics, the latter set of metrics are proxy variables – they’re not especially interesting in and of themselves, but are assumed to correlate highly with other characteristics that are important. For example, if the victim of an accident has normal and stable vital signs and shows no clear indications of serious injury, 2 she may be sent home by medics rather than hospitalized. Proxy variables are employed widely in evaluation—we use data that are readily available (e.g. vital signs), under the presumption that they will, in essence, tell us what we really want

to know (i.e. the presence or absence of an injury). The potentially messy problem of assessing the health of a person or a nonprofit organization requires the use of proxies. There is always risk in using proxies; the patient in our example above may have a significant problem that was not detected by measurements at the scene of the accident. And so performance measurement becomes an inexact science.

But it is science nonetheless—that is, it must be if we want a systematic way to answer a donor’s basic questions.

Performance measurement has several potential pitfalls:

 Miscasting the problem

If an evaluator asks the wrong fundamental questions or designs the task in ways that don’t truly address the right questions, evaluation will fail, regardless how well it is executed.

 Choosing the wrong metrics

Even with sound design, the mechanics of evaluation can go awry if the chosen metrics are

not accurate proxies of the information we really want to know.

 Making the task overly simple or complex 6

An overly simple measurement system can miss important nuances. A complex system can become burdensome to operate, and may produce unintended consequences. Sound evaluation design resides in between—it should be agile to operate and straightforward to interpret, yet nuanced enough to provide depth of understanding.

 Using inaccurate, incomplete or inconsistent data

Even well-conceived evaluation systems will be derailed by unreliable data. The data used to analyze performance should be unbiased, collected through a clear and consistent process, and directly comparable within the studied population.

In the context of multiple challenges and potential pitfalls, Charity Navigator has undertaken

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC the task of devising a holistic system for understanding and comparing nonprofits’ perfor- PUBLIC mance. CN was founded to fill a gap. The IRS establishes charitable tax-exempt status of nonprofits early in their histories, but the agency conducts only basic compliance activity after approval, and does not actively evaluate organizations’ performance. No definitive system for addressing key donor questions existed before CN, and donors have historically relied on their own assessments and/or information provided by nonprofits themselves. In concept, CN has thus offered charitable donors a new tool for decision-making.

We will now explore CN’s evaluation methodology.

Overview of the Charity Navigator Evaluation System

Charity Navigator evaluates U.S. public charities (501(c)(3) nonprofits) on two broad categories: “financial health” and “accountability and transparency.” A third category, “results and impact,” is in development, and not currently in use. The organization asserts that its ratings “show givers how efficiently we believe a charity will use their support today, how well it has sustained its programs and services over time and [its] level of commitment to being account- able and transparent.”5

CN has strict limitations on which organizations it evaluates. Primary criteria include budget size (total annual revenue must be greater than $1 million), longevity of reporting (at least seven years of IRS tax returns must be available), and sources of revenue (organizations receiving a majority of funding from government sources or program fees are excluded). CN evaluates over 8,000 nonprofits, a small but influential segment of the U.S. total.

Nonprofits that qualify for evaluation are rated on a series of criteria under each of the two broad categories. These ratings are then combined into a single index under each of the two categories, with scores converted to numbers up to 100. A rated organization may receive, for example, a financial health score of 84 and an accountability and transparency score of

92. These scores are accompanied by ratings of one to four “stars”; in this example, the rated

organization would receive three stars for financial health, and four stars for accountability and transparency. ngCharity Navigator's Rating System Financial Health Rating

To assess a nonprofit’s financial health, CN pulls seven data elements from an organization’s annual tax returns (IRS form 990):6

1. Program expenses (as a % of total operating expenses) 2. Administrative expenses (as a % of total operating expenses)* 3. Fundraising expenses (as a % of total operating expenses)* 4. Fundraising efficiency (amount spent as a % of amount raised)*

valuatingthe Evaluator:Unpacki 7. Working capital ratio (net available assets as a % of annual operating expenses).

E PUBLIC PUBLIC

CN considers the first four metrics to be measures of an organization’s “financial efficiency.” Metrics 5 through 7 are identified as “financial capacity” measurements, relating to an organization’s ability to sustain its services. The directional preference established by CN (in other words, the end of the spectrum for which CN rewards the rated nonprofit with a higher score) is that a higher value is generally better, except for those metrics (numbers 2-4) marked with an asterisk (*) above, in which cases a lower value is better. For example, a higher percentage of program expenses is rewarded with a higher metric value, while a lower percentage of fundraising expenses is rewarded by a higher value.

4 5 CN rating overview, http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=1284#.Vs9p8Xn2apo 6 CN Financial Health Rating, http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=35#.Vs9qHXn2apo

For each of these metrics, CN converts the data value to a score, based either on a mathe- matical adjustment or CN-defined bands along a range of possible values. There are also a variety of adjustments that CN may make to values, based on the type of work the nonprofit performs—CN classifies nonprofits using 11 service categories and two to four “causes,” or service subgroups, within each category.7 Each evaluated nonprofit receives a zero to 10 rating on each of the seven metrics, these ratings are added together, and a flat adjustment of 30 is added to this cumulative number, creating a Financial Health Rating value between 30 and 100.8

Accountability and Transparency Rating

To assess a nonprofit’s accountability (“an obligation or willingness by a charity to explain its actions to its stakeholders”) and transparency (“an obligation or willingness by a charity to publish and make available critical data about the organization”), CN gauges a total of 17 metrics from an organization’s tax return and website.9

From tax returns:

1. Independent board (at least five independent members, constituting a majority)

2. Material diversion of assets*

3. Independently audited financial statements 6 4. Loans to or from related parties* 5. Documented board meeting minutes 6. Provided 990 to board in advance of filing 7. Conflict of interest policy 8. Whistleblower policy 9. Records retention and destruction policy 10. CEO listed with salary 11. Process for determining CEO’s compensation 12. Board listed/members not compensated

From organizations’ websites:

13. Board members listed 2016 © GROUP MANAGEMENT INTEREST

14. Key staff listed

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC 15. Audited financials available PUBLIC 16. Form 990 available 17. Privacy policy stated.

7 CN classification system, http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=34#.VtZdGXn2apo 8 CN Financial Rating Tables, 5 http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=48#.Vs9qfnn2apo 9 CN Accountability and Transparency Rating, http://www.charitynavigator.org/ index.cfm?bay=content.view&cpid=1093#.Vs9rQHn2apo; Accountability and Transparency Rating tables, http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=1283#.Vs9sHXn2apo

The directional preference established by CN for the metrics above is that the existence of the characteristic, policy or practice is desirable, except for those indicated with an asterisk (*), where the stated practice is undesirable.

The scoring methodology here is as follows: Each rated organization starts with 100 points, and varying quantities of points are deducted for undesirable attributes in each of the 17 categories. As with financial health, an organization has a maximum Accountability and Transparency Rating value of 100. A nonprofit would receive a perfect 100 rating if all criteria are satisfied. The minimum score in this category is negative 16, as the maximum number of

point deductions is 116.

Results and Impact Assessment

On its website, CN states that the organization is developing a methodology to assess nonprofits’ mission-related results, as part of a multi-year process. Comparative ratings are not yet available for individual nonprofits, and the nature of the end product is, at this stage, unknown. CN published a Results Reporting Concept Note in January 2013, laying out an approach to evaluating organizations; criteria include the existence of a logic framework, specific outcome measures, third-party validators and a process for constituent feedback.10 A results and impact indicator may, in the future, constitute a third dimension of organiza-

tional evaluation data available to CN constituents.

ngCharity Navigator's Rating System

valuatingthe Evaluator:Unpacki

E PUBLIC PUBLIC

10 Charity Navigator Results Reporting Concept Note, http://www.charitynavigator.org/__asset__/_etc_/ CN_Results_Reporting_Concept_Note.pdf

Strengths and Weaknesses of the Charity Navigator Methodology

Charity Navigator’s evaluation system brings some valuable information on nonprofits to the public. The CN methodology, however, has several critical flaws and gaps. In this section we summarize strengths and weaknesses of the current methodology.

 CN’s categories of evaluation are well chosen to address basic donor needs.

CN has identified three categories of measurement that logically address the needs of donors. Financial health addresses the issue of competence. Accountability and, to lesser

extent, transparency address the issue of integrity. Results and impact can address the issue of quality and social value of work. Donors may, in principle, use this information alongside their own priorities and values to make educated choices. The CN evaluation system’s framework thus rests on a conceptually strong foundation.

 The Accountability and Transparency Rating offers useful insights and establishes a valuable standard.

CN’s accountability and transparency metrics establish a reasonable and needed standard

for public charities to meet. These are key elements of effective nonprofit management, given

the public roles and responsibilities of tax-exempt organizations. The website content metrics 6 could be criticized for being overly prescriptive, given the many functions a website may play for a nonprofit; other information, such as clear detail on services and evidence of program effectiveness, may be considered to be of greater value to a prospective donor and others users of a website. A uniform accountability standard is a nonetheless a valuable innovation for both management practice and donor communication.

 The Financial Health Rating does not actually gauge nonprofits’ financial health.

CN, in effect, defines financial health as a fusion of financial “efficiency” and “capacity.” Finan- cial efficiency refers to the proportion of funds directed to mission-related activity. Capacity, as defined by CN, is a hybrid of demonstrated growth and liquidity.

Financial efficiency may, in principle, be an interesting characteristic to assess for nonprofits 2016 © GROUP MANAGEMENT INTEREST

(we address the specific metrics below), but it is not an indicator of an organization’s financial

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC health. Nor does it necessarily correlate to results, or even operational efficiency. In fact, a PUBLIC growing body of literature suggests that so-called measures of financial efficiency not only measure relatively little of value, they reinforce a widely held misconception that can harm nonprofits. (See discussion of the “Overhead Myth” below.) Organizations that strive for high financial efficiency may be underinvesting in needed infrastructure—that is, administrative staff, operational systems and technology—which, in turn, can limit quality of service, support for staff in conducting their work and scalability. These constraints work counter to long-term financial health.

Further, nonprofits that achieve high fundraising efficiency (more precisely termed return on investment) may be reaping only “low-hanging fruit,” and may be underperforming in addres- 7 sing their missions as a result. While efficiency is clearly better than inefficiency, the over- arching goal of charitable fundraising is to maximize mission-related results. The pursuit of fundraising efficiency as an objective in and of itself can compromise the larger goal.

As CN defines it, financial capacity can also be tangential to an organization’s financial health. Two of the three metrics CN associates with financial capacity relate to growth, specifically expansion of primary revenues (which include most operating income sources for nonprofits) and program expenses. While building scale of an organization’s programs and services may be desirable from a mission standpoint (such as when a nonprofit is addressing a large, unmet community need), neither financial growth metric implies that the organization is on fiscally sound footing. The problem is that organizations can grow financially without strengthening their positions.

Consider this example, which is illustrative of a common experience in the sector: Middletown Social Services (MSS) obtained a large, restricted multi-year grant that did not cover the full costs of delivering services. MSS experienced rapid growth, while also expanding its need for unrestricted funds, which are hard to obtain. Though the organization grew, its infrastructure remained modest, and did not adequately support a larger operation. This infrastructure gap compromised MSS’s ability to operate effectively and raise more money. The organization has grown, while digging itself into an operational and financial hole. In this example, growth can correlate to a decline in financial health and sustainability. This situation is common enough that there’s a name for this type of development, which can choke and even kill

organizations: the “Nonprofit Starvation Cycle.”11

A second example illustrates the flawed construct of using financial growth metrics in deter- mination of financial health: Eastside Animal Welfare (EAW) broke even in 2013. EAW’s

primary revenues increased by 10% per year in 2014 and 2015. The same organization’s ngCharity Navigator's Rating System program, administrative and fundraising expenses increased by 15% in both 2014 and 2015. This nonprofit has had two consecutive large deficits and may be heading toward insolvency— yet it has achieved the maximum score of 10 in both CN growth metrics.

A nonprofit’s financial health, much like that of a for-profit business, is determined by its profitability (the ability to generate operating surpluses), balance sheet strength and future revenue outlook.12 Only one of CN’s financial metrics—working capital ratio—actually measures an element of financial health. The other six proxy variables are tangential to their stated purpose. Since working capital ratio is buried among extraneous financial measures in INTEREST MANAGEMENT GROUP © 2016 © GROUP MANAGEMENT INTEREST CN’s composite formula for financial health, its impact is watered down to just 14% of an

organization’s overall financial score. Further, CN excludes common measures of profitability,

valuatingthe Evaluator:Unpacki

E PUBLIC PUBLIC such as operating surplus and change in unrestricted net assets. The upshot is that CN’s Financial Health Rating tells us little about a nonprofit’s financial health.

 CN’s financial efficiency metrics have shortcomings, and their use reinforces the “Overhead Myth,” which harms nonprofits.

As noted above, the inclusion of “efficiency” and growth measurements within the Financial Health Rating constitutes a key flaw in the design of the composite financial health indicator.

8 11 Gregory, Ann G., and Howard, Don, “The Nonprofit Starvation Cycle,” Stanford Social Innovation Review, Fall 2009 12 Profitability and balance sheet data is available on 990 forms. Future revenue outlook is related to predictability and stability of funding, which vary by type and diversity of income sources; this data is not fully available through 990 forms, and would require a supplemental reporting mechanism.

There’s a second problem with CN’s chosen efficiency metrics: they are weak proxy variables for financial efficiency.

A nonprofit’s operating expenses are allocated among three categories in the 990 form: program services, fundraising and administration. No definition exists for these categories, however, so each organization must establish its own method for allocation. Auditors often assist with this process, but many gray areas exist; two otherwise similar organizations may have different expense allocation methods and, therefore, non-comparable figures. Further, it is possible to intentionally adopt allocation methods that favor program service expenses over fundraising and administrative expenses for the purpose of improving “optics,” without

violating any standard.

The concept behind metrics 1-4 (program expenses, administrative expenses, fundraising expenses and fundraising efficiency), is that it is good when a nonprofit spends as much money as possible on its program, and as little as necessary on anything else. This is exactly how CN’s formula treats these allocations, mathematically. This line of thinking, part of the “Overhead Myth,” has been widely and increasingly criticized in recent years.13 The basis for this criticism is the existence of a commonly held “false contention that financial ratios are a proxy for nonprofit performance.”14 Several national organizations prominently endorsed an

end to the practice of using such ratios in this way in 2013. These entities include the Better

Business Bureau, GuideStar and, notably, Charity Navigator. The idea expressed in their 6 joint endorsement of change is that the use of measures such as CN’s financial metrics 1-4 does not in fact measure efficiency in any meaningful sense, regardless of the accuracy of the allocation method.16

The apparent paradox of CN’s continued use of metrics it publicly disavowed several years ago is a separate issue, outside the scope of this paper.

 Assessment of results and impact is critical for donors, and a recognized gap in the current system.

One weakness of the present framework is the absence of data on results; a nonprofit’s work is the reason for its charitable status and the presumed motivation for donor involvement. As such, results data are ultimately what donors need most. They are also the hardest to obtain, 2016 © GROUP MANAGEMENT INTEREST

as CN has discovered. Not only can social mission results be difficult to measure (true

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC outcomes may take years and costly efforts to uncover), nonprofits often lack genuine data PUBLIC on outcomes (they often measure outputs, or measures of activity, instead). Further, the process of comparing two organizations offering different services is inherently challenging.

In order to address donor Question 1 above (Is the organization effective in conducting its work?), donors must currently rely on nonprofits’ self-reporting, media coverage and/or their

13 One recent example is, Knowlton, Claire, “Why Funding Overhead is Not the Real Issue: The Case to Cover Full Costs,” The Nonprofit Quarterly, Winter 2015. 14 Dan Palotta is one prominent writer/speaker in opposition to the Overhead Myth; see his TED talk here. 15 Overhead Myth website, http://overheadmyth.com/ 9 16 While we agree with the view that CN’s financial efficiency metrics are problematic, we do not believe that all financial ratios are tangential to nonprofit performance; true indicators of financial health, such as working capital ratio and profitability expressed as a percentage of operating expenses, offer value to donors and nonprofit leaders.

own intuition. CN’s high-level approach of assessing an organization’s logic model, defined outcomes and evidence of third-party confirmation establishes a sound standard, and a potential foundation for a comparative assessment of nonprofits’ results and impact. CN is on a promising path, but the system has limited value without this third dimension.

 Much of CN’s source data is reliable, with exceptions.

Most of the data CN uses to compute its ratings comes from nonprofits’ self-reporting in IRS tax returns. These returns are often prepared by independent auditors, and are approved by organizations’ leadership before submission. Form 990 data should agree with audited financial statements and meet generally accepted accounting procedures. In principle, this data should have high integrity, given the potential consequences to organizations and auditors of providing inaccurate information.

As noted above, the data used to calculate values for financial metrics 1-4 are not subject to a unified standard, and therefore cannot be assumed to be either accurate or comparable between organizations—and comparing nonprofits is a central purpose of CN’s rating system.

Other financial data and the full set of accountability and transparency data are presumed accurate, though transparency metrics derived from nonprofit websites (metrics 13-17) are

subject to being out of date quickly, given the frequency and ease of website changes.

 The methodology’s complexity addresses some challenges, but creates others.

ngCharity Navigator's Rating System CN blends various metrics into two composite indicators, with the purpose of simplifying the task donors face in researching nonprofits. This approach is appropriate—many familiar statistics (e.g. the Dow Jones Industrial Average, the Consumer Price Index and the “wind- chill factor”) are composite indices that serve their functions well. The process of calculating the composites requires (a) converting data into common units, (b) making meaningful distin- ctions between different data values, and (c) weighting the various ingredients to balance differing levels of importance to the result. By definition, the process requires value judge- ments, and indexes are thus not as objective as they may seem. This is the “sausage-

CN is transparent about most of the evaluation system’s mechanics on its website, including

valuatingthe Evaluator:Unpacki

E PUBLIC PUBLIC the measurement of component metrics and the basis for tiered rating levels. Some elements of the methodology’s complexity, such as the use of composite indicators and differentiated value ranges for scoring individual metrics, add clear value. Others do not, for example:

- The scoring for individual financial metrics has numerous special-case exceptions, based on the type of work nonprofits do (e.g., food banks, art museums, etc.). While these exceptions are research-based, they do not take into effect the scale of an organization’s operation, which may have a larger impact on its financial data than does a nonprofit’s field of work.17

17 Public Interest Management Group’s research suggests this is the case. A 2015 exploratory study of 40 nonprofits identified scale of operations as one of a number of characteristics correlating with nonprofit organizational success.

- Validity concerns aside, financial metrics 1-3 (program, fundraising and administration expense) are dependent on, and a function of, one other, because they sum to 100% of an organization’s expenses, by definition. These three metrics are therefore essentially a single data element—a nonprofit’s expense allocation. CN, in effect, assigns triple value (which constitutes 43% of the Financial Health Rating) to this variable.

- CN uses a high number of component metrics, arguably more than is necessary to establish financial health, accountability or transparency. The Financial Health Rating has seven ingredients, where two or three appropriate metrics could provide a clearer picture of financial health. The Accountability and Transparency Rating has 17 components. The

higher the number of component metrics, in general, the lower the impact individual metrics will have on the rating. We mentioned working capital ratio above, with an impact diffused to just 14% of the Financial Health Rating. A second example is the existence of a conflict of interest policy, which accounts for less than 3% of the Accountability and Transparency Rating, but could be a flag for a serious internal problem.

- The weighting schemes CN uses to blend component metrics into composite indicators are neither explained, nor clear in their logic.18 The seven financial metrics are equally weighted in value as components of the Financial Health Rating, meaning that they are

considered by CN to be of equal relevance to a nonprofit’s financial health; this is appar-

ently arbitrary, and does not reflect individual metrics’ disproportionate relationships to 6

fiscal strength. (This formula also fails to reflect the interdependence of three of the seven metrics mentioned above.) The accountability and transparency metrics have score adjustment values that vary between three and 15, with no apparent basis for differential magnitudes other than the subjective judgments of CN staff.

These complexities can compromise the value of the composite indicators and make it harder for nonprofit leaders to understand how their CN ratings correlate to their performance.

 Restrictions in the types and number of evaluated nonprofits currently limit CN’s value to donors.

One of CN’s selection criteria sharply constrains the number of nonprofits eligible for rating. INTEREST MANAGEMENT GROUP © 2016 © GROUP MANAGEMENT INTEREST By excluding organizations that receive a majority of their funding from government sources

or program fees, CN makes numerous nonprofits with large-scale fundraising operations

Evaluatingthe Evaluator:Unpacking Charity Navigator's Rating System PUBLIC INTEREST MANAGEMENT GROUP © 201 © GROUP MANAGEMENT INTEREST PUBLIC ineligible—particularly in the fields of health, social services and education. Given that earned PUBLIC income is an important component of many sustainable nonprofit business models, this restriction limits the CN system’s impact and value to donors.

11 18 PIMG queried Charity Navigator about its underlying logic behind metric weighting; CN staff responded that the equal financial metric weighting has been in place since CN’s inception, with changes possible in the future, and that accountability and transparency metric weights are determined by staff, without further elaboration.

Recommendations

Charity Navigator’s evaluation system is an ambitious undertaking that aims to provide charitable donors with a research tool to inform their decisions. CN’s system has added tangible value, primarily through its Accountability and Transparency Rating, but the system as a whole has clear limitations and needs for future development. Donors, nonprofits and other users of the CN system should be aware of these gaps.

The current system suffers from all four of the potential measurement pitfalls summarized above, to varying degrees: CN has miscast the problem of measuring financial health, and has incorporated several invalid metrics; its Financial Health Rating is designed, essentially, as a collage of financial efficiency and growth, which often do not correlate with financial strength. CN’s composite indicators may be overly complex, and some data the system draws on is inconsistent, and thus not comparable, among evaluated nonprofits. The current lack of data on the results and impact of nonprofits’ work is a major gap that CN has acknow- ledged. As a result, CN’s evaluation process does not address donor Question 1 (Is the organization effective in conducting its work?), only partially addresses Question 2 (Is it managed with integrity and competence?), and cannot reliably address Question 3 (Should

I direct money to this particular entity, given other alternatives?).

CN is reviewing its methodology, with a planned future iteration it calls “CN 3.0,” which the organization anticipates will incorporate a process for rating organizations on impact and

results. In addition, CN is reviewing its existing evaluation methods, with no specific date ngCharity Navigator's Rating System identified for revision. These processes may result in substantive changes, and we will publish an addendum to this paper to reflect updates.

At this point in time, we offer several specific recommendations to those impacted by CN’s system, based on the analysis presented herein.

For individual and institutional donors:

 Utilize CN’s Accountability and Transparency Ratings, which offer valuable

valuatingthe Evaluator:Unpacki  Avoid using CN’s current Financial Health Ratings. As a substitute, examine

E PUBLIC PUBLIC nonprofits’ past three years’ IRS 990 forms (or independent financial audit reports), looking for trends in financial bottom line and strength in cash position.19

 Research organizations’ mission-related results and impact by inquiring about programmatic theories of change and evidence of short- and long-term results, including any third-party evaluations that may be available.

 Gain familiarity with two concepts—the nonprofit “Overhead Myth” and “Starvation Cycle,” with an eye toward developing a full picture of nonprofits’ financial needs.20

19 IRS 990 forms of many nonprofits are publicly available on Guidestar.org. In addition, nonprofits and the IRS are required by law to make 990 forms publicly available. 20 See footnotes 11 and 14, above.

For nonprofit leaders:

 Meet CN’s accountability and transparency standards, which constitute a sound set of organizational practices.21

 Engage with and educate donors about the organization’s full costs of services, operational infrastructure requirements, unrestricted funding needs and strategies for sustainability.

 Proactively offer financial status information to supporters as an alternative to CN’s

Financial Health Rating.

For management consultants:

 Help nonprofit clients build strong, sustainable business models.

 Help clients present clear information to funders on financial health, results and impact, infrastructure needs and the full costs of services.

 Discourage clients from prioritizing financial efficiency and/or growth, when doing so

may compromise efforts to build robust operational capacity. 6

For Charity Navigator:

 Continue efforts to revise the Financial Health Rating methodology; replace six of the seven current metrics with specific measures of financial health from nonprofits’ balance sheets, income statements and appropriate supplemental sources, consistent with these findings.

 Continue the important task of developing an indicator for results and impact.

 Simplify the evaluation methodology, where possible.

program fee income from evaluation eligibility.

Charity Navigator was established to address an important need in the nonprofit sector. While we believe the CN system currently has several substantial problems, we recognize that these issues can be resolved, and note that CN is working on a new iteration. CN’s results reporting work may offer a promising future data source for donors seeking to assess organizational effectiveness, and an improved CN system can favorably influence public engagement in socially beneficial work. We hope the concerns and recommendations expressed here will constructively help the nonprofit community and its donors make well- informed performance management and resource allocation decisions in the future. 13

21 CN Accountability and Transparency metrics (see also footnote 9)

Addendum – June 2016

Charity Navigator issued revisions to its methodology three months after publication of this paper. Here we summarize changes and their impact on the findings above.

Specifically, CN has changed aspects of the Financial Health Rating:

 Program revenue growth has been removed as a component metric of the Financial Health Rating.

 A new metric, liabilities to assets ratio, has been added to the Financial Health Rating.

 Data for the four financial efficiency metrics, all of which continue from the existing methodology, are now computed based on three-year averages, rather than just the most recent year.

 The basis for scoring nonprofits on the administrative expenses metric has been adjusted, with the highest score achieved in a range (0-15% for most nonprofits, with several specific subsector exceptions) rather than simply 0% administrative expenses.

 Adjustments have been added for two uncommon special cases in nonprofit

accounting: indirect cost allocation and joint cost allocation (see CN website).

These changes constitute small, incremental improvements. For example, liabilities to assets ngCharity Navigator's Rating System ratio is a second tangible financial health metric within the Financial Health Rating. However, the sum total of these changes is not substantive in light of the issues raised in this paper.

The Financial Health Rating remains fundamentally flawed, for the reasons raised in the discussion in the body of this paper. No measure of profitability is included in the revised formula, a critical oversight. Five of the seven current component metrics are tangential to financial health, and can (and frequently do) flow counter to financial performance. Since the seven metrics remain equally weighted in the Financial Health Rating formula, this means that only 29% of the rating’s content is valid. The use of these metrics continues to mis-

growth and effectiveness—as negative behaviors.

valuatingthe Evaluator:Unpacki

E PUBLIC PUBLIC No other methodology changes have been announced, and no additional data is available through CN on nonprofits’ mission-related impact.

Despite the apparently well-intentioned revisions summarized here, Public Interest Manage- ment Group re-emphasizes our recommendations presented on pages 12-13 above.