All Metrics Are Not Created Equal: A Closer Look at Measuring Severity of Need and Well‐Being
JESH HARBAUGH, ASST. DIRECTOR OF BUSINESS INTELLIGENCE SARAH ACCOMAZZO, ASST. DIRECTOR OF RESEARCH AND EVALUATION Today’s Session
•Overview of aggregate need & strength metrics •Important nuances •Findings: how similar/different are these metrics? •Relationship between needs‐ and strengths‐based metrics? Why Use Aggregate Metrics?
Single CANS Overall Need Metric Assessment
Aggregate All “TAI” Item Scores 34 (Total Actionable Items)
•Enables many different types of analyses • I.e. Easily identify clients as having “high” or “low” overall need compared to other individuals or groups •Highly desired by stakeholders Background •Prompted by our study of whether certain item scores predict greater overall need (Identifying Patterns of Service Need, 2017).
•Realized that the entire study hinged on the way we calculated severity of need (TAI)
•Do other metrics express severity of need differently when applied to the same CANS assessment? Different Types of Aggregate CANS Metrics
Metric Calculation
Total Actionable Items (TAI) Count of items with item score of 2 or 3
Sum of Item Scores Sum total of all item scores (including 1s)
% Actionable TAI / Number of item scores Count of items with item score of 0 or 1 in Number of Strengths Strengths domain Sum total of all item scores in Strengths Sum of Strength Domain Item Scores domain. (Lower sum = more strengths) Different CANS Versions and Item Subsets
The same metric will produce different results when run on different sets of items.
CANS Version / Item Subset Notes (Seneca’s CANS in CA)
Full CANS (Including Ext. Modules) 96 core domain items, up to 189 items total
Core Domain Items Only 96 core domain items
CANS Core 50 50 items, no extension modules
Strengths Domain Only 13 items (9 in Core 50) Using the CANS to Measure “Well‐Being”
•What is “well‐being”? • Most broadly, the presence of positive states/outlooks and absence of negative states. Largely based on the individual’s self‐perception. • Harvard’s “Flourishing” Measure • 2 self‐report questions in each of the 5 domains:
Happiness and Life Satisfaction Mental and Physical Health Meaning and Purpose
Character and Virtue Close Social Relationships • We decided to focus on “Strength‐related” metrics due to challenges in measuring self‐perception in these domains
VanderWeele, T.J. (2017). On the promotion of human flourishing. Proceedings of the National Academy of Sciences, U.S.A., 31:8148‐8156. Why Sweat the Small Stuff? Findings: Metric Comparison Our Dataset •Sample: • 2,302 CANS assessments • Administered between Nov 2015‐Mar 2018 • Youth aged 6‐17 •CANS Details • All assessments had exactly 96 “core domain” item scores • Caveat: Our Core 50‐based metrics only include 42 items which were present in Seneca’s full CANS assessments. Analytic Approach •Assess correlation of metrics: run Pearson tests of correlation
•Explore similarity of metrics on individual youth‐level: convert each metric into percentile rank to identify where each metric places each youth in the continuum of need within the sample Needs‐Related Metrics Summary Stats: Needs Metrics Metric Name Mean St Dev. Range TAI –Full CANS 24.9 15.3 0 –90 TAI –Core Domains Only 15.4 8.2 0 –51 TAI –Core 50 8.8 5 0 –28 Sum –Full CANS 85.9 39.8 7 – 258 Sum –Core Domains Only 52.5 19.8 6 – 138 Sum –Core 50 30 11.6 2 ‐ 71 Distributions: Need Metrics Correlation Between Needs Metrics •Generally, strong correlation across all needs metrics as assessed by Pearson tests of correlation. • Correlation coefficient values ~0.9
•Slightly less strong correlations when differences in metric type and CANS type are both present
•BUT, this can look different at the individual level Median Difference in Percentile Rank Between Two Named Scales 5.7% 8.2% 6.0% 6.0% 7.2% 5.6% 5.6% 6.1% 6.3% Average Difference in Percentile Rank Between Two Named Scales 7.8% 10.1% 7.6% 7.9% 9.3% 7.7% 7.3% 7.9% 8.6% Percentile Ranks ‐ Severity of Need Metrics Difference in Percentile Ranks: Need Metrics Diff by Metric Type Diff by CANS Type Core Core Domain Full TAI ‐ Core Full Sum ‐ Domain TAI ‐ Core 50 Percentile Percentile Percentile Percentile Percentile Percentile Core Full TAI ‐ Domain Core Full Sum ‐ Sum ‐ Core TAI ‐ Percentile Ranks: Rank ‐ Full Rank ‐ Full Rank ‐ Core Rank ‐ Core Rank ‐ Core Rank ‐ Core Domain Core 50 TAI ‐ Core Domain Core 50 Core 50 Full TAI ‐ Domain Core 50 TAI CANS Sum Domain TAI Domain Sum 50 TAI 50 Sum TAI TAI 50 TAI Sum Sum Sum Full Sum Sum Sum 95% 94% 92% 89% 95% 94% 3% 0% 4% 5% 0% 5% 1% 2% 1% 68% 56% 68% 56% 74% 54% 0% 6% 6% 1% 2% 1% 12% 12% 19% Need Metrics 71% 65% 71% 72% 68% 64% 0% 3% 3% 6% 2% 8% 6% 0% 4% 74% 90% 46% 73% 61% 82% 28% 13% 15% 17% 8% 9% 16% 27% 21% Placing a client on the continuum of 79% 72% 75% 69% 61% 57% 4% 18% 13% 4% 15% 11% 7% 6% 4% need, relative to others 99% 98% 99% 99% 100% 100% 0% 1% 1% 0% 1% 1% 1% 1% 1% 95% 91% 95% 94% 95% 96% 1% 0% 0% 3% 5% 2% 4% 1% 1% 57% 67% 64% 79% 61% 75% 7% 4% 3% 13% 9% 4% 10% 15% 14% 48% 68% 36% 59% 29% 57% 12% 20% 8% 9% 10% 1% 19% 22% 29% 52% 65% 56% 67% 45% 73% 4% 7% 11% 2% 8% 6% 13% 11% 27% 90% 94% 85% 93% 74% 84% 5% 17% 12% 1% 10% 9% 3% 8% 11% 90% 94% 85% 98% 89% 93% 5% 2% 3% 4% 1% 5% 4% 13% 4% 96% 93% 96% 95% 97% 95% 0% 1% 1% 2% 2% 0% 3% 1% 2% 68% 70% 78% 72% 82% 82% 11% 14% 3% 1% 12% 10% 2% 7% 0% 64% 61% 92% 90% 89% 86% 28% 25% 3% 29% 24% 5% 2% 1% 3% 64% 78% 51% 73% 45% 69% 12% 19% 6% 5% 9% 4% 15% 22% 24% Very consistent 40% 46% 56% 59% 53% 64% 16% 14% 3% 13% 18% 5% 6% 3% 10% 57% 75% 41% 59% 53% 73% 16% 4% 12% 17% 3% 14% 18% 17% 19% across metrics 55% 65% 36% 64% 45% 64% 18% 10% 9% 1% 2% 0% 11% 28% 19% 34% 34% 28% 33% 21% 41% 7% 14% 7% 1% 6% 7% 0% 6% 20% 99% 99% 100% 100% 99% 100% 1% 0% 1% 1% 0% 0% 1% 0% 1% 74% 84% 85% 88% 89% 94% 11% 14% 3% 4% 10% 6% 10% 2% 5% 82% 84% 90% 89% 89% 91% 9% 7% 2% 5% 7% 2% 2% 1% 2% 71% 67% 87% 82% 86% 80% 16% 14% 2% 16% 13% 2% 5% 5% 6% 71% 89% 71% 88% 78% 84% 0% 7% 7% 1% 5% 4% 17% 17% 6% Less Consistent 62% 77% 32% 64% 29% 60% 30% 33% 3% 13% 17% 4% 15% 32% 32% 37% 46% 46% 56% 61% 75% 10% 25% 15% 10% 30% 20% 9% 9% 14% 57% 67% 51% 66% 61% 67% 6% 4% 10% 2% 1% 1% 10% 14% 5% 93% 96% 95% 98% 94% 97% 2% 1% 1% 2% 1% 0% 3% 3% 3% 88% 95% 81% 90% 86% 90% 7% 2% 4% 5% 5% 1% 7% 9% 4% 55% 46% 51% 43% 53% 51% 3% 1% 2% 2% 6% 8% 9% 8% 2% 88% 85% 93% 91% 91% 91% 5% 4% 1% 7% 6% 0% 3% 1% 0% 96% 95% 94% 93% 94% 93% 2% 3% 0% 2% 3% 0% 1% 1% 1% 74% 74% 68% 70% 74% 69% 6% 1% 6% 4% 5% 1% 0% 2% 4% 10% 20% 9% 26% 3% 14% 0% 6% 6% 6% 6% 12% 10% 16% 10% 21% 35% 36% 52% 38% 64% 15% 16% 1% 16% 29% 12% 14% 15% 26% 21% 46% 32% 49% 45% 57% 10% 24% 14% 3% 12% 8% 24% 17% 12% Key Takeaways: Needs Metrics •In the big picture, these metrics are generally similar
•On the individual level, some metrics will assess youth as having different levels of need relative to others. Strength‐Related Metrics Summary Stats: Strength Metrics Metric Name Mean St Dev. Range Num of Centerpiece Strengths –Full CANS 2.9 3 0 –13 Num of Centerpiece Strengths –Core 50 1.9 2 0 –8 Num of Strengths –Full CANS 7.6 3.3 0 –13 Num of Strengths –Core 50 4.8 2.1 0 –8 Sum of Strength Domain Item Scores – 17.2 6.9 0 –35 Full CANS Sum of Strength Domain Item Scores – 10.2 4.5 0 ‐ 23 Core 50 Distributions: Strength Metrics Correlation Between Strength Metrics •Generally, strong correlation between these strength‐related metrics • Correlation coefficient of ~0.9 for non‐centerpiece strength metrics
•Weaker, but still relatively strong, correlations present between centerpiece and non‐centerpiece strength metrics • Correlation coefficient of ~0.67 for centerpiece strength metrics Median Difference in Percentile Rank Between Two Named Scales 4.4% 5.5% 4.8% 13.1% 10.9% 7.2% 15.4% 12.3% 9.7% Average Difference in Percentile Rank Between Two Named Scales 7.1% 7.5% 6.4% 18.1% 14.5% 9.0% 19.8% 17.2% 11.4% Percentile Ranks ‐ Strength Metrics Difference in Percentile Ranks: Strength Metrics Diff by CANS Type Diff by Metric Type Percentile Rank ‐ Percentile Percentile Rank ‐ Percentile Core 50 Sum of Full Core 50 Rank ‐ Percentile Sum of Strengths Rank ‐ Core 50 Percentile Strengths Full #Center #Strengths ‐ Full Sum ‐ Full #Center ‐ Full Core 50 #Center ‐ Core 50 #Centerpiece Rank ‐ Domain (Inverse #Centerpiece Rank ‐ Core Domain (Inverse ‐ Core 50 Core 50 Core 50 Sum Full Full #Center #Strengths ‐ #Center ‐ Core Core 50 #Strengths ‐ Strengths #Strengths Scale) Strengths 50 #Strengths Scale) #Center #Strengths (Str) #Strengths ‐ Full Sum Full Sum 50 #Strengths Sum Core 50 Sum 0% 7% 96% 0% 2% 98% 0% 5% 2% 7% 4% 3% 2% 2% 0% 44% 4% 89% 33% 2% 88% 10% 1% 1% 40% 33% 7% 31% 21% 10% 44% 13% 78% 33% 15% 76% 10% 2% 2% 31% 21% 9% 18% 9% 9% Percentile Ranks: 0% 13% 86% 0% 15% 83% 0% 2% 3% 13% 14% 1% 15% 17% 2% 0% 49% 50% 0% 59% 41% 0% 11% 8% 49% 51% 2% 59% 59% 0% 25% 4% 78% 33% 2% 76% 8% 1% 2% 22% 3% 19% 31% 9% 22% 25% 1% 82% 33% 2% 76% 8% 1% 6% 24% 7% 17% 31% 9% 22% Strengths Metrics 0% 28% 73% 0% 29% 68% 0% 1% 5% 28% 27% 0% 29% 32% 4% 0% 37% 56% 0% 59% 41% 0% 22% 15% 37% 44% 7% 59% 59% 0% 25% 20% 67% 0% 29% 68% 25% 9% 0% 5% 7% 13% 29% 32% 4% 0% 28% 67% 0% 42% 51% 0% 15% 17% 28% 33% 5% 42% 49% 7% 0% 7% 86% 0% 7% 83% 0% 0% 3% 7% 14% 7% 7% 17% 11% 0% 7% 82% 0% 7% 76% 0% 0% 6% 7% 18% 11% 7% 24% 18% 0% 1% 89% 0% 2% 88% 0% 1% 1% 1% 11% 10% 2% 12% 10% 44% 4% 97% 33% 2% 96% 10% 1% 2% 40% 41% 1% 31% 29% 2% 0% 49% 56% 0% 59% 41% 0% 11% 15% 49% 44% 4% 59% 59% 0% 0% 59% 50% 0% 42% 51% 0% 17% 1% 59% 51% 8% 42% 49% 7% 0% 69% 38% 0% 73% 33% 0% 4% 5% 69% 62% 7% 73% 67% 6% 25% 37% 56% 0% 42% 51% 25% 5% 5% 12% 19% 7% 42% 49% 7% 0% 69% 43% 0% 73% 41% 0% 4% 2% 69% 57% 12% 73% 59% 15% 0% 7% 82% 0% 7% 83% 0% 0% 1% 7% 18% 11% 7% 17% 11% 44% 49% 38% 0% 29% 59% 44% 20% 22% 5% 19% 14% 29% 41% 12% 0% 4% 94% 0% 2% 93% 0% 1% 1% 4% 6% 2% 2% 7% 5% 0% 4% 82% 0% 2% 83% 0% 1% 1% 4% 18% 15% 2% 17% 15% 0% 87% 29% 0% 73% 33% 0% 14% 4% 87% 72% 16% 73% 67% 6% Less consistent 25% 49% 43% 33% 59% 33% 8% 11% 10% 23% 32% 9% 26% 34% 8% 0% 37% 62% 0% 42% 59% 0% 5% 3% 37% 38% 1% 42% 41% 2% 44% 69% 29% 55% 59% 27% 12% 10% 2% 25% 28% 3% 4% 18% 14% 0% 7% 86% 0% 7% 83% 0% 0% 3% 7% 14% 7% 7% 17% 11% 25% 37% 50% 33% 42% 41% 8% 5% 8% 12% 25% 14% 9% 26% 17% Very consistent 57% 37% 43% 55% 29% 41% 1% 9% 2% 20% 0% 20% 27% 3% 30% 0% 1% 97% 0% 2% 96% 0% 1% 2% 1% 3% 2% 2% 4% 2% 0% 7% 92% 0% 7% 88% 0% 0% 4% 7% 8% 1% 7% 12% 6% across metrics 25% 7% 78% 0% 7% 76% 25% 0% 2% 18% 3% 15% 7% 24% 18% 96% 87% 3% 93% 73% 6% 3% 14% 2% 9% 1% 10% 19% 2% 21% 0% 7% 99% 0% 7% 99% 0% 0% 0% 7% 1% 6% 7% 1% 6% 25% 13% 86% 33% 7% 93% 8% 7% 7% 12% 11% 1% 27% 26% 0% 0% 4% 99% 0% 2% 99% 0% 1% 0% 4% 1% 2% 2% 1% 1% 0% 4% 96% 0% 2% 96% 0% 1% 0% 4% 4% 1% 2% 4% 2% 76% 87% 12% 70% 73% 17% 6% 14% 5% 11% 12% 1% 3% 13% 10% Key Takeaways: Strength Metrics •Metrics which do not exclusively involve “centerpiece” strengths are generally similar.
•As with needs metrics, on the individual level, some metrics will assess youth as having different levels of strength relative to others. Needs vs. Strengths Metrics Correlation: Needs and Strength Metrics? •Moderately strong negative correlations between needs‐ and strength‐related metrics do exist • Correlation coefficient values range from 0.53 ‐ 0.73
•Higher overall need is generally associated with the absence of strengths. Implications and Next Steps •Aggregate measures are generally similar
•Challenge of meaningful aggregation
•Further exploration of using differences in CANS forms
•Deep dive into measuring “well‐being” Questions or thoughts?