Annex 3: Analysis of Knowledge Study Freedom from Hunger R2-125

Innovations for Poverty Action

Health Microinsurance Education Project Evaluation ,

Knowledge Test Results August 2011

IPA Ghana IPA Headquarters Osu PMB 57, , Ghana 101 Whitney Avenue, 2nd Floor Tel: 021 774570, 021 765040 New Haven, CT 06510-1256 USA www.poverty-action.org www.poverty-action.org Contents Executive Summary ...... 3 1. Introduction ...... 4 2. Background ...... Error! Bookmark not defined. Health Insurance in Ghana ...... Error! Bookmark not defined. 3. Description of Study ...... Error! Bookmark not defined. Health Microinsurance Education ...... Error! Bookmark not defined. Partnership Roles ...... Error! Bookmark not defined. Evaluation Design ...... Error! Bookmark not defined. Data Collection and Analysis ...... Error! Bookmark not defined. 4. Knowledge Quiz Administration ...... 4 Protocols ...... 4 Date Quality Considerations ...... 5 5. Knowledge Quiz Results ...... 5 Baseline Results ...... 5 Post-Education Knowledge Quiz Results ...... 7 6. Conclusion ...... 12

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 2 Executive Summary

National health insurance that provides a comprehensive set of health care services has been available to the formal and informal sectors in Ghana since 2003. However, coverage is far from universal, especially in rural areas. Freedom from Hunger, Sinapi Aba Trust (SAT) and Innovations for Poverty Action have formed a partnership to create, implement, and evaluate a program to educate microfinance clients in Ghana’s Northern Region about health insurance provided through the National Health Insurance Scheme (NHIS) of Ghana. The evaluation will assess if education has a positive effect on insurance take-up and retention, and the impact of insurance on use of health services, health spending, and financial security. Designed as a randomized control trial, the evaluation will compare outcomes for microfinance clients randomly assigned to receive the education treatment to outcomes for clients randomly assigned to the control group. In October 2010, Innovations for Poverty Action completed the baseline survey of all clients in the sample. From January to March of 2011, SAT loan officers conducted knowledge tests with a subset of the SAT clients in the study. This report summarizes the results from those knowledge tests.

The research design includes a sample of 300 microfinance groups in four different areas of the Northern Region: Tamale, Walewale, , and Bole. Forty percent of these groups were assigned to the control and will receive no education. Thirty percent of the groups were assigned each to two different types of education treatment: a treatment of short sessions conducted over several weeks and a treatment of one long session conducted once. The sessions were given to the groups by their SAT microfinance loan officer. Of the groups receiving treatments, half will also receive a reminder session a year after the completion of the education sessions, to remind them they must re-enroll to keep their insurance status current.

The knowledge tests were not conducted according to strict data collection protocols. Nevertheless, the results suggest that the education sessions had impacts on some aspects of client knowledge and attitudes about insurance. In particular, clients in the treatment groups scored significantly better on the knowledge questions, had significantly higher improvements in scores on the knowledge questions, and among those clients who did not have insurance, they were significantly more likely to say the reason was simply that they had not yet had time to do it.

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 3 1. Introduction Although Ghana introduced a national health insurance program in 2003, enrollment rates remain low, particularly in rural areas. In 2010, Freedom from Hunger entered into a partnership with Sinapi Aba Trust (SAT), a Ghanaian microfinance institute, and Innovations for Poverty Action, an NGO specializing in impact evaluation, to design, implement and evaluate a program to teach microfinance clients about health insurance. The key questions of this evaluation are to determine whether the program increases up-take of insurance, and how insurance enrollment affects use of health services, health spending, and indicators of financial security. The project is receiving funding from the International Labor Organization.

The baseline survey for this project was conducted in September and October of 2010, just before the education program began implementation. The survey questioned five members of each of the 300 microfinance groups included in the sample. Data entry for the survey was completed in February of 2011. A preliminary analysis of these data was completed in April 2011; a final baseline analysis was completed in September 2011

This report is analysis of the data collected in knowledge quizzes conducted by the SAT loan officers with their loan groups. The knowledge tests were administered immediately following the completion of the education sessions. For more information about health insurance in Ghana and the details of the Health Microinsurance Education project, as well as an update on the status of the project, please refer to the Final Baseline Survey Report.

2. Knowledge Quiz Administration Protocols The knowledge quizzes were administered by the SAT loan officers immediately following the completion of the education sessions. The knowledge quizzes were administered to a subset of the original sample of 1500; two respondents were randomly chosen from each loan group to be given the knowledge test for a total of 600 respondents. The SAT loan officers were provided with a list of SAT clients to give the knowledge test to. For each loan group, a third respondent name was also provided as an alternate, in case one of the original two could not be located or chose not to consent to the quiz.

The knowledge quiz was conducted in paper form. The SAT loan officer was instructed to ask for consent, and then read the questions and record the responses. Each quiz was estimated to take no more than 10 minutes to complete.

The completed questionnaires were delivered to IPA for data entry and analysis, which was conducted by two IPA interns. After the questionnaires were handed over to IPA, they were subject to standard IPA procedures for data security and confidentiality.

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 4 Date Quality Considerations A number of issues arose with the collection of the knowledge quiz data which could have implications for data quality, and consequently confidence in the results of the data analysis.

 The SAT loan officers did not survey the entire subsample. IPA received only 155 surveys out of 600.

 The SAT loan officers did not follow the randomization. A much higher than expected number of respondents were alternates, suggesting that the SAT loan officers may have conducted the survey with any of the three respondents listed on their roster, rather than focusing on the first two before using the alternate. In some cases, the respondent interviewed was not among the three selected in the randomization at all. Because of the small total number of surveys received, all quizzes were included in the analysis, regardless of whether the respondent was among the list of three respondents given to SAT to survey.

 The SAT loan officers were evaluating themselves. Best practice would be to have independent evaluators with no stake in the findings. Loan officers may have felt pressure to get results showing improvement as a result of their efforts. To minimize the risk of this, we tried to emphasize to both the loan officers and the respondents that the quiz was intended to test the program, not the people implementing it or receiving it. 3. Knowledge Quiz Results Baseline Results

Respondents who had at least household member not registered in NHIS were asked why household members were not registered (Table 1). Multiple answers were allowed. Table 1. Reasons Reported for not Registering for Insurance Number of Percent of respondents respondents reporting each reporting each reason reason Didn't know about it 4 0.5% Don't know how to register 5 0.6% Too difficult to register 60 7.2% The premium is too expensive 316 38.0% Don't think will get sick 27 3.2% Services too far away 20 2.4% Services are not good 32 3.8% Intend to, but just haven't done it yet 403 48.4% Other 92 11.1% The most common answer was that the household intended to register everyone, but had not gotten around to doing so. The second most common response was that the premium

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 5 was too expensive. Very few had household members who were unregistered because they did not know about insurance or did not know how to register. Among respondents who selected “other” and specified an alternative, a common answer was “not interested in insurance”, with no given reason for the disinterest.

Respondents’ knowledge of health insurance before the education sessions was also tested. The knowledge test consisted of six true-or-false questions. Table 2 lists each question, the correct answer, and the percent of respondents who were able to give the correct answer.

Table 2. Knowledge Test Correct Responses by Question

Percent Respondents Correct with Correct Answer Answer T or F: After registering for insurance for the first time, I can use insurance to pay for health care immediately. F 56.58% T or F: Transportation costs and lost work time are part of the costs of being sick. T 56.49% T or F: I must re-enroll in insurance every year in order to access services using my insurance card. T 93.43% T or F: There is a limit to how many times I can use my insurance each year. F 73.96% T or F: People with health insurance must still pay the doctor or the hospital before they can get covered services. F 89.31% T or F: If I do not use health services this year, I will get back the money I paid for insurance. F 95.28%

Table 3 reports the distribution of respondent scores on the knowledge test. Most respondents performed well. Of the respondents whose surveys had data for every question on the knowledge test, 88 percent of respondents missed 2 or fewer questions on the test.

Table 3. Respondent Performance on Knowledge Test

Number of Percent of Respondents Respondents Number Achieving that Achieving Correct Score that Score 0 14 0.9% 1 17 1.1% 2 34 2.3%

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 6 3 113 7.6% 4 375 25.4% 5 591 40.0% 6 335 22.7% 1479 100.0%

The Knowledge Section also asked respondents questions about their attitudes towards toward insurance. Table 4 shows the percent of respondents who responded “agree”, “disagree”, or “don’t know” to the two statements about health insurance. Responses suggest very favorable attitudes toward insurance.

Table 4. Attitudes About Health Insurance Don't Agree Disagree Know N I would rather risk paying for health expenses cash and carry than pay for health insurance 6.1% 92.1% 1.8% 1501 Insurance is not a good value for the money 16.5% 74.7% 8.8% 1503

Of the respondents with data, 92 percent indicated they would rather pay for insurance than risk paying cash when they need health services, and 75 percent indicated that they though insurance was a good value for the cost.

Table 5 shows the percent of respondents who report they have discussed insurance with at least one of their household members. A large majority of respondents, 87 percent, report discussing insurance with family.

Table 5. Respondents Who Have Discussed Insurance with Household Yes No N

Have you discussed health insurance with any of your household members? 87.0% 13.0% 1502

Post-Education Knowledge Quiz Results Table 6 reports the share of respondents who reported that all of their household members are registered for insurance, by treatment group: short education sessions, consolidated education sessions, and control group. Respondents who did not know the status of all their household members are excluded from this summary statistic. The share of respondents reporting complete household coverage was not significantly different between respondents who received short education sessions and those who were in the control. Respondents in the consolidated session treatment group were significantly less likely to

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 7 have complete household coverage than those in the control group. Significance was determined using equal variance t-test.

Table 6. Percent of Households with all Members Registered, by Treatment Group Short Consolidated Sessions Sessions Control

Percent Answering Yes: Are all of your household members registered for insurance? 42% 33%* 53% *Significantly different from the control group at the 5% level.

Table 7 shows the percent of respondents who did not have complete insurance coverage in their households who reported each reason for not having insurance. Multiple responses were allowed. Respondents in the short session treatment group were significantly less likely to report that members of their household did not have insurance because the premium was too expensive. Respondents in either treatment group—short session or consolidated session—were more likely to report that household members did not have insurance simply because they had not yet had time to go register.

Table 7. Reasons Reported for not Registering for Insurance, by Treatment Group Percent of respondents lacking complete Consolidated coverage who reported each reason Short Sessions Sessions Control Didn't know about it 13% 18% 12% Don't know how to register 16% 32% 18% Too difficult to register 41% 50% 59% The premium is too expensive 28%* 49% 53% Don't think will get sick 16% 14% 18% Services too far away 31% 46% 47% Services are not good 22% 36% 12% Intend to, but just haven't done it yet 63%* 57%* 29%

*Significantly different from the control group at the 5% level.

Although respondents in the treatment groups were more likely to report that members of their household did not have insurance because they hadn’t had time to register yet, there was no significant difference in the number of respondents answering that they were “likely” to enroll in health insurance in the next 6 months between the treatment and control groups. Among the 73 respondents who said not all of their household members were registered, only 3 indicated they were “unlikely” to register; the remaining 70 indicated that they were “likely” to register. Of the three indicating they did not plan to register, one was in a consolidated treatment group, the other two were in groups that were either back-up groups or were replaced, and were not included in the final treatment or control groups.

Table 8 reports correct responses the insurance knowledge questions for before the education (results from the baseline survey) and after the education, for those respondents

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 8 included in the post-education knowledge quiz subset. Note that because this does not include all of the original baseline respondents—only those who also participated in the post-education quiz—the results for “before” will differ from the results reported above in Table 2.

Table 8. Knowledge Test Correct Responses by Question, Before and After Treatment

Correct Pre- Post- Answer Education Education T or F: After registering for insurance for the first time, I can use insurance to pay for health care immediately. F 58% 59% T or F: Transportation costs and lost work time are part of the costs of being sick. T 62% 79% T or F: I must re-enroll in insurance every year in order to access services using my insurance card. T 96% 90% T or F: There is a limit to how many times I can use my insurance each year. F 75% 75% T or F: People with health insurance must still pay the doctor or the hospital before they can get covered services. F 87% 89% T or F: If I do not use health services this year, I will get back the money I paid for insurance. F 96% 89%

Table 9 reports the same post-education results, by treatment group. The treatment groups generally performed better than the control group, except on the second knowledge question, which asked the respondent whether transportation costs and lost work time were part of the costs of being sick; respondents in the control group were significantly more likely to correctly identify this as true.

Table 9. Knowledge Test Correct Responses by Question

Short Consolidated Correct Sessions Sessions Control Answer (N=57) (N=42) (N=40) T or F: After registering for insurance for the first time, I can use insurance to pay for health care immediately. F 61%* 71%* 43% T or F: Transportation costs and lost work time are part of the costs of being sick. T 75%* 74%* 90%

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 9 T or F: I must re-enroll in insurance every year in order to access services using my insurance card. T 89% 90% 87% T or F: There is a limit to how many times I can use my insurance each year. F 84%* 83%* 55% T or F: People with health insurance must still pay the doctor or the hospital before they can get covered services. F 89% 93%* 80% T or F: If I do not use health services this year, I will get back the money I paid for insurance. F 91% 95%* 80% *Significantly different from the control group at the 5% level.

Table 10 shows the average score, out of 6 questions, on the knowledge quiz, by treatment group. Both treatment groups had significantly higher average scores than the control group.

Table 10. Average Score on Knowledge Questions, by Treatment Group

Short Consolidated Sessions Sessions Control (N=57) (N=42) (N=40) Average number of correct answers, out of 6 4.9* 5.0* 4.3 *Significantly different from the control group at the 5% level.

Table 11 shows the average improvement in score on the knowledge quiz by treatment group. This was calculated by subtracting each respondent’s score on the knowledge questions in the baseline survey from his or her score on the same questions in the post- education knowledge quiz. Both treatment groups had significantly higher average scores than the control group.

Table 11. Average Improvement on Knowledge Questions, by Treatment Group

Short Consolidated Sessions Sessions Control (N=57) (N=42) (N=40) Average number of correct answers, out of 6 0.4* 0.3* -0.6 *Significantly different from the control group at the 5% level.

Respondents were also asked two questions intended to reveal their attitudes towards insurance:  “Agree or disagree: I would rather risk paying for health expenses cash and carry than pay for health insurance.”

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 10  “Agree or disagree: Insurance is not good value for the money”

Responses to both questions suggest generally favorable attitudes towards health insurance (See figures 12 and 13). There were no significant differences in the answers to these questions between respondents in the control and treatment groups.

Lastly, respondents were asked if they had talked about insurance with any of their household members; 95% reported doing so. There was no significant difference in the answer to this question between respondents in the control and treatment groups.

Figure 12. "Agree or disagree: I would rather risk paying for health expenses cash and carry than pay for health insurance." Don't Know 3%

Agree 14%

Disagree 83%

Figure 13.

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 11 "Agree or disagree: Insurance is not a good value for the money. " Don't Know 4%

Agree 21%

Disagree 75%

4. Conclusion

The data from these knowledge tests should be interpreted cautiously, taking into consideration the challenges of collecting data through SAT’s loan officers, who have not had the benefit of being trained in data collection. However, these data are suggestive of some positive impacts of education. Most notably:

 Respondent in the treatment groups scored significantly better on the knowledge questions  Respondents in the treatment groups showed significantly higher improvements in scores on the knowledge questions  Among those clients who did not have insurance, respondents in the treatment groups were significantly more likely to say the reason was simply that they had not yet had time to register for insurance

If positive impacts on insurance registration are found in the endline data collection, these findings will support the conclusion that the increased take-up was due in part to better knowledge about and understanding of health insurance.

Innovations for Poverty Action | Osu PMB 57 | Accra, Ghana | 021 774570 | www.poverty-action.org 12