Evaluation of a Critical Appraisal Program for Clinical Nurses: A

Home , Medical literature

Evaluation of a Critical Appraisal Program for Clinical Nurses: A Controlled Before-and-After Study Shu-Chen Chang, RN, MSN, Chin-Yi Huang, MHA, Shu-Yu Chen, RN, MSN, Yi-Cheng Liao, MD, Ching-Hsiung Lin, MD, PhD, and Hsiu-Hung Wang, RN, PhD, FAAN

abstract paigns, has had a positive effect, increasing knowledge and skills in evidence-based practice and accelerating Background: Evidence-based practice is important to the dissemination of evidence-based practice in regional clinical health care professionals. Clinical nurses can make hospitals in Taiwan (Chiu, Weng, Lo, Shih, et al., 2010). informed decisions by applying the best evidence to their Many health care institutions in Taiwan have recognized care. However, standardized curricula on evidence-based the importance of evidence-based practice and have medicine are lacking in nursing education programs. This implemented information systems to support physi- study evaluated a critical appraisal education program and cians who practice evidence-based medicine (Hung, Ku, assessed its value in increasing nurses’ knowledge of criti- & Chien, 2012). However, Chiu, Weng, Lo, Shih, et al. cal appraisal and confidence in their critical appraisal skills. (2010) found that many nurses reported that less sup- Methods: A controlled before-and-after study design port by health care administrators would be a barrier to was used. The education program integrated lectures, their access to educational programs on evidence-based practice, and group discussion. A questionnaire was used practice. to evaluate participants’ knowledge and confidence in critical appraisal pre- and postintervention. Results: Participants’ knowledge and confidence in Ms. Chang is Director, Department of Nursing, Changhua Christian critical appraisal improved significantly postintervention Hospital, Changhua; and PhD candidate, College of Nursing, Kaohsi- ung Medical University, Kaohsiung, Taiwan. Ms. Huang is Administra- (both p < .001). tor, Center for Evidence-Based Medicine, Changhua Christian Hospi- Conclusion: A 1-day, small group discussion educa- tal, Changhua, Taiwan. Ms. Chen is Supervisor, Department of Nursing, tion program can effectively improve nurses’ knowledge Changhua Christian Hospital, Changhua; and doctoral student, De- and confidence in critical appraisal. Educators and ad- partment of Nursing, National Taiwan University, Taipei, Taiwan. ministrators may replicate this education program to im- Dr. Liao is Director, Center for Evidence-Based Medicine, Changhua Christian Hospital, Changhua, Taiwan. Dr. Lin is Director, Division of prove the quality of nursing care. Chest Medicine, Department of Internal Medicine, Changhua Christian J Contin Educ Nurs 2013;44(X):xx-xx. Hospital, Changhua; Assistant Professor, Department of Respiratory Care, College of Health Sciences, Chang Christian University, Tainan; and Lecturer, School of Medicine, Chung Shan Medical University, Tai- ritical appraisal of the relevant published literature chung, Taiwan. Dr. Wang is Professor, College of Nursing, Kaohsiung Medical University, Kaohsiung, Taiwan. Cis an essential competence for nurses and is regarded The authors have disclosed no potential conflicts of interest, financial by medical professionals as the standard for health care or otherwise. (Bernardo, Matthews, Kaufmann, & Yang, 2008). It is Address correspondence to Hsiu-Hung Wang, RN, PhD, FAAN, believed that the practice of evidence-based medicine College of Nursing, Kaohsiung Medical University, No. 100, Shih- would help health professionals to improve the quality Chuan 1st Rd., Kaohsiung City, Taiwan. E-mail: [email protected]. tw. of health care. The program conducted by the National Received: April 15, 2012; Accepted: October 10, 2012; Posted: No- Health Research Institutes of Taiwan, which includes vember 8, 2012. information, resource support, and multifaceted cam- doi:10.3928/00220124-20121101-51

The Journal of Continuing Education in Nursing · Vol 44, No X, 2013 1 Evidence-based practice is highly recommended in cal nurses at all levels of the clinical nursing ladder who the nursing literature. However, many nurses lack con- were interested in evidence-based nursing. The clinical fidence in their own appraisal skills to evaluate evidence nursing ladder system refers to a hierarchical structure and determine what could be implemented in practice, proposed by the Taiwan Nurses Association that divides mainly because of the lack of evidence-based nursing nurses into four levels according to their clinical abili- training in clinical practice (Tagney & Haines, 2009). ties and proficiency (Leu, Liao, Chang, & Su, 2009). A Chiu, Weng, Lo, Hsu, et al. (2010) found that signifi- description of the program was distributed by the Evi- cant discrepancies exist between physicians and nurses dence-Based Nursing Working Team. The head nurse of in their awareness of, attitude toward, knowledge of, every nursing unit recommended a voluntary participant skills in, behavior toward, and barriers regarding evi- for the program. dence-based practice in Taiwan (Chiu, Weng, Lo, Hsu, The difference in the total score of knowledge test et al., 2010). Even nurses who underwent a research of matched pairs (pre- to postintervention) is normally course might still lack skills in appraising the value of distributed. If the true difference in the mean response research (Duffy, 2005). Therefore, developing objective of matched pairs is 1.5, with a standard deviation of 2.0, and structured guidelines for critical appraisal is impor- then at least 36 subjects must be included in the study. tant to influence clinical decision making. By developing The type II error (power) is 0.85 and the type I error such appraisal skills, nurses can access and evaluate the probability is 0.05. To allow for a 10% dropout rate, the most up-to-date resources and make appropriate clinical actual minimal sample size is 40. A total of 49 partici- decisions according to the validity of the research. Skills pants were enrolled in the study, and all 49 completed and knowledge in critical appraisal are important aspects the education program. Participants’ knowledge of criti- of implementing evidence into clinical practice (Milne, cal appraisal and confidence in their appraisal skills were Krishnasamy, Johnston, & Aranda, 2007). evaluated pre- and postintervention. Inconsistencies have been found in the amount and quality of different teaching approaches, and the lack Educational Intervention of evidence supporting the approaches affects the use- The critical appraisal educational program consisted fulness of such training (Harewood & Hendrick, 2010). of three sessions. Each session included three parts and Therefore, well-designed continuing education pro- lasted for 2.5 hours. This 1-day education program was grams for the appraisal and use of research are needed to designed by the Evidence-Based Nursing Working Team improve nurses’ knowledge and skills and enhance their of Changhua Christian Hospital and included a small use of evidence in clinical practice. group discussion. The participants were assigned to read This study was undertaken to develop and conduct three papers and other materials that were distributed a critical appraisal education program for nurses, based before the program. They were also expected to be fully on social learning theory, and to assess the improvement prepared to take part in the small group discussions. in nurses’ knowledge and confidence in critical appraisal First, the lecturer introduced the topic for that ses- after completion of the education program. sion. All lectures were delivered by the same speaker, a physician who had experience in evidence-based prac- METHODS tice and was qualified by the Taiwan Evidence-Based This study and the questionnaires used were approved Medicine Association. Each lecture was followed by a by the Evidence-Based Practice Committee of Chang- small group discussion to reinforce the information. A hua Christian Hospital, Changhua, Taiwan. Participants tutor, who helped guide the small group discussion, was were recruited voluntarily after the research was an- assigned to each group by the Evidence-Based Nursing nounced. Before the program was started, the content Working Team. Each group was assigned to read three and purpose of the pre- and postintervention were de- different types of clinical research papers: treatment scribed to participants. To protect the rights of partici- studies, systematic reviews, and clinical practice guide- pants, the pre- and postintervention questionnaires were lines. After the participants read and appraised an article processed anonymously and no personal information by themselves, each group held a 50-minute discussion was collected. using the critical appraisal users’ guidelines and an appraisal instrument. Participants shared their appraisals Research Setting and Study Design of the paper with the other members of the small group. This study was conducted in a 1,676-bed medical cen- Finally, in a 40-minute session, the lecturer assisted the ter located in central Taiwan. A controlled before-and- participants in reaching a consensus on the format of a after study design was used. The participants were clini- larger group review of the critical appraisal users’ guide-

2 Copyright © SLACK Incorporated lines and the appraisal instrument for the guidelines. Before the first session, participants were asked to com- TABLE 1 plete the preintervention evidence-based nursing ques- DEMOGRAPHIC CHARACTERISTICS OF THE tionnaire on critical appraisal. They were also asked to PARTICIPANTS (N = 49) complete the immediate postintervention questionnaire Characteristic n (%) at the end of the education program. Age (years) 26 to 30 12 (24.5) Instruments 31 to 35 26 (53.1) An evidence-based nursing questionnaire was developed by the research team, based on the literature (Cul- 36 to 40 5 (10.2) lum, 2000, 2001; Guyatt & Rennie, 2008) and the critical > 40 6 (12.2) appraisal users’ guidelines to assess the effectiveness of Education the education program, including participants’ knowl- Postgraduate 2 (4.0) edge of critical appraisal and their perceived confidence Undergraduate 38 (77.6) in appraising evidence. The questionnaire included three major sections: (1) College or less 9 (18.4) basic demographic information; (2) evaluation of basic Motivation for attending programa knowledge of critical appraisal; and (3) self-evaluation of Interested 8 (16.3) confidence in appraisal (appraisal of literature on treat- Needed for work 11 (22.4) ment and intervention, systematic reviews, and clinical Recommended 29 (59.2) practice guidelines). The knowledge scale consisted of Clinical nursing ladderb 8 multiple-choice questions, such as “Please arrange the levels of evidence of the results of ‘treatment’ studies Level 1 1 (2.0) from high to low”; and 12 true-or-false questions, such as Level 2 13 (26.5) “If the 95% confidence interval of relative risk includes Level 3 32 (65.3) 0, it indicates that the treatment results are not statisti- Level 4 2 (4.1) cally significant” and “The number needed to treat refers Note. aOne participant had another motivation and selected none of to the number of patients who must be treated to prevent the three choices. bOne participant had a missing response (did not one adverse result (i.e., to promote good results).” The record) on the item regarding the clinical nursing ladder. confidence scale consisted of 27 statements, such as “Is the research literature randomized?” and “This systematic review stated a specific clinical question clearly.” In 49 participants, including four sections: knowledge test, knowledge questions, correct responses were scored as confidence in appraisal of the treatment study, confi- 1 and incorrect responses were scored as 0. The knowl- dence in appraisal of the systematic review, and con- edge scores were then summed and ranged from 0 to fidence in appraisal of the clinical practice guidelines. 20. Internal consistency reliability, measured by Kuder- Postintervention scores for all sections were increased Richardson Formula 20, was 0.5 for dichotomous choice compared with preintervention scores (all p values < items. .001) (Table 2). Confidence statements were scored on a five-point Before the education program, the participants Likert scale, with a score of 5 representing “strongly achieved a mean score of 14.0 (SD = 2.0). The postinter- confident,” or a positive confidence statement, and a vention mean score increased to 15.8 (SD = 1.4), showing score of 1 representing “strongly not confident,” or a a significant improvement in participants’ knowledge of negative confidence statement. The number of confi- appraisal and confidence in their appraisal skills. On the dence statements was 9 (scored 9 to 45), 10 (scored 10 to postintervention questionnaire, participants rated their 50), and 8 (scored 8 to 40). Cronbach’s alpha coefficient confidence in appraisal of the treatment study as moder- values of the scales were 0.94, 0.95, and 0.95. ate (M = 32.4, SD = 5.8) and rated their confidence in appraisal of the systematic review (M = 33.9, SD = 6.6) RESULTS and the clinical practice guidelines (M = 27.1, SD = 4.8) All participants were female. The characteristics of as lower. the 49 study participants are shown in Table 1. Table 2 Participants’ motivations for attending the appraisal summarizes and compares the differences in the scores education program appeared to affect the pre- and post- of the pre- and postintervention questionnaires for the program results. Repeated measure analysis of variance

The Journal of Continuing Education in Nursing · Vol 44, No X, 2013 3 TABLE 2 DIFFERENCES IN SUBCATEGORY SCORES BETWEEN PRE- AND POSTINTERVENTION EVIDENCE-BASED NURSING QUESTIONNAIRES (N = 49) Evidence-Based Nursing Questionnaire Preprogram Postprogram pa Total score of knowledge test, M ± SD 14.0 ± 2.0 15.8 ± 1.4 < .001 Mean change from pre- to postintervention (95% CI) 1.8 (1.2 to 2.4) Confidence in appraisal of treatment study, M ± SD 24.1 ± 8.6 32.4 ± 5.8 < .001 Mean change from pre- to postintervention (95% CI) 8.3 (6.0 to 10.6) Confidence in appraisal of systematic review, M ± SD 23.5 ± 8.8 33.9 ± 6.6 < .001 Mean change from pre- to postintervention (95% CI) 10.4 (8.1 to 12.7) Confidence in appraisal of clinical practice guidelines, M ± SD 18.1 ± 7.9 27.1 ± 4.8 < .001 Mean change from pre- to postprogram (95% CI) 9.0 (6.7 to 11.3) Note. CI = confidence interval.a p values were derived via paired t test of confidence in appraisal of the treatment study and confidence in appraisal of the systematic review; Wilcoxon signed-rank test was used for total scores of knowledge and confidence of appraisal of the clinical practice guidelines as a result of abnormal distribution.

ever, no significant effects were found on confidence in appraisal skills by the types of motivation. Participants’ age, education, and level on the clinical nursing ladder did not influence their mean knowledge or confidence scores.

DISCUSSION Nurses who had been recommended to take the education program showed greater improvement in knowledge and confidence scores than those who took the education program because of their own interest in appraisal or because they believed that they needed appraisal skills for work. Those recommended to take the education program may have been given the responsibil- Figure. Pre- to postintervention change in total score of knowledge ity to implement evidence-based nursing in their wards test according to motivation. Data are shown as a bar chart showing M ± SD. Differences between pre- and postintervention scores or to serve as tutors in hospital-led education programs, for each type of motivation were compared with the Wilcoxon and this may have motivated them to expend greater ef- signed-rank test. ***p < .001, indicating significant differences fort and to learn more effectively during the program. between pre- and postintervention scores. Total score on the The intent of the appraisal education program re- knowledge test was evaluated by participants’ motives—interested: 15.8 ± 1.8 vs. 16.1 ± 1.9, p = .317; needed for work: 14.4 ± 1.4 vs. ported in this study was to increase nurses’ knowledge 15.5 ± 1.6, p = .136; recommended: 13.4 ± 2.0 vs. 15.8 ± 1.2, p of critical appraisal as well as to increase their confidence < .001 (n = 48; one study subject with another motivation was not in their appraisal skills. The most effective appraisal edu- included). cation programs integrate lectures, practice, and small group discussion, which enhance the appraisal compe- shows that the total score on the knowledge test might tencies of nurses (Bernardo et al., 2008; Bradley et al., be associated with participants’ motivation for attending 2005; Brancato, 2006). the education program. The changes in the total scores Therefore, this education program included lectures, on the knowledge test from pre- to postintervention practice sessions, and small group discussion that fo- according to motivation are shown in the Figure. The cused on critical appraisal of treatment studies, system- participants who were recommended to attend the edu- atic reviews, and clinical practice guidelines. cation program showed a significant improvement (pre- A review of the literature found that perceived lack of intervention 13.4 ± 2.0 vs. postintervention 15.8 ± 1.2, time is the main barrier for nurses to access information p < .001) in their knowledge of critical appraisal. How- and apply evidence-based approaches in their clinical

4 Copyright © SLACK Incorporated care (Bertulis, 2008). Other barriers include insufficient computer skills to search for studies, poor search and key points appraisal skills, poor understanding of statistics, and high pressure on nursing staff in most clinical settings Critical Appraisal Program Chang, S.-C., Huang, C.-Y., Chen, S.-Y., Liao, Y.-C., Lin, C.-H., Wang, (Bertulis, 2008). Gerrish, Ashworth, Lacey, and Bai- H.-H. (2013). Evaluation of a Critical Appraisal Program for ley (2008) also reported barriers to access to evidence- Clinical Nurses: A Controlled Before-and-After Study. The Jour- based information, such as nurses’ lack of confidence nal of Continuing Education in Nursing, 44(X), xxx-xxx. and lack of authority. Tagney and Haines (2009) stated that, to ensure that quality evidence-based practice is Critical appraisal of relevant published literature is an essential implemented into clinical nursing care, a “realistic and 1 competence for nurses and is regarded as the standard of health practical structure” must be applied. Implementing a care by medical professionals. realistic research framework into clinical nursing prac- A well-designed critical appraisal education program combines tice may lead to a more active research culture (Tagney lectures, practice sessions, and group discussions focused on & Haines, 2009). Continuing education remains an ac- 2 critical appraisal of treatment studies, systematic reviews, and cepted and effective way to promote evidence-based clinical practice guidelines. practice. Problem-based learning and models based on so- A 1-day, small-group discussion education program can ef- cial learning theory have been effective in transferring 3 fectively improve nurses’ knowledge of critical appraisal and knowledge to health care personnel. Social learning confidence in their critical appraisal skills. theory increases learners’ incentives by applying interactive lectures, clear educational goals, stepwise educational programs, interactive group work, and feedback limitation was that only short-term study outcomes (Bradley et al., 2005). Results of a 1-day workshop were evaluated, so the long-terms effects would not be showed that a short-term intervention could enhance validated. A more comprehensive evidence-based prac- school nurses’ appraisal skills (Bernardo et al., 2008). tice education program, including 3- to 6-month and Compared with other programs, the duration of the even 1-year follow-up, should be designed to examine program reported in this study was short. Nevertheless, the long-term effects of the education program. the results suggest that the benefits of the education program could be achieved in the short term, which is CONCLUSION important for busy clinical nurses who report that they A 1-day education program was an efficient and ef- have inadequate time to attend a long-term education fective way to improve nurses’ knowledge of critical program. After the education program, participants appraisal and confidence in their critical appraisal skills. responded that most of the sessions met or exceeded Education programs may help and encourage nurses to their expectations and had improved their abilities. In engage in evidence-based nursing practice and help clini- addition, the participants reported that the critical ap- cal professionals to overcome commonly reported barri- praisal users’ guidelines provided a convenient tool to ers to applying evidence-based practice. Administrators increase their awareness of appraisal. These guidelines, can provide evidence-based practice education programs which are based on research and include an appraisal specifically designed for nurses instead of those designed checklist, can help participants to appraise different as- for physicians. A comprehensive education program can pects of research. be included in the advanced curriculum of continuing Although nurses did not completely understand all nursing education. Clinicians who have critical appraisal of the research terminology, their pre- and postinterven- skills will be able to determine the importance of research tion scores indicated improved knowledge and improved studies and improve their clinical decision making. confidence in their critical appraisal skills. Clearly, if clinical nurses are willing to apply critical appraisal of REFERENCES evidence-based medicine in their daily clinical practice, Bernardo, L. M., Matthews, J. T., Kaufmann, J. A., & Yang, K. (2008). more appraisal-related programs must be developed to Promoting critical appraisal of the research literature: A workshop enhance their competency in this respect. for school nurses. The Journal of Continuing Education in Nursing, 39(10), 461-467. One limitation of this study was that the research Bertulis, R. (2008). Barriers to accessing evidence-based information. design contained only a single group and did not in- Nursing Standard, 22(36), 35-39. clude random assignment of participants. This would Bradley, P., Nordheim, L., Innvær, S., & Thompson, C. (2005). A sys- limit the applicability of the study outcomes. The other tematic review of qualitative literature on educational interventions

The Journal of Continuing Education in Nursing · Vol 44, No X, 2013 5 for evidence-based practice. Learning in Health & Social Care, evidence-based practice: Experiences of senior and junior clinical 4(2), 89-109. nurses. Journal of Advanced Nursing, 62, 62-73. Brancato, V. C. (2006). An innovative clinical practicum to teach evi- Guyatt, G., & Rennie, D. (2008). Users’ guides to the medical literature dence-based practice. Nurse Educator, 31(5), 195-199. (2nd ed.). New York, NY: McGraw-Hill. Chiu, Y. W., Weng, Y. H., Lo, H. L., Hsu, C. C., Shih, Y. H., & Kuo, K. Harewood, G. C., & Hendrick, L. M. (2010). Prospective, controlled N. (2010). Comparison of evidence-based practice between physi- assessment of the impact of formal evidence-based medicine teach- cians and nurses: A national survey of regional hospitals in Taiwan. ing workshop on ability to appraise the medical literature. Irish Journal of Continuing Education for Health Professionals, 30(2), Journal of Medical Science, 179(1), 91-94. 132-138. Hung, S. Y., Ku, Y. C., & Chien, J. C. (2012). Understanding physi- Chiu, Y. W., Weng, Y. H., Lo, H. L., Shih, Y. H., Hsu, C. C., & Kuo, K. cians’ acceptance of the Medline system for practicing evidence- N. (2010). Impact of a nationwide outreach program on the diffu- based medicine: A decomposed TPB model. International Journal sion of evidence-based practice in Taiwan. International Journal of of Medical Informatics, 81(2), 130-142. Quality Health Care, 22(5), 430-436. Leu, L. J., Liao, H. C., Chang, I. C., & Su, Z. Y. (2009). Applying non- Cullum, N. (2000). Evaluation of studies of treatment or prevention synchronized e-learning to the nursing clinical ladder system. Journal interventions. Evidence-Based Nursing, 3(4),100-102. of Medical Systems, 34(5), 909-917. doi:10.1007/s10916-009-9306-5 Cullum, N. (2001). Evaluation of studies of treatment or prevention in- Milne, D. J., Krishnasamy, M., Johnston, L., & Aranda, S. (2007). Pro- terventions: Part 2. Applying the results of studies to your patients. moting evidence-based care through a clinical research fellowship Evidence-Based Nursing, 4(1), 7-8. programme. Journal of Clinical Nursing, 16(9), 1629-1639. Duffy, J. R. (2005). Critically appraising quantitative research. Nursing Tagney, J., & Haines, C. (2009). Using evidence-based practice to ad- and Health Sciences, 7(4), 281-283. dress gaps in nursing knowledge. British Journal of Nursing, 18(8), Gerrish, K., Ashworth, P., Lacey, A., & Bailey, J. (2008). Developing 484-489.