Competence Assessment Instruments in Perianesthesia Nursing Care

Competence Assessment Instruments in Perianesthesia Nursing Care: A Scoping Review of the Literature Yunsuk Jeon, MSc, RN, Riitta-Liisa Lakanmaa, PhD, RN, Riitta Meretoja, PhD, RN, Helena Leino-Kilpi, PhD, RN, FEANS

Purpose: To identify competence assessment instruments in perianesthesia nursing care and to describe the validity and reliability of the instruments. Design: A scoping review in a systematic manner. Methods: A search in CINAHL, MEDLINE, and ERIC was carried out to identify empirical studies from 1994 to 2015. A narrative synthesis approach was undertaken to analyze the data. Findings: Nine competence assessment instruments in perianesthesia nursing care were identiﬁed. The instruments used three types of data collection methods: Self-report, observation, and written examinations. The most commonly reported validity method was content validity involving expert panels and reliability tests for internal consistency and inter-rater’s consistency. Conclusions: Integrating more than one data collection method may give support to overcoming some of the limitations, such as lack of objectivity and misinterpretation of the assessment results. In an ever- changing environment, perianesthesia nursing competence requires constant reassessment from the perspective of content validity, scoring methods, and reliability. Keywords: competence assessment instrument, perianesthesia nursing care, literature review. Ó 2016 by American Society of PeriAnesthesia Nurses

PERIANESTHESIA NURSING CARE is commonly Yunsuk Jeon, PhD Candidate, Clinical Researcher, MSc, RN, regarded as a nursing specialty concerned with Department of Nursing Science, University of Turku, Finland; providing nursing care to patients undergoing or Corporate Headquarters, Helsinki University Hospital, Hel- recovering from anesthesia. Internationally, in a sinki, Finland; Riitta-Liisa Lakanmaa, PhD, RN, Senior large umbrella of the perianesthesia specialty, peri- Lecturer, Department of Nursing Science, University of Turku, anesthesia nursing includes planning and caring Finland; Turku University of Applied Sciences, Turku, Finland; Riitta Meretoja, PhD, RN, Development Manager, for preanesthesia, intra-anesthesia, and postanes- Adjunct Professor, Department of Nursing Science, University thesia patients until the patient is discharged into of Turku, Finland; Corporate Headquarters, Helsinki Univer- a ward setting or home.1,2 Furthermore, sity Hospital, Helsinki, Finland; and Helena Leino-Kilpi, PhD, perianesthesia nursing care can be expanded RN, FEANS, Professor and Chair, Nurse Director (part-time), from the operating theater environment to Department of Nursing Science, University of Turku, Finland; Turku University Hospital, Turku, Finland. various clinical settings such as diagnostic, Conflict of interest: None to report. therapeutic, obstetrical procedures, and pain Address correspondence to Yunsuk Jeon, Department of management.1,2 In various environments, Nursing Science, University of Turku, Joukahaisenkatu 3-5, perianesthesia nurses are expected to be experts, 20014 Turku, Finland; e-mail address: yunsuk.jeon@utu.fi. to be able to make flawless and rapid judgments, Ó 2016 by American Society of PeriAnesthesia Nurses 1089-9472/$36.00 and to deal professionally with the ethical 2-4 http://dx.doi.org/10.1016/j.jopan.2016.09.008 issues. Therefore, they require a set of

542 Journal of PeriAnesthesia Nursing, Vol 32, No 6 (December), 2017: pp 542-556 PERIANESTHESIA COMPETENCE ASSESSMENT INSTRUMENTS 543 specialized knowledge and skills to satisfy the competence assessment instruments in perianes- complicated needs of a patient in perianesthesia thesia nursing care and to describe the validity care.1,5 and reliability of the instruments.

International council of nurses defined competence Methods as ‘‘the effective application of a combination of knowledge, skill and judgment demonstrated A scoping review refers to a process of mapping or by an individual in daily practice or job perfor- summarizing the existing literature to understand mance.6’’ In the case of the United States of the range of the field.15-19 In this study, a scoping America, nurses in perianesthesia care periodically literature review methodology based on the take a certain number of supplementary courses, framework (five stages) outlined by Arksey and continuing education credits, or web-based O’Malley20 was undertaken to identify the litera- learning to maintain their certification.3,7,8 In ture available in the field of perianesthesia nursing Europe, there is no specific regulation for care. recertification of nurses in perianesthesia care and nursing training varies a great deal both nationally Stage 1. Identifying the Research Questions and internationally.9 Therefore, it is essential to provide a more accurate assurance when evaluating A scoping review was carried out to answer whether a professional has achieved an acceptable following questions: level of knowledge and skills.10 1. What instruments were used to assess competence in perianesthesia nursing care? To provide safe care for patients and maintain the 2. How were the validity and reliability of the credibility of nurses by providing the capabilities assessment instruments reported in the liter- to correspond to dynamic circumstances, a system ature? of competence assessment in perianesthesia care is demanded. However, the competence assessment instruments currently used in perianesthesia Stage 2. Identifying Relevant Studies nursing education and practice may lack validity or reliability, or both. A competence assessment in- A literature search was carried out using the elec- strument developed by a valid and reliable process tronic databases, CINAHL, MEDLINE (Ovid), and would be beneficial for a more rigorous evaluation ERIC in September, 2015. Single and combined of the competence of nurses and enable the iden- search terms included: competen*, assess*, tification of deficiencies in professional develop- scale*, tool*, evaluat*, measure*, nurs*, peri- ment and educational needs.11-14 an(a)esth*, prean(a)esth*, intraan(a)esth*, postan(a)esth*, an(a)esthe*, PACU, perioper- To date, little information is available regarding ati*, preoperati*, intraoperati*, postoperati*, what competence assessment instruments exist recovery, valid*, and reliab*. To uncover any in perianesthesia nursing care, and how valid and additional publications or gray areas in the litera- reliable they are. Thus, comprehensive research ture, hand searching of reference lists for key arti- focusing on competence assessment in perianes- cles and a search using a general internet browser thesia nursing care as well as the validity and reli- (Google Scholar) was undertaken. The search ability of the assessment instruments is necessary. covered all countries, but only the English language For this purpose, a scoping review methodology from the last two decades (1994 to 2015) were has been conducted to provide an overview of included. During this period, nursing education the current literature related to competence assess- has been under reform in Europe and research con- ment in perianesthesia nursing care. cerning competence in nursing has increased.21

Purpose Stage 3. Study Selection

The purpose of this review was to describe how For the study selection, the scoping review adop- competence in perianesthesia nursing care has ted similar methods to a systematic review. Refer- been assessed. The review aimed to identify ences were included if they measured any 544 JEON ET AL aspect of the scoping in relation to nurses’ compe- Stage 4. Charting the Data tence in perianesthesia care on the basis of the inclusion criteria (Table 1). In the search, The charting of the studies provided an overview perianesthesia nursing care was operationally of the existing literature. A spreadsheet was defined as nursing care related to patients undergo- created to chart relevant data based on the focus ing procedures requiring sedation, analgesia, and of the scoping question (Y.J.). To identify the range anesthesia in the operating unit, anesthesia proce- of the study, 13 articles were summarized by the dural areas, or postanesthesia care units (PACUs). author with regards to year of publication, country, purpose of study, design/sample/participant/ The initial search generated 232 research articles. ethics, instrument, and key findings. On the basis Two researchers (Y.J. and R.L.) independently re- of the summary of the data, competence assess- viewed and applied the selection criteria to all ti- ment instruments were identified and charted spe- tles and abstracts. During the title and abstract cifically by a data collection method, the domain/ screening process, references were marked as category (item), a scale/scoring, and the validity either ‘‘include,’’ ‘‘unclear,’’ or ‘‘excluded,’’ and the and reliability of the instruments. former two categories marked on the abstract screening were included for the full-text review. A scoping study tends to present broader themes After the titles (deletion of 132 references, which for which the various types of study might be did not meet the inclusion criteria based on the ti- applicable and is less likely to seek to assess the 20 tle) and abstracts (deletion of 71 references, which quality of the studies included. As a scoping did not meet the inclusion criteria based on the ab- study, this review did not conduct a quality stracts) of the articles were scrutinized, 19 articles appraisal process. Instead of a quality appraisal of remained for the full-text review. In addition, 18 the studies included, this review focused on references identified by manual searches were describing the quality of the instruments identified included for full-text review to determine their to find the answers to the research questions. study eligibility. In all, 37 articles were assessed as relevant based on the inclusion criteria. After re- Stage 5. Collating, Summarizing, and viewing the full text, 24 references were excluded Reporting of the Results because their participants did not comprise perianesthesia nurses (n 5 4), they were evaluations of On the basis of the charting, commonalities, teaching methods or orientation programs (n 5 themes, and gaps in the literature were identified. 9), they were not empirical studies but guidelines A narrative synthesis approach was undertaken to or standards (n 5 5), and they did not evaluate analyze the data. To summarize the validities and competence in perianesthesia nursing (n 5 6). reliabilities of the instruments, the ‘‘Guidelines re- Finally, 13 articles met the selection criteria and porting the psychometric soundness of instru- 22 were included in the analysis (Figure 1). ments’’ were used. One (Y.J.) of the researchers

Table 1. Inclusion and Exclusion Criteria

Review question Inclusion Criteria Exclusion Criteria Participants Perianesthesia nurse, anesthesia nurse/nurse Operating room nurses without anesthesia anesthetist, anesthesiology nurse, anesthesia nurses (eg, focus only on scrub nurses and/or care team, OR nurse, student nurse circulation nurses) anesthetist, recovery room nurse, Anesthesia care team without anesthesia nurses postanesthesia care unit nurse, perioperative (eg, focus only on physicians), air force nurse nurse anesthetists Interest Perianesthesia nurse’s competence assessment Assessment instrument for other purposes (eg, instrument competence of preceptors or competence- based education) Study design Original empirical studies and mixed method, Literature reviews, editorial, discussion articles, instrument validation studies guideline, standards, qualitative study PERIANESTHESIA COMPETENCE ASSESSMENT INSTRUMENTS 545

Records idenƟﬁed through database searching MEDLINE = 60 CINAHL = 127 ERIC =45 (n =232) Manual search (n=18)

Duplicates excluded (n =10)

Titles screened (n =222) Records excluded based on the Ɵtle (n =132)

Abstracts screened (n =90) Records excluded based on the abstract (n =71)

Full text arƟcles assessed for eligibility Full texts excluded (n =24) (n=19+18=37)

Included in review (n = 13)

Figure 1. Flowchart of the search and selection process. charted the data, collated, and summarized the of PACU nurses. Six articles measured the compe- commonalities, themes, and gaps. The consistency tence of nurse anesthetists or nurse anesthetist stu- and accuracy of the summary were checked by dents. Six articles focused on perioperative nurses’ other researchers (R.L., R.M., and H.L.). Any competence. Eleven articles reported the ethical changes were discussed by all the researchers, aspects such as voluntary nature, informed con- and a ﬁnal decision arrived at through consensus. sent, conﬁdentiality, anonymity, and ethics committee’s approval, whereas two did not clarify Results the ethical issues.

Thirteen relevant articles were reviewed (Table 2). As a result of the analysis of these 13 articles, nine Among the reviewed articles, seven articles were competence assessment instruments were ﬁnally from the United States, two from Australia, two identiﬁed. from Finland, and two from the UK. Ten of the total 13 articles reported were within the last 5 years. To examine the nature and methodological This indicates that the concern for competence features of the instruments, nine instruments assessment in perianesthesia care has increased were described by types of data collection since 2010. One article measured the competence methods, the domain/category, number of items, Table 2. Summary of the 13 Included Studies 546

Design/Sample Authors/Year/Country Purpose (Participant) Ethics Instrument Key Findings Collins and Callahan/2014/ To test a clinical evaluation tool Ex post facto cross-sectional Clinical evaluation Instrument 17-Item clinical evaluation tool United States30 in terms of validity and to study design/N 5 137 measured only three identify the relationship (certified registered nurse underlying constructs, between the clinical anesthetist (CRNA) technical skill, patient focus evaluation score and NCE students) and evaluators concept, and manage (National Certification (N 5 not mentioned)/ethical resource. No correlation Examination) score issues: Not mentioned between clinical evaluation score and NCE Cook et al/2013/United States29 To assess recent CRNA Cross-sectional survey Professional competence Most of new graduates enter graduate’s preparation and design/N 5 560 into nurse anesthesia performance for entry into (CRNAs) and N 5 696 practice prepared with the practice (employers)/ethical issues: required knowledge and Not mentioned skills to practice as safe, competent providers Gaba et al/1998/United States31 To assess anesthesia care Quasi experimental Crisis management behaviors Successful at implementing provider’s performance on study/N 5 72 (residents, tool appropriate technical action . technical skills and behavior faculties, and Technical action check-list in general performing ( 80% when responding to critical CRNAs) and N 5 5 of checklist). Crisis events (evaluators)/ethical issues: management behavior varied Anonymity and informed with some teams rated as consent minimally acceptable or poor Gabriel/2013/United States7 To determine relationships Validation studies of three Knowledge examination Written examination 67% between written measurements/N 5 18 Mini-CEX (clinical exercise) correct. Simulation 5 examination scores, (CRNAs) and N 2 Technical action checklist performance 77.28%. self-assessment score, and (evaluators)/ethical issues: Negative correlation between performance scores in a Voluntary participation and written examination and simulated environment informed consent performance scores in simulation Gillespie et al/2011/Australia23 To describe the influence of Cross sectional survey/N 5 345 PCS-R (Perceived Competence More experienced nurses and years of operating room (OR) (perioperative nurses) Scale-Revised) those with specialty experience and specialty ethical issues: Voluntary education reported higher education on nurses’ participation, anonymity, competence scores ENE AL ET JEON perioperative competence informed consent, and ethics committee (Continued) EINSHSACMEEC SESETINSTRUMENTS ASSESSMENT COMPETENCE PERIANESTHESIA Table 2. Continued

Design/Sample Authors/Year/Country Purpose (Participant) Ethics Instrument Key Findings Gillespie et al/2012/Australia24 To describe the development Cross sectional survey PPCS-R (perceived 40 Items were grouping six and validation of the revised design/N 5 1,138 perioperative competence dimensions. OR nurses were perioperative competence (perioperative nurses)/ scale-revised) more likely to perceive high scale ethical issues: Ethics level perioperative committee’s approval, competence (total score anonymity, voluntary 174.7(min 0-max 200) nature, and informed consent Gillespie and Pearson/2013/ To compare operating Cross-sectional survey/N 5 214 PPCS-R Both groups reported their UK25 department practitioner (perioperative nurses)/ competence high across all (ODP) and operating theater ethical issues: Informed subscales. There were (OT) nurses’ perception of consent and ethics significant differences in their perioperative committee’s approval dimension, foundational competence knowledge and skill, and empathy between two groups Greenfield et al/2014/United To identify encounters with 14 Cross-sectional survey NCS (nurse competence scale) Surgical PACU nurses showed States28 topics (eg, basic life support design/N 5 54 (obstetric high encounters each topic and advanced cardiac life postanesthesia care unit in their practice and support) related to (PACU) nurses) and N 5 68 indicated high competency postanesthesia and assess (surgical PACU nurses)/ perianesthesia nurses’ ethical issues: Voluntary competence nature, confidential, and anonymity Henrichs et al/2009/United To determine whether Prospective, randomized, Technical action checklist CRNA group achieved States33 experienced anesthesia single-blinded study/N 5 61 maximum scores on teams have comparable skill (anesthesiologists and bronchospasm and loss of levels in managing acute CRNAs) and N 5 2 pipeline oxygen, whereas conditions (evaluators)/ethical issues: they had difficulty in MH Ethics committee’s approval (malignant hyperthermia) and informed consent and hyperkalemia Meretoja et al/2004/Finland26 To examine nurses’ perception Cross-sectional survey/N 5 498 NCS Operation room unit nurses of competence in different (perioperative nurses) showed high competence university hospital work ethical issues: Informed level in managing situations, environments consent, voluntary nature, whereas diagnostic and anonymity, confidentiality, teaching-coaching was lower 547 and ethics committee’s than the ward nurses approval (Continued) 548 Table 2. Continued

Design/Sample Authors/Year/Country Purpose (Participant) Ethics Instrument Key Findings Meretoja and Koponen/2012/ To develop a model to compare Qualitative and quantitative/ NCS Optimal competence was Finland27 nurses’ optimal and actual N 5 24 (experts), N 5 87 higher than the nurses’ competencies in the clinical (perioperative nurses), and self-reported actual setting N 5 88 (nurse managers)/ competence and nurse ethical issues: Informed manager’s assessed level of consent, voluntary nature, actual competence and hospital approval Murray et al/2005/United To evaluate scenario content, to Validation test of a simulation Technical action checklist Most educated and experienced States32 provide further validation of based acute care received high scores, a simulation based acute care assessment/N 5 58 (nurse providing to support the assessment, and to compare anesthetist students and validity of simulation the acute care skills of residents) and N 5 6 anesthesia trainees (evaluators)/ethical issues: Ethics approval for the protocol Robertson et al/2014/UK34 To describes and evaluate the Observation study in real Oxford NOTECHS II Most operations were new scale of nontechnical setting/N 5 297 (OR cases) performed by skill (NOTECHS II) of an and N 5 2 (evaluators)/ well-coordinated and entire operating theater team ethical issues: Ethics functioning teams committee’s approval and informed consent ENE AL ET JEON PERIANESTHESIA COMPETENCE ASSESSMENT INSTRUMENTS 549 the scale/scoring system, validity (content, crite- inter-rater consistency were commonly reported rion, and construct), and reliability (internal con- for reliability tests. Four instruments reported sistency, inter-rater, and intra-rater/test-retest) Cronbach’s coefficient as the internal consis- (Table 3). tency7,23-27 and the direct observation instrument stressed inter-rater reliability among evalua- Three types of data collection methods were iden- tors.7,31-34 Not all instruments documented tified: Self-reporting, observation (direct or validity and reliability. Two instruments used in reflect), and written examinations. Four assess- simulated settings paid attention to reliability ment instruments used a self-report method, rather than validity. One instrument reported which was the most frequently conducted data neither validity nor reliability tests.29 collection method in this review.7,23-29 In addition to self-reporting, two of the four instru- Discussion ments used preceptors or managers’ assessment by reflective observation.27,29 One instrument A scoping review was undertaken to describe what used a reflective observation method alone competence assessment instruments exist in assessed by faculty members.30 Two assessment in- perianesthesia nursing care, and how valid and reli- struments used a direct observation method in able they are. In the review, three assessment simulated situations,7,31-33 and one instrument in instruments, nurse competence scale (NCS) and a real clinical setting.34 One instrument was de- perceived perioperative competence scale - Revised signed as a written examination to test the knowl- (PPCS-R), and nontechnical skills (NOTECHS II) edge of nurse anesthetists7 (Table 3). were considered as psychometrically sound as they clarified the issues of validity and reliability suf- Most of the instruments were composed of several ficiently (Table 3). Because these assessment instru- competence areas called domains, categories, ments vary in content, purpose, and data dimensions, or competencies. As the result of measurement method, the results of the narrative charting the domains of competence from nine in- synthesis showed that perianesthesia nursing care struments, frequently measured domains were still requires sound competence assessment instru- ‘‘managing situation,’’ ‘‘collaboration/team work,’’ ments from the perspective of validity and reliability. ‘‘knowledge,’’ ‘‘practice skills,’’ and ‘‘assessing/judgment.’’ The range of the number of items was be- Competence Assessment Instruments in tween 3 and 73. The types of scales used were Perianesthesia Nursing the Likert scale and ordinary scale (n 5 6), visual analog scale (n 5 1), a binary scale (n 5 1), and The NCS measured the competence of operating multiple choice (n 5 1). room nurses.26 Because NCS was originally developed to measure the generic competence of prac- In the process of using competence assessment ticing nurses in different phases of their career and instruments, many studies took into account issues in a variety of clinical settings, it might be an appro- related to validity and reliability (Table 3). In this re- priate instrument to compare competence of peri- view,validity was described from the perspective of anesthesia nursing with those of other fields of content, criterion, and construct validity. The most nursing. Gillespie et al23-25 benchmarked the frequently reported content validity method was minimum standards of clinical performance from the Delphi technique (n 5 4).7,23-28,34 Two types generic competence assessments and developed of criterion validity were predictive validity (n 5 a self-assessment tool, the PPCS-R, which enables 1)30 and concurrent validity (n 5 1).34 Construct the special contexts of perioperative care to be validity was tested by several techniques such as depicted. Perioperative nursing is a complex field exploratory factor analysis (n 5 1),23-25 where perianesthesia nurses strive to cover surgi- confirmatory factor analysis (n 5 2),23-25,30 and cal intervention, anesthesiology, and post anes- principal component analysis (n 5 1).23-25 thetic care. In addition, perianesthesia nurses practice in collaboration with other health profes- For the reliability test, this review focused on sionals such as other operating room nurses, internal consistency, inter-rater, and intrarater/ surgeons, and anesthesiologists. Therefore, the test-retest technique. Internal consistency and PPCS-R is a likely instrument to assess competence Table 3. Psychometric Properties of the Competence Assessment Instruments 550

Validity Reliability

Instrument (Data Domain/ Intrarater Collection Category Internal Test- Method) (Item) Scale/Scoring Content Criterion Construct Consistency Inter-rater retest Articles

Mini-clinical Seven domains (7): 9-Point scale: 1-3 — — — 0.95 — — Gabriel 20137 exercise (S) Medical Unsatisfactory, 4-5 interviewing skills, satisfactory, 7-9 physical superior examination skills, humanistic qualities/ professionalism, clinical judgment, counseling skills, organization/ efficiency, overall clinical competences PPCS-R Six domains (40): 5 Likert: 1 5 Never, to Delphi, — PCA, EFA, CFA 0.95-0.98 — — Gillespie et al 201123 , (perceived Foundational skills 5 5 always CVI 0.97 Gillespie et al perioperative and knowledge (9), 201224 , Gillespie competence leadership (8), and Pearson 201325 scale-revised) (S) collaboration (6), empathy (5), proficiency (6), professional development (6) NCS (nurse Seven competence Visual Analog Scale Content Refer to Refer to previous Good — — Meretoja et al 200426 , competence categories (73): (0-100): 0 5 A analysis, previous studies Meretoja and scale) (S 1 Helping role (7), very low level, Delphi studies Koponen 201227 , RO by teaching-coaching 100 5 very high Greenfield et al managers) (16), diagnostic level. Frequency of 201428 function (7), use 0 5 not managing applicable. situations (8), 1 5 very seldom to therapeutic 3 5 very often in interventions (10), my work AL ET JEON ensuring quality (6), and work role (19) (Continued) EINSHSACMEEC SESETINSTRUMENTS ASSESSMENT COMPETENCE PERIANESTHESIA Table 3. Continued

Validity Reliability

Instrument (Data Domain/ Intrarater Collection Category Internal Test- Method) (Item) Scale/Scoring Content Criterion Construct Consistency Inter-rater retest Articles

Professional Clinical practice, Five Likert scale — — — — — — Cook et al 201329 competence independence, (S 1 RO by knowledge, employers) teamwork, judgment/critical thinking (17) Clinical Not clearly explained. Four Likert scale: — PRE, with CFA — — — Collins and Evaluation Patient assessment 1 5 Failure, National Callahan 201430 Instrument and anesthetic 2 5 below Certiﬁcation (RO by faculty plan, didactic expectations, Examination members) transfer of 3 5 meets knowledge, expectations perianesthetic 4 5 above management, expectations communication skills/professional role, and care and equipment (17) Crisis management 10 Domains (12): Five-point ordinary — — — — Satisfactory — Gaba et al 199831 behaviors tool Orientation to scale: 1 5 Poor, (DO in case, inquiry/ 2 5 minimally simulation) assertion, acceptable, communication, 3 5 standard, feedback, 4 5 good, and leadership, group 5 5 outstanding climate, anticipation/ planning, work load distribution, vigilance, and re- evaluation (Continued) 551 552 Table 3. Continued

Validity Reliability

Instrument (Data Domain/ Intrarater Collection Category Internal Test- Method) (Item) Scale/Scoring Content Criterion Construct Consistency Inter-rater retest Articles

NOTECHS II Four domains (16): Eight-point scale Delphi CON Tested — Good — Robertson (Nontechnical Leadership and et al 201433 Skills) (DO) management, teamwork and cooperation, problem solving and decision- making, situation awareness Technical action Items depend on the Binary scale: — — — — Good, — Gaba et al 199831 , checklist (DO scenarios (3-7): eg, 0 5 Absence, 0.80 Henrichs et al in simulation) bronchospasm (5) 1 5 present 200933 Acute Hemorrhage 0.85 Gabriel 20137 (6) Hyperkalemia (6) — Moderately Murray reliable et al 200532 Knowledge Airway management, Multiple choice Delphi — — 0.60 — Tested Gabriel 20137 examination clinical (W) pharmacology, physiology, and anesthesia technology (30) CFA, conﬁrmatory factor analysis; CON, concurrent validity; CVI, content validity index; EFA, exploratory factor analysis; PCA, principal component analysis; PRE, predictive validity. Data collection method: (S) 5 self-report, (RO) 5 reﬂective observation, (DO) 5 direct observation, (W) 5 written examination. ENE AL ET JEON PERIANESTHESIA COMPETENCE ASSESSMENT INSTRUMENTS 553 in the nature of perioperative care consisting of and to clarify the ethical issues such as recruiting not only foundational knowledge and skills but participants, consent, approval process, recoding, also collaboration, empathy, and leadership. How- and storing data. All the studies considered that ever, further validation tests are needed to increase used the direct observation method did take into the accuracy of the instruments to measure core account such ethical issues and described them competence in perianesthesia nursing. in detail.7,31-34

Self-directed assessment has a weakness because In the nursing context, there is a tendency to of the fact that there is little public account- consider that knowledge is equal to competence; ability.35 In response to this weakness, and as an nurses who have a high score in written tests alternative to self-reporting, NOTECHS II used were treated as being competent and having met direct observation to evaluate nontechnical skills the approved standards despite no measurement such as teamwork and cooperation, leadership being made of their actual performance. Researches and management, problem solving, decision- pointed out that the correlation between written making, and situation awareness in the real clinical examinations and performance scores is low.38,39 setting.34 Critical incidents caused by insufficient Gabriel7 supported this finding by showing a low competence in nontechnical skills have been to moderate correlation between written examina- paid increasing attention. NOTECHS II reported tion scores and performance scores in competence validity and reliability appropriately, and it can be measurement. This is evidence that competence used as an important instrument to measure assessment requires multiple assessment methods nontechnical skills in perianesthesia nursing. to reduce the possibility of misinterpreting the result and to increase the validity of the measure- In addition to the real clinical setting, there are two ment tool. However, combined approaches need instruments (crisis management behaviors tool more time and resources, and compound-errors and technical action checklist) used for direct might occur when analyzing the results.36 observation to assess nurses’ performance with regard to their technical skills and behavior in Validity and Reliability of Competence 7,31-33 a simulated environment. In a simulated Assessment Instruments setting, observers or examiners evaluate nurse’s performance as an outcome of nurse’s One of the essential issues in evaluating the quality competence in simulated conditions.36 One of of an assessment instrument is validity; the ability the advantages of a simulated environment is that to which an instrument measures what it is in- it enables assessment of a nurse’s skills and tended to measure.22 In this review, content, crite- behavior in recognizing and responding to crucial rion, and construct-related validity were changes such as patients in critical conditions, considered to describe the validity of the instru- problem solving, and communication manage- ments. Content validity involves the processes of ment.7 However, the simulated setting is not natu- pooling instrument items through searching the ral and a nurse may not perform in the same way as literature, seeking expert opinions (Delphi they would in real situation.37 Furthermore, peria- method), pilot testing, or qualitative research.22 nesthesia professionals have an integrated ability The most commonly reported content validity as regards knowledge, skills, and roles that encom- was seeking expert opinion (Delphi method). passes the preoperative phases to the postopera- However, it was noted that the Delphi method tive phases, where they work together as a team used in this study was not explained clearly in with other health professionals. When considering terms of the process of reduction of items and competence assessment in a simulated environ- the agreement among expert panel’s opinion. Clar- ment, examiners need to plan carefully how well ifying the Delphi method with objective measures the use of simulation can be controlled and how such as content validity ratio or content validity in- well the assessment outcome matches real peria- dex is recommended in reporting content validity nesthesia nursing practice. Because the subject (eg, content validity index 5 0.97).24 of a competence assessment study using a direct observation method is human, it is important to The criterion validity is divided into two types of conduct studies with ethically accepted methods validity: Concurrent validity and predictive 554 JEON ET AL validity. Concurrent validity indicates a measure of to good).7,31,33,34 However, the assessment how one test correlates with another test that mea- results might fluctuate by the consistency within sures the same variable.40 Predictive validity indi- an evaluator (intra-raters’ reliability). The cates a measure of how one test predicts an consistencies of the intrarater reliability were outcome based on information from other test.40 tested by one study.32 For instance, Collins and Callahan30 tested whether student nurse anesthetist’s clinical scores It was noted that four instruments (four of nine) measured by the ‘‘clinical evaluation instrument’’ did not address either the validity or reliability or were predictive of National Certification Examina- both. Other literature review studies concerning tion. It found that the instrument did not signifi- assessment had the same difficulty of a lack of in- cantly correlate with the National Certification formation on validity and reliability.40,42 The Examination scores, and a further test may be clarification as how to develop and test the needed. instruments in terms of validity and reliability is essential information to determine the quality of Construct validity is known as one of most difficult the instruments. To minimize the probability of processes because it focuses on the theoretical missing information and to clarify the meanings of measurements, the logical relation- instrument’s validity and reliability systematically, ship between one concept and other concepts, the use of structured reporting framework such and the link of theory with the empirical world.41 as Guidelines Reporting the Psychometric Collin and Callahan30 showed the importance of Soundness22 or consensus-based standards for construct validity tests. They conducted factor the selection of health measurement instruments analysis for a clinical evaluation instrument used (COSMIN)43 is recommended. to measure clinical performance of student nurse anesthetists in six competence areas and revealed In the review, some researchers treated an instru- that it actually measured only three competence ment as a valid and reliable tool when it had areas. They suggested not only a redefinition of been tested in other studies, and used it without key concepts, but also a revision of the instrument further psychometric tests. Because it is not through further validation tests. If a certain clinical possible to use the instrument in exactly the evaluation tool is used as evidence to decide a stu- same setting as those for which it was developed, dent’s success or failure, ensuring the instrument’s psychometric properties such as validity, reli- validity is critical. The gap between scores ob- ability, cultural difference, language translation, tained from an evaluation tool and the true values and feasibility for the participants should be tested should be minimal.30 again.

The second issue was reliability. Cronbach’s alpha Limitations coefficient was the most frequently used reliability test in this review. A coefficient alpha of 0.70 is re- Regarding the selection process, the inclusion and garded as an acceptable level for new scales, and exclusion criteria may have influenced the out- the number of items is recommended to be short- comes of this research. This is especially because ened if the alpha coefficient value is greater than of the fact that the search was limited to the En- 0.90.42 For example, the PPCS-R Cronbach’s alpha glish language. In perianesthesia nursing practice, coefficient was 0.95 to 98. It seems that some most countries tend to develop instruments in redundancy remains and the reliability might be their own languages. This can be one explanation optimized (between 0.70 and 0.90), and the feasi- why the quantity of studies conducted in the bility might be increased by reducing the quantity United States is high. of items. Nursing practices in perianesthesia care vary One important issue in the direct observation among countries. To increase the possibility of method is the reliability of evaluators. All observa- capturing perianesthesia nursing competence in- tion studies having multievaluator demonstrated struments, nurses caring patients undergoing or inter-rater reliability. Most researchers were well recovering from anesthesia such as perianesthesia aware of the reliability of inter-raters (satisfactory nurses, PACU nurses, nurse anesthetist, anesthesia PERIANESTHESIA COMPETENCE ASSESSMENT INSTRUMENTS 555 nurses, and perioperative nurses were included as continuing education programs to improve inclusion criteria. These various professional nursing competence. In particular, nurses in peri- groups might cause issues with the validity in anesthesia care have a primary responsibility for this search result. providing safe care through continuous professional development. This review may be used to This scoping review did not conduct a quality support nurses when reflecting on their practices appraisal process of the studies included. This to identify areas of strength and weakness. might cause a bias in the interpretation of the research results. However, the psychometric prop- Future research should focus on the development erties of instruments discussed by the recommen- of more valid assessment instruments for perianes- dations of DeVon et al22 can be partial evidence of thesia nursing care. For perianesthesia nursing a quality appraisal of this review. Some instrument care, competence assessment studies need to be psychometrics have been reported in previous replicated to improve psychometric properties of studies but they have not been included in this re- the assessment tool; this can be done through review process. exploring content validity, scoring methods, and reliability. Because perianesthesia nursing care is Conclusions always evolving, there is a need for a study of future competence covering health information This scoping review found that at least nine assess- competence, ethical competence, and cultural ment instruments have been used to assess peria- competence in perianesthesia nursing. nesthesia nurses’ competence by means of self- report, observation, and written examination Acknowledgments methods. This review concluded that perianesthesia nursing care is still lacking in sound assessment Author contributions: Y.J. contributed to the conceptions and instruments. Integrating more than one data design of the study, data collection and analysis, drafting, and collection method is recommended to overcome writing of the manuscript. R.L. contributed to data collection, the limitations such as the lack of objectivity and data analysis, and drafting of the manuscript. R.M. and H.L. to provide a more comprehensive view of compe- contributed to the study conception, design, drafting of the manuscript, and overall supervision. All authors contributed tence measurement. to the critical revision of the manuscripts. The final version of the manuscript has been accepted by all authors. This review provides valuable information for Funding: This study was funded by the foundation of Finnish nurse educators who are developing evaluation Nurse Education. methods for student nurses, nurse managers who Ethical approval: In this study, formal ethical approval or con- sents were not sought because literature reviews and the use are helping nurses to identify professional of routinely available data with open access do not normally development and educational needs, and perianes- require ethical approval. However, all steps for literature review thesia nursing organizations who are planning have met the ethical guidelines for the systematic review.

References

1. Ead H. Perianesthesia nursing-beyond the critical care 7. Gabriel N. A Study of Relationships Among Three Assess- skills. J Perianesth Nurs. 2014;29:36-49. ment Methods for Nurse Anesthetists [dissertation]. San Fran- 2. Mamaril ME. Standards of perianesthesia nursing practice: cisco: Nursing in the Graduate Division, University of Advocating patient safety. J Perianesth Nurs. 2003;18:168-172. California; 2013. 3. Godden B. Orientation, competencies, skills fairs: Sorting 8. National Board of Certification and Recertification for it all out. J Perianesth Nurs. 2011;26:107-109. Nurses Anesthetists. Overview of the continued professional 4. Plaus K, Muckle TJ, Henderson JP. Advancing recertifica- certification (CPC) program. Available at: http://www.nbcrna. tion for nurse anesthetists in an environment of increased com/Pages/default.aspx. Accessed June 10, 2015. accountability. AANA J. 2011;79:413-418. 9. International Federation of Nurse Anesthetists. 5. Jeon Y, Lahtinen P, Meretoja R, Leino-Kilpi H. Anaesthesia Available at: http://ifna-int.org/ifna/news.php. Accessed June nursing education in the Nordic countries: Literature review. 10, 2015. Nurse Educ Today. 2015;35:680-688. 10. European Union Directive. Directive 2013/55/EU of the 6. International Council of Nurses. ICN Framework of Com- European Parliament and of the Council. Available at: http:// petencies for the Nurse Specialist. Geneva, Switzerland: Inter- eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri5CELEX:320 national Council of Nurse; 2009. 40/978-92-95065-55-0. 13L0055&from5EN. Accessed April 18, 2015. 556 JEON ET AL

11. McGrath P, Anastasi J, Fox-Young S, Gorman D, 28. Greenfield M, O’Brien D, Kofflin S. A cross-sectional sur- Moxham L, Tollefson J. Collaborative voices: Ongoing reflec- vey study of nurses’ self-assessed competencies in obstetric and tions on nursing competencies. Contemp Nurse. 2006;22: surgical postanesthesia care units. J Perianesth Nurs. 2014;29: 46-58. 385-396. 12. Meretoja R, Leino-Kilpi H. Comparison of competence 29. Cook KA, Marienau MS, Wildgust B, Gerbasi F, Watkins J. assessments made by nurse managers and practising nurses. J Assessment of recent graduates preparedness for entry into Nurs Manag. 2003;11:404-409. practice. AANA J. 2013;81:341-345. 13. Dellai M, Mortari L, Meretoja R. Self-assessment of 30. Collins S, Callahan M. A call for change: Clinical evalua- nursing competencies—validation of the Finnish NCS instru- tion of students registered nurse anesthetists. AANA J. 2014; ment with Italian nurses. Scand J Caring Sci. 2009;23:783-791. 82:65-72. 14. Lakanmaa RL, Suominen T, Perttil€a J, Ritmala-Castren M, 31. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Vahlberg T, Leino-Kilpi H. Basic competence in intensive and Botney R. Assessment of clinical performance during simulated critical care nursing: Development and psychometric testing crises using both technical and behavioural ratings. Anesthesi- of a competence scale. J Clin Nurs. 2014;23:799-810. ology. 1998;89:8-18. 15. Davis K, Drey N, Gould D. What are scoping studies? A 32. Murray D, Boulet J, Kras J, McAllister J, Cox T. A review of the nursing literature. Int J Nurs Stud. 2009;46: simulation-based acute skills performance assessment for anes- 1386-1400. thesia training. Anesth Analg. 2005;101:1127-1134. 16. Levac D, Colquhoun H, O’Brien KK. Scoping studies: 33. Henrichs BM, Avidan MS, Murray DJ, et al. Performance Advancing the methodology. Implement Sci. 2010;5:1-9. of certified registered nurse anesthetists and anesthesiologists 17. Armstrong R, Hall BJ, Doyle J, Waters E. Cochrane update in a simulation-based skills assessment. Anesth Analg. 2009; ‘Scoping the scope’ of a Cochrane review. J Public Health. 108:255-262. 2011;33:147-150. 34. Robertson ER, Hadi M, Morgan LJ, et al. Oxford 18. Griffiths P,Bridges J, Sheldon H, Thompson R. The role of NOTECHS II: A modified theatre team non-technical skills the dementia specialist nurse in acute care: A scoping review. J scoring system. PLOS One. 2014;9:1-8. Clin Nurs. 2015;24:1394-1405. 35. Burns B. Continuing competency—What’s ahead? J Peri- 19. Hussein ME, Hirst S, Salyers V. Factors that contribute to nat Neonatal Nurs. 2009;23:218-227. underrecognition of delirium by registered nurses in acute care 36. Fotheringham D. Triangulation for the assessment of clin- settings: A scoping review of the literature to explain this phe- ical nursing skills: A review of theory, use and methodology. Int nomenon. J Clin Nurs. 2015;24:906-915. J Nurs Stud. 2010;47:386-391. 20. Arksey H, O’Malley L. Scoping studies: Towards a meth- 37. Watson R, Stimpson A, Topping A, Porock D. Clinical odological framework. Int J Soc Res Methodol. 2005;8:19-32. competence assessment in nursing: A systematic review. J 21. A€€ari RL, Suominen T, Leino-Kilpi H. Competence in Adv Nurs. 2002;39:421-431. intensive and critical care nursing: A literature review. Intensive 38. Crawford SW, Colt HG. Virtual reality and written assess- Crit Care Nurs. 2008;24:78-89. ments are of potential value to determine knowledge and skill in 22. DeVon HA, Block ME, Moyle-Wright P,et al. A psychomet- flexible bronchoscopy. Respiration. 2004;71:269-275. ric toolbox for testing validity and reliability. J Nurs Sch. 2007; 39. Nunnink L, Venkatesh B, Krishnan A, Vidhani K, Udy A. A 39:155-164. prospective comparison between written examination and 23. Gillespie BM, Chaboyer W, Wallis M, Werder H. Educa- either simulation-based or oral viva examination of intensive tion and experience make a difference: Results of a predictor care trainees’ procedural skills. Anaesth Intensive Care. 2010; study. AORN J. 2011;94:78-90. 38:876-882. 24. Gillespie BM, Polit D, Hamlin L, Chaboyer W.Developing 40. Edler AA, Fanning RG, Chen MI, et al. Patient simulation: a model of competence in the operating theatre: Psychometric A literary synthesis of assessment tools in anesthesiology. J Educ validation of the perceived perioperative competence scale- Eval Health Prof. 2009;6:3. revised. Int J Nurs Stud. 2012;49:90-101. 41. Frank-Stromborg M, Olsen SJ. In Instruments for Clin- 25. Gillespie BM, Pearson E. Perceptions of self-competence ical Health-Care Research, 3rd ed. London: Jones and Bartlett in theatre nurses and operating department practitioners. Publishers; 2004. ACORN J. 2013;26:29-34. 42. Zwakhalen S, Hamers J, Abu-Saad H, Berger M. Pain in 26. Meretoja R, Leino-Kilpi H, Kaira AM. Comparison of elderly people with severe dementia: A systematic review of be- nurse competence in different hospital work environments. J havioural pain assessment tools. BMC Geriatr. 2006;6:3. Nurs Manag. 2004;12:329-336. 43. Mokkink LB, Terwee CB, Patrick DL, et al. COSMIN 27. Meretoja R, Koponen L. A systematic model to compare checklist manual. Available at: http://www.cosmin.nl/images/ nurses’ optimal and actual competencies in the clinical setting. upload/files/COSMIN%20checklist%20manual%20v9.pdf. Ac- J Adv Nurs. 2012;68:414-422. cessed May 7, 2016.