F14 - Are You Reliable? How to Establish Interrater Reliability

Abstract : Feedback and evaluation are essential components of competency-based education. The use of simulation for formative feedback is well established. While the use of simulation for summative and high stakes evaluation can be achieved, the processes for this requires significant time and facilitator development. As outlined in the Healthcare Simulation Standards of Best PracticeTM: Evaluation of Learning and Performance, summative and high stakes evaluation should include facilitators trained in principles of simulation-based education, evaluation techniques, and the use of valid and reliable tools. Establishing interrater reliability for standardized, quality feedback improves student outcomes, consistency, and credibility of the evaluation process.
Interrater reliability and the importance of training simulation facilitators when using simulation for grading purposes will be identified. Workshop activities will allow participants to examine the interaction between Facilitator and Educational Strategies described in the NLN Jeffries Simulation Theory and how they are associated with participant outcomes. These potential issues include academic and personal elements that impact interpreting student behavior and clinical judgement.
Workshop activities (video, rating tools, discussions) will guide participants through the process of interrater reliability and establishing evaluation standards among peers. Debriefing, guided by open-ended discussion questions, will allow participants to share their perceptions of student performance and negotiating group interrater reliability, and reflect on challenges and success in establishing interrater reliability for summative and high stakes grading.

Please include a short summary of your presentation that highlights why an attendee would want to participate in your session.: Workshop will address the process of establishing interrater reliability:
-Brief presentation of interrater reliability process and measurement
-Video of nursing care/skill and evaluation of student performance using a rubric
-Groups establish interrater reliability within small groups
-Examine rater scores for P/F and grading
-Discuss challenges and success in establishing interrater reliability