Skip to main content Skip to main navigation menu Skip to site footer

Content validity and inter-rater reliability of procedural skill checklists used in the online OSCE scoring management system

  • Ferika Indarwati ,
  • Yanuar Primanda ,
  • Fahni Haris ,
  • Resti Yulianti Sutrisno ,
  • pdf  |
  • Published: 2023-01-30


Abstract

Introduction: The use of online Objective Structured Clinical Examination (OSCE) scoring system in medical and nursing education is emerging. To ensure valid student’ score, the OSCE check lists need to be assessed for its validity and reliability. This study aims to test the validity and reliability of several procedural check lists commonly used in nursing profession such as peripheral intravenous insertion, electrocardiogram placement, nasogastric tube insertion, urinary catheter insertion and oxygenation procedure.

Methods: The expert consensus was used to generate items, content validity index and inter-rater reliability was used to evaluate the validity and reliability of the check lists. Five experts assessed the content validity of the checklists and five raters used the check list to evaluate performance of 11 students. Data were collected from April to May 2022. The panel experts rated the content relevance of each instrument using a four-point rating scale. Item level and scale level content validity index were calculated. Inter-rater reliability was calculated using Fleiss Kappa.

Results: The item validity index for the five check lists reviewed showed relatively high content validity among experts. I-CVI for each tool was very good, ranging from 0.8 to 1. The average content agreement (S-CVI/Ave) and the universal agreement (S-CVI/UA) for each check list were also very good. The inter-rater reliability results indicated that the agreement among raters was ranged from moderate to very good/excellent. The lowest Kappa value was for the nasogastric insertion check list (0.40, 95%CI 0.40 – 0.41) and the highest Kappa value was for the oxygenation check list (1, 95% CI 0.99 – 1).

Conclusion: The face validity was reported as easy to understand and presented logically. Nonetheless, re-formatting of some items and addition of details in the checklists are needed to avoid ambiguity, which could lead to confusion for the examiner and examinee.

Keywords: content validity inter-rater reliability online osce skill

References

  1. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education. 1979;13(1):39-54. PMID: 763183. https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2923.1979.tb00918.x
  2. Luimes JD, Labrecque ME. Implementation of electronic objective structured clinical examination evaluation in a nurse practitioner program. Journal of Nursing Education. 2018 Aug 1;57(8):502-5. doi:10.3928/01484834-20180720-10
  3. Primanda Y, Sutrisno RY, Haris F, editors. The development of online OSCE prototype for OSCE in school of nursing: Lesson learned. The Third International Conference on Sustainable Innovation 2019 – Health Science and Nursing (IcoSIHSN 2019); 2019; Indonesia: Atlantis Press; 2019. https://doi.org/https://doi.org/10.2991/icosihsn-19.2019.47
  4. Sutrisno RY, Primanda Y, Haris F. Student's Satisfaction on Online Nursing OSCE (ON-OSCE) Assessment Application. IJNP (Indonesian Journal of Nursing Practices). 2020;4(2):70-6. doi: 10.18196/ijnp.v4i2.10142
  5. Meskell P, Burke E, Kropmans TJ, Byrne E, Setyonugroho W, Kennedy KM. Back to the future: An online OSCE Management Information System for nursing OSCEs. Nurse Educ Today. 2015;35(11):1091-6. https://doi.org/10.1016/j.nedt.2015.06.010
  6. Lara S, Foster CW, Hawks M, Montgomery M. Remote assessment of clinical skills during COVID-19: A virtual, high-stakes, summative pediatric Objective Structured Clinical Examination. Acad Pediatr. 2020;20(6):760-1. https://www.academicpedsjnl.net/article/S1876-2859(20)30239-4/pdf
  7. Shorbagi S, Sulaiman N, Hasswan A, Kaouas M, Al-Dijani MM, El-hussein RA, et al. Evaluating the feasibility and effectiveness of e-OSCE in the COVID-19 era. Research Square. 2021:1-21. https://doi.org/10.21203/rs.3.rs-506145/v1
  8. Yazbeck Karam V, Park YS, Tekian A, Youssef N. Evaluating the validity evidence of an OSCE: results from a new medical school. BMC medical education. 2018;18(1):1-7. https://doi.org/10.1186/s12909-018-1421-x
  9. Kolivand M, Esfandyari M, Heydarpour S. Examining validity and reliability of objective structured clinical examination for evaluation of clinical skills of midwifery undergraduate students: a descriptive study. BMC Medical Education. 2020;20(1):1-7. https://doi.org/10.1186/s12909-020-02017-4
  10. Sattelmayer KM, Jagadamma KC, Sattelmayer F, Hilfiker R, Baer G. The assessment of procedural skills in physiotherapy education: a measurement study using the Rasch model. Arch Physiother. 2020;10:9. https://doi.org/10.1186/s40945-020-00080-0
  11. Jelovsek JE, Kow N, Diwadkar GB. Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review. Med Educ. 2013;47(7):650-73. doi: 10.1111/medu.12220
  12. Pugh D, Halman S, Desjardins I, Humphrey-Murto S, Wood TJ. Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests. Teaching and Learning in Medicine. 2016;28(4):406-14. https://doi.org/10.1080/10401334.2016.1218337
  13. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.
  14. Downing SM, Haladyna TM. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ. 2004;38(3):327-33. https://onlinelibrary.wiley.com/doi/pdf/10.1046/j.1365-2923.2004.01777.x
  15. Schuster C, Stahl B, Murray C, Glover K. Development and testing of an instrument to measure short peripheral catheter insertion confidence. Journal of infusion nursing : the official publication of the Infusion Nurses Society. 2016;39(3):159-65. doi: 10.1097/NAN.0000000000000166
  16. Canton SP, Foley CE, Fulcher I, Newcomb LK, Rindos N, Donnellan NM. Reliability and Discriminant Validity of a Checklist for Surgical Scrubbing, Gowning and Gloving. International Journal of Medical Students. 2022;10(1):18-24. doi: 10.5195/ijms.2021.1221
  17. Setyonugroho W, Kennedy KM, Kropmans TJ. Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: a systematic review. Patient education and counseling. 2015;98(12):1482-91. https://doi.org/10.1016/j.pec.2015.06.004
  18. Falzarano M, Zipp GP. Seeking consensus through the use of the Delphi technique in health sciences research. Journal of allied health. 2013 May 31;42(2):99-105.
  19. Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29(5):489–97. https://doi.org/10.1002/nur.20147
  20. Paulsen R, Gallu T, Gilkey D, Reiser II R, Murgia L, Rosecrance J. The inter-rater reliability of Strain Index and OCRA Checklist task assessments in cheese processing. Applied ergonomics. 2015 Nov 1;51:199-204.
  21. Gorski LA, Hadaway L, Hagle ME, Broadhurst D, Clare S, Kleidon T, Meyer BM, Nickel B, Rowley S, Sharpe E, Alexander M. Infusion therapy standards of practice. Journal of Infusion Nursing. 2021 Jan 1;44(1S):S1-224. doi: 10.1097/NAN.0000000000000396
  22. Vallverdú Vidal M, Barcenilla Gaite F. Antiseptic urinary catheterization and maintenance of the bladder catheter. Med Intensiva (Engl Ed). 2019;43 Suppl 1:48-52. doi: 10.1016/j.medin.2018.09.014
  23. Darpo B. Clinical ECG Assessment. Handb Exp Pharmacol. 2015;229:435-68. https://doi.org/10.1007/978-3-662-46943-9_17
  24. Irving SY, Rempel G, Lyman B, Sevilla WMA, Northington L, Guenter P. Pediatric Nasogastric Tube Placement and Verification: Best Practice Recommendations From the NOVEL Project. Nutrition in clinical practice : official publication of the American Society for Parenteral and Enteral Nutrition. 2018;33(6):921-7. doi: 10.1002/ncp.10189
  25. Ahlin C, Klang-Söderkvist B, Johansson E, Björkholm M, Löfmark A. Assessing nursing students' knowledge and skills in performing venepuncture and inserting peripheral venous catheters. Nurse Educ Pract. 2017;23:8-14. https://doi.org/10.1016/j.nepr.2017.01.003
  26. Hurley KF, Giffin NA, Stewart SA, Bullock GB. Probing the effect of OSCE checklist length on inter-observer reliability and observer accuracy. Med Educ Online. 2015;20(1):29242. https://doi.org/10.3402/meo.v20.29242
  27. Daniels VJ, Pugh D. Twelve tips for developing an OSCE that measures what you want. Medical teacher. 2018;40(12):1208-13. https://doi.org/10.1080/0142159X.2017.1390214
  28. Indarwati F, Munday J, Keogh S. Adaptation and validation of pediatric peripheral intravenous catheter insertion and care practices audit tools. Int J Nurs Sci. 2022;9(2):179-86. https://doi.org/10.1016/j.ijnss.2022.03.005

How to Cite

Indarwati, F., Primanda, Y. ., Haris, F. ., & Yulianti Sutrisno, R. . (2023). Content validity and inter-rater reliability of procedural skill checklists used in the online OSCE scoring management system. Bali Medical Journal, 12(1), 456–461. Retrieved from https://balimedicaljournal.ejournals.ca/index.php/bmj/article/view/3760
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver
Download Citation
Endnote/Zotero/Mendeley (RIS) BibTeX

HTML
5

Total
2

Share

Search Panel

Ferika Indarwati
Google Scholar
Pubmed
BMJ Journal

Yanuar Primanda
Google Scholar
Pubmed
BMJ Journal

Fahni Haris
Google Scholar
Pubmed
BMJ Journal

Resti Yulianti Sutrisno
Google Scholar
Pubmed
BMJ Journal