Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review | BMC Medical Education
de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care. 2008;17(3):216–23.
Google Scholar
Memon MA, Joughin GR, Memon B. Oral assessment and postgraduate medical examinations: establishing conditions for validity, reliability and fairness. Adv Health Sci Educ Theory Pract. 2010;15(2):277–89.
Google Scholar
Thiessen N, Fischer MR, Huwendiek S. Assessment methods in medical specialist assessments in the DACH region – overview, critical examination and recommendations for further development. GMS J Med Educ. 2019;36(6):Doc78–Web.
Wijnen-Meijer M, Burdick W, Alofs L, Burgers C, Ten Cate O. Stages and transitions in medical education around the world: Clarifying structures and terminology. Med Teach. 2013;35(4):301–7.
Google Scholar
Cassel CK, Holmboe ES. Professionalism and accountability: the role of specialty board certification. Trans Am Clin Climatol Assoc. 2008;119:295–303 (discussion 303-304).
Sharp LK, Bashook PG, Lipsky MS, Horowitz SD, Miller SH. Specialty board certification and clinical outcomes: The missing link. Acad Med. 2002;77(6):534–42.
Google Scholar
Chen J, Rathore SS, Wang Y, Radford MJ, Krumholz HM. Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction. J Gen Intern Med. 2006;21(3):238–44.
Google Scholar
Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon’s training, certification, and experience. Surgery. 2002;132(4):663–70 (discussion 670-662).
Google Scholar
Reid RO, Friedberg MW, Adams JL, McGlynn EA, Mehrotra A. Associations between physician characteristics and quality of care. Arch Intern Med. 2010;170(16):1442–9.
Google Scholar
Lipner RS, Hess BJ, Phillips RL Jr. Specialty board certification in the United States: issues and evidence. J Contin Educ Health Prof. 2013;33(Suppl 1):S20–35.
Google Scholar
Institute of Medicine Committee on Quality of Health Care in A. In: To Err is Human: Building a Safer Health System. edn. Edited by Kohn LT, Corrigan JM, Donaldson MS. Washington (DC): National Academies Press (US). Copyright 2000 by the National Academy of Sciences. All rights reserved.; 2000.
Weiss KB. Future of board certification in a new era of public accountability. J Am Board Fam Med. 2010;23(Suppl 1):S32–39.
Google Scholar
McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. The quality of health care delivered to adults in the United States. New Engl J Med. 2003;348(26):2635–45.
Google Scholar
James JT. A New, Evidence-based Estimate of Patient Harms Associated with Hospital Care. J Patient Saf. 2013;9(3):122–8.
Makary MA, Daniel M. Medical error—the third leading cause of death in the US. BMJ. 2016;353:i2139.
Google Scholar
Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–14.
Google Scholar
Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay MF, Roberts T, Swanson D. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102–9.
Google Scholar
Spike N. Is medical postgraduate certification improving health outcomes? Med Educ. 2002;36:7–8.
Google Scholar
Hutchinson L, Aitken P, Hayes T. Are medical postgraduate certification processes valid? A systematic review of the published evidence. Med Educ. 2002;36(1):73–91.
Google Scholar
Herb U, Geith U. Kriterien der qualitativen Bewertung wissenschaftlicher Publikationen: Befunde aus dem Projekt visOA. Information – Wissenschaft & Praxis. 2020;71(2–3):77–85.
Google Scholar
Cook DA, Reed DA. Appraising the quality of medical education research methods: the Medical Education Research Study Quality Instrument and the Newcastle-Ottawa Scale-Education. Acad Med. 2015;90(8):1067–76.
Google Scholar
Berkenstadt H, Ziv A, Gafni N, Sidi A. The validation process of incorporating simulation-based accreditation into the anesthesiology Israeli national board exams. Israel Med Assoc J. 2006;8(10):728–33.
Sun H, Warner DO, Patterson AJ, Harman AE, Rathmell JP, Keegan MT, Dainer RJ, McLoughlin TM, Fahy BG, MacArio A. The American Board of Anesthesiology’s Standardized Oral Examination for Initial Board Certification. Anesth Analg. 2019;129(5):1394–400.
Google Scholar
Warner DO, Lien CA, Wang T, Zhou Y, Isaak RS, Peterson-Layne C, Harman AE, Macario A, Gaiser RR, Suresh S, et al. First-Year Results of the American Board of Anesthesiology’s Objective Structured Clinical Examination for Initial Certification. Anesth Analg. 2020;23:1412–8.
Warner DO, Isaak RS, Peterson-Layne C, Lien CA, Sun H, Menzies AO, Cole DJ, Dainer RJ, Fahy BG, Macario A, et al. Development of an objective structured clinical examination as a component of assessment for initial board certification in anesthesiology. Anesth Analg. 2020;130(1):258–64.
Google Scholar
Gali A, Roiter H, de Mollein D, Swieszkowskimtsac S, Atamañuk N, Guerreromtsac AA, Grancellimtsac H, Barero C. Evaluation of the quality of multiple-choice questions used in cardiology certification and recertification exams. Revista Argentina de Cardiologia. 2011;79(5):419–22.
Tan LT, McAleer JJA. The Introduction of Single Best Answer Questions as a Test of Knowledge in the Final Examination for the Fellowship of the Royal College of Radiologists in Clinical Oncology. Clin Oncol. 2008;20(8):571–6.
Google Scholar
O’Leary F. Simulation as a high stakes assessment tool in emergency medicine. EMA – Emerg Med Australas. 2015;27(2):173–5.
Google Scholar
Bianchi L, Gallagher EJ, Korte R, Ham HP. Interexaminer agreement on the American Board of Emergency Medicine oral certification examination. Ann Emerg Med. 2003;41(6):859–64.
Google Scholar
Slovut DP, Saiar A, Gray BH. Endovascular medicine certification 2005–2014: report from the american board of vascular medicine. Vascular Medicine (United Kingdom). 2015;20(3):245–50.
Google Scholar
Khafagy G, Ahmed M, Saad N. Stepping up of MCQs’ quality through a multi-stage reviewing process. Educ Prim Care. 2016;27(4):299–303.
Google Scholar
Weingarten MA, Polliack MR, Tabenkin H, Kahan E. Variations among examiners in family medicine residency board oral examinations. Med Educ. 2000;34(1):13–7.
Google Scholar
O’Neill TR, Royal KD, Puffer JC. Performance on the American Board of Family Medicine (ABFM) certification examination: are superior test-taking skills alone sufficient to pass? J Am Board Fam Med. 2011;24(2):175–80.
Google Scholar
O’Neill TR, Peabody MR, Stelter KL, Puffer JC, Brady JE. Validating the Test Plan Specifications for the American Board of Family Medicine’s Certification Examination. J Am Board Fam Med. 2019;32(6):876–82.
Google Scholar
Greco M, Spike N, Powell R, Brownlea A. Assessing communication skills of GP registrars: a comparison of patient and GP examiner ratings. Med Educ. 2002;36(4):366–76.
Google Scholar
Munro N, Denney ML, Rughani A, Foulkes J, Wilson A, Tate P. Ensuring reliability in UK written tests of general practice: the MRCGP examination 1998–2003. Med Teach. 2005;27(1):37–45.
Google Scholar
Simpson RG, Ballard KD. What is being assessed in the MRCGP oral examination? A qualitative study. Br J Gen Pract. 2005;55(515):430–6.
Sandars J, Coughlin S, Foulkes J. The assessment of skills in evidence-based medicine: The MRCGP examination approach. Educ Prim Care. 2004;15:550–63.
Dixon H. The multiple-choice paper of the MRCGP examination: a study of candidates’ views of its content and effect on learning. Educ Prim Care. 2005;16(6):655–62.
Siriwardena AN, Edwards AGK, Campion P, Freeman A, Elwyn G. Involve the patient and pass the MRCGP: investigating shared decision making in a consulting skills examination using a validated instrument. Br J Gen Pract. 2006;56(532):857–62.
Dixon H. Candidates’ views of the MRCGP examination and its effects upon approaches to learning: a questionnaire study in the Northern Deanery. Educ Prim Care. 2003;14:146–57.
Wass V, Wakeford R, Neighbour R, Van Der Vleuten C. Achieving acceptable reliability in oral examinations: an analysis of the Royal College of General Practitioners membership examination’s oral component. Med Educ. 2003;37(2):126–31.
Google Scholar
Dixon H, Blow C, Milne P, Siriwardena N, Milne H, Elfes C. Quality assurance of the Applied Knowledge Test (AKT) of the MRCGP examination – an immediate post-test questionnaire evaluation of the candidates’ views. Educ Prim Care. 2015;26(4):223–32.
Google Scholar
Partridge J. Feedback from candidates sitting the written module of the Membership of the Royal College of General Practitioners examination in Spring 2006: a satisfactory conclusion. Educ Prim Care. 2008;19(2):165–72.
Google Scholar
Dixon H, Blow C, Irish B, Milne P, Siriwardena AN. Evaluation of a postgraduate examination for primary care: perceptions and performance of general practitioner trainers in the multiple choice paper of the Membership Examination of the Royal College of General Practitioners. Educ Prim Care. 2007;18(2):165–72.
Google Scholar
Bourque J, Skinner H, Dupré J, Bacchus M, Ainslie M, Ma IWY, Cole G. Performance of the Ebel standard-setting method in spring 2019 royal college of physicians and surgeons of canada internal medicine certification examination consisted of multiple-choice questions. J Educ Eval Health Prof. 2020;17:12–Web.
Chierakul N, Danchaivijitr S, Kontee P, Naruman C. Reliability and validity of long case and short case in internal medicine board certification examination. J Med Assoc Thai. 2010;93(4):424–8.
McManus IC, Mooney-Somers J, Dacre JE, Vale JA. Reliability of the MRCP(UK) Part I Examination, 1984–2001. Med Educ. 2003;37(7):609–11.
Google Scholar
McManus IC, Elder AT, Dacre J. Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations. BMC Med Educ. 2013;13:103.
Google Scholar
McManus IC, Thompson M, Mollon J. Assessment of examiner leniency and stringency (‘hawk-dove effect’) in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Med Educ. 2006;6:42. https://rdcu.be/dkHg6.
Atsawarungruangkit A: Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam. Med Educ Online. 2015;20(1):28631.
Marques TR, Lains I, Martins MJ, Goiana-Da-Silva F, Sampaio F, Pessanha I, Fernandes DH, Brandao M, Pinto Teixeira P, de Oliveira SM, et al. Evaluation of the medical board exam in Portugal. Acta Med Port. 2018;31(11):670–9.
Google Scholar
Burch VC, Norman GR. Turning words into numbers: establishing an empirical cut score for a letter graded examination. Med Teach. 2009;31(5):442–6.
Google Scholar
Burch VC, Norman GR, Schmidt HG, van der Vleuten CP. Are specialist certification examinations a reliable measure of physician competence? Adv Health Sci Educ. 2008;13(4):521–33.
Google Scholar
Cookson J. A critique of the specialty certificate examinations of the Federation of Royal Colleges of Physicians of the UK. Clin Med J R Coll Phys Lond. 2010;10(2):141–4.
Mucklow J. Development and implementation of the specialty certificate examinations. Clinical Medicine. J R Coll Phys Lond. 2011;11(3):235–8.
Raddatz MM, Royal KD, Pennington J. Evaluating the Systematic Validity of a Medical Subspecialty Examination. Online Submission Paper presented at the Midwestern Educational Research Association Annual Meeting. 2012.
Lunz ME, Bashook PG. Relationship between candidate communication ability and oral certification examination scores. Med Educ. 2008;42(12):1227–33.
Google Scholar
Houston JE, Myford CM. Judges’ perception of candidates’ organization and communication, in relation to oral certification examination ratings. Acad Med. 2009;84(11):1603–9.
Google Scholar
Mathysen DG, Aclimandos W, Roelant E, Wouters K, Creuzot-Garcher C, Ringens PJ, Hawlina M, Tassignon MJ. Evaluation of adding item-response theory analysis for evaluation of the European Board of Ophthalmology Diploma examination. Acta Ophthalmol. 2013;91(7):e573–577.
Google Scholar
Mathysen DGP, Aclimandos W, Roelant E, Wouters K, Creuzot-Garcher C, Ringens PJ, Hawlina M, Tassignon MJ. History and future of the European Board of Ophthalmology Diploma examination. Acta Ophthalmol. 2013;91(6):589–93.
Google Scholar
Chow R, Zhang L, Soong IS, Mang OWK, Lui LCY, Wong KH, Siu SWK, Lo SH, Yuen KK, Yau YSH, et al. Inter-rater Reliability of Examiners in the Hong Kong College of Radiologists’ Palliative Medicine Oral Examination. Hong Kong J Radiol. 2017;20(3):232–6.
Althouse LA, Du Y, Ham HP. Confirming the validity of the general pediatrics certification examinations: a practice analysis. J Pediatr. 2009;155(2):155–156.e151.
Google Scholar
Emadzadeh A, Ravanshad Y, Makarem A, Azarfar A, Ravanshad S, Aval SB, Mehrad-Majd H, Alizadeh A. Challenges of OSCE national board exam in Iran from participants’ perspective. Electronic Phys [Electronic Resource]. 2017;9(4):4195–201.
Raddatz MM, Robinson LR. Demonstrating construct validity of the american board of physical medicine and rehabilitation part i examination: an analysis of dimensionality. PM and R. 2017;9(10):985–9.
Google Scholar
Tibbo P, Templeman K. The RCPSC oral examination: patient perceptions and impact on participating psychiatric patients. Can J Psychiat. 2004;49(7):480–6.
Google Scholar
Tong E, Spooner M, Van Delden O, Uberoi R, Sheehan M, O’Neill DC, Lee M. The European board of interventional radiology examination: a cross-sectional web-based survey. Cardiovasc Intervent Radiol. 2018;41(1):21–6.
Google Scholar
Yeung A, Booth TC, Larkin TJ, McCoubrie P, McKnight L. The FRCR 2B oral examination: is it reliable? Clin Radiol. 2013;68(5):466–71.
Google Scholar
Yeung A, Booth TC, Jacob K, McCoubrie P, McKnight L. The FRCR 2B examination: a survey of candidate perceptions and experiences. Clin Radiol. 2011;66(5):412–9.
Google Scholar
Yang JC, Wallner PE, Becker GJ, Bosma JL, Gerdeman AM. Reliability of oral examinations: Radiation oncology certifying examination. Pract Radiat Oncol. 2013;3(1):74–8.
Google Scholar
Kerridge WD, Gunderman RB. The validity and timing of the ABR core exam. Acad Radiol. 2016;23(9):1176–9.
Google Scholar
Yang JC, Gerdeman AM, Becker GJ, Bosma JL. American Board of Radiology diagnostic radiology initial qualifying (written) examinations. AJR Am J Roentgenol. 2010;195(1):10–2.
Google Scholar
Pascual-Ramos V, Guilaisne Bernard-Medina A, Flores-Alvarado DE, Portela-Hernández M, Maldonado-Velázquez MDR, Jara-Quezada LJ, Amezcua-Guerra LM, Rubio-Judith López-Zepeda NE, Álvarez-Hernandez E, Saavedra MÁ, et al. The method used to set the pass mark in an objective structured clinical examination defines the performance of candidates for certification as rheumatologists. Reumatologia Clinica. 2018;14(3):137–41.
Google Scholar
Smith JD, Prideaux D, Wolfe CL, Wilkinson TJ, Sen Gupta T, DeWitt DE, Worley P, Hays RB, Cowie M. Developing the accredited postgraduate assessment program for Fellowship of the Australian College of Rural and Remote Medicine. Rural Remote Health. 2007;7(4):805.
Beasley SW, Wannan C, Hardware N. Justification and implications of the introduction of an expanded Close Marking System for the Fellowship Examination. ANZ J Surg. 2013;83(6):444–7.
Google Scholar
De Montbrun S, Roberts PL, Satterthwaite L, Macrae H. Implementing and evaluating a national certification technical skills examination. Ann Surg. 2016;264(1):1–6.
Google Scholar
Lineberry M, Park YS, Hennessy SA, et al. The Fundamentals of Endoscopic Surgery (FES) skills test: factors associated with first-attempt scores and pass rate. Surg Endosc. 2020;34:3633–43. https://doi.org/10.1007/s00464-020-07690-6.
Motoyama S, Yamamoto H, Miyata H, Yano M, Yasuda T, Ohira M, Kajiyama Y, Toh Y, Watanabe M, Kakeji Y, et al. Impact of certification status of the institute and surgeon on short-term outcomes after surgery for thoracic esophageal cancer: evaluation using data on 16,752 patients from the National Clinical Database in Japan. Esophagus. 2020;17(1):41–9.
Google Scholar
Montbrun d, Lynn S. High Stakes Technical Skill Assessments in Surgery: Development, Implementation and Predicting Performance. 2017. p. 2017.
Crisostomo AC. The Effect of Standardization on the Reliability of the Philippine Board of Surgery Oral Examinations. J Surg Educ. 2011;68(2):138–42.
Google Scholar
Rhodes RS, Biester TW, Bell RH Jr, Lewis FR Jr. Assessing Surgical Knowledge: a Primer on the Examination Policies of the American Board of Surgery. J Surg Educ. 2007;64(3):138–42.
Google Scholar
Cundy P. Examining the orthopaedic examiners: Reliability of the Part 2 Orthopaedic Clinical Fellowship Examination in Australia. ANZ J Surg. 2012;82(9):607–11.
Google Scholar
Hohmann E, Tetsworth K. Fellowship exit examination in orthopaedic surgery in the commonwealth countries of Australia, UK, South Africa and Canada. Are they comparable and equivalent? A perspective on the requirements for medical migration. Med Educ Online. 2018;23(1):1537429–7.
Gillis ME, Scott SA, Richardson CG, Oxner WM, Gauthier L, Wilson DA, Glennie RA. Developing and assessing the feasibility of implementing a Surgical Objective Structured Clinical Skills Examination (S-OSCE). J Surg Educ. 2020;77(4):939–46.
Google Scholar
Ullmann Y, Fodor L, Meilick B, Eshach H, Ramon Y, Meilick A. The oral board examination for plastic surgery: Seeking a better way. Med Teach. 2006;28(4):360–4.
Google Scholar
Dwyer T, Chahal J, Murnaghan L, Theodoropoulos J, Cheung J, McParland A, Ogilvie-Harris D. Development of a certification examination for orthopedic sports medicine fellows. Can J Surgery Journal canadien de chirurgie. 2020;63(2):E110–7.
Google Scholar
Payne SR, Pickard RS, O’Flynn KJ, Winton EP. Does the Intercollegiate Specialty Examination in urology (FRCS Urol) assess the breadth of the urology syllabus? Br J Med Surg Urol. 2011;4(4):139–47.
Google Scholar
Hohmann E, Tetsworth K. Fellowship exit examination in orthopaedic surgery in the commonwealth countries of Australia, UK, South Africa and Canada. Are they comparable and equivalent? A perspective on the requirements for medical migration. Med Educ Online. 2018;23(1):1537429.
Google Scholar
Practitioners RCoG. The RCGP Curriculum. Being a General Practitioner. 2019.
Lockyer J, Carraccio C, Chan M-K, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR. Core principles of assessment in competency-based medical education. Med Teach. 2017;39(6):609–16.
Google Scholar
de Montbrun S, Louridas M, Grantcharov T. Passing a technical skills examination in the first year of surgical residency can predict future performance. J Grad Med Educ. 2017;9(3):324–9.
Google Scholar
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–67.
Google Scholar
Van Der Vleuten CPM, Schuwirth LWT. Assessing professional competence: From methods to programmes. Med Educ. 2005;39(3):309–17.
Google Scholar
Witheridge A, Ferns G, Scott-Smith W. Revisiting Miller’s pyramid in medical education: the gap between traditional assessment and diagnostic reasoning. Int J Med Educ. 2019;10:191–2.
Google Scholar
Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med. 1998;129(1):42–8.
Google Scholar
Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 2013;35(7):564–8.
Google Scholar
Kogan JR, Holmboe E. Realizing the Promise and Importance of Performance-Based Assessment. Teach Learn Med. 2013;25(sup1):S68–74.
Google Scholar
Misra S, Iobst WF, Hauer KE, Holmboe ES. The importance of Competency-Based Programmatic Assessment in Graduate Medical Education. J Grad Med Educ. 2021;13(2s):113–9.
Google Scholar
van der Vleuten CPM, Schuwirth LWT, Driessen EW, Dijkstra J, Tigelaar D, Baartman LKJ, van Tartwijk J. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205–14.
Google Scholar
Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226–31.
Google Scholar
Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. Twelve tips for programmatic assessment. Med Teach. 2015;37(7):641–6.
Google Scholar
MRCGP Clinical Skills Assessment (CSA) https://www.rcgp.org.uk/training-exams/mrcgp-exam/mrcgp-clinical-skills-assessment-csa.aspx.
WPBA assessments. https://www.rcgp.org.uk/training-exams/training/workplace-based-assessment-wpba/assessments.aspx.
MRCGP: Recorded Consultation Assessment (RCA). https://www.rcgp.org.uk/mrcgp-exams/recorded-consultation-assessment.
Silber JH, Kennedy SK, Even-Shoshan O, Chen W, Mosher RE, Showan AM, Longnecker DE. Anesthesiologist board certification and patient outcomes. Anesthesiology. 2002;96(5):1044–52.
Google Scholar
Wallace A, McFarland BH, Selvam N, Sahota G. Quality of care provided by board-certified versus non-board-certified psychiatrists and neurologists. Acad Med. 2017;92(1):108–15.
Google Scholar
Norcini J, Lipner R, Kimball H. The certification status of generalist physicians and the mortality of their patients after acute myocardial infarction. Acad Med. 2001;76(10 Suppl):S21–23.
Google Scholar
link