Harmonized Milestones

The Milestones were not intended to remain a document set in stone, but rather to be reviewed and revised as adequate feedback is obtained, graduate medical education evolves, and the specialty advances and changes. Experts from each specialty or subspecialty creating subcompetencies resulted in marked variability between specialties and subspecialties with respect to content and how educational progression was operationalized across Milestone levels.⁶ In 2016 at the 2nd ACGME Milestones Summit, stakeholders from all ACGME core specialties and subspecialties expressed dissatisfaction with inconsistencies in the Milestones and subcompetencies. For example, the subcompetencies tended to detail a variety of differing skills across milestone levels, rather than describing the expected spectrum of growth from beginning to advanced resident or fellow of one particular skill. Additional problems that needed to be addressed were that there were too many subcompetencies for some specialties, the language at times was too complex, some of the guidelines were not practical, and there was too much material in a milestone set. It was reported that these differences had “complicated efforts to share assessment tools across programs and provide comprehensive faculty development across specialties.”⁷ In other words, the Milestones needed to have more national stakeholder input and overall simplification.

To evaluate and address these concerns, the ACGME Department of Research, Milestone Development, and Evaluation reviewed data collected from the field, data submitted for the biannual milestone reporting, as well as learning from quantitative and qualitative research. Analysis of non-PC and non-MK domains showed marked variation between 26 core specialties and the transitional year; there were more than 230 different ways of describing PROF, 171 for PBLI, 176 for ICS, and 122 for SBP.⁸ As a result, the ACGME created 4 stakeholder groups consisting of content experts, directors, interprofessional team members, and other faculty; each group was tasked to develop 2 to 3 subcompetencies that applied to all specialties and subspecialties, thus creating Harmonized Milestones. Terminology in these Harmonized Milestones could be modified if needed to pertain to a particular specialty or subspecialty during the Milestones 2.0 workgroup meetings.

Original Neuroradiology Milestones

The Neuroradiology Fellowship Milestones were initially published in February 2014, followed by a minor edit to the introductory pages in July 2015; however, the content remained the same.⁹ The original task force, chaired by Jim Anderson, MD, consisted of a panel of experts from the ACGME Radiology Review Committee, the American Board of Radiology, the American Society of Neuroradiology, and additional volunteers. The group met over 2 days at the ACGME headquarters in Chicago. There were 14 separate subcompetencies included in the initial document that were based in the 6 core competencies (Table). The workgroup focused on those milestones that were specific to neuroradiology training under the PC and MK domains. Those milestones within SBP, PBLI, PROF, and ICS were modelled after the Diagnostic Radiology Residency Milestones, which had been published in December 2012. Following the initial draft and edits, the document was posted for public commentary, approved, and published approximately 1 year later.

The main goal of the original task force was to create a list of subcompetencies that were applicable to all neuroradiology fellowships and representative of the expected abilities of graduating fellows. There was a general interest to keep the number to a minimum, which ultimately led to grouping of subcompetencies under larger headings.

The major criticism of the Neuroradiology Milestones 1.0 is the challenge in assessing a learner in a Milestone when more than one element is included. For example, “Contrast Agents,” “Radiation Safety,” and “Magnetic Resonance Safety” are all included as separate elements under Patient Care 3 in Milestones 1.0. The more elements represented within a single Milestone has made it challenging for programs to accurately assess its learners. The other major shortcoming of the process is inherent in applying a 5-point scale to show Milestone progression in a 1-year fellowship with midpoint and end of year assessment time points.

Development of Neuroradiology Milestones 2.0

The Neuroradiology Milestones 2.0 workgroup was created with the intention of involving neuroradiologists from various backgrounds; as such, representatives were appointed by the ACGME Review Committee and American Board of Radiology, and there was a call for volunteers in late 2019; a total of 9 neuroradiologists and the ACGME Vice President of Milestones Development were part of the working group. The original plan was to have 3 meetings in Chicago, with a target implementation date of the Milestones 2.0 in either 2021 or 2022, based on public commentary. However, due to the COVID-19 pandemic, the ACGME converted from in-person to virtual meetings. The group work was extended to 4 virtual meetings with individual or small group work between meetings. As with in-person meetings, the virtual meetings included development and thorough analysis of the Milestones and Supplemental Guide and individual off-line reviews for accuracy and agreement on the draft.

The first meeting was August 17, 2020, followed by August 18, October 20, and November 5. Before the first meeting, reading materials were provided to serve as background and a basis for discussion in revising the Milestones. Meetings 1 and 2 focused on creating a shared mental model of the task, then selecting topics important to the graduate of 2025 and deciding if they were at the level of a subcompetency, a theme within a subcompetency, something to be used as an example, or if it was not something to be evaluated regularly. The group also started developing subcompetencies. Before meeting 3, the group reviewed the changes discussed and made edits as necessary. Meeting 3 was utilized to complete development of the Milestones and work on the Supplemental Guide. At the end of meeting 3, the team was divided into 4 groups (3 groups of 2 and 1 group of 3) to work on refining or developing examples within each subcompetency, describing the expected growth from beginning to advanced fellow of a particular skill. These smaller groups also reviewed or updated suggested reading material and references included in the Supplemental Guide. Meeting 4 was used for consensus review, minor adjustments, and approval of each group's examples and suggested reading material, thus completing the Supplemental Guide. Following this meeting, each team member reviewed both the Milestones 2.0 and Supplemental Guide documents for accuracy and final endorsement, before public commentary.

Major Changes Between Milestones 1.0 and 2.0

The Neuroradiology Milestones 2.0 workgroup was tasked with several areas of improvement to focus upon. First, since many evaluators utilize both the Radiology Milestones and the Neuroradiology Milestones, the workgroup aimed to revise the Neuroradiology Milestones 2.0 such that they align with the Radiology Milestones 2.0 put into effect several months before the Neuroradiology workgroup convened. Radiology Milestones 2.0 was used as a starting point in writing the Neuroradiology Milestones 2.0, which results in commonalities in language and organizational structure between the documents, thus harmonizing and simplifying the radiology specialty and neuroradiology subspecialty Milestones documents. An effort was made to keep the overall number of milestones manageable while still capturing the core assessable components of neuroradiology practice and fellowship training. The Radiology Milestones 2.0 contains 24 milestones, and the Neuroradiology Milestones 2.0 was expanded from the original 14 milestones to 21 milestones. The differences between Radiology Milestones 2.0 and Neuroradiology Milestones 2.0 are primarily under the MK competency. MK2, MK3, and MK4 of the Radiology Milestones 2.0, which assess application of physics to imaging, appropriate selection of protocol and contrast, and knowledge of imaging technology, respectively, were combined to form MK2 of the Neuroradiology Milestones 2.0, assessing knowledge of all 3 topics in a format appropriate for an advanced subspecialty trainee.

The workgroup assessed the current and projected future practice of neuroradiology and training to ensure that the Milestones remain relevant. Using the first Neuroradiology Milestones as a starting point, we identified the most important areas concerning the MK and PC competencies that needed to be included in the Neuroradiology 2.0 Milestones. The increasing adoption of lexicons and structured reporting, skills reflecting the importance of the role of the neuroradiologist as a consultant to subspecialized clinicians, and the need to apply concepts regarding imaging physics, image processing, and image acquisition to optimize imaging were incorporated into Neuroradiology Milestones 2.0 as changes from the original version. The non-PC, non-MK milestones are derived from the harmonized milestones across all medical specialties in the Neuroradiology Milestones 2.0 document, with additional milestones specific to medical imaging added to the Radiology and Neuroradiology SBP domain (SBP5, SBP6, SBP7).

The workgroup simplified the Milestones by reducing the number of rows within each milestone set and the complexity of the milestone wording. Subcompetencies were limited to no more than 3 rows, with most only containing 1 or 2. Also, within each subcompetency, there are no longer any random milestones (evaluation or introduction of a different skill at a different level within a particular subcompetency), meaning that each subcompetency follows expected growth of 1 particular skill from level 1 to level 5 (Figure). While limiting the number of subcompetencies led to an increase in overall number of Milestones, the new Milestones create concise benchmarks. Often with the original milestones, the separate subcompetencies may have been attained at different levels of training for individual trainees. This led to difficulties in assigning a single Milestone level to any given trainee. With the more granular milestones, while there are overall more milestones to score by the Clinical Competency Committee, the time spent deciding between Milestone levels will be decreased due to the uniformity of each Milestone.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE.

Comparison between Neuroradiology Milestones 1.0 and 2.0. This figure shows some of the major differences between the original Neuroradiology Milestones and the Neuroradiology Milestones 2.0. The original milestone (top left) for Patient Care 3 contains up to 5 subcompetencies, some of which did not follow through each of the levels of the milestone. In the Neuroradiology Milestones 2.0, the milestone is more simplified, with the evaluation of only 1 subcompetency and with the single subcompetency having a skillset under each level, followed through level 1 to 5.

The Supplemental Guide for Neuroradiology Milestones 2.0 is a separate document available through the ACGME website that replaces the footnotes at the bottom of the original Neuroradiology Milestones. The Supplemental Guide was created to provide insights into the intent of the subcompetencies, with examples for each level, sample assessment methods, and other resources. The Supplemental Guide for the Radiology Milestones 2.0 was also used as a starting point for the Neuroradiology Milestone 2.0 Supplemental Guide, with general radiology clinical examples replaced with neuroradiology subspecialty-specific content.