Discussion

The practice of MR imaging radiology is fairly heterogeneous depending on country, size of practice, and personal preference, with studies reported in some instances by general radiologists and at other times by subspecialty radiologists. Conventional MS follow-up study reading techniques have been shown to be subjective and dependent on the skill and consistency of the reader.¹⁰ There have been many proposed solutions to facilitate time-efficient, reproducible, and accurate lesion detection, including semiautomated and fully automated techniques. Although many of these various proposed methods show promising results, the results are usually in small samples and generally have not entered routine clinical use.¹¹

A previous study has shown that a semiautomated assistive platform is fast, robust, and detects many more new lesions on MR imaging of patients with MS than conventional side-by-side comparison.⁹ The same software has also been shown to perform similarly when used by readers of different radiology experience, including readers with minimal radiology experience.¹² This new approach allowed us to use it as a new “gold standard” against which we can measure the performance of radiologists of various levels of subspecialty experience.

Other studies have shown no significant difference in NR versus NNR in stroke CT imaging,¹³ significant discrepancy between NNR and second-opinion NR reporting,⁷ and increased accuracy of interpretation in brain MR imaging by tertiary center NRs compared with external institutions.⁸ Some of the aforementioned studies have showed the utility of second-opinion subspecialty reporting in neuroradiology.^7,8 However, in these studies, the original reports were over-read by an NR, whereas in our study, the 2 groups are independent of one another.

To our knowledge, no previous study has directly compared the performance of NNRs with subspecialty NRs in reporting a cohort of MR imaging for the same indication against a validated external “gold standard.”

The main finding of our study is that NRs performed better than NNRs in the conventional reading of MS studies in terms of better sensitivity, better negative predictive value, and lower false-negative rate. Specificity and positive predictive value were at an expected 100%, as there were no false-positives in our cohort. Among individual radiologists, the false-negative rate ranged from 0%–29% for NRs and 19%–100% for NNRs. The caveat is that the radiologist with a 100% false-negative rate had only reported 2 MS studies during the recruitment period.

In addition, we have also demonstrated that even when only grouping results into 2 or more lesions, versus 1 or 0 lesions, there is a much higher false-negative rate among NNRs compared with NRs (OR, 7.88) when taking into account other variables such as number of studies and patient demographics. This is potentially an important finding because, in some circumstances, a solitary asymptomatic lesion may not be sufficient for neurologists to consider treatment change.

There is a trend toward subspecialty reporting in North America. This is also the case in many other countries, though it is our direct experience that until all neuroradiology studies are reported by subspecialty-trained radiologists, general radiologists will choose to report studies that are perceived to require less subspecialty expertise. Follow-up MS studies are anecdotally often thought of as only requiring identification of new lesions, a task that is felt to be both cognitively simple and requiring less interpretation. Our findings suggest that this assumption is incorrect and that experience has a significant impact on the accuracy of detecting new lesions. As such, we believe this study not only supports the general move toward subspecialty reporting, but also suggests that there may not be such a thing as an “easy” study.

This study has a number of weaknesses. The largest and most difficult to control for is the fact that subspecialty-trained NRs also tend to report the greatest number of neuroradiology studies and would, in most cases, have done so more consistently over a longer period of time. We attempted to control for the overall number of neuroradiology studies reported by the included radiologist, though this proved to be impractical because of differences in years of experience, number of days worked during the study period, and variability in the type of studies some radiologists chose to report. As such, establishing the effect size of subspecialty training versus experience was not possible. Having said that, if looking at only the 2 most prolific reporters of MS studies from each group, each with more than 20 studies (having reported 64/96 of the studies from the NR group and 45/63 from the NNR group), the results are fairly similar to the entire cohort in that the false-negative rate in the NR group is much lower (6.3% versus 22%). Thus, we feel that the conclusion that NRs are more accurate than NNRs is broadly valid, even if the reason for this is not clear. It is important to note that we are not suggesting a simple direct causative link between subspecialty fellowship training and diagnostic accuracy, but rather using fellowship training as a pragmatic marker of a number of related factors that are likely to be contributory.

Another limitation of this study is that each study pair was reviewed by a single radiologist. Ideally, all 159 study pairs would have been read by each of the involved radiologists. However, this was beyond the scope of this study, which used pre-existing, retrospective data from an earlier study. In some ways, this is not a weakness, in that the clinical reports used are truly reflective of everyday practice rather than a somewhat artificial trial setting.