Inter- and Intrarater Agreement on the Outcome of Endovascular Treatment of Aneurysms Using MRA
===============================================================================================

* S. Jamali
* R. Fahed
* J.-C. Gentric
* L. Letourneau-Guillon
* H. Raoult
* F. Bing
* L. Estrade
* T.N. Nguyen
* É. Tollard
* J.-C. Ferre
* D. Iancu
* O. Naggara
* M. Chagnon
* A. Weill
* D. Roy
* A.J. Fox
* D.F. Kallmes
* J. Raymond

## Abstract

**BACKGROUND AND PURPOSE:** Patients treated with coiling are often followed by MR angiography. Our objective was to assess the inter- and intraobserver agreement in diagnosing aneurysm remnants and recurrences by using multimodality imaging, including TOF MRA.

**MATERIALS AND METHODS:** A portfolio composed of 120 selected images from 56 patients was sent to 15 neuroradiologists from 10 institutions. For each case, raters were asked to classify angiographic results (3 classes) of 2 studies (32 MRA-MRA and 24 DSA-MRA pairs) and to provide a final judgment regarding the presence of a recurrence (no, minor, major). Six raters were asked to independently review the portfolio twice. A second study, restricted to 4 raters having full access to all images, was designed to validate the results of the electronic survey.

**RESULTS:** The proportion of cases judged to have a major recurrence varied between 16.1% and 71.4% (mean, 35.0% ± 12.7%). There was moderate agreement overall (κ = 0.474 ± 0.009), increasing to nearly substantial (κ = 0.581 ± 0.014) when the judgment was dichotomized (presence or absence of a major recurrence). Agreement on cases followed-up by MRA-MRA was similarly substantial (κ = 0.601 ± 0.018). The intrarater agreement varied between fair (κ = 0.257 ± 0.093) and substantial (κ= 0.699 ± 0.084), improving with a dichotomized judgment concerning MRA-MRA comparisons. Agreement was no better when raters had access to all images.

**CONCLUSIONS:** There is an important variability in the assessment of angiographic outcomes of endovascular treatments. Agreement on the presence of a major recurrence when comparing 2 MRA studies or the MRA with the last catheter angiographic study can be substantial.

Time-of-flight MR angiography is a noninvasive, radiation-free follow-up technique that has been shown to be sensitive (>85%) and specific (>85%) in identifying incomplete aneurysm occlusion after endovascular treatment.1 MRA is replacing conventional angiography in following patients after coiling in many centers.2,3 Interobserver agreement in the diagnosis of a residual or recurrent aneurysm has been reviewed recently.4 Mean interrater agreement of the pooled studies was substantial (κ = 0.65; 95% confidence interval, 0.60–0.69). However, authors concluded that because all 4 studies primarily focusing on reliability concerned conventional angiography, the reliability of MRA must still be considered unclear.4

Multiple MR angiographic scales have been proposed, but few have been tested for reliability.5 Interobserver agreement by using a catheter angiographic scale directly applied to MR angiographic results has been reported as “substantial” and “similar” to angiography, but the number of readings was limited to duplicates.3,6 A more rigorous evaluation of the agreement in the MR angiographic diagnoses of residual or recurrent aneurysms in patients treated by endovascular coiling by a larger number of observers with various expertise is needed to establish the reliability of MRA diagnoses.7

In clinical practice, the evolution of treated aneurysms is often followed by comparing the follow-up MRA with the final catheter angiogram of the embolization procedure. The reliability of the diagnoses of a stable occlusion or a recurrent lesion when the verdict depends on a comparison between 2 different imaging modalities (the MRA and the catheter angiographic results) has so far not been studied, to our knowledge.

Our objective was to assess inter- and intraobserver agreement in diagnosing aneurysm remnants and recurrences by using multimodality imaging, including DSA-MRA and MRA-MRA comparisons in close-to-clinical conditions.

## Materials and Methods

The present report was written in compliance with the Guidelines for Reporting Reliability and Agreement studies.8 The evaluation of the intra- and interobserver variability in adjudicating outcomes of endovascular treatment was primarily done by electronic survey by using a portfolio of selected images (DSA-MRA and MRA-MRA) to ease the participation of multiple readers from various backgrounds, institutions, and countries. A second study restricted to expert readers having full access to the set of angiographic and MRA data on the same patients on the hospital PACS system of a single institution was designed to validate the results of the electronic survey and resemble clinical working conditions.

### Cases

On the basis of Donner and Rotondi9 (where for an expected K of 0.600 with a prevalence of 0.3 and 5 raters, 24 subjects are sufficient for the lower limit of a 95% 1-sided confidence limit to be no less than 0.400), we estimated that 24 cases per group (MRA-MRA and DSA-MRA comparisons for a total of 48 cases) would suffice to provide meaningful results. The number of cases was increased to 56 to account for potential missing responses and to include a spectrum of patients followed by using various follow-up methods. Images were retrieved from 52 patients with 56 coiled aneurysms (all platinum coils) followed between May 2012 and June 2013 in 1 center. Cases were selected to include at least 2 comparable images from 1.5T or 3T MRI or angiographic series either immediately following treatment or later. Two authors (J.-C.G. and S.J.) selected the cases, aiming to include approximately 50% of easily replicable verdicts (25% of large recurrences, 25% of stable occlusions) and 50% of less clear cases. Cases and proportions were chosen to mimic a typical endovascular case series and to aim for a prevalence of approximately 30%–40% major recurrences, to minimize paradoxes of κ statistics.10,11 The characteristics of patients are summarized in Table 1. Detailed characteristics for each patient can be found in On-line Table 1.

View this table:
[Table 1:](http://www.ajnr.org/content/37/5/879/T1)

Table 1: 
Characteristics of patients

### Portfolio of Images

A portfolio composed of 120 images from 56 cases (typically 1 pair of images per case) was assembled. On each page of the electronic survey, 1 postembolization and 1 follow-up image (at least) or 2 follow-up MRA images were displayed side by side. Most MR images (*n* = 92) were time-of-flight source images, but 8 were 3D reconstructions.

There were 32 MRA-MRA pairs (22 comparing 1.5T–3T; three, 1.5T–1.5T; five, 3T–3T; and 2 comparing 3T–1.5T) and 24 DSA-MRA pairs (14 comparing DSA with 3T and 10 with 1.5T MRA). All MR imaging examinations were performed in 1 center; MR imaging protocols used for postcoiling surveillance included 3D TOF angiography of the circle of Willis on either a 1.5T Avanto (Siemens, Erlangen, Germany) or a 3T Achieva X system (Phillips Healthcare, Best, the Netherlands). The 1.5T 3D TOF sequence was performed with the following parameters: TR, 25 ms; TE, 7 ms; section thickness, 0.7 mm; FOV, 200 mm. The 3T 3D TOF sequence was performed with the following parameters: TR, 25 ms; TE, 3.5 ms; section thickness, 1.0 mm; FOV, 200 mm.

No clinical information was provided. Observers had to independently grade each image according to a 3-value scale (complete occlusion, residual neck, residual aneurysm), graphically displayed on each page.5,12 They were also asked to make a final judgment regarding the presence of a recurrence, according to a 3-value scale (no recurrence, minor recurrence, major recurrence) by comparing the 2 images. The definition of a major recurrence was “a saccular recurrence of a size sufficient to allow retreatment.” Any other increase in the residuum was to be labeled a minor recurrence.5,13 The portfolio was provided electronically (On-line Appendix).

### Raters

The portfolio was sent to 15 participants, selected because they had served as a core lab for endovascular trials (*n* = 5), participated in ongoing trials (*n* = 5), or were on lists of potential participants (*n* = 5). There were 14 interventionists (13 neuroradiologists) working in 10 different centers from 3 different countries (United States, France, and Canada). There were 8 senior (4 with >10 and 4 with >20 years; maximum, 40 years of experience) and 7 junior observers (<10 years of experience; minimum, 3 years).

The portfolio was sent twice electronically at least 3 months apart to 6 raters, blinded to their previous responses, who agreed to participate in the intraobserver agreement study.

### PACS Study

Because agreement on judgments based on pairs of selected images differs from the normal clinical context, the same cases were independently reviewed, in a random order provided by another investigator, by 4 observers (all interventional neuroradiologists) having access to all images. Two senior observers also independently assessed the same cases twice on the PACS system >3 months apart in a different random order.

### Statistics

The interrater agreement regarding the angiographic results (in 3 categories at 2 points in time [A and B]) and the final judgment regarding the presence of a recurrence (no, minor, or major recurrence) for the 15 raters by using the portfolio were estimated by using generalized κ. The 95% confidence intervals are reported. Stratified analyses according to follow-up imaging methods (DSA-MRA or MRA-MRA) or experience (all, seniors, juniors) were performed. For the 6 judges with a replicated judgment on the portfolio, intrarater agreement was also estimated by using κ statistics. The observed minimum, median, and maximum values of κ statistics are reported, including the 95% confidence interval. Because the primary angiographic end point of many trials has been the occurrence of a major recurrence, agreement was also analyzed in 2 categories (major recurrence, yes or no). For the PACS study, the inter- and intrarater agreement was estimated similarly by using κ statistics. All analyses were performed by using SAS, Version 9.2 (SAS Institute, Cary, North Carolina). All categories, such as “fair,” “moderate,” or “substantial” agreement, were qualified according to Landis and Koch.14

## Results

Detailed statistical results can be found in On-line Tables 2–6.

### Angiographic Results and Final Judgment on the Electronic Survey

There was a wide variability in angiographic results, with 2–23 cases (3.6%–41.1%) being judged as having a residual aneurysm on the first posttreatment evaluation. On the second set of images, between 12 and 39 cases (21.4%–69.6%) were judged as presenting a residual aneurysm. The proportion of portfolio cases judged to have a major recurrence at follow-up MRA varied between 16.1% and 71.4% (mean, 35.0% ± 12.7%) (On-line Table 2).

### Interrater and Intrarater Agreement on the Electronic Survey

Interrater agreement regarding the 3 different categories of angiographic results for the 2 sets of images is summarized in On-line Tables 3–6. The category showing a lesser degree of agreement was “residual neck.” Agreement regarding the presence of a recurrence (3 categories) varied greatly between raters from slight (κ = 0.156 ± 0.069 [95% CI, 0.021–0.292]) to almost perfect agreement (κ = 0.874 ± 0.049 [95% CI, 0.777–0.971]). There was moderate overall agreement among raters regarding the final verdict (κ = 0.474 ± 0.009); the category showing a lesser degree of agreement was “minor recurrence.” Agreement improved to nearly substantial (κ = 0.581 ± 0.014) when the judgment was dichotomized as the presence or absence of a major recurrence (Table 2). Overall agreement regarding dichotomized verdicts on cases followed-up by MRA-MRA comparisons was similarly substantial (κ = 0.601 ± 0.018).

View this table:
[Table 2:](http://www.ajnr.org/content/37/5/879/T2)

Table 2: 
Summary of agreements on the electronic survey

Agreement regarding a final dichotomized judgment among experienced raters (≥10 years; *n* = 8; κ = 0.501 ± 0.027) was no better than the agreement among less experienced observers (0–9 years of experience; *n* = 7; κ = 0.660 ± 0.031).

The intrarater agreement regarding final angiographic verdicts varied between fair (κ = 0.257 ± 0.093 [95% CI, 0.074–0.440]) and substantial (κ = 0.699 ± 0.084 [95% CI, 0.534–0.864]), improving with a dichotomized judgment, becoming moderate to almost perfect when a dichotomized judgment concerned MRA-MRA comparisons (Table 2). Three of 6 raters reached a “substantial” level of agreement between 2 interpretations.

### PACS Reading

Raters having access to all images on the PACS system found major recurrences in 13.2%–57.4% of cases (mean, 32.4% ± 15.3%). Agreement was no better when raters had full access to all images than among raters having access to only the selected images of the portfolio (Table 3). Interobserver agreement regarding the presence or absence of a major recurrence varied from slight to substantial, with a median κ of 0.455. Examples are illustrated in Fig 1.

View this table:
[Table 3:](http://www.ajnr.org/content/37/5/879/T3)

Table 3: 
Summary of agreement on PACS reading

![Fig 1.](http://www.ajnr.org/http://asnr-client.stage.highwire.org/content/ajnr/37/5/879/F1.medium.gif)

[Fig 1.](http://www.ajnr.org/content/37/5/879/F1)

Fig 1. 
Illustrative cases from the electronic survey. Case number 10 is a 66-year-old woman with an unruptured aneurysm of the left ICA bifurcation. All 15 readers agreed on score A (complete occlusion) on first follow-up 1.5T MRA (*A*) and second follow-up 3T MRA (*B*). All 15 readers agreed on the final result, “no recurrence.” Case number 31 is a 59-year-old woman with an unruptured aneurysm of the left ICA. All 15 readers agreed on score A (complete occlusion) on first follow-up DSA (*C*), and 13/15 readers agreed on score B (residual neck) on second follow-up 3T MRA (*D*). Finally, 14/15 readers agreed on the final result “minor recurrence.” Case number 53 is a 61-year-old woman with an unruptured aneurysm of the anterior communicating artery. On the first follow-up DSA (*E*), 10/15 readers answered “residual neck” (score B), and 5/15 readers, “residual aneurysm” (score C). On the second follow-up 3T MRA (*F*), 7/15 readers answered “residual neck” (score B), and 8/15 readers, “residual aneurysm” (score C). The final angiographic result was “no recurrence” for 2/15 readers, “minor recurrence” for 6/15 readers, and “major recurrence” for 7/15 readers.

### MRA-MRA Comparisons

Inter- and intrarater agreement regarding the presence of a recurrence in the subgroup of 32 cases that were studied twice with MRA was very similar to the overall results (Table 2). Power was insufficient to study whether agreement differed with MRAs of different field strengths.

## Discussion

There was wide variability in the adjudication of outcomes of endovascular coiling on selected images from catheter angiographic or TOF MRA studies. Agreement was moderate at best and did not improve with rater experience. Disagreement among observers is not explained by divergences in the interpretations of the meaning of the various categories of the scale because agreement between 2 observations from the same raters was also fair to moderate in most cases. Reassuringly, agreement nearly reached a “substantial” level (κ ≥ 0.600) when the final verdict regarding the presence of a major recurrence was dichotomized (present-absent). Agreement when comparing the last angiographic result with the first MRA follow-up study (DSA-MRA), as commonly performed in clinical practice, was similar to MRA-MRA agreement. Agreement was not improved when raters had full access to all images.

### Use of MRA for Aneurysm Follow-Up

Previous studies on the reliability of postcoiling follow-up imaging studies have been systematically reviewed.4 The overall diagnostic reliability was substantial (κ = 0.65), but there was significant heterogeneity. Furthermore, the reliability of MRA follow-up examinations, now frequently replacing catheter angiographic studies, was not severely tested, for the primary aim of most studies was to assess diagnostic accuracy; interobserver agreement and reliability were conducted with few raters and reported as secondary quality measures.4

By contrast, our primary aim was to test the reproducibility of angiographic outcomes as they are assessed in clinical practice with a variety of modalities (DSA, 1.5T and 3T TOF MRA) that are actually used in realistic cases selected to mimic a clinical series. Although Pierot et al6 have proposed that contrast-enhanced MRA performed better than TOF-MRA, most studies did not find a significant difference in accuracy between modalities or field strengths (3T or 1.5T), and for pragmatic reasons, patients cannot always be followed on the same equipment. The strengths of the present study also include a large number of observers with various experience and from various institutions and the inclusion of DSA-MRA comparisons that are often used in clinical practice.

### Purpose of Follow-Up after Coiling

Endovascular treatment of ruptured intracranial aneurysms was shown to improve patient outcomes compared with surgery.15 It is also commonly used to treat unruptured aneurysms, even though it has not been shown superior to clipping or to conservative management.16

Angiography is a pragmatic way to assess the results of treatment of unruptured aneurysms because trials powered to show a decreased incidence of clinical ruptures would necessitate a large number of patients followed for a long time.16 The main drawback of coiling, compared with surgery, is the risk of angiographic recurrence, reported to occur in 10%–20% of patients.17⇓–19 Second-generation coils20⇓⇓⇓⇓–25 and, in some centers, stents and flow diverters26 have been proposed to improve the stability of treatment. Angiographic results are still the main surrogate end point of ongoing trials on modified coils27,28 or of other studies performed for the approval of endovascular devices.29 However, the clinical significance of angiographic recurrences remains controversial.30 Aneurysm ruptures have been rare, and whether treated patients should even be followed has recently been questioned.30 Some authors believe that most recurrences occur early and that follow-up imaging of aneurysms shown to be stable at 6 months is not necessary.2,31

A randomized trial, following versus not following patients with imaging, may be indicated to settle the issue. What to do when a recurrence is identified is even more controversial. There is little agreement regarding indications for retreatment,32 though substantial agreement between decisions based on MRA and conventional angiography has been shown in 1 study.33

### Various Scales

At least 21 different grading scales have been reported,4 but most can be translated into the 3-value scale we have previously proposed.11,34 It has been observed that agreement increases as the number of categories is decreased.35 Other ways to improve the reliability of angiographic judgments have been proposed, by using volumetric measurements of residual lesions,4 or by increasing the precision of nominal definitions,24 but the success of such strategies remains to be demonstrated. The present work supports the idea that agreement among observers can reach an acceptable level when the scale is translated into a simple dichotomous verdict (presence or absence of a major recurrence).

### Limitations

Our study has several limitations. First, we were careful to include a wide variety of cases that would test agreement in circumstances that were close to clinical conditions with multimodality imaging, but the selection of cases, images, and image parameters was artificial; a different set of cases could have led to different results. Second, we did not include patients treated with stents or stent-assisted coiling; this choice may affect MRA interpretations. Third, aneurysms were, on average, larger; and posterior circulation aneurysms were more frequent than those in typical endovascular series, perhaps because we included a sufficient number of major recurrences to minimize the paradoxes of κ statistics.10,11 The variability we observed in judging the extent of angiographic occlusion of treated aneurysms and the presence of a recurrence at follow-up was probably underestimated by the portfolio method we used to multiply the number of raters. Many potential sources of discrepancies (selection of images, series, or sequences; diverse techniques; and equipment from various centers) were absent. Nevertheless, the substudy on a smaller number of readers having access to all images from the PACS yielded similar results. Finally, how seriously observers worked to come to verdicts can always be questioned, and the context of assessment certainly differed from the normal clinical context.

## Conclusions

There is an important variability in the assessment of angiographic outcomes of endovascular treatments. Agreement on the presence of a major recurrence when comparing 2 MRA studies or the MRA with the last catheter angiographic study can be substantial.

## Footnotes

*   Disclosures: Robert Fahed—*RELATED*: *Grant*: Fondation pour la Recherche Médicale, grant number DEA20140630151, *Comments*: The research grant was accorded to me (Dr. Robert Fahed) from November 2014 to October 2015 as a full-time research year. Laurent Letourneau-Guillon—*UNRELATED*: *Payment for Lectures (including service on Speakers Bureaus)*: Societe Canadienne Francaise de Radiologie, *Comments*: honorarium for 1 educational lecture at a radiology meeting. Allan J. Fox—*UNRELATED*: *Expert Testimony*: occasional medicolegal case (not relevant to the topic in question); *Stock/Stock Option*s: none related to companies on this topic. David F. Kallmes—*UNRELATED*: *Board Membership*: GE Healthcare (Cost-Effectiveness Board membership),* *Consultancy*: Medtronic (planning and implementing clinical trials)*; *Grants/Grants Pending*: MicroVention,* Medtronic,* Codman,* Sequent Medical,* NeuroSigma,* Surmodics*; *Royalties*: University of Virginia Patent Foundation. Jean Raymond—*UNRELATED*: *Grants/Grants Pending*: Canadian Institutes of Health research (Canadian UnRuptured Endovascular Versus Surgery Trial),* Codman (Does Embolization With Larger Coils Lead to Better Treatment of Aneurysms Trial).* *Money paid to the institution.

*   Robert Fahed is the recipient of a research scholarship delivered by the Fondation pour la Recherche Médicale, Paris, France (grant number DEA20140630151).

## References

1.  1. van Amerongen MJ, Boogaarts HD, de Vries J, et al. MRA versus DSA for follow-up of coiled intracranial aneurysms: a meta-analysis. AJNR Am J Neuroradiol 2014;35:1655–61 doi:10.3174/ajnr.A3700 pmid:24008171
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIzNS85LzE2NTUiO3M6NDoiYXRvbSI7czoxOToiL2FqbnIvMzcvNS84NzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  2. Sprengers ME, Schaafsma J, van Rooij WJ, et al. Stability of intracranial aneurysms adequately occluded 6 months after coiling: a 3T MR angiography multicenter long-term follow-up study. AJNR Am J Neuroradiol 2008;29:1768–74 doi:10.3174/ajnr.A1181 pmid:18583406
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIyOS85LzE3NjgiO3M6NDoiYXRvbSI7czoxOToiL2FqbnIvMzcvNS84NzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  3. Schaafsma JD, Velthuis BK, Majoie CB, et al. Intracranial aneurysms treated with coil placement: test characteristics of follow-up MR angiography—multicenter study. Radiology 2010;256:209–18 doi:10.1148/radiol.10091528 pmid:20505063
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.10091528&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=20505063&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000279106900025&link_type=ISI) 

4.  4. Ernst M, Yoo AJ, Kriston L, et al. Is visual evaluation of aneurysm coiling a reliable study end point? Systematic review and meta-analysis. Stroke 2015;46:1574–81 doi:10.1161/STROKEAHA.114.008513 pmid:25944331
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjQ2LzYvMTU3NCI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

5.  5. Tollard É, Darsaut TE, Bing F, et al. Outcomes of endovascular treatments of aneurysms: observer variability and implications for interpreting case series and planning randomized trials. AJNR Am J Neuroradiol 2012;33:626–31 doi:10.3174/ajnr.A2848 pmid:22194386
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzMy80LzYyNiI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

6.  6. Pierot L, Portefaix C, Gauvrit JY, et al. Follow-up of coiled intracranial aneurysms: comparison of 3D time-of-flight MR angiography at 3T and 1.5T in a large prospective series. AJNR Am J Neuroradiol 2012;33:2162–66 doi:10.3174/ajnr.A3124 pmid:22678846
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czoxMDoiMzMvMTEvMjE2MiI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.  7. Bankier AA, Levine D, Halpern EF, et al. Consensus interpretation in imaging research: is there a better way? Radiology 2010;257:14–17 doi:10.1148/radiol.10100252 pmid:20851935
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.10100252&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=20851935&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000282527200004&link_type=ISI) 

8.  8. Kottner J, Audigé L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011;64:96–106 doi:10.1016/j.jclinepi.2010.03.002 pmid:21130355
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.03.002&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21130355&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 

9.  9. Donner A, Rotondi MA. Sample size requirements for interval estimation of the kappa statistic for interobserver agreement studies with a binary outcome and multiple raters. Int J Biostat 2010;6:Article 31 pmid:21969984
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21969984&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 

10. 10. Feinstein AR, Cicchetti DV. High agreement but low kappa, I: the problems of two paradoxes. J Clin Epidemiol 1990;43:543–49 doi:10.1016/0895-4356(90)90158-L pmid:2348207
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/0895-4356(90)90158-L&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=2348207&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=A1990DG55500002&link_type=ISI) 

11. 11. Cicchetti DV, Feinstein AR. High agreement but low kappa, II: resolving the paradoxes. J Clin Epidemiol 1990;43:551–58 pmid:2189948
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/0895-4356(90)90159-M&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=2189948&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=A1990DG55500003&link_type=ISI) 

12. 12. Roy D, Milot G, Raymond J. Endovascular treatment of unruptured aneurysms. Stroke 2001;32:1998–2004 doi:10.1161/hs0901.095600 pmid:11546888
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjMyLzkvMTk5OCI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. 13. Raymond J, Guilbert F, Weill A, et al. Long-term angiographic recurrences after selective endovascular treatment of aneurysms with detachable coils. Stroke 2003;34:1398–403 doi:10.1161/01.STR.0000073841.88563.E9 pmid:12775880
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjM0LzYvMTM5OCI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14. 14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74 doi:10.2307/2529310 pmid:843571
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 

15. 15. Molyneux A, Kerr R, Stratton I, et al. International Subarachnoid Aneurysm Trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomized trial. Lancet 2002;360:1267–74 doi:10.1016/S0140-6736(02)11314-6 pmid:12414200
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S0140-6736(02)11314-6&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=12414200&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000178827200007&link_type=ISI) 

16. 16. Darsaut TE, Findlay JM, Raymond J. The design of the Canadian UnRuptured Endovascular versus Surgery (CURES) trial. Can J Neurol Sci 2011;38:236–41 doi:10.1017/S0317167100011392 pmid:21320826
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1017/S0317167100011392&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21320826&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000288350300012&link_type=ISI) 

17. 17. Crobeddu E, Lanzino G, Kallmes DF, et al. Review of 2 decades of aneurysm-recurrence literature, part 1: reducing recurrence after endovascular coiling. AJNR Am J Neuroradiol 2013;34:266–70 doi:10.3174/ajnr.A3032 pmid:22422180
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzNC8yLzI2NiI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

18. 18. Crobeddu E, Lanzino G, Kallmes DF, et al. Review of 2 decades of aneurysm-recurrence literature, part 2: managing recurrence after endovascular coiling. AJNR Am J Neuroradiol 2013;34:481–85 doi:10.3174/ajnr.A2958 pmid:22422182
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzNC8zLzQ4MSI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

19. 19. Lecler A, Raymond J, Rodriguez-Régent C, et al. Intracranial aneurysms: recurrences more than 10 years after endovascular treatment: a prospective cohort study, systematic review, and meta-analysis. Radiology 2015;277:173–80 doi:10.1148/radiol.2015142496 pmid:26057784
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.2015142496&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26057784&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 

20. 20. White PM, Lewis SC, Gholkar A, et al; HELPS trial collaborators. Hydrogel-coated coils versus bare platinum coils for the endovascular treatment of intracranial aneurysms (HELPS): a randomised controlled trial. Lancet 2011;377:1655–62 doi:10.1016/S0140-6736(11)60408-X pmid:21571149
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S0140-6736(11)60408-X&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21571149&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000290777700030&link_type=ISI) 

21. 21. White PM, Lewis SC, Nahser H, et al; HELPS Trial Collaboration. HydroCoil Endovascular Aneurysm Occlusion and Packing Study (HELPS trial): procedural safety and operator-assessed efficacy results. AJNR Am J Neuroradiol 2008;29:217–23 doi:10.3174/ajnr.A0936 pmid:18184832
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIyOS8yLzIxNyI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

22. 22. McDougall CG, Claiborne Johnston S, Gholkar A, et al; MAPS Investigators. Bioactive versus bare platinum coils in the treatment of intracranial aneurysms: the MAPS (Matrix and Platinum Science) trial. AJNR Am J Neuroradiol 2014;35:935–42 doi:10.3174/ajnr.A3857 pmid:24481333
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzNS81LzkzNSI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

23. 23. Coley S, Sneade M, Clarke A, et al. Cerecyte coil trial: procedural safety and clinical outcomes in patients with ruptured and unruptured intracranial aneurysms. AJNR Am J Neuroradiol 2012;33:474–80 doi:10.3174/ajnr.A2836 pmid:22207299
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzMy8zLzQ3NCI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

24. 24. Molyneux AJ, Clarke A, Sneade M, et al. Cerecyte coil trial: angiographic outcomes of a prospective randomized trial comparing endovascular coiling of cerebral aneurysms with either Cerecyte or bare platinum coils. Stroke 2012;43:2544–50 doi:10.1161/STROKEAHA.112.657254 pmid:22836352
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjEwOiI0My8xMC8yNTQ0IjtzOjQ6ImF0b20iO3M6MTk6Ii9ham5yLzM3LzUvODc5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

25. 25. Rezek I, Lingineni RK, Sneade M, et al. Differences in the angiographic evaluation of coiled cerebral aneurysms between a core laboratory reader and operators: results of the Cerecyte coil trial. AJNR Am J Neuroradiol 2014;35:124–27 doi:10.3174/ajnr.A3623 pmid:23868159
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzNS8xLzEyNCI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

26. 26. Brinjikji W, Murad MH, Lanzino G, et al. Endovascular treatment of intracranial aneurysms with flow diverters: a meta-analysis. Stroke 2013;44:442–47 doi:10.1161/STROKEAHA.112.678151 pmid:23321438
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjg6IjQ0LzIvNDQyIjtzOjQ6ImF0b20iO3M6MTk6Ii9ham5yLzM3LzUvODc5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

27. 27. Raymond J, Klink R, Chagnon M, et al; PRET Collaborative Group. Patients prone to recurrence after endovascular treatment: periprocedural results of the PRET randomized trial on large and recurrent aneurysms. AJNR Am J Neuroradiol 2014;35:1667–76 doi:10.3174/ajnr.A4035 pmid:24948508
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIzNS85LzE2NjciO3M6NDoiYXRvbSI7czoxOToiL2FqbnIvMzcvNS84NzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

28. 28.ClinicalTrials.gov. Hydrogel Endovascular Aneurysm Treatment Trial (HEAT). [https://clinicaltrials.gov/ct2/show/NCT01407952](https://clinicaltrials.gov/ct2/show/NCT01407952). Accessed July 23, 2015. 
    
    

29. 29. Becske T, Kallmes DF, Saatci I, et al. Pipeline for uncoilable or failed aneurysms: results from a multicenter clinical trial. Radiology 2013;267:858–68 doi:10.1148/radiol.13120099 pmid:23418004
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.13120099&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=23418004&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000319445400023&link_type=ISI) 

30. 30. Molyneux AJ, Birks J, Clarke A, et al. The durability of endovascular coiling versus neurosurgical clipping of ruptured cerebral aneurysms: 18 year follow-up of the UK cohort of the International Subarachnoid Aneurysm Trial (ISAT). Lancet 2015;385:691–97 doi:10.1016/S0140-6736(14)60975-2 pmid:25465111
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S0140-6736(14)60975-2&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=25465111&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 

31. 31. van Rooij WJ, Sluzewski M. Opinion: imaging follow-up after coiling of intracranial aneurysms. AJNR Am J Neuroradiol 2009;30:1646–48 doi:10.3174/ajnr.A1673 pmid:19617448
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIzMC85LzE2NDYiO3M6NDoiYXRvbSI7czoxOToiL2FqbnIvMzcvNS84NzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

32. 32. McDonald JS, Carter RE, Layton KF, et al. Interobserver variability in retreatment decisions of recurrent and residual aneurysms. AJNR Am J Neuroradiol 2013;34:1035–39 doi:10.3174/ajnr.A3326 pmid:23099500
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIzNC81LzEwMzUiO3M6NDoiYXRvbSI7czoxOToiL2FqbnIvMzcvNS84NzkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

33. 33. Schaafsma JD, Velthuis BK, van den Berg R, et al. Coil-treated aneurysms: decision making regarding additional treatment based on findings of MR angiography and intraarterial DSA. Radiology 2012;265:858–63 doi:10.1148/radiol.12112608 pmid:23012464
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.12112608&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=23012464&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 

34. 34. Raymond J, Roy D, Bojanowski M, et al. Endovascular treatment of acutely ruptured and unruptured aneurysms of the basilar bifurcation. J Neurosurg 1997;86:211–19 doi:10.3171/jns.1997.86.2.0211 pmid:9010422
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.3171/jns.1997.86.2.0211&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=9010422&link_type=MED&atom=%2Fajnr%2F37%2F5%2F879.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=A1997WD67700008&link_type=ISI) 

35. 35. Cloft HJ, Kaufmann T, Kallmes DF. Observer agreement in the assessment of endovascular aneurysm therapy and aneurysm recurrence. AJNR Am J Neuroradiol 2007;28:497–500 pmid:17353321
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIyOC8zLzQ5NyI7czo0OiJhdG9tIjtzOjE5OiIvYWpuci8zNy81Lzg3OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

*   Received July 23, 2015.
*   Accepted after revision September 14, 2015.


*   © 2016 by American Journal of Neuroradiology