ARTICLE

Changes in Breast Cancer Detection and Mammography Recall Rates After the Introduction of a Computer-Aided Detection System

David Gur, Jules H. Sumkin, Howard E. Rockette, Marie Ganott, Christiane Hakim, Lara Hardesty, William R. Poller, Ratan Shah, Luisa Wallace

Affiliations of authors: Department of Radiology, University of Pittsburgh, and Magee-Womens Hospital of the University of Pittsburgh Medical Center, Pittsburgh, PA (DG, JHS, HER, MG, CH, LH, WRP, RS, LW); Department of Biostatistics, University of Pittsburgh, Pittsburgh, PA (HER).

Correspondence to: David Gur, ScD, Imaging Research, Suite 4200, Department of Radiology, University of Pittsburgh, 300 Halket St., Pittsburgh, PA 15213 (e-mail: gurd{at}upmc.edu)


    ABSTRACT
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Background: Computer-aided mammography is rapidly gaining clinical acceptance, but few data demonstrate its actual benefit in the clinical environment. We assessed changes in mammography recall and cancer detection rates after the introduction of a computer-aided detection system into a clinical radiology practice in an academic setting. Methods: We used verified practice- and outcome-related databases to compute recall rates and cancer detection rates for 24 Mammography Quality Standards Act-certified academic radiologists in our practice who interpreted 115 571 screening mammograms with (n = 59 139) or without (n = 56 432) the use of a computer-aided detection system. All statistical tests were two-sided. Results: For the entire group of 24 radiologists, recall rates were similar for mammograms interpreted without and with computer-aided detection (11.39% versus 11.40%; percent difference = 0.09, 95% confidence interval [CI] = -11 to 11; P = .96) as were the breast cancer detection rates for mammograms interpreted without and with computer-aided detection (3.49% versus 3.55% per 1000 screening examinations; percent difference = 1.7, 95% CI = -11 to 19; P = .68). For the seven high-volume radiologists (i.e., those who interpreted more than 8000 screening mammograms each over a 3-year period), the recall rates were similar for mammograms interpreted without and with computer-aided detection (11.62% versus 11.05%; percent difference = -4.9, 95% CI = -21 to 4; P = .16), as were the breast cancer detection rates for mammograms interpreted without and with computer-aided detection (3.61% versus 3.49% per 1000 screening examinations; percent difference = -3.2, 95% CI = -15 to 9; P = .54). Conclusion: The introduction of computer-aided detection into this practice was not associated with statistically significant changes in recall and breast cancer detection rates, both for the entire group of radiologists and for the subset of radiologists who interpreted high volumes of mammograms.



    INTRODUCTION
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
A mounting body of evidence suggests that early detection of breast cancer through periodic mammography screening reduces the morbidity and mortality associated with this disease (1,2). Mammography screening is rapidly gaining acceptance worldwide, and the number of mammography procedures performed continues to increase (3,4). However, mammography screening has a relatively low cancer detection rate of only two to six cancers per 1000 mammograms after the first 2 years of screening (5).

The performance levels among radiologists who read and interpret mammograms vary widely. Several factors may account for this variability. These include, but are not limited to, the low incidence of breast cancer, the difficulty in identifying suspicious (i.e., potentially malignant) regions in the surrounding breast tissue, and the tedious and somewhat repetitious nature of the task of reading mammograms (57).

In recent years, a major effort has been expended to develop computer-aided detection systems to assist radiologists with the diagnostic process. The hope is that these computer-aided detection systems will improve the sensitivity of mammography without substantially increasing mammography recall rates, in addition to possibly decreasing inter-reader variability. These systems are intended for the early detection of breast cancer and, accordingly, are designed to assist the radiologist in the identification (i.e., detection) of suspicious regions (i.e., findings), such as clustered microcalcifications and masses (810). Computer-aided diagnosis (discrimination) systems are currently being developed to help radiologists determine whether an identified suspicious region is likely to represent a benign or a malignant finding (1113).

The U.S. Food and Drug Administration (FDA) has approved several computer-aided detection systems for clinical use, and Medicare and many insurance companies have approved reimbursement for the use of these systems in clinical practice. The initial FDA approval process for these systems included retrospective interpretations of select groups of cases in a laboratory environment (9,14,15). Results of these studies (9,15) suggest that the use of computer-aided detection systems can potentially increase cancer detection rates by approximately 20% without substantially increasing recall rates. However, there are only limited data on the impact of such systems when used prospectively in a clinical environment (1619). We used large, prospectively ascertained databases to evaluate the recall and cancer detection rates in our clinical breast imaging practice in an academic setting for a 3-year period during which a computer-aided diagnosis system was introduced.


    METHODS
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Subjects and General Procedures

All screening mammography examinations performed in our facilities at Magee-Womens Hospital of the University of Pittsburgh Medical Center (Pittsburgh, PA) and its five satellite breast imaging clinics during 2000, 2001, and 2002 were included in this study. Our study was carried out under an institutional review board-approved protocol.

The data sources for our analysis were databases that contained information on procedure scheduling, procedure completion, radiology reporting, and procedure-related outcomes as determined from relevant pathology reports. These databases were assembled from the original reports for quality assurance purposes, as required by the Mammography Quality Standards Act (MQSA) (20), among other reasons. The same computerized reporting system was in use throughout the study period.

In the second quarter of 2001, we introduced a computer-aided detection system (R2 Technologies, Los Altos, CA) into our clinical practice at the main facility, where most of the screening mammograms in our practice were read in batch mode. By the third quarter of 2001, more than 70% of the screening mammograms were interpreted with use of the computer-aided detection system. By the fourth quarter of 2001, more than 80% of the screening mammograms were interpreted with the assistance of the computer-aided detection system. The radiologists in our practice could not select which mammograms would be interpreted with or without the computer-aided detection system. After training on the computer-aided detection system was completed (June 2001), all screening mammograms interpreted in our main facility were processed by and interpreted with the assistance of the computer-aided detection system. Radiologists at the five satellite clinics sometimes reviewed screening mammograms if time allowed, but the number of these cases was small, and there was no selection process that could bias the analyses performed in this study. Knowing the schedule for radiologists' presence at the remote sites, we assembled a batch of serially acquired mammograms for them to read in the same way they would be read at the central facility, and those mammograms were interpreted and reported in the same manner (with the exception of the use of computer-aided detection). This set of mammograms was not specifically selected because of suspicious findings by the technologists. To reduce possible biases, an individual not involved in this investigation was asked to examine summaries of time-dependent recall rates for all radiologists in our practice for the study period. A different team examined all cancers detected throughout our practice as a result of screening mammography during the same period.

During the study period, our practice performed a total of 115 571 screening examinations that were interpreted by 24 radiologists, 18 of whom interpreted more than 1000 mammograms each. All radiologists were members of the Breast Imaging Section of the Department of Radiology and would be considered breast imaging specialists in an academic practice. We also repeated our analysis by using only data for the seven highest volume radiologists, all of whom read more than 8000 mammograms each over a 3-year period. These seven radiologists, who were with our institution throughout the study period, performed the most readings, both with and without computer-aided detection assistance.

For the purpose of computing recall rates, mammograms were considered to be positive if recall for additional imaging evaluation was recommended (i.e., mammograms classified as Breast Imaging Reporting and Data System [BI-RADS] category 0) and negative if a 1-year follow-up was recommended (i.e., mammograms classified as either BI-RADS category 1 or 2) (21). Radiologists at these facilities did not use BI-RADS assessment categories 3, 4, or 5 for screening examinations. Positive outcome was defined as breast cancer detected as a result of the diagnostic work-up initiated by a positive screening mammogram.

Computation of Mammography Recall Rates

Recall rates for each radiologist and for the group of 24 radiologists were computed directly from mammographic interpretation records. In all of our analyses, we excluded recommendations for recall that were due to technical reasons, such as image artifacts (<1%). Recalls due to palpable findings identified during clinical breast examinations performed on all women by the technologist were included in our analyses because the majority of these findings were also marked on the mammograms. Such recalls amounted to approximately 1% of the screening examinations; hence, the underlying rates attributable to mammography interpretations alone are approximately 1% lower than those reported here. The women in this group of recalls are not the same as the group of women with palpable findings discovered by the woman herself or by a physician during a breast physical examination. Women in the latter group were scheduled for diagnostic examinations and were not included in our study. In our practice, palpable findings that are discovered by the technologists are noted during the physical examination and the procedure continues as a screening examination (including the use of computer-aided detection). The interpreting radiologists are aware of the technologists' findings and recall the women for additional procedures as needed. We recognize that this practice may not be a common one. We assumed that the effects of recalling this group of women due to palpable findings, if any, on the recall rates of individual radiologists would be proportional to the overall volume of mammograms read by each radiologist; hence, it should not substantially affect the results.

A small percentage (<4%) of the examinations in our practice classified as BI-RADS category 0 were scheduled for an interpretation at a later date because the needed comparison films were missing during the originally scheduled interpretation. Those cases were distributed proportionally to the volume of mammograms read by each radiologist and were included in the recall rates because it was not clear how many of them would have been recalled anyway.

Each mammography examination was identified in our database as to whether computer-aided detection was used during the interpretation. We therefore analyzed the data according to whether cases were interpreted with computer-aided detection.

Computation of Breast Cancer Detection Rates

Breast cancer detection rates were computed as follows: For every breast cancer detected, we found the most recent screening mammogram that identified a finding that led to a diagnostic follow-up and ultimately resulted in a biopsy that was positive for cancer. Only the interpreter of the original screening mammogram that led to the detection of breast cancer was credited with the finding (i.e., invasive and ductal carcinoma in situ). Findings of lobular carcinoma in situ were not attributed to the interpreting radiologist as a cancer detected in the analyses. If a woman was recommended for a biopsy directly as a result of the screening examination, the interpreter was credited with the finding as well. Cases were excluded from the analysis if the most recent screening mammogram prior to biopsy had been performed more than 180 days before the biopsy or if the original interpreter had not recommended a recall (i.e., false-negative cases). We chose a cutoff of 180 days because we have found that, in the vast majority of cases, women are lost to follow-up or ignore the recall recommendation altogether if the recommended follow-up diagnostic procedure is not scheduled within 90 days or performed within 180 days of the original mammogram. We attributed any subsequent findings associated with recalls for diagnostic work-ups that did not take place within 180 days of the original mammogram to the subsequent examination. We included all examinations that that had been originally scheduled as screening procedures but were diagnosed during the same visit and during which a diagnosis was made that resulted in a positive outcome (i.e., converted into a diagnostic procedure that led to a finding of cancer). However, these cancer cases (n = 30) were excluded from the computed breast cancer detection rates in our analysis (both nominator and denominator) because they were all diagnosed by a radiologist without the use of computer-aided detection, and we therefore could not determine whether these cases would have been detected had they undergone routine interpretation (with or without computer-aided detection) as a routine screening procedure. In addition, all breast cancer patients who were referred to us from other facilities and for whom the diagnosis did not originate from a screening examination done at one of our facilities were excluded from the analysis.

Statistical Methods

Recall and detection rates with and without computer-aided detection were compared by using a generalized estimating equations (GEE) logistic regression model that accounts for clustering of findings within each reader (22). In addition, we asked an independent team of investigators to evaluate the numbers of cancer cases that were detected with and without computer-aided detection by the type of abnormality(s) noted in the original report. Those findings were assigned to one of the following categories: 1) mass(es) only; 2) clustered microcalcifications only; 3) mass(es) and clustered microcalcifications; and 4) other findings. Because the performance levels of computer-aided detection systems are generally outstanding for detecting microcalcifications (16), we used the GEE model to analyze our findings with respect to possible changes in the percentage of cancer detections attributable to microcalcification clusters associated with the use of computer-aided detection. In addition, all analyses were repeated using a mixed-effect logistic regression model in which readers were considered a random effect, and modality (i.e., with or without computer-aided detection) was considered a fixed effect (23). We also examined data from the seven high-volume radiologists (i.e., those who interpreted more than 8000 mammograms each during the study period). Because of the serial nature of the analysis (namely, this was not a randomized study), we repeated the analyses with respect to the timing of the major use of computer-aided detection in our practice by comparing the results for all cases interpreted without computer-aided detection from January 1, 2000, through June 30, 2001, when computer-aided diagnosis was used in only a small percentage of cases (<0.2%) at our facilities, with results for all cases interpreted with computer-aided detection from October 1, 2001, through December 31, 2002, when most (>93%) of the cases at our facilities were interpreted with computer-aided detection. All statistical tests were two-sided.


    RESULTS
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
The mean age of the screened population (n = 115 571) during the study period was 50.05 years (standard deviation = 11.17 years). During the study period, the percentage of women who were screened for the first time gradually decreased from approximately 40% in 2000 to 30% in the last quarter of 2002, whereas the percentage of women who had repeated screenings gradually increased.

Table 1 summarizes our data for the 24 radiologists who interpreted screening mammograms at our facility with and without the use of a computer-aided detection system. Among the 115 571 examinations in our database, 56 432 (48.8%) were interpreted without the use of the computer-aided detection system and 59 139 (51.2%) were interpreted with the use of the computer-aided detection system. Recall rates for the entire group of 24 radiologists were 11.39% for mammograms interpreted without computer-aided detection and 11.40% for mammograms interpreted with it (percent difference = 0.09, 95% confidence interval [CI] = -11 to 11; P = .96). Recall rates for the 18 radiologists who interpreted more than 1000 mammograms each during the study period ranged from 7.7% to 17.2% (data not shown). Recall rates for the seven high-volume radiologists who interpreted more than 8000 mammograms each during the study period ranged from 7.7% to 14.9% (data not shown). Among this latter group of radiologists, there was no statistically significant correlation (rho = -0.21, P = .64) between recall rate and the total number of screening mammograms interpreted by individual radiologists. In our practice, approximately 3.0% of the cases recommended for recall are typically lost to follow-up because the woman either undergoes re-screening at another institution or ignores our recommendations. This group remained relatively constant as a percentage of recalled women over the period in question.


View this table:
[in this window]
[in a new window]
 
Table 1. Mammography recall rates and breast cancer detection rates for 24 radiologists performing screening mammograms without and with computer-aided detection*

 
Table 2 summarizes our data for the seven high-volume radiologists who interpreted more than 8000 screening mammograms each with and without the use of a computer-aided detection system. During the study period, these radiologists interpreted a total of 82 129 screening mammograms and were credited with the detection of 292 breast cancers as a result of these screening procedures. In this group, the recall rates decreased from 11.62% for mammograms interpreted without computer-aided detection to 11.05% for mammograms interpreted with computer-aided detection (percent difference = -4.9, 95% CI = -21 to 4; P = .16).


View this table:
[in this window]
[in a new window]
 
Table 2. Mammography recall rates and breast cancer detection rates for the seven high-volume radiologists performing screening mammograms without and with computer-aided detection

 
Breast cancer detection rates for the entire group of 24 radiologists were 3.49 per 1000 screening examinations for mammograms interpreted without computer-aided detection and 3.55 per 1000 screening examinations for mammograms interpreted with it (percent difference = 1.7, 95% CI = -11 to 19; P = .68) (Table 1). Breast cancer detection rates for the seven high-volume radiologists were 3.61 per 1000 screening examinations for mammograms interpreted without computer-aided detection and 3.49 per 1000 screening examinations for mammograms interpreted with computer-aided detection (percent difference = -3.2, 95% CI = -15 to 9; P = .54) (Table 2).

The cancer detection rates associated with recalls due to the detection of clustered microcalcifications alone were 1.35 per 1000 mammograms interpreted without computer-aided detection and 1.44 per 1000 mammograms interpreted with computer-aided detection (P = .66) (data not shown). We observed no trend in breast cancer detection rates over time when we reviewed average detection rates for all 24 radiologists by calendar quarter (data not shown). We repeated our analyses using a random-effects logistic regression model and found that there were no statistically significant changes in recall rates or detection rates for all measurements presented above. Our results were not substantially affected when we compared only mammograms interpreted without computer-aided detection prior to July 1, 2001, with only those interpreted with computer-aided detection after October 1, 2001.


    DISCUSSION
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
The introduction of computer-aided detection into our practice was not associated with statistically significant changes in recall and breast cancer detection rates for the entire group of radiologists as well as for the subset of seven radiologists who interpreted high volumes of mammograms. The magnitudes of the improvements we observed were substantially less than those reported in the literature as the range of possible improvements based on retrospective analyses and limited prospective data (9,17,18). The improvements we observed may be attributable to the better detection of clustered microcalcifications associated with malignancy. Our findings are consistent with the range of improvement in detection rates estimated and reported by others (9,1618). However, our large confidence intervals reflect the relatively low number of breast cancers detected with and without computer-aided detection and the large inter-reader variability among the radiologists in our practice. Because there were no repeat measures in this database—that is, each of the examinations was interpreted only once by one radiologist—we could not assess intra-reader variability.

It should be noted that we could not provide detailed information for individual radiologists without providing individually traceable data because each staff radiologist knows his or her reading volume and approximate recall rate. Our data are not adjusted for any learning effect: namely, the majority of interpretations made without computer-aided detection occurred chronologically prior to those made with computer-aided detection. We also did not account for any effect that may have resulted from a continuous effort to improve performance (in particular, sensitivity) by group reviews of all false-negative cases or from the steps undertaken to reduce recall rates through various actions, such as monthly performance reviews and direct consultation with interpreters who had higher-than-average recall rates.

Although one could argue that some or all of the reduction in recall rates we observed for the high-volume radiologists may be attributable to the use of computer-aided detection, the corresponding decrease in cancer detection rates we observed among the radiologists in this group is not easily explained by expected practice variations. An assessment of whether the small improvement we observed in cancer detection is due to learning effects—namely, that our radiologists had substantially more overall experience interpreting mammograms without computer-aided detection than with computer-aided detection—is beyond the scope of this investigation.

This investigation covered a period during which conventional film mammography was performed in all of our screening procedures. Hence, we cannot comment on the possible effect of computer-aided detection in a digital mammography environment. In our study, we did not account for women who had decided to follow up on our recommendations elsewhere. However, because compliance in patient follow-up was relatively constant during the study period, any bias in the results due to changes in patient loss to follow-up is likely to be small.

There are limited reported data concerning the actual effect of computer-aided detection on breast cancer detection and mammography recall rates. The prospective data reported by Freer and Ulissey (16), which suggested a substantial improvement (19.5%) in breast cancer detection rates associated with the use of computer-aided detection systems, may have been affected by the fact that the results of mammographic interpretations without and with computer-aided detection were reported on the same cases (i.e., mammograms were read in one sitting, first without computer-aided detection then immediately afterward with the use of a computer-aided detection system). Another prospective study performed in a similar manner reported a 12% improvement in detection rates associated with the use of a computer-aided detection system (18). This type of protocol, namely reading mammograms without computer-aided detection followed immediately by readings of the same mammograms with the use of a computer-aided detection system and a reassessment of the original finding without computer-aided detection, may have introduced a lower level of vigilance among radiologists during the initial interpretation without computer-aided detection, because they knew that computer-aided detection would be available to them for the final recommendation and that the initial interpretation did not constitute a formal clinical recommendation.

Results of the only study similar to ours, albeit on a substantially smaller group of patients and under a different set of circumstances, suggested that computer-aided detection was associated with a 13% improvement in breast cancer detection rates (17). One of the advantages of the approach taken in our investigation is that the radiologists' interpretations were performed and recorded prospectively in a clinical setting and data were collected primarily for quality-assurance purposes (24).

Our results for the interpretations made with computer-aided detection may be marginally biased because the outcomes of as many as nine recommendations for recalls and three recommendations for biopsies during the last quarter of 2002 are not yet available. Although some of these follow-up procedures or biopsies may ultimately be performed at our institution, we assume that the women who underwent the original mammograms have been lost to follow-up. However, on the basis of our typical recall-to-cancer-detection ratios (approximately 1 of 32 cases) and biopsy-to-confirmed cancer ratios (approximately 1 of 5 cases), we suspect that this bias would not substantially affect our findings or conclusions. It is possible that the gradually increasing fraction of women who had prior screening examinations created a bias in our results. Repeat screening examinations have a slightly lower number of cancers present as more are detected during the first screen, and on average, cancers detected on repeat mammograms may be more "difficult" to detect because more of the "easier" (e.g., larger) cancers are detected during the initial screen. Repeat mammograms have a lower recall rate, as the radiologists have prior films for comparison, to help inform their decision. The availability of prior examinations for comparison (in the repeat examinations) should have aided in the interpretation of these mammograms and offset the possible effect (if any) on the interpretations due to an increase in the "average case difficulty." The fact that our recall rates and detection rates remained virtually constant over time suggests that the possible bias due to a gradual increase in repeat examinations is not a statistically significant factor. We suspect that this increasing availability of prior examinations for comparison is a general phenomenon that is observed by most mammography screening practices and that there is not a simple way to account for it in an analysis such as the one we performed. When we included the 30 examinations that had been originally scheduled as screening procedures but were diagnosed during the same visit and resulted in a positive outcome in the estimation, our actual cancer detection rate attributable to screening was 3.8 per 1000 examinations, which is reasonable for a population in which the majority of women had undergone several screening procedures prior to the study period (19).

On the basis of published performance levels of other computer-aided detection systems (25), we believe that our results are not unique to the specific computer-aided detection system that is used at our institution. It is possible, however, that in clinical practices with substantially lower recall rates than ours, computer-aided detection would have larger effects on mammography recall rates and detection rates than what we observed. Such an improvement in detection rates would be consistent with results of a study (17) that reported lower recall rates without computer-aided detection (8.02%) than with computer-aided detection (8.43%).

The financial implications of our findings are beyond the scope of this work. However, a simple assessment of the additional estimated cost of using computer-aided detection per additional cancer detected in our practice (approximately $150 000 per additional detected cancer, assuming a reimbursement rate of $10 per case for professional and technical components combined) clearly indicates that more rigorous evaluations of the cost effectiveness of this practice are needed.

Our observations with respect to recall and detection rates may be exceptions (stemming from large inter-practice variations) that highlight the need for additional recall and detection rate data from multiple clinical practices and different reading environments. However, until such data clearly demonstrate that our experience is indeed an exception, these results represent an important first step.

This analysis of our practice was designed to assess the changes, if any, that occurred in recall and breast cancer detection rates with the introduction of computer-aided detection. Our results suggest that, in our practice, neither recall rates nor breast cancer detection rates changed with the introduction of this technology at its current level of performance, particularly as related to the detection of abnormalities other than clustered microcalcifications. Due to large confidence intervals, our results are statistically consistent with the possibility of large improvements in cancer detection rates with computer-aided detection. Yet, actually observed changes in our practice were substantially lower than expected. This is not to say that the use of computer-aided detection would not be beneficial or cost-effective in other practices. Rather, we suggest that, at its current level of performance, computer-aided detection may not improve mammography recall or breast cancer detection rates (especially as related to the detection of masses) in academic practices similar to ours that employ specialists for interpreting screening mammograms.


    NOTES
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
Supported in part by Public Health Service grants CA85241, CA67947, and CA77850 (to the University of Pittsburgh) from the National Cancer Institute, National Institutes of Health, Department of Health and Human Services and by the U.S. Army Medical Research Acquisition Center (Fort Detrick, MD) contract DAMD17-00-1-0410 (to the University of Pittsburgh).

We thank Jennifer Herrmann, Jill King, Amy Klym, Christopher Traylor, and Andriy Bandos for their diligent and tireless work on this project.


    REFERENCES
 Top
 Notes
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 

1 Humphrey LL, Helfand M, Chan BK, Woolf SH. Breast cancer screening: a survey of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med 2002;137:347–60.[Abstract/Free Full Text]

2 Tabár T, Duffy SW, Vitak B, Chen HH, Prevost TC. The natural history of breast carcinoma: what have we learned from screening? Cancer 1999;86:449–62.[CrossRef][ISI][Medline]

3 Hendrick RE, Klabunde C, Grivegnee A, Pou G, Ballard-Barbash R. Technical quality control practices in mammography screening programs in 22 countries. Int J Qual Health Care 2002;14:219–26.[Abstract/Free Full Text]

4 Ng EH, Ng FC, Tan PH, Low SC, Chiang G, Tan KP, et al. Results of intermediate measures from a population-based, randomized trial of mammographic screening prevalence and detection of breast carcinoma among Asian women: the Singapore Breast Screening Project. Cancer 1998;82:1521–8.[CrossRef][ISI][Medline]

5 Sickles EA, Wolverton DE, Dee KE. Performance parameters for screening and diagnostic mammography: specialist and general radiologists. Radiology 2002;224:861–9.[Abstract/Free Full Text]

6 Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med 1994;331:1493–9.[Abstract/Free Full Text]

7 Beam CA, Conant EF, Sickles EA. Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. J Natl Cancer Inst 2003;95:282–90.[Abstract/Free Full Text]

8 Doi K, Giger ML, Nishikawa RM, Schmidt RA. Computer-aided diagnosis of breast cancer on mammograms. Breast Cancer 1997;4:228–33.[Medline]

9 Warren Burhenne LJ, Wood SA, Orsi CJ, Feig SA, Kopans DB, O'Shaughnessy KF, et al. Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology 2000;215:554–62.[Abstract/Free Full Text]

10 Brem RF, Schoonjans JM. Radiologist detection of microcalcifications with and without computer-aided detection: a comparative study. Clin Radiol 2001;56:150–4.[CrossRef][ISI][Medline]

11 Leichter I, Fields S, Nirel R, Bamberger P, Novak B, Lederman R, et al. Improved mammographic interpretation of masses using computer-aided diagnosis. Eur Radiol 2000;10:377–83.[CrossRef][ISI][Medline]

12 Chan HP, Sahiner B, Helvie MA, Petrick N, Roubidoux MA, Wilson TE, et al. Improvement of radiologists' characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology 1999;212:817–27.[Abstract/Free Full Text]

13 Jiang Y, Nishikawa RM, Schmidt RA, Toledano AY, Doi K. Potential of computer-aided diagnosis to reduce variability in radiologists' interpretations of mammograms depicting microcalcifications. Radiology 2001;220:787–94.[Abstract/Free Full Text]

14 Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol 2002;9:1264–77.[CrossRef][ISI][Medline]

15 Brem RF. Enhancement of mammographic interpretation with computer-aided detection (CAD): a multi-institutional trial. Presented at the 87th Scientific Assembly and Annual Meeting of the Radiological Society of North America, November 25–30, 2001, Chicago, IL. Radiology 2001;221(P):472.

16 Freer TW, Ulissey MJ. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology 2001;220:781–6.[Abstract/Free Full Text]

17 Cupples TE. Impact of computer-aided detection (CAD) in a regional screening mammography program. Presented at the 87th Scientific Assembly and Annual Meeting of the Radiological Society of North America, November 25–30, 2001, Chicago, IL. Radiology 2001;221(P):520.

18 Bandodkar P, Birdwell RL, Ikeda DM. Computer aided detection (CAD) with screening mammography in an academic institution: preliminary findings. Presented at the 88th Scientific Assembly and Annual Meeting of the Radiological Society of North America, December 1–6, 2002, Chicago, IL. Radiology 2002;225(P):458.

19 Young WW, Destounis SV, Bonaccio E, Zuley ML. Computer-aided detection in screening mammography: can it replace the second reader in an independent double read? Preliminary results of a prospective double blinded study. Presented at the 88th Scientific Assembly and Annual Meeting of the Radiological Society of North America, December 1–6, 2002, Chicago, IL. Radiology 2002;225(P):600.

20 Food and Drug Administration. Quality Standards and Certification Requirements for Mammography Facilities (21 CFR Part 900). Federal Register 1993;58:67565.[Medline]

21 American College of Radiology (ACR). Breast imaging reporting and data system (BI-RADS). Reston (VA): American College of Radiology, 1998. Available at http://www.acr.org/departments/stand_accred/birads/contents.html. [Last accessed: 12/9/03.]

22 Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika 1986;73:13–22.[ISI]

23 Brown H, Prescott R. Applied mixed models in medicine. Chichester (NY): J. Wiley & Sons; 1999, pages 104–11

24 Yankaskas BC, Cleveland RJ, Schell MJ, Kozar R. Association of recall rates with sensitivity and positive predictive values of screening mammography. AJR Am J Roentgenol 2001;177:543–9.[Abstract/Free Full Text]

25 Hoffmeister JW, Rogers SK, DeSimio MP, Brem RF. Determining efficacy of mammographic CAD systems. J Digit Imaging 2002;15 Suppl 1:198–200.

Manuscript received May 14, 2003; revised October 5, 2003; accepted November 26, 2003.


This article has been cited by other articles in HighWire Press-hosted journals:


Correspondence about this Article

Editorial about this Article

             
Copyright © 2004 Oxford University Press (unless otherwise stated)
Oxford University Press Privacy Policy and Legal Statement