Artificial intelligence (AI) continues to make a major impact on medical imaging, particularly in computer-aided cancer detection. Multiple studies have demonstrated that using AI-based computer-aided detection (CAD) for mammography significantly improves radiologists’ ability to detect breast cancers^[1] and reduces the number of false positives recalled for unnecessary screening^[2].

Trust in AI^[3] is growing across industries, not just the medical field. However, many clinicians are rightfully careful about adopting new tools and technologies; they want to ensure that they will offer the best outcomes possible for their patients and practices before making them a part of their diagnostic workflow.  

That’s why CureMetrix invited radiologists to test AI-based mammography solutions for themselves this past October as part of an “AI Test Drive” challenge and study. CureMetrix presented these findings to attendees at the virtual 2020 Radiological Society of North America (RSNA) Scientific Assembly and Annual Meeting. Here’s what we shared.

Objectives

The objective of the CureMetrix “AI Test Drive Challenge” was to give radiologists a hands-on chance to test AI CAD for themselves, and see the difference it made in their performance. It was also another excellent opportunity for CureMetrix to observe how radiologists reacted to using this type of technology.

Methods & Solutions Used

The Challenge presented hundreds of registered radiologists with 25 mammogram cases that were a mixture of complex and routine cases, including six breast cancer cases that were ultimately confirmed with a biopsy. Some participants reviewed the cases with the help of CureMetrix AI solutions cmTriage™ and cmAssist®, and another group reviewed the cases without the aid of artificial intelligence. Both groups conducted the readings in the online Ambra Health platform.

cmTriage™ is the first FDA-cleared product for mammography. It automatically sorts the suspicious mammograms to the top of a radiologist’s worklist, which expedites recall of suspicious mammogram cases—and can also be used for intelligent distribution of mammograms to multiple readers in a large practice. Users can set the sensitivity threshold of cmTriage™ to an ultra-high 99%, to flag even very low suspicion mammograms.

cmAssist is CureMetrix’s investigational AI-CAD software that helps radiologists identify, mark and score regions of interest on mammogram scans. Once a case has been determined as suspicious, cmAssist then marks the mammogram, highlighting identified anomalies. The software also generates a unique, data-driven neuScore™ which provides the radiologist a quantitative measure of suspiciousness of a marked region of interest, ranging from 0 (least suspicious) to 100 (highly suspicious). This score can be tracked over time to evaluate any changes in lesions or marks.

Results

The radiologists who read mammograms with the assistance of cmTriage and cmAssist were able to diagnose with significantly more sensitivity, specificity and overall accuracy.

Some of the top findings:

• Sensitivity: All of the radiologists in our study are highly-trained, experienced clinicians, and therefore already had a high rate of sensitivity when reading mammograms. However, when these doctors used CureMetrix AI, they saw a five percent increase in sensitivity.

• Specificity: CureMetrix’s solutions were clearly beneficial for making accurate diagnoses and efficiently screening out likely negative cases. When using CureMetrix AI-based mammography tools, the false recall rate at which they recalled normal patients for further screening decreased by 34 percent, sparing both patient and clinic time and resources and cost.

• Overall Accuracy: Simply put, these radiologists were even more accurate in their diagnoses when aided by CureMetrix AI. The group using cmTriage and cmAssist were 25 percent more accurate in their diagnoses. And that that translates into finding an additional 23 cancers per 100,000 cases with a lower overall false recall rate.

• Read time: CureMetrix AI had the added benefit of allowing doctors to evaluate images accurately in far less time. On average, the participating radiologists were able to read normal cases 40 percent faster when aided by CureMetrix

Two examples in particular illustrate how powerful the advantage is for radiologists when they read mammograms using AI-based tools.

The above image came from a patient who ultimately turned out to be negative for breast cancer. Without AI, radiologists were able to correctly identify this case as a true negative 14 percent of the time. With AI, however, that number nearly quintupled; 63 percent were able to identify the case as negative when using cmTriage and cmAssist.  The markings shown, all false, are the actual areas marked by study participants.

The patient in this image was diagnosed with biopsy confirmed cancer. When the participating radiologists first read this mammogram without the help of AI, 57 percent of them correctly identified the case as a true positive. However, with AI, a stunning 93 percent correctly identified the case as a true positive.  Further, it is worth noting the concentration of correct marks by the participating radiologists and the relationship to the mark from CureMetrix cmAssist.

AI’s Benefit for All Radiologists and Clinical Practices

The “AI Test Drive” challenge clearly demonstrated that AI-based CAD for mammography improves radiologists’ sensitivity and specificity in diagnosing breast cancers, while reducing false positives. Additionally, the study also revealed that regardless of the number of years’ experience reading mammograms or the number of mammograms read annually, CureMetrix AI significantly enhanced performance across all of the participants in the study.

Solutions like cmTriage and cmAssist are an extra pair of eyes that help all radiologists accurately diagnose more breast cancers and catch them even earlier. Statistics^[4] show that women whose breast cancer is detected at an early stage have a 93 percent or higher survival rate in the first five years. So AI is a boon for patient outcomes—and it has multiple positive implications for clinical practices.

When reading mammograms, every second counts and every second saved through the use of AI helps radiologists focus their time on the most suspicious cases.  When using AI, participating radiologists read normal cases up to 2 minutes faster, on average, and those seconds add up once multiplied by dozens of patients per day over several years. At a hospital such as MD Anderson where over 100,000 mammograms are conducted each year, and based on the study that was conducted at MD Anderson showing that CureMetrix AI-CAD had 69% fewer false positive marks than traditional CAD up to 400 hours of reading time could be saved per year. 

Additionally, identifying cancers earlier isn’t just good for patients. In fact, treating late-stage cancer literally costs the healthcare system twice as much as treating the disease in its early stages. Using AI-based CAD[5], meanwhile, can save millions in treatment costs; other radiology experts have asserted that “the potential cost savings are immense.” 

Better patient outcomes also often translate into more frequent follow-ups and referrals of new patients, so in addition to better clinical outcomes, AI can also increase patient compliance have a positive impact on a practice’s image in the community it serves.

Conclusion

All radiologists who work in breast imaging can improve their efficiency, accuracy, overall practice value and job satisfaction by incorporating AI-based mammography solutions into their daily workflow. CureMetrix AI truly puts radiologists in the driver’s seat, empowering them to focus on the most urgent cases, and diagnose more quickly and accurately, saving more lives.

^[1] https://link.springer.com/epdf/

^[2] https://link.springer.com/article/10.1007/s10278-018-0168-6

^[3] https://www.inc.com/joel-comm/why-ai-may-be-more-trustworthy-than-you-think.html

^[4] Early Detection is Key | Carol Milgard Breast Center

^[5] https://towardsdatascience.com/5-radiology-artificial-intelligence-companies