DASA, headquartered in Sao Paolo, Brazil, is the 4th largest diagnostics company in the world and the largest in Brazil. They are unique in that they are the both, the largest in-vitro and radiology diagnostics company in Brazil. They invest heavily in cutting edge technologies and have research collaborations with institutes such as Harvard and Stanford Universities.
Being one of the world’s largest medical imaging service providers, and given the innovation DNA of the organisation itself, radiologists at DASA are constantly evaluating newer tools to help them in their day to day clinical practice. Needless to say, many of these tools are Artificial Intelligence (AI) based products which automate parts of the radiology workflow.
One such tool is QXR from Qure.ai (India). Qure.ai is one of the world’s leading medical imaging AI companies building algorithms that automate the reporting of Chest X-Rays and Head CT scans. QXR is an AI system that automatically reports Chest X-Rays, and even discerns normal from abnormal. In early 2020, Qure.ai developed a new version of QXR which could do normal-vs-abnormal classification for pediatric Chest X-Rays, something that has always been a challenge for AI in general.
How does DASA evaluate Qure.ai’s pediatric X-ray algorithm in a simple yet statistically and clinically significant manner without allocating significant resources or budget?
DASA runs the radiology at the Sabara Children’s Hospital in Sao Paolo, Brazil. This is one of the leading pediatric hospitals in Brazil and also one of the few with a dedicated pediatric radiology department. Dr Marcelo Straus Takahashi, Radiologist at Sabara Children’s Hospital used CARPL to test Qure.ai’s pediatric X-ray algorithm on ~3,300 Chest X-rays for its ability to discern normal from abnormal.
CARPL makes this process very easy:
On CARPL, the following analysis was obtained for those cases which were labeled as normal or abnormal by the AI:
An astoundingly high AUC of 0.98 was obtained, and at a threshold of 50, there was only one false negative and 50 false positives. Note that the false positives would have been read by a radiologist any way, avoiding any error.
Upon digging deeper into the one false negative case, it was noted that it was clinically insignificant and hence the AI was indeed correct:
CARPL allowed Dr Takahashi to quickly and effectively test Qure.ai’s QXR solution on a pediatric population without requiring additional assistance from a statistics or data science team. Additionally he was able to deep dive into the false positive and false negative cases to see whether the errors were true errors or not, and taking a more informed decision on the performance of the algorithm. From Qure.ai’s point of view, they were able to run a retrospective validation study on a new version of their algorithm, in a true independent test setting, without any effort whatsoever.
This work was presented at RSNA 2020 by Dr Takahashi as an example of successful validation of an AI algorithm on pediatric Chest X-rays
