Speeding up pediatric AI CXR Validation at DASA, Brazil

Speeding up pediatric AI CXR Validation at DASA, Brazil

About the Customer

DASA, headquartered in Sao Paolo, Brazil, is the 4th largest diagnostics company in the world and the largest in Brazil. They are unique in that they are the both, the largest in-vitro and radiology diagnostics company in Brazil. They invest heavily in cutting edge technologies and have research collaborations with institutes such as Harvard and Stanford Universities.

The Pain Point

Being one of the world’s largest medical imaging service providers, and given the innovation DNA of the organisation itself, radiologists at DASA are constantly evaluating newer tools to help them in their day to day clinical practice. Needless to say, many of these tools are Artificial Intelligence (AI) based products which automate parts of the radiology workflow.

One such tool is QXR from Qure.ai (India). Qure.ai is one of the world’s leading medical imaging AI companies building algorithms that automate the reporting of Chest X-Rays and Head CT scans. QXR is an AI system that automatically reports Chest X-Rays, and even discerns normal from abnormal. In early 2020, Qure.ai developed a new version of QXR which could do normal-vs-abnormal classification for pediatric Chest X-Rays, something that has always been a challenge for AI in general.

How does DASA evaluate Qure.ai’s pediatric X-ray algorithm in a simple yet statistically and clinically significant manner without allocating significant resources or budget?

CARPL Impact

DASA runs the radiology at the Sabara Children’s Hospital in Sao Paolo, Brazil. This is one of the leading pediatric hospitals in Brazil and also one of the few with a dedicated pediatric radiology department. Dr Marcelo Straus Takahashi, Radiologist at Sabara Children’s Hospital used CARPL to test Qure.ai’s pediatric X-ray algorithm on ~3,300 Chest X-rays for its ability to discern normal from abnormal.

CARPL makes this process very easy:

  • Load the X-rays onto CARPL using CARPL’s front-end
  • Run Qure.ai’s algorithm on those X-rays (in this case, to speed up the process, the X-rays were sent to Qure.ai’s API). Qure.ai’s algorithm gave an output of:
    • Normal
    • Abnormal
    • To be read (where the AI is ‘not sure’)
  • Load the normal vs abnormal status of each X-ray onto CARPL (this is a simple csv upload)

On CARPL, the following analysis was obtained for those cases which were labeled as normal or abnormal by the AI:

An astoundingly high AUC of 0.98 was obtained, and at a threshold of 50, there was only one false negative and 50 false positives. Note that the false positives would have been read by a radiologist any way, avoiding any error.

Upon digging deeper into the one false negative case, it was noted that it was clinically insignificant and hence the AI was indeed correct:

 

Outcome

CARPL allowed Dr Takahashi to quickly and effectively test Qure.ai’s QXR solution on a pediatric population without requiring additional assistance from a statistics or data science team. Additionally he was able to deep dive into the false positive and false negative cases to see whether the errors were true errors or not, and taking a more informed decision on the performance of the algorithm. From Qure.ai’s point of view, they were able to run a retrospective validation study on a new version of their algorithm, in a true independent test setting, without any effort whatsoever.

This work was presented at RSNA 2020 by Dr Takahashi as an example of successful validation of an AI algorithm on pediatric Chest X-rays

Real-time validation of AI in production

Real-Time Validation Of AI In Production

Note: The images used below are for representational purposes only

About the Customer

Qure.ai is the world’s leading developers of Artificial Intelligence solutions for medical imaging and radiology applications. Pioneers in the field, they are amongst the most published AI research groups in the world having more than 30 publications and presentations in leading journals and conferences, including the first paper on AI in the prestigious journal – The Lancet.

 

The Pain Point

With tens of thousands of chest X-rays passing through the Qure.ai algorithms every day, it becomes critical for the Qure data science leadership to know the real-time performance of their algorithms across their user base. It is a well known fact that the performance of AI can vary dramatically based on patient ethnicity and equipment vendor characteristics – as an AI developer’s user-base scales, the likelihood of an error creeping through the system increases. The challenge becomes the orchestration of a mechanism where randomly picked Chest X-rays are double-read by a team of radiologists, labels established during these reads are compared against the AI outputs, and a dashboard is presented which contains real-time performance metrics (Area Under Curve, Sensitivity, Specificity etc.) with the ability to deep dive into the false positives / negatives.

How does the leadership team at Qure.ai create such a system without investing significant engineering effort?

CARPL Impact

CARPL’s Real-Time AI validation workflow allows AI developers to monitor the performance of their algorithms in real-time. Reshma Suresh, the Chief Operating Officer at Qure.ai, uses CARPL to get real-time ground truth inputs from radiologists and then compares the radiologist reads to the AI outputs, subsequently creating a real-time performance dashboard for QXR – Qure.ai’s Chest X-Ray AI algorithm.

CARPL makes the process very easy:

  • Create a Dataset on CARPL
  • Create a “Classification” Testing project on CARPL → choose the appropriate algorithm, i.e. QXR, and the Dataset created above
  • Create an Annotation Template on CARPL → create whatever fields that need to get annotated
  • Create an Annotation Project on CARPL → select the dataset and the annotation project created above
    • Link the annotation project to the Testing project
    • Select to “Auto-Assign” cases to radiologist(s)
  • Now as data keeps getting added to the Dataset, either manually or through CARPL’s APIs, the data gets inferred on the AI and gets assigned to the radiologist(s), and as the radiologist(s) keep reading the scans, the real-time dashboard keeps getting populated!

CARPL is deployed on Qure.ai’s infrastructure allowing Qure to take control of all the data that comes onto CARPL!

Example of a case which is otherwise normal, but was wrongly classified by AI as an abnormal case possibly due to poor image quality and a coiled naso-gastric tube in the eosophagus

Example of a case where the radiologist identified cardiomegaly

Representational Image of a Real-Time Validation Project on CARPL

Impact

While much has been spoken about monitoring of AI in clinical practice at the hospital level, it is even more important for AI developers themselves to monitor AI in real-time so that they may detect shifts in model performance, and intermediate as and when needed. This moves the process of AI monitoring and consequent improvement from a retrospective and post-facto process to a proactive approach. As we go and build on our vision to make CARPL the singular platform behind all clinically useful and successful AI, working with AI developers on helping them establish robust and seamless processes for monitoring of AI is key.

 

Anonymised chart to show a fall in AUC detected by real-time monitoring of AI by an AI developer – image for representational purposes only – not reflective of real world data.

We stay true to our mission of Bringing AI from Bench to Clinic.

CARPL.ai raises $6M from Stellaris Venture Partners

CARPL.ai raises $6M from Stellaris Venture Partners

 

We are excited to announce the seed funding round of $6M led by Stellaris Venture Partners, a leading enterprise software investor, with participation from leading strategic angel investors. Our heartfelt gratitude to all those who’ve been part of this incredible journey! We look forward to expanding the team in North America and continuing to build our tech stack.

CARPL.ai is used by the world’s top healthcare organizations like the Singapore Government, Massachusetts General Hospital, Radiology Partners, University Hospitals, I-MED Radiology, Albert Einstein Hospital, and Clinton Health Access Initiative, to name a few.

We are also creating an impact in the public health space by working with the Government of India to enable large-scale Tuberculosis Screening Programmes in the most remote regions of the country.

“Over the past two years, we have onboarded more than 50 AI developers having 100+ AI applications, which made us the largest AI marketplace in terms of number of AI applications offered to customers. We are proud that some of the largest healthcare enterprises in the world have vetted our technology and trust us to be their partner in their AI and automation journeys,” said Dr. Vidur Mahajan, Chief Executive Officer of CARPL.ai.

A recent joint statement by the world’s top radiology associations also brings to light the importance of validation, deployment and monitoring of AI while being used in clinical practice. We address this need through our proprietary DEV-D framework allowing healthcare providers to first Discover (D), Explore (E) and Validate (V) AI applications from the CARPL.ai marketplace, and subsequently Deploy (D) the most appropriate application across their clinical workflows.

Alok Goyal, a partner at Stellaris Venture Partners, says new technology in radiology is badly needed. “The volume of imaging scans shows a steady 9% year-on-year growth, outpacing the 1.8% growth in the number of radiologists; bridging this demand gap is a crucial challenge for healthcare providers, and we believe AI will be the key,” he says. “CARPL’s integrated platform, designed for testing, deploying, and monitoring radiology AI applications, is poised to empower healthcare providers by seamlessly integrating AI into their clinical workflows.”

To learn the full story, read more.

University Hospitals uses CARPL.ai as the AI validation and deployment partner

University Hospitals Uses CARPL.Ai As The AI Validation And Deployment Partner

CARPL.ai x UH RadiCLE

CARPL.ai is proud to announce that the University Hospitals Department of Radiology has signed a multi-year strategic agreement with CARPL.ai as the AI validation and deployment partner for the UH RadiCLE AI Lab, RadiCLE – UH Radiology A.I. and Diagnostic Innovation Collaborative.

CARPL\’s D.E.V.-D framework – Discovery, Exploration, Validation & Deployment – allows UH Radiology Innovation team to quickly access, assess and integrate AI solutions from all over the world in the RadiCLE ecosystem.

This relationship will enhance UH\’s current capabilities to assess 3rd party AI, provide a structured approach to offer validation services toward attaining FDA clearance, and become a central repository of curated datasets for AI creation.

“We are excited to have CARPL.ai as a platform for executing our projects in developing, validating, integrating and assessing radiology AI solutions at UH RadiCLE.” – Leonardo Kayat Bittencourt, MD PhD, Director of RadiCLE and Vice-Chair of Innovation at UH Radiology.

 

CARPL.ai has partnered with NTT DATA for the US market

CARPL.Ai Has Partnered With NTT DATA For The US Market

CARPL.ai x NTT DATA

CARPL.ai is delighted to announce our partnership with NTT DATA, for the US market!

Through this partnership, we seek to accelerate the deployment of AI at scale across provider and life sciences clinical imaging domains in the US.

NTT DATA’s Advocate AI services empower CARPL.ai’s users to leverage data assets and enable improved data insights for a tailored, data-driven approach to providing value-based care and achieving positive patient outcomes.

By using CARPL’s cutting-edge platform, healthcare providers can seamlessly explore, discover, validate and deploy single or multiple AI solutions across their workflows.

 

CARPL.ai Joins Stanford Affiliates Program to Expand Deployment of Medical Imaging AI in the Clinical Realm

CARPL.Ai Joins Stanford Affiliates Program To Expand Deployment Of Medical Imaging AI In The Clinical Realm

Stanford, CA, June 16, 2022: CARPL.ai, a technology platform that connects Artificial Intelligence (AI) applications and healthcare providers announced their participation in Stanford University’s prestigious Industry Affiliates Program through the Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI).

The announcement came as part of Stanford AIMI Symposium 2022, one of the world’s leading conferences on AI in medicine. The 3rd annual symposium was a hybrid event, held both in person at Stanford and live streamed for online attendees.

Curt Langlotz, Professor of Radiology and Biomedical Informatics and Director of Stanford AIMI, said, “AIMI faculty have been working with CARPL for almost three years, including our most recent work on cryptographic inferencing for AI models. We are excited to formalize our relationship with CARPL.ai and look forward to our affiliation with them.”

Speaking at the AIMI symposium, Dr. Vidur Mahajan, Chief Executive Officer of CARPL.ai, said, “Stanford is the undoubted leader in the field, having sparked the AI in imaging revolution by publishing some of the first research, and open sourcing some of the first datasets in medical imaging. We consider it our privilege to be part of their ecosystem.”

As the number of AI applications coming into the clinical realm increases, it is becoming nearly impossible for healthcare providers to access, assess and then integrate these solutions into their clinical workflows. CARPL bridges this gap by acting as an intermediary platform for development, testing, and distribution of these AI applications.

“CARPL is unique in the combined capabilities that the platform offers – their data management, search, cohort creation, annotation, validation and deployment modules provide a holistic approach to translation of AI models into the clinic. We look forward to a successful long-term partnership. It will be exciting to see how CARPL supports researchers and clinicians.”, said Johanna Kim, Executive Director of Stanford AIMI.

About CARPL.ai

CARPL is an end-to-end technology platform for the development, testing, and distribution of medical imaging AI applications in clinical workflows. Used by some of the world’s top AI researchers and health systems, it connects AI applications and healthcare providers helping improve access, affordability, and quality of medical care. More Information: www.carpl.ai

About the Stanford AIMI Center

The Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI) was established in 2018 with the primary mission to solve clinically important problems in medicine using AI. Drawing on Stanford’s interdisciplinary expertise in clinical medical imaging, bioinformatics, statistics, electrical engineering, and computer science, the AIMI Center supports the development, evaluation and dissemination of new AI methods applied across the medical imaging life cycle. Its mission is to develop and support transformative medical AI applications and the latest in applied computational and biomedical imaging research to advance patient health. More info: https://aimi.stanford.edu

CONTACT

For CARPL.ai

Vidur Mahajan | Chief Executive Officer | vidur.mahajan@carpl.ai

For Stanford AIMI

Johanna Kim | Executive Director | johannakim@stanford.edu

CARPL wins Economic Times Health Tech Startup of the Year 2021!

CARPL Wins Economic Times Health Tech Startup Of The Year 2021!

CARPL – the world’s first testing and deployment platform for radiology automation has recently been awarded as the Health Tech Startup of the Year by Economic Times! We are proud to partner with some of the world’s leading AI developers, helping healthcare providers improve access, affordability and quality of medical care globally.


 

Using Traditional Machine Learning Techniques for Predicting the Best Clinical Outcomes On Improvement In Vas, Mod And Ncos Based Upon Clinical And Imaging Features

Using Traditional Machine Learning Techniques for Predicting the Best Clinical Outcomes On Improvement In Vas, Mod And Ncos Based Upon Clinical And Imaging Features

PURPOSE OR LEARNING OBJECTIVE:

To use machine learning techniques for predicting the clinical outcomes on improvement in VAS, MOD, and NCOS based on the managements like Spinal decompression without fusion (open discectomy/Laminectomy), Spinal compression with fusion, and Conservative Management.

METHODS OR BACKGROUND:

Patients with symptomatic lumbar spine disease with back pain with or without radiculopathy and neurological deficit were enrolled. The primary outcome measures were Visual analogue scale (VAS), Modified Oswestry Disability Index (MOD), and Neurogenic Claudication Outcome Score (NCOS) collected at pre-operatively and at 3 months post-operatively. The further analysis studied the following factors to determine if any are predictive of outcomes: sex, BMI, occupation, involvement in sports, herniation type, depression, work status, herniation level, duration of symptoms, and history of past spine surgery. The features were selected and machine learning models were trained to predict the improvement in the primary outcome measures. The results were evaluated on the basis of the ROC-AUC score for different classes.

RESULTS OR FINDINGS:

There were a total of 200 entries of patients with Lumbar Spine Disease between age 18 and above. Among the various Machine Learning Models, Random Forest Classifier gave the best ROC-AUC score in all three classes. The AUC score for VAS, MOD, and NCOS was 0.877, 0.8215, and 0.830 respectively and the macro-average AUC score was found to be 0.84 (see Fig.1).

 Fig 1: ROC-AUC scores of different Machine Learning Algorithms on Test Dataset.

 

CONCLUSION:

Machine Learning model could be used as a predictive tool for deciding the type of management that a patient should undergo to achieve the best results. Based on the predicted improvement in different indices for the particular management type, the predictions could help Surgeons for deciding the type of management that would be most beneficial for the patient.

Estimation of bias of deep learning-based chest X-ray classification algorithm

Estimation Of Bias Of Deep Learning-Based Chest X-Ray Classification Algorithm​

PURPOSE OR LEARNING OBJECTIVE:

To evaluate the bias in the diagnostic performance of a deep learning-based chest X-ray classification algorithm on previously unseen external data.

METHODS OR BACKGROUND:

632 chest X-rays were randomly collected from an academic centre hospital and anonymised selectively, leaving out fields needed for the bias estimation (manufacturer name, age, and gender). They were from six different vendors – AGFA (388), Carestream (45), DIPS (21), GE (31), Philips (127), and Siemens (20). The male and female distribution was 376 and 256. The X-rays were read for consolidation ground truth establishment on CARING analytics platform (CARPL). These X-rays were run on open-sourced chest X-ray classification model. Inferencing results were analysed using Aequitas, an open-source python-based package to detect the presence of bias, fairness of algorithms. Algorithms’ performance was evaluated on the three metadata classes – gender, age group, and brand of equipment. False omission rate (FOR) and false-negative rate (FNR) metrics were used for calculating the inter-class scores of bias.

RESULTS OR FINDINGS:

AGFA, 60 to 80 age group, and male were the dominant entities and hence considered as baseline for evaluation of bias towards other classes. Significant false omission rate (FOR) and false negative rate (FNR) disparities were observed for all vendor classes except Siemens as compared to AGFA. No gender disparity was seen. All groups show FNR parity whereas all classes showed disparity with respect to false omission rate for age.

CONCLUSION:

We demonstrate that AI algorithms may develop biases, based on the composition of training data. We recommend bias evaluation check to be an integral part of every AI project. Despite this, AI algorithms may still develop certain biases, some of those difficult to evaluate.

LIMITATIONS:

Limited pathological classes were evaluated.