In this exhibit, we discuss the six key must-have features on any analytical platform that is intended for the validation of AI algorithms. With the recent developments in machine learning and especially deep learning, a lot of companies are trying to develop solutions for assisting radiologists in medical imaging. We have developed a system that combines statistics with medical inputs to provide insights and validate deep learning algorithms at scale. One of the key challenges is the variety of output from these algorithms. The output could be a binary variable if the model is trying to predict whether the patient is suffering from a disease or not. Some algorithms, for example, predicting the nodule size, have a continuous variable as the output. Other algorithms can have even more complex outputs like the 3D boundary of intracranial hemorrhage. Our system presents the data to data scientists and medical practitioners in the simplest form possible with all the important insights. In addition to this, an integrated arbitration tool helps in validating the output with just a few clicks.
We believe that the usage of such tools will decrease the time required for validating the deep learning algorithms in healthcare setup and at the same time, will provide useful insights to the companies which will help them in improving the algorithm further.
Ability to fetch data from PACS: To conduct a study, the hospital/clinic should be able to easily search and extract cases. The tool should have features to filter cases on the basis of modalities, diseases, and other related fields. In addition to these, advanced features like semantic search can be really useful to capture the diversity of diseases and modalities. Our system provides features to include/exclude modalities and diseases.
Client-side anonymization: Data privacy and security is an important aspect of any validation study. The system should have the ability to anonymize DICOM images on the browser side only. Uploading files on the cloud system and then anonymization could have serious implications in terms of data privacy.
Cloud-Based computation: To conduct the studies at scale, it is essential to have a cloud-based deployment where the configuration of the system could be increased in real-time depending on the usage. Apart from computational flexibility, it also provides wider accessibility. Anyone with internet can access the system and conduct the study. As soon as the arbitrator uploads DICOM images, the processing of the cases starts automatically on our cloud-based system.
Visualization: To present the data in the simplest and most meaningful form is the biggest challenge for any study. With a wide variety of outputs(for example, binary variable, a continuous variable, masking areas, etc), it is essential to present the data in a form that can help arbitrator to gauge the model accuracy in the best possible way. For example, for chest x-ray algorithms, our system provides the following scatter plot. Different colors and the number of cases of each type(Abnormal, Normal, Mismatch and Not Reported) help the arbitrator in understanding the crux in a single look. Apart from the plot, the ability to change the threshold provides an ability to the arbitrator to test the algorithm at different thresholds in real-time. ROC/AUC curve plays a key role in deciding the threshold for algorithms with binary output.
Arbitration: For the cases where there is a mismatch between radiologists (ground truth) and algorithm’s output, it is essential to have a third-eye looking at the data to minimize any human error. The tool should have an interface where the arbitrator can see the ground truth and algorithm’s output and can act as a moderator. Ideally, a system should have a DICOM viewer integrated with an ability to input the arbitrator’s feedback.
Summary Report: Once the arbitration process is done, the system should generate a summary of the algorithm’s performance on a set of parameters. These parameters can vary depending on the type of modality and the study.
Once the arbitration process is done, the system should generate a summary of the algorithm’s performance on a set of parameters. These parameters can vary depending on the type of modality and the study.
The poster can be viewed here: http://dx.doi.org/10.26044/ecr2020/C-14851
