DIAGNOSTIC ACCURACY OF ARTIFICIAL INTELLIGENCE FOR ANALYSIS OF 1.3 MILLION MEDICAL IMAGING STUDIES: THE MOSCOW EXPERIMENT ON COMPUTER VISION TECHNOLOGIES medrxiv.org/content/10.1101/20

DIAGNOSTIC ACCURACY OF ARTIFICIAL INTELLIGENCE FOR ANALYSIS OF 1.3 MILLION MEDICAL IMAGING STUDIES: THE MOSCOW EXPERIMENT ON COMPUTER VISION TECHNOLOGIES

Objective: to assess the diagnostic accuracy of services based on computer vision technologies at the integration and operation stages in Moscow's Unified Radiological Information Service (URIS). Methods: this is a multicenter diagnostic study of artificial intelligence (AI) services with retrospective and prospective stages. The minimum acceptable criteria levels for the index test were established, justifying the intended clinical application of the investigated index test. The Experiment was based on the infrastructure of the URIS and United Medical Information and Analytical System (UMIAS) of Moscow. Basic functional and diagnostic requirements for the artificial intelligence services and methods for monitoring technological and diagnostic quality were developed. Diagnostic accuracy metrics were calculated and compared. Results: based on the results of the retrospective study, we can conclude that AI services have good result reproducibility on local test sets. The highest and at the same time most balanced metrics were obtained for AI services processing CT scans. All AI services demonstrated a pronounced decrease in diagnostic accuracy in the prospective study. The results indicated a need for further refinement of AI services with additional training on the Moscow population datasets. Conclusions: the diagnostic accuracy and reproducibility of AI services on the reference data are sufficient, however, they are insufficient on the data in routine clinical practice. The AI services that participated in the experiment require a technological improvement, additional training on Moscow population datasets, technical and clinical trials to get a status of a medical device. ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement No funding. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The study was approved by the Independent Ethics Committee of the Moscow Society of Radiology on February 20, 2020. The principles for conducting the Experiment corresponded to the Joint Statement of the European and North American Society of Radiologists and others on the Ethics of Artificial Intelligence in Radiology. All patients signed a special informed consent form for voluntary participation. A specially designed brochure was provided for the additional information. The Experiment was registered in the ClinicalTrials with the assigned ID [NCT04489992][1]. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes All data produced in the present study are available upon reasonable request to the authors [1]: /lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT04489992&atom=%2Fmedrxiv%2Fearly%2F2023%2F08%2F31%2F2023.08.31.23294896.atom

www.medrxiv.org
Sign in to participate in the conversation
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.