We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Diagnosis of malignancy in oropharyngeal confocal laser endomicroscopy using GPT 4.0 with vision.
- Authors
Sievert, Matti; Aubreville, Marc; Mueller, Sarina Katrin; Eckstein, Markus; Breininger, Katharina; Iro, Heinrich; Goncalves, Miguel
- Abstract
Purpose: Confocal Laser Endomicroscopy (CLE) is an imaging tool, that has demonstrated potential for intraoperative, real-time, non-invasive, microscopical assessment of surgical margins of oropharyngeal squamous cell carcinoma (OPSCC). However, interpreting CLE images remains challenging. This study investigates the application of OpenAI's Generative Pretrained Transformer (GPT) 4.0 with Vision capabilities for automated classification of CLE images in OPSCC. Methods: CLE Images of histological confirmed SCC or healthy mucosa from a database of 12 809 CLE images from 5 patients with OPSCC were retrieved and anonymized. Using a training data set of 16 images, a validation set of 139 images, comprising SCC (83 images, 59.7%) and healthy normal mucosa (56 images, 40.3%) was classified using the application programming interface (API) of GPT4.0. The same set of images was also classified by CLE experts (two surgeons and one pathologist), who were blinded to the histology. Diagnostic metrics, the reliability of GPT and inter-rater reliability were assessed. Results: Overall accuracy of the GPT model was 71.2%, the intra-rater agreement was κ = 0.837, indicating an almost perfect agreement across the three runs of GPT-generated results. Human experts achieved an accuracy of 88.5% with a substantial level of agreement (κ = 0.773). Conclusions: Though limited to a specific clinical framework, patient and image set, this study sheds light on some previously unexplored diagnostic capabilities of large language models using few-shot prompting. It suggests the model's ability to extrapolate information and classify CLE images with minimal example data. Whether future versions of the model can achieve clinically relevant diagnostic accuracy, especially in uncurated data sets, remains to be investigated.
- Subjects
LANGUAGE models; GENERATIVE pre-trained transformers; SQUAMOUS cell carcinoma; IMAGE recognition (Computer vision); TRANSFORMER models
- Publication
European Archives of Oto-Rhino-Laryngology, 2024, Vol 281, Issue 4, p2115
- ISSN
0937-4477
- Publication type
Article
- DOI
10.1007/s00405-024-08476-5