Voice-Assisted Image Labeling for Endoscopic Ultrasound Classification Using Neural Networks

Bonmati Coll, E., Hu, Y., Grimwood, A., Johnson, G.J., Goodchild, G., Keane, M.G., Gurusamy, K., Davidson, B., Clarkson, M.J., Pereira, S.P. and Barratt, D.C. 2022. Voice-Assisted Image Labeling for Endoscopic Ultrasound Classification Using Neural Networks. IEEE Transactions on Medical Imaging. 41 (6), pp. 1311-1319. https://doi.org/10.1109/tmi.2021.3139023

TitleVoice-Assisted Image Labeling for Endoscopic Ultrasound Classification Using Neural Networks
TypeJournal article
AuthorsBonmati Coll, E., Hu, Y., Grimwood, A., Johnson, G.J., Goodchild, G., Keane, M.G., Gurusamy, K., Davidson, B., Clarkson, M.J., Pereira, S.P. and Barratt, D.C.
Abstract

Ultrasound imaging is a commonly used technology for visualising patient anatomy in real-time during diagnostic and therapeutic procedures. High operator dependency and low reproducibility make ultrasound imaging and interpretation challenging with a steep learning curve. Automatic image classification using deep learning has the potential to overcome some of these challenges by supporting ultrasound training in novices, as well as aiding ultrasound image interpretation in patient with complex pathology for more experienced practitioners. However, the use of deep learning methods requires a large amount of data in order to provide accurate results. Labelling large ultrasound datasets is a challenging task because labels are retrospectively assigned to 2D images without the 3D spatial context available in vivo or that would be inferred while visually tracking structures between frames during the procedure. In this work, we propose a multi-modal convolutional neural network (CNN) architecture that labels endoscopic ultrasound (EUS) images from raw verbal comments provided by a clinician during the procedure. We use a CNN composed of two branches, one for voice data and another for image data, which are joined to predict image labels from the spoken names of anatomical landmarks. The network was trained using recorded verbal comments from expert operators. Our results show a prediction accuracy of 76% at image level on a dataset with 5 different labels. We conclude that the addition of spoken commentaries can increase the performance of ultrasound image classification, and eliminate the burden of manually labelling large EUS datasets necessary for deep learning applications.

JournalIEEE Transactions on Medical Imaging
Journal citation41 (6), pp. 1311-1319
ISSN0278-0062
Year2022
PublisherIEEE
Publisher's version
Digital Object Identifier (DOI)https://doi.org/10.1109/tmi.2021.3139023
Web address (URL)https://doi.org/10.1109/TMI.2021.3139023
Publication dates
Published in printJun 2022
Published online28 Dec 2021

Related outputs

Active learning using adaptable task-based prioritisation
Shaheer U. Saeed, João Ramalhinho, Mark Pinnock, Ziyi Shen, Yunguan Fu, Nina Montaña-Brown, Ester Bonmati, Dean C. Barratt, Stephen P. Pereira, Brian Davidson, Matthew J. Clarkson and Yipeng Hu 2024. Active learning using adaptable task-based prioritisation. Medical Image Analysis. 95 103181. https://doi.org/10.1016/j.media.2024.103181

Fan-Slicer: A Pycuda Package for Fast Reslicing of Ultrasound Shaped Planes
João Ramalhinho, Thomas Dowrick, Bonmati Coll, E. and Matthew J. Clarkson 2023. Fan-Slicer: A Pycuda Package for Fast Reslicing of Ultrasound Shaped Planes. Journal of Open Research Software. 11 (1), p. 3. https://doi.org/10.5334/jors.422

Assisted Probe Positioning for Ultrasound Guided Radiotherapy Using Image Sequence Classification
Grimwood, A., McNair, H., Hu, Y., Bonmati Coll, E., Barratt, D. and Harris, E.J. 2020. Assisted Probe Positioning for Ultrasound Guided Radiotherapy Using Image Sequence Classification. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 23rd International Conference. Lima, Peru 04 - 08 Oct 2020 Springer. https://doi.org/10.1007/978-3-030-59716-0_52

Novel Brain Complexity Measures Based on Information Theory
Bonmati Coll, E., Bardera, A., Feixas, M. and Boada, I. 2018. Novel Brain Complexity Measures Based on Information Theory. Entropy. 20 (7) 491. https://doi.org/10.3390/e20070491

Determination of optimal ultrasound planes for the initialisation of image registration during endoscopic ultrasound-guided procedures
Bonmati Coll, E., Hu, Y., Gibson, E., Uribarri, L., Keane, G., Gurusami, K., Davidson, B., Pereira, S.P., Clarkson, M.J. and Barratt, D.C. 2018. Determination of optimal ultrasound planes for the initialisation of image registration during endoscopic ultrasound-guided procedures. International Journal of Computer Assisted Radiology and Surgery. 13, pp. 875-883. https://doi.org/10.1007/s11548-018-1762-2

Automatic Multi-Organ Segmentation on Abdominal CT with Dense V-Networks
Gibson, E., Giganti, F., Hu, Y., Bonmati Coll, E., Bandula, S., Gurusamy, K., Davidson, B., Pereira, S.P., Clarkson, M.J. and Barratt, D.C. 2018. Automatic Multi-Organ Segmentation on Abdominal CT with Dense V-Networks. IEEE Transactions on Medical Imaging. 37 (8), pp. 1822-1834. https://doi.org/10.1109/tmi.2018.2806309

Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalizing neural network
Bonmati Coll, E., Hu, Y., Sindhwani, N., Dietz, H.P., D'hooge, J., Barratt, D., Deprest, J. and Vercauteren, T. 2018. Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalizing neural network. Journal of Medical Imaging. 5 (2) 021206. https://doi.org/10.1117/1.jmi.5.2.021206

Brain parcellation based on information theory
Bonmati Coll, E., Bardera, A. and Boada, I. 2017. Brain parcellation based on information theory. Computer Methods and Programs in Biomedicine. 151, pp. 203-212. https://doi.org/10.1016/j.cmpb.2017.07.012

Towards image-guided pancreas and biliary endoscopy: Automatic multi-organ segmentation on abdominal CT with dense dilated networks
Gibson, E., Giganti, F., Hu, Y., Bonmati Coll, E., Bandula, S., Gurusamy, K., Davidson, B.R., Pereira, S.P., Clarkson, M.J. and Barratt, D.C. 2017. Towards image-guided pancreas and biliary endoscopy: Automatic multi-organ segmentation on abdominal CT with dense dilated networks. Medical Image Computing and Computer Assisted Intervention − MICCAI 2017. Quebec City, QC, Canada 11 - 13 Sep 2017 Springer. https://doi.org/10.1007/978-3-319-66182-7_83

2D-3D Registration Accuracy Estimation for Optimised Planning of Image-Guided Pancreatobiliary Interventions
Hu, Y., Bonmati Coll, E., Gibson, E., Hipwell, J.H., Hawkes, D.J., Bandula, S., Pereira, S.P. and Barratt, D.C. 2016. 2D-3D Registration Accuracy Estimation for Optimised Planning of Image-Guided Pancreatobiliary Interventions. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016. MICCAI 2016. Athens, Greece 17 - 21 Oct 2016 Springer. https://doi.org/10.1007/978-3-319-46720-7_60

Assessment of Electromagnetic Tracking Accuracy for Endoscopic Ultrasound
Bonmati Coll, E., Hu, Y., Gurusamy, K., Davidson, B., Pereira, S.P., Clarkson, M.J. and Barratt, D.C. 2016. Assessment of Electromagnetic Tracking Accuracy for Endoscopic Ultrasound. Computer-Assisted and Robotic Endoscopy. CARE 2016. Athens, Greece 17 Oct 2016 Springer. https://doi.org/10.1007/978-3-319-54057-3_4

Measuring Complex Brain Networks Structure
Bonmati Ester, Bardera Anton, Boada Imma and Bonmati Coll, E. 2016. Measuring Complex Brain Networks Structure. Frontiers in Neuroinformatics. Conference Abstract: Neuroinformatics 2016. https://doi.org/10.3389/conf.fninf.2016.20.00012

Hierarchical clustering based on the information bottleneck method using a control process
Bonmati Coll, E., Bardera, A., Boada, I., Feixas, M. and Sbert, M. 2015. Hierarchical clustering based on the information bottleneck method using a control process. Pattern Analysis and Applications (PAA). 18, pp. 619-637. https://doi.org/10.1007/s10044-015-0467-1

Permalink - https://westminsterresearch.westminster.ac.uk/item/vwyzy/voice-assisted-image-labeling-for-endoscopic-ultrasound-classification-using-neural-networks


Share this

Usage statistics

48 total views
19 total downloads
These values cover views and downloads from WestminsterResearch and are for the period from September 2nd 2018, when this repository was created.