Thesis Description
Coronary Artery Disease (CAD) is one the most predominant contributors to cardiovascular disease
that stands as the major cause of death globally [1]. Usually, it manifests itself as narrowed or blocked
arteries caused by plaque buildup, a condition known as Atherosclerosis. Percutaneous Coronary In-
tervention (PCI) is a frequently used treatment for CAD in which the narrowed arteries are widened.
A typical type of PCI is revascularization using angioplasty with a stent [2]. Generally, as part of this
treatment, a thin flexible tube is inserted into the femoral artery through the groin. Once the tip is
properly positioned in the blockage site, a balloon which is surrounded by a stent graft is inflated to
compress the plaque against the arterial walls. After the procedure is completed and the balloon is
removed, the stent keeps the artery open and supports the blood flow.
Real-time 2D X-ray projections serve as the guidance method of choice for catheter-based interventions
like PCI to help the physicians visually determine the position and extent of the stents [3]. However,
visualizing stents in conventional X-ray images is challenging because of their low radio-opacity. Hence,
digital stent enhancement (DSE) methods have been developed to enhance the stent visibility in X-ray
image sequences [4]. Angioplasty balloons often incorporate two highly radio-opaque markers [5]. De-
tecting and tracking these markers enables DSE methods by registering all frames within the sequence
followed by a mean intensity projection [6]. Therefore, an accurate and robust detection of the stent
markers is a crucial component of all DSE methods. This task is usually performed automatically
using machine learning (ML). An additional challenge in this domain is that due to the high frame
rates of up to 30 frames per second the marker detection needs high computational efficiency.
Possible ML techniques for this task include landmark detection and object detection approaches.
These play a crucial role in the computer vision area, particularly in the field of medical image pro-
cessing [7]. They facilitate the identification of anatomical features, recognition and precise localization
of pathological conditions, and the accurate delineation of structures of interest within medical im-
ages [8]. A few of these methodologies have been employed for balloon marker detection and stent
localization [9–15]. Some approaches rely on conventional ML approaches such as template-based-
matching [14] and adaptive thresholding [9]. On the other hand, the continuous progress in deep
learning offers promising potential for further advancements in stent localization. U-net, a widely
adopted CNN architecture in medical image segmentation, has been employed as the backbone of
several state-of-the-art models to segment the catheter shaft [12] or generate markers heatmap by
treating each landmark as a 2D Gaussian distribution [15]. Although these approaches have improved
stent visualization significantly, they primarily focus on detecting and tracking single balloon marker
pairs in each frame. To address the challenge of stabilizing multiple stents, some researchers have
employed object detection methods through guidewire endpoint localization employing an extended
variant of the Faster R-CNN model [10, 11]. However, detecting objects with common architectures
like R-CNN family models is computationally demanding. Since real-time performance is crucial for
providing instant information about the position of stents to physicians, faster models like YOLO [16]
are needed to accelerate the process. Therefore, a model based on YOLOv3 is proposed that meets
the requirement for real-time guidewire detection and endpoint localization [13].
Coronary Artery Disease (CAD) is one the most predominant contributors to cardiovascular disease
that stands as the major cause of death globally [1]. Usually, it manifests itself as narrowed or blocked
arteries caused by plaque buildup, a condition known as Atherosclerosis. Percutaneous Coronary In-
tervention (PCI) is a frequently used treatment for CAD in which the narrowed arteries are widened.
A typical type of PCI is revascularization using angioplasty with a stent [2]. Generally, as part of this
treatment, a thin flexible tube is inserted into the femoral artery through the groin. Once the tip is
properly positioned in the blockage site, a balloon which is surrounded by a stent graft is inflated to
compress the plaque against the arterial walls. After the procedure is completed and the balloon is
removed, the stent keeps the artery open and supports the blood flow.
Real-time 2D X-ray projections serve as the guidance method of choice for catheter-based interventions
like PCI to help the physicians visually determine the position and extent of the stents [3]. However,
visualizing stents in conventional X-ray images is challenging because of their low radio-opacity. Hence,
digital stent enhancement (DSE) methods have been developed to enhance the stent visibility in X-ray
image sequences [4]. Angioplasty balloons often incorporate two highly radio-opaque markers [5]. De-
tecting and tracking these markers enables DSE methods by registering all frames within the sequence
followed by a mean intensity projection [6]. Therefore, an accurate and robust detection of the stent
markers is a crucial component of all DSE methods. This task is usually performed automatically
using machine learning (ML). An additional challenge in this domain is that due to the high frame
rates of up to 30 frames per second the marker detection needs high computational efficiency.
Possible ML techniques for this task include landmark detection and object detection approaches.
These play a crucial role in the computer vision area, particularly in the field of medical image pro-
cessing [7]. They facilitate the identification of anatomical features, recognition and precise localization
of pathological conditions, and the accurate delineation of structures of interest within medical im-
ages [8]. A few of these methodologies have been employed for balloon marker detection and stent
localization [9–15]. Some approaches rely on conventional ML approaches such as template-based-
matching [14] and adaptive thresholding [9]. On the other hand, the continuous progress in deep
learning offers promising potential for further advancements in stent localization. U-net, a widely
adopted CNN architecture in medical image segmentation, has been employed as the backbone of
several state-of-the-art models to segment the catheter shaft [12] or generate markers heatmap by
treating each landmark as a 2D Gaussian distribution [15]. Although these approaches have improved
stent visualization significantly, they primarily focus on detecting and tracking single balloon marker
pairs in each frame. To address the challenge of stabilizing multiple stents, some researchers have
employed object detection methods through guidewire endpoint localization employing an extended
variant of the Faster R-CNN model [10, 11]. However, detecting objects with common architectures
like R-CNN family models is computationally demanding. Since real-time performance is crucial for
providing instant information about the position of stents to physicians, faster models like YOLO [16]
are needed to accelerate the process. Therefore, a model based on YOLOv3 is proposed that meets
the requirement for real-time guidewire detection and endpoint localization [13].
This thesis aims to investigate a set of research questions with respect to real-time models for detecting
multiple balloon markers in fluoroscopic images. Firstly, the most promising approaches from the
literature need to be compared, also with the most recent developments in the field included. These
include the object detection networks of the YOLO family [17] as well as the heatmap-based point
regression approaches.
Secondly, the pre-processing of the fluoroscopic images can vary to a large extent as multiple algorithms
with a plethora of parameters can be altered. Therefore, it is of high interest to evaluate which influence
different pre-processing parameterization has on the performance of a marker detector.
Finally, the training of a robust network requires a large collection of data covering a large variation
of potential physical influences. To mitigate the need to collect a large amount of clinical data, an
evaluation of whether the use of suitable phantom data is sufficient shall be conducted.
The thesis will comprise the following work items:
• Literature overview of state-of-the-art automated landmark and object detection approaches
• Balloon marker detection
– Data annotation and preprocessing
– Train a network of the YOLO-family on phantom data
– Train a U-net model for heatmap regression
– Possibly: post-processing
• Analysis of the Deep Learning models
– Evaluate the effect of various pre-processing parameterizations on the marker detector
performance
– Evaluate and compare the performance of implemented algorithms on both phantom and
clinical data
multiple balloon markers in fluoroscopic images. Firstly, the most promising approaches from the
literature need to be compared, also with the most recent developments in the field included. These
include the object detection networks of the YOLO family [17] as well as the heatmap-based point
regression approaches.
Secondly, the pre-processing of the fluoroscopic images can vary to a large extent as multiple algorithms
with a plethora of parameters can be altered. Therefore, it is of high interest to evaluate which influence
different pre-processing parameterization has on the performance of a marker detector.
Finally, the training of a robust network requires a large collection of data covering a large variation
of potential physical influences. To mitigate the need to collect a large amount of clinical data, an
evaluation of whether the use of suitable phantom data is sufficient shall be conducted.
The thesis will comprise the following work items:
• Literature overview of state-of-the-art automated landmark and object detection approaches
• Balloon marker detection
– Data annotation and preprocessing
– Train a network of the YOLO-family on phantom data
– Train a U-net model for heatmap regression
– Possibly: post-processing
• Analysis of the Deep Learning models
– Evaluate the effect of various pre-processing parameterizations on the marker detector
performance
– Evaluate and compare the performance of implemented algorithms on both phantom and
clinical data
References
[1] Gregory A Roth, George A Mensah, Catherine O Johnson, Giovanni Addolorato, Enrico Ammi-
rati, et al. Global burden of cardiovascular diseases and risk factors, 1990–2019: update from the
GBD 2019 study. Journal of the American College of Cardiology, 76(25):2982–3021, 2020.
[2] Javaid Iqbal, Julian Gunn, and Patrick W Serruys. Coronary stents: historical development,
current status and future directions. British Medical Bulletin, 106(1), 2013.
[3] Ardit Ramadani, Mai Bui, Thomas Wendler, Heribert Schunkert, Peter Ewert, et al. A survey of
catheter tracking concepts and methodologies. Medical Image Analysis, page 102584, 2022.
[4] Vincent Bismuth, R ́egis Vaillant, Fran ̧cois Funck, Niels Guillard, and Laurent Najman. A com-
prehensive study of stent visualization enhancement in X-ray images by image processing means.
Medical Image Analysis, 15(4):565–576, 2011.
[5] Robert A Close, Craig K Abbey, and James Stuart Whiting. Improved image guidance of coronary
stent deployment. In Medical Imaging 2000: Image Display and Visualization, volume 3976, pages
301–304. SPIE, 2000.
[6] Kyle McBeath, Krishnaraj Rathod, Matthew Cadd, Anne-Marie Beirne, Oliver Guttmann, et al.
Use of enhanced stent visualisation compared to angiography alone to guide percutaneous coro-
nary intervention. International Journal of Cardiology, 321:24–29, 2020.
[7] Zhuoling Li, Minghui Dong, Shiping Wen, Xiang Hu, Pan Zhou, et al. CLU-CNNs: Object
detection for medical images. Neurocomputing, 350:53–59, 2019.
[8] Andreas Maier, Christopher Syben, Tobias Lasser, and Christian Riess. A gentle introduction
to deep learning in medical image processing. Zeitschrift f ̈ur Medizinische Physik, 29(2):86–101,
2019.
[9] Negar Chabi, Oliver Beuing, Bernhard Preim, and Sylvia Saalfeld. Automatic stent and catheter
marker detection in X-ray fluoroscopy using adaptive thresholding and classification. In Current
Directions in Biomedical Engineering, volume 6. De Gruyter, 2020.
[10] Xiaolu Jiang, Yanqiu Zeng, Shixiao Xiao, Shaojie He, Caizhi Ye, et al. Automatic detection of
coronary metallic stent struts based on YOLOv3 and R-FCN. Computational and Mathematical
Methods in Medicine, 2020, 2020.
[11] Rui-Qi Li, Xiao-Liang Xie, Xiao-Hu Zhou, Shi-Qi Liu, Zhen-Liang Ni, et al. A unified framework
for multi-guidewire endpoint localization in fluoroscopy images. IEEE Transactions on Biomedical
Engineering, 69(4):1406–1416, 2021.
[12] Ina Vernikouskaya, Dagmar Bertsche, Tillman Dahme, and Volker Rasche. Cryo-balloon catheter
localization in X-ray fluoroscopy using U-Net. International Journal of Computer Assisted Radi-
ology and Surgery, 16:1255–1262, 2021.
[13] Rui-Qi Li, Xiao-Liang Xie, Xiao-Hu Zhou, Shi-Qi Liu, Zhen-Liang Ni, et al. Real-time multi-
guidewire endpoint localization in fluoroscopy images. IEEE Transactions on Medical Imaging,
40(8):2002–2014, 2021.
[14] Ahmed G Kotb, Ahmed M Mahmoud, and Muhammad A Rushdi. Template-based balloon-
marker and guidewire detection for coronary stents in cardiac fluoroscopy. In 2022 44th Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages
2199–2202. IEEE, 2022.
[15] Luojie Huang, Yikang Liu, Li Chen, Eric Z Chen, Xiao Chen, et al. Robust landmark-based
stent tracking in X-ray fluoroscopy. In European Conference on Computer Vision, pages 201–216.
Springer, 2022.
[16] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified,
real-time object detection. In Proceedings of the IEEE Conference On Computer Vision and
Pattern Recognition, pages 779–788. IEEE, 2016.
[17] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. YOLOv7: Trainable bag-of-
freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 7464–7475. IEEE, 2023
to deep learning in medical image processing. Zeitschrift f ̈ur Medizinische Physik, 29(2):86–101,
2019.
[9] Negar Chabi, Oliver Beuing, Bernhard Preim, and Sylvia Saalfeld. Automatic stent and catheter
marker detection in X-ray fluoroscopy using adaptive thresholding and classification. In Current
Directions in Biomedical Engineering, volume 6. De Gruyter, 2020.
[10] Xiaolu Jiang, Yanqiu Zeng, Shixiao Xiao, Shaojie He, Caizhi Ye, et al. Automatic detection of
coronary metallic stent struts based on YOLOv3 and R-FCN. Computational and Mathematical
Methods in Medicine, 2020, 2020.
[11] Rui-Qi Li, Xiao-Liang Xie, Xiao-Hu Zhou, Shi-Qi Liu, Zhen-Liang Ni, et al. A unified framework
for multi-guidewire endpoint localization in fluoroscopy images. IEEE Transactions on Biomedical
Engineering, 69(4):1406–1416, 2021.
[12] Ina Vernikouskaya, Dagmar Bertsche, Tillman Dahme, and Volker Rasche. Cryo-balloon catheter
localization in X-ray fluoroscopy using U-Net. International Journal of Computer Assisted Radi-
ology and Surgery, 16:1255–1262, 2021.
[13] Rui-Qi Li, Xiao-Liang Xie, Xiao-Hu Zhou, Shi-Qi Liu, Zhen-Liang Ni, et al. Real-time multi-
guidewire endpoint localization in fluoroscopy images. IEEE Transactions on Medical Imaging,
40(8):2002–2014, 2021.
[14] Ahmed G Kotb, Ahmed M Mahmoud, and Muhammad A Rushdi. Template-based balloon-
marker and guidewire detection for coronary stents in cardiac fluoroscopy. In 2022 44th Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages
2199–2202. IEEE, 2022.
[15] Luojie Huang, Yikang Liu, Li Chen, Eric Z Chen, Xiao Chen, et al. Robust landmark-based
stent tracking in X-ray fluoroscopy. In European Conference on Computer Vision, pages 201–216.
Springer, 2022.
[16] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified,
real-time object detection. In Proceedings of the IEEE Conference On Computer Vision and
Pattern Recognition, pages 779–788. IEEE, 2016.
[17] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. YOLOv7: Trainable bag-of-
freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 7464–7475. IEEE, 2023