Index

Multi-Task Learning for Glacier Segmentation and Calving Front Detection with the nnU-Net Framework

With the nnU-Net, Fabian Isensee et al. [1] provide a training framework for the widely applied
segmentation network U-Net [2]. The framework automates the tedious adjustment of hyperparameters
(e.g. number of layers, learning rate, batch size, etc.) that is needed to optimize the performance of
the U-Net. The framework takes a fingerprint of the given dataset and adjusts the hyperparameters
accordingly. This approach achieved stunning results on multiple independent datasets in the medical
domain without manual adjustments [1].

This thesis evaluates the performance of nnU-Net on the segmentation of synthetic aperture radar
(SAR) images of glaciers taken by satellites. Additionally to the segmentation of the different regions
(glacier, ocean, rock outcrop, SAR shadow), precise detection of the calving front position provides
important information for the glacier surveillance [3]. The relatedness of these two problems suggests
the application of multi-task learning (MTL) [4]. MTL with a single input image and multiple output
labels can be divided into late branching and early branching networks [5].

Late branching: The network is trained with multiple output channels for each task. This approach is widely used, because of its straightforward implementation [6, 7]. Dot et al. already uses the nnU-Net framework for segmentation of craniomaxillofacial structures in CT scans [8].
Early branching: Two separate Decoders are trained for each task with a common Encoder. This approach was used by Amyar et al. [9] to jointly segment lesion from lung CT scans and identify COVID-19 patients.

The nnU-Net framework will be extended for the application of MTL. Both approaches will be implemented
and evaluated on a dataset of glacier images. The dataset contains SAR images of seven
glaciers, their corresponding segmentation masks, and calving front positions. To emphasize the effect
of MTL on the performance of the nnU-Net, additional tasks like reconstruction of the input image will
be integrated into the training. The resulting models will be compared quantitatively and qualitatively
with the single-task networks and state-of-the-art in calving front detection.

[1] Fabian Isensee et al. “nnU-Net: a self-configuring method for deep learning-based biomedical
image segmentation”. In: Nature Methods 18.2 (Feb. 2021), pp. 203–211.

[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. “U-net: Convolutional networks for
biomedical image segmentation”. In: International Conference on Medical image computing and
computer-assisted intervention. Springer. 2015, pp. 234–241.

[3] Celia A. Baumhoer et al. “Environmental drivers of circum-Antarctic glacier and ice shelf front
retreat over the last two decades”. In: The Cryosphere 15.5 (May 2021), pp. 2357–2381.

[4] Konrad Heidler et al. “HED-UNet: Combined Segmentation and Edge Detection for Monitoring
the Antarctic Coastline”. In: IEEE Transactions on Geoscience and Remote Sensing (Mar. 2021),
pp. 1–14.

[5] Kelei He et al. “HF-UNet: Learning Hierarchically Inter-Task Relevance in Multi-Task U-Net for
Accurate Prostate Segmentation in CT Images”. In: IEEE transactions on medical imaging 40.8
(Aug. 2021), pp. 2118–2128.

[6] Ava Assadi Abolvardi, Len Hamey, and Kevin Ho-Shon. “UNET-Based Multi-Task Architecture
for Brain Lesion Segmentation”. In: Digital Image Computing: Techniques and Applications
(DICTA). Melbourne, Australia: IEEE, Nov. 2020, pp. 1–7.

[7] Soumyabrata Dev et al. “Multi-label Cloud Segmentation Using a Deep Network”. In: USNCURSI
Radio Science Meeting (Joint with AP-S Symposium). July 2019, pp. 113–114.

[8] Gauthier Dot et al. “Fully automatic segmentation of craniomaxillofacial CT scans for computerassisted
orthognathic surgery planning using the nnU-Net framework”. In: medRxiv (2021).

[9] Amine Amyar et al. “Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia:
Classification and segmentation”. In: Computers in Biology and Medicine 126 (Nov. 2020),
p. 104037.

Virtual Dynamic Contrast Enhanced Image Prediction of Breast MRI using Deep Learning Architectures

Test

The UKER BrainMet Dataset: A brain metastasis dataset from University Hospital Erlangen

Brain Metastasis Synthesis Using Deep Learning in MRI Images

GAN-based Synthetic Chest X-ray Generation for Training Lung Disease Classification Systems

Project description

With the rise and ever-growing potential of Deep Learning (DL) techniques in recent years, completely new opportunities have emerged in the field of medical image processing, in particular in fundamental application areas such as image detection and recognition, image segmentation, image registration, and computer-aided diagnosis. However, DL techniques are known to require very large amounts of data to train the neural networks (NN), which can sometimes be a problem due to limited data availability. In recent years, the public release of medical image data has increased and has led to significant advances in the scientific community. For instance, publicly availabe large-scale chest X-ray datasets enabled the development of novel systems for automated lung abnormality classification [1, 2]. In recent work, however, it has been shown that DL techniques can also be used maliciously, e. g., for linkage attacks on public chest X-ray datasets [3]. This constitutes a tremendous issue in terms of data security and patient privacy, as a potential attacker may leak available information (e. g. age, gender, diseases, and more) about a specific patient present in a public dataset. To alleviate privacy concerns, the question now arises whether the exlusive use of synthetically generated images can represent a serious alternative for the development of diagnostic algorithms in the medical field.

In this work, we investigate whether synthetically generated chest X-ray images can be used to train a reliable classification system for lung diseases. Therefore, we will use different approaches, e. g., [4–6], to synthesize realistic looking chest X-ray scans from a real data distribution. In doing so, we will focus on ensuring that characteristic disease patterns will be preserved in the generated images. For our experiments, we will employ the NIH ChestX-ray14 [7] dataset, a collection of 112,120 frontal-view chest X-ray images from 30,805 unique patients with the text-mined fourteen disease image labels.

The Master’s thesis covers the following aspects:

Overview of the current state-of-the-art in DL for the generation of synthetic medical image data.
Building one or multiple GAN-based image generation networks which includes:
- Hyper-parameter tuning
- Analyzing the performance of the networks
- Analyzing the realism of the generated images
Evaluating the feasibility of using synthetically generated chest X-ray images for training a lung disease classification system.
Outlining strategies and research directions to enhance the preservation of patient privacy in public datasets (optional).

All DL implementations will be implemented using PyTorch [8].

References

[1] S. Gündel, S. Grbic, B. Georgescu, S. Liu, A. Maier, and D. Comaniciu, “Learning to Recognize Abnormalities in Chest X-Rays with Location-Aware Dense Networks,” in Iberoamerican Congress on Pattern Recognition, pp. 757–765, Springer, 2018.

[2] S. Gündel, A. A. Setio, F. C. Ghesu, S. Grbic, B. Georgescu, A. Maier, and D. Comaniciu, “Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment,” Medical Image Analysis, vol. 72, p. 102087, 2021.

[3] K. Packhäuser, S. Gündel, N. Münster, C. Syben, V. Christlein, and A. Maier, “Is Medical Chest X-ray Data Anonymous?,” arXiv preprint arXiv:2103.08562, 2021.

[4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” Advances in Neural Information Processing Systems, vol. 27, 2014.

[5] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” arXiv preprint arXiv:1411.1784, 2014.

[6] A. Odena, C. Olah, and J. Shlens, “Conditional Image Synthesis with Auxiliary Classifier GANs,” in International Conference on Machine Learning, pp. 2642–2651, PMLR, 2017.

[7] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[8] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” Advances in Neural Information Processing Systems, vol. 32, pp. 8026–8037, 2019.

Exploring Style-transfer techniques on Greek vase paintings for enhancing pose-estimation

Multi-stage Patch based U-Net for Text Line Segmentation of Historical Documents

Deep Learning-based Bleed-through Removal in Historical Documents

Disentangling Visual Attributes for Inherently Interpretable Medical Image Classification

Project description:

Interpretability is essential for a deep neural network approach when applied to crucial scenarios such as medical image processing. Current gradient-based [1] and counterfactual image-based [2] interpretability approaches can only provide information of where the evidence is. We also want to know what the evidence is. In this master thesis project, we will build an inherently interpretable classification method. This classifier can learn disentangled features that are semantically meaningful and, in the future, corresponding to related clinical concepts.

This project based on a previous proposed visual feature attribution method in [3]. This method can generate class relevant attribution map for a given input disease image. We will extend this method to generate class relevant shape variations and design an inherently interpretable classifier only using the disentangled features (class relevant intensity variation and shape variation). This method can be further extended by disentangling more semantically meaningful and causal independent features such as texture, shape, and background as the work in [4].

References

[1] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.

[2] Cher Bass, Mariana da Silva, Carole Sudre, Petru-Daniel Tudosiu, Stephen M Smith, and Emma C Robinson. Icam: Interpretable classifi- cation via disentangled representations and feature attribution mapping. arXiv preprint arXiv:2006.08287, 2020.

[3] Christian F Baumgartner, Lisa M Koch, Kerem Can Tezcan, Jia Xi Ang, and Ender Konukoglu. Visual feature attribution using wasserstein gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8309–8319, 2018.

[4] Axel Sauer and Andreas Geiger. Counterfactual generative networks. arXiv preprint arXiv:2101.06046, 2021.