Index

Deep Learning for Cancer Patient Survival Prediction Using 2D Portrait Photos Based on StyleGAN Embedding

Current deep-learning-based classification methods require large amounts of data for training, and in certain scenarios such as in the surveillance imaging there is only a limited amount of data. The aim of the research is to generate new training images of vehicles with the same characteristics as the training data but from novel view points and investigate its suitability for fine-grained classification of vehicles.

Generative models such as generative adversarial networks (GANs) [1] allow for customization of images. However, adjusting the perspective through methods such as conditional GANs for unsupervised image-to-image translation has proven to be particularly difficult [1]. Methods such as StyleGANs [2] or neural radiance fields (NeRFs) [3] are relevant approaches to generate images with different styles and perspectives.
StyleGAN is an extension to the GAN architecture that proposes changes to the generator model such as the introduction of a mapping network. The mapping network generates intermediate latent codes which are transformed into styles that is integrated at each point in the generator network. It also includes a progressive growing approach for training generator models capable of synthesizing very large high-quality images.
NeRF can generate novel views of complex 3D scenes based on a partial set of 2D images. It is trained to directly map from spatial location and viewing direction (5D input) to opacity and color, using volume rendering [4] to render new views.

The thesis consists of the following milestones:

Literature review on the state-of-the-art approaches for GAN- and neural radiance fields-based
image synthesis
Adoption of existing GAN- and neural radiance fields-based image synthesis methods to generate
car images using different styles and camera poses [5]
Experimental evaluation and comparison of different image synthesis methods
Investigate the suitability of the generated images for fine-grained vehicle classification using
different classification methods [6], [7]

The implementation will be done in Python.

References
[1] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, “Generative Adversarial Networks ”, in NIPS, 2014
[2] Tero Karras, Samuli Laine, Timo Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks ”, in proceedings of the IEEE/CVF Conference on CVPR, 2019
[3] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis ”, in ECCV 2020
[4] Robert A. Drebin, Loren Carpenter, Pat Hanrahan, “Volume Rendering ”, in Proceedings of SIGGRAPH 1988
[5] Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt, “StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis ”, in ICLR 2022
[6] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows ”, in IEEE/CVF conference on ICCV, 2021
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Deep residual learning for image recognition ”, Proceedings of the IEEE conference on CVPR, 2016

Modelling of the breast during the mammography examination

Metal-conscious Transformer Enhanced CBCT Projection Inpainting

Computed tomography device (CT) is a means of tomographic imaging technology, and it has been developed
rapidly. Due to beam hardening effect, metallic artifacts occur and degrade the quality of CT images. Metal
artifacts have been the focus and difficulty in the field of CT imaging research because of their direct impact on
clinical diagnosis and the diversity of manifestations and causes [1]. In order to reconstruct metal-free CT
images, the inpainting task is an essential part.
The traditional method of inpainting replaces the metal-affected region of the projected data by interpolation
[2][3]. Recently, deep convolutional networks (CNNs) have shown strong potential in all computer vision tasks,
including image inpainting. Several approaches have been proposed for image restoration using CNN based
encoder-decoder network. Shift-Net based on U-Net architecture is one of these approaches, which has good
restoration accuracy in structure and texture [4]. Zeng et al. [5] built a pyramidal-context architecture called
PEN-NET for high-quality image inpainting. Liao et al. [6] proposed a new generative mask pyramid network
to reduce for CT/CBCT Metal Artifact Reduction. Although CNNs have many advantages, their field of
perception is usually small and not conducive to capturing global features. On the contrary, Vision Transformer
(ViT) uses attention to model long-term dependencies among image patches. The shifted window Transformer
(Swin Transformer) is proposed to adapt to the high resolution of images in vision tasks [8], taking into account
the translational invariance of CNNs, the perceptual field and the hierarchical relationship.
To overcome the shortage of medical image data and the domain shift problem in the field of deep learning, this
research is based on simulated X-ray images using ViT as the encoder and CNN as the decoder for image
inpainting. In order to further improve the inpainting performance, some variants of the backbone network are
considered, such as using Swin Transformer instead of ViT and adding the adversarial loss.
The paper will include the following points:
• Literature review in inpainting and metal artifacts reduction.
• Traditional method and CNN based model implementation.
• ViT-based model construction; parameter optimization and incorporation with adversarial loss; results
evaluation.
• Thesis writing.

References
[1] Netto, C., Mansur, N., Tazegul, T., Lalevee, M., Lee, H., Behrens, A., Lintz, F., Godoy-Santos, A., Dibbern,
K., Anderson, D. Implant Related Artifact Around Metallic and Bio-Integrative Screws: A CT Scan 3D
Hounsfield Unit Assessment. Foot & Ankle Orthopaedics. 7, 2473011421S00174 (2022)
[2] Kalender WA, Hebel R, Ebersberger J. Reduction of CT artifacts caused by metallic implants. Radiology.
1987 Aug;164(2):576-7. doi: 10.1148/radiology.164.2.3602406. PMID: 3602406.
[3] Meyer E, Raupach R, Lell M, Schmidt B, Kachelriess M. Normalized metal artifact reduction (NMAR) in
computed tomography. Med Phys. 2010 Oct;37(10):5482-93. doi: 10.1118/1.3484090. PMID: 21089784.
[4] Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via
deep feature rearrangement, 2018.
[5] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder
network for high-quality image inpainting, 2019.
[6] Haofu Liao, Wei-An Lin, Zhimin Huo, Levon Vogelsang, William J Sehnert, S Kevin Zhou, and Jiebo Luo.
Generative mask pyramid network for ct/cbct metal artifact reduction with joint projection-sinogram
correction. In International Conference on Medical Image Computing and Computer-Assisted Intervention,
pages 77–85. Springer, 2019.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth
16×16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[8] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin
transformer: Hierarchical vision transformer using shifted windows, 2021.

Detection and Classification of Photovoltaic Modules in Electroluminescence Videos

The hippocampus and language: Word to word prediction in terms of the successor representation

The theoretical background of the master thesis is formed by the place and grid cells of the hippocampus, which are responsible for a wide variety of navigation tasks. This ranges from classical spatial navigation in a city or a building to abstract assignments in cognitive rooms, like the maximum speed of a vehicle based on engine power and weight. Since basic place cell firing patterns have already been investigated by machine learning, the thesis will focus on whether this method can also be used to process speech in order to draw conclusions about the involvement of place and grid cells in this domain. For this purpose, the theory of cognitive maps and its mathematical formulation the Successor Representation will be used.
To apply this concept to language, different techniques of Natural Language Processing as well as a neural network will be used. The former are mainly used to provide the training data for the network. These consist of successive pairs of words, one serving as input, the other as output. The goal is to infer the grammatical structure from the word-by-word predictions. To achieve this, several configurations are investigated, with the main focus on processing books that are used as a proxy for valid language data.

Fully Automated Segmentation of Subcutaneous Fat in CT Images

Thesis description

Given that obesity is as a major global health issue, as well as the fact that body fat is an important
risk factor for cancer, many cardiovascular and metabolic diseases [1, 2], providing a precise measuring
tool for the distribution of adipose tissue is of high interest. Adipose tissue is also associated with
many physiological functions as the principal energy storage organ and due to its endocrine activity [3].
Computed tomography (CT) and magnetic resonance imaging (MRI) are both utilized to localize and
quantify body fat, however, in contrast to CT, MRI is more challenging with the current segmentation
problem because of the image intensity inhomogeneities [4]. In addition, MRI is slower, more expensive
and thus less clinically available [5].

To the best of our knowledge, all approaches of the most recent publications addressing the problem
of subcutaneous adipose tissue (SAT) segmentation still lack one or more important aspects. The vast
majority of related approaches are only semi-automatic, requiring a carefully chosen user input to
reach their goals [6] or are rather mostly manual [1, 7, 8]. Several convolutional neural network-based
methods [9, 10], together with an active-contour-based method [11] for fully automatic subcutaneous
fat segmentation were already proposed, but they are either limited to the abdominal region or are
only operating on 2D image slices, which is sub-optimal for 3D image data. A novel neural network
architecture was found to have achieved accurate 3D segmentation results on volumetric CT data for
both thorax and abdomen [5]. However, the remaining drawbacks are the mislabeled annotations for
certain thoracic slices and the relatively small training dataset (only 18 images), which will restrict
the model generalizability. Therefore, the contribution of this thesis is intended to fill the gaps in the
previous publications by introducing a fully automatic, more reliable and reproducible framework for
3D segmentation of the abdominal and thoracic SAT in CT images.

Semantic segmentation networks have become a powerful tool for segmenting spatially structured
images and thus play an essential role for biomedical image data. However, since there is no dataset
available with the required ground truth annotations of SAT, these annotation masks have to be
generated first. Manual delineation of the inner and outer contours that are defining SAT on axial
images is an inefficient and time-consuming process, so some semi-automatic algorithms are used to
accelerate the generation of initial segmentation masks for our dataset. The dataset originally consists
of selected CT images from Siemens internal database. In principle, the work of the thesis will be
twofold. In the first phase, active contours (AC) [12, 13] will be used as a baseline algorithm for SAT
segmentation. Nevertheless, several improvement steps, such as finding optimum AC parameters,
helpful preprocessing and having consistent 3D masks [14], need to be implemented to get satisfactory
segmentations. Final annotations can be then used for the training after some manual corrections.
The second phase is to train a deep neural network using the annotated dataset in order to have a
fully automatic 3D SAT segmentation. For this task, nnU-Net [15] as an advanced state-of-the-art
deep learning-based segmentation tool is going to be applied. Furthermore, different training schemes
that rely on anatomical prior knowledge (i.e. two different segmentation networks for thorax and
abdomen) and ground truth-driven patch sampling will be implemented and evaluated.

The thesis will comprise the following work items:
• Literature review of most efficient active contour segmentation algorithms, fully- and semiautomated
segmentation methods for subcutaneous fat in CT images
• Utilization of the improved active contour algorithms besides some manual corrections to generate
an annotated dataset
• Implementation and training of a deep neural network to fully automate SAT segmentation
• Quantitative assessment and evaluation of the developed method
• Encapsulation of the new pipeline into usable MeVisLab modules (www.mevislab.de) for later
development of the company’s current prototype software

References

[1] Amir A. Mahabadi, Joseph M. Massaro, Guido A. Rosito, Daniel Levy, Joanne M. Murabito,
Philip A. Wolf, Christopher J. O’Donnell, Caroline S. Fox, and Udo Hoffmann. Association of
pericardial fat, intrathoracic fat, and visceral abdominal fat with cardiovascular disease burden:
the framingham heart study. European Heart Journal, 64(7):850–856, 2009. https://academic.
oup.com/eurheartj/article-lookup/doi/10.1093/eurheartj/ehn573.
[2] Giuliano Enzi, Mauro Gasparo, Pietro Raimondo Biondetti, Davide Fiore, Marcello Semisa, and
Francesco Zurlo. Subcutaneous and visceral fat distribution according to sex, age, and overweight,
evaluated by computed tomography. The American Journal of Clinical Nutrition, 44(6):739–746,
2009. https://academic.oup.com/ajcn/article/44/6/739/4692311.
[3] Jules Dichamp, Corinne Barreau, ChristopheGuissard, Audrey Carri´ere, Yves Martinez, Xavier
Descombes, Luc Penicaud, Jacques Rouquette, LouisCasteilla, Franck Plouraboue, and Anne
Lorsignol. 3D analysis of the whole subcutaneous adipose tissue reveals a complex spatial network
of interconnected lobules with heterogeneous browning ability. Scientific Reports, 9(1):6684, 2019.
http://www.nature.com/articles/s41598-019-43130-9.
[4] Hans-Peter M¨uller, Florian Raudies, Alexander Unratha, Heiko Neumann, Albert C. Ludolph,
and Jan Kassubek. Quantification of human body fat tissue percentage by MRI: Quantification
of human body fat tissue. NMR in Biomedicine, 24(1):17–2–4, 2011. https://onlinelibrary.
wiley.com/doi/10.1002/nbm.1549.
[5] Tiange Liu, Junwen Pan, Drew A. Torigian, Pengfei Xu, Qiguang Miao, Yubing Tong, and
Jayaram K. Udup. ABCNet: A new efficient 3D dense-structure network for segmentation and
analysis of body tissue composition on body-torso-wide CT images. American Association of
Physicists in Medicine, 47(7):2986–2999, 2020. https://doi.org/10.1002/mp.14141.
[6] Robin F. Gohmann, Sebastian Gottschling, Patrick Seitz, Batuhan Temiz, Christian Krieghoff1,
Christian L¨ucke, Matthias Horn, and Matthias Gutberlet. 3D-segmentation and characterization
of visceral and abdominal subcutaneous adipose tissue on CT: influence of contrast medium and
contrast phase. Quantitative Imaging in Medicine and Surgery, 11(2):697–705, 2021. http:
//qims.amegroups.com/article/view/55860/html.
[7] Won G. Kwack, Yun-Seong Kang, Yun J. Jeong, Jin Y. Oh, Yoon K. Cha, Jeung S. Kim, and
Young S. Yoon. Association between thoracic fat measured using computed tomography and lung
function in a population without respiratory diseases. Journal of Thoracic Disease, 11(12):5300–
5309, 2019. http://jtd.amegroups.com/article/view/34021/html.
[8] Yubing Tong, Jayaram K. Udupa, Drew A. Torigian, Dewey Odhner, CaiyunWu, Gargi Pednekar,
Scott Palmer, Anna Rozenshtein, Melissa A. Shirk, John D. Newell, Mary Porteous, Joshua M.
Diamond, Jason D. Christie, and David J. Lederer. Chest fat quantification via CT-based on
standardized anatomy space in adult lung transplant candidates. PLOS ONE, 12(1), 2017. https:
//dx.plos.org/10.1371/journal.pone.0168932.
[9] Zheng Wang, Yu Meng, Futian Weng, Yinghao Chen, Fanggen Lu, Xiaowei Liu, Muzhou Hou,
and Jie Zhang. An effective CNN method for fully automated segmenting subcutaneous and
visceral adipose tissue on ct scans. Annals of Biomedical Engineering, 48(1):312–328, 2020.
http://link.springer.com/10.1007/s10439-019-02349-3.
[10] Sebastian Nowak, Anton Faron, Julian A. Luetkens, Helena L. Geißler, Michael Praktiknjo, Wolfgang
Block, Daniel Thomas, and Alois M. Sprinkart. Fully automated segmentation of connective
tissue compartments for CT-based body composition analysis. Investigative Radiology, 55(6):357–
366, 2020. http://journals.lww.com/10.1097/RLI.0000000000000647.
[11] Scott J. Lee, Jiamin Liu, Jianhua Yao, Andrew Kanarek, Ronald M. Summers, and Perry J.
Pickhardt. Fully automated segmentation and quantification of visceral and subcutaneous fat at
abdominal CT: application to a longitudinal adult screening cohort. The British Journal of Radiology,
91:20170968, 2018. http://www.birpublications.org/doi/10.1259/bjr.20170968.
[12] Fabien Pierre, Mathieu Amendola, Cl´emence Bigeard, Timoth´e Ruel, and Pierre-Fr´ed´eric Villard.
Segmentation with active contours. Image Processing On Line, 11:120–141, 2021. https://www.
ipol.im/pub/art/2021/298/?utm_source=doi.
[13] Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. IEEE Comput.
Soc. Press, pages 694–699, 1995. http://ieeexplore.ieee.org/document/466871/.
[14] Huiyan Jiang and Qingshui Cheng. Automatic 3D segmentation of CT images based on active
contour models. 11th IEEE International Conference on Computer-Aided Design and Computer
Graphics (CAD/Graphics), pages 540–543, 2009. http://ieeexplore.ieee.org/document/
5246845/.
[15] Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, , and Klaus H. Maier-Hein. nnUNet:
a self-configuring method for deep learning-based biomedical image segmentation. Nature
Methods, 18(2):203–211, 2021. http://www.nature.com/articles/s41592-020-01008-z.