Index
Fitting a 2D to 3D Transformation with Neural Fields for Vessel Unfolding
Thesis Description
“Time is brain” is a frequently used term associated with the fast diagnosis and treatment of cerebrovascular
disease, especially with stroke. In fact, cerebral cell death in the affected areas starts within only a few
minutes after the impairment of the vessel [vB¨ud22]. A distinction is made between ischemic (occlusion of
vessels that results in blood shortage) and hemorrhagic strokes (injury or rupture of a vessel that results in a
bleeding) [Kha+23]. The relevant anatomical structures for this include the circle of Willis (CoW) and the main
surrounding arteries. For the diagnostic process, the tracking of all of these vessels in the volumetric CT data
can be time consuming. In order to improve this process, it would be beneficial to unfold these vessels from the
3D CT data onto a 2D image plane. However, the complex geometry of the CoW and the almost perpendicular
orientation of some vessels to each other make it infeasible for the whole structure to be unfolded properly by
common visualization techniques like the curved planar reformation [Kan+02]. A heuristic mesh-based approach
to solve this problem has been presented with CeVasMap [Ris+23], but the unfolding and merging of all major
vessels creates strong distortions in some areas, especially for the internal carotid artery and the basilar artery.
Instead of a heuristic method, the transformation itself can be seen as an optimization problem, which allows
for a more flexible transformation and a better incorporation of constraints through the loss function.
Neural fields are currently gaining more and more popularity in computer vision due to their ability to
provide a continuous approximation of a field function, which enables sampling at arbitrary points and resolutions
[Ram+22]. The core element is a simple multi-layer perceptron which typically gets coordinates as
input and outputs the respective field value at those coordinates [Xie+22]. Since these values are lying in the
reconstruction domain, a mapping to the sensor domain is needed in order to evaluate the results and compute
the loss [Xie+22]. A few examples for fields that can be described by this are 2D images, 3D shapes or even
full 3D scenes [Xie+22; Mil+20]. Outside of the field of computer vision, neural fields have also gained popularity
in robotics, audio processing, physics and medical imaging [Xie+22]. In the latter, many applications
have emerged, including CT and MRI image reconstruction [Zan+21; Sun+21] and medical image segmentation
[Kha+22]. In addition to that, the implicit deformable image registration model proposed by Wolterink et
al. [Wol+22] fits a 3D deformation vector field for the registration of CT images, which has similarities to the
transformation, that this thesis aims to find. However, in this work the neural field is trained to produce the
3D coordinates that need to be sampled into the respective 2D image coordinates, such that the resulting image
contains the unfolded vessels. The goal is to find a suitable transformation for each sample in the dataset, such
that the neural field is trained to overfit on only one sample, unlike a typical neural network which needs a
large amount of data for training and testing. In order to achieve a visually pleasing training result, different
quality criteria have to be evaluated and incorporated into the loss function. These include multiple constraints
for the transformation, such as diffeomorphism and isometry. Due to the high degrees of freedom for the transformation,
a good initialization has to be found as well. The mentioned constraints and defined loss terms are
iteratively improved to further enhance the quality of the unfolded image. First, the neural field is trained to
unfold one singular vessel, to reduce the complexity of the geometric structure that has to be unfolded. Once
this yields satisfying results, the neural field can be extended to unfold multiple vessels at the same time.
Summary:
1. Implement a basic neural field architecture to fit a 2D to 3D transformation inspired by Wolterink et
al. [Wol+22]
2. Investigate and tune different loss terms and quality measures for the neural field to unfold phantom
geometries and individual vascular structures
3. Assess and discuss the quality of the resulting images
4. If possible, extend the approach to unfold multiple vessels
References
[Kan+02] A. Kanitsar, D. Fleischmann, R. Wegenkittl, P. Felkel, and E. Groller. CPR – curved planar reformation.
In IEEE Visualization, 2002. VIS 2002. Pages 37–44, 2002. doi: 10.1109/VISUAL.2002.
1183754.
[Kha+22] M. O. Khan and Y. Fang. Implicit Neural Representations for Medical Imaging Segmentation. In L.
Wang, Q. Dou, P. T. Fletcher, S. Speidel, and S. Li, editors, Medical Image Computing and Computer
Assisted Intervention – MICCAI 2022, pages 433–443, Cham. Springer Nature Switzerland, 2022.
isbn: 978-3-031-16443-9.
[Kha+23] A. S. Khaku and P. Tadi. Cerebrovascular Disease. https://www.ncbi.nlm.nih.gov/books/
NBK430927/, January 2023.
[Mil+20] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF:
Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
[Ram+22] S. Ramasinghe and S. Lucey. Beyond Periodicity: Towards a Unifying Framework for Activations
in Coordinate-MLPs. In S. Avidan, G. Brostow, M. Ciss´e, G. M. Farinella, and T. Hassner, editors,
Computer Vision – ECCV 2022, pages 142–158, Cham. Springer Nature Switzerland, 2022. isbn:
978-3-031-19827-4.
[Ris+23] L. Rist, O. Taubmann, H. Ditt, M. S¨uhling, and A. Maier. Flexible Unfolding of Circular Structures
for Rendering Textbook-Style Cerebrovascular Maps. In H. Greenspan, A. Madabhushi, P.
Mousavi, S. Salcudean, J. Duncan, T. Syeda-Mahmood, and R. Taylor, editors, Medical Image
Computing and Computer Assisted Intervention – MICCAI 2023, pages 737–746, Cham. Springer
Nature Switzerland, 2023. isbn: 978-3-031-43904-9.
[Sun+21] Y. Sun, J. Liu, M. Xie, B. Wohlberg, and U. S. Kamilov. CoIL: Coordinate-Based Internal Learning
for Tomographic Imaging. IEEE Transactions on Computational Imaging, 7:1400–1412, 2021. doi:
10.1109/TCI.2021.3125564.
[vB¨ud22] H. J. von B¨udingen. Was Bedeutet ”Zeit ist Hirn”? https : / / schlaganfallbegleitung . de /
wissen/zeit-ist-hirn, July 2022.
[Wol+22] J. M. Wolterink, J. C. Zwienenberg, and C. Brune. Implicit Neural Representations for Deformable
Image Registration. In E. Konukoglu, B. Menze, A. Venkataraman, C. Baumgartner, Q. Dou, and
S. Albarqouni, editors, Proceedings of The 5th International Conference on Medical Imaging with
Deep Learning, volume 172 of Proceedings of Machine Learning Research, pages 1349–1359. PMLR,
June 2022. url: https://proceedings.mlr.press/v172/wolterink22a.html.
[Xie+22] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann,
and S. Sridhar. Neural Fields in Visual Computing and Beyond. Computer Graphics Forum, 2022.
issn: 1467-8659. doi: 10.1111/cgf.14505.
[Zan+21] G. Zang, R. Idoughi, R. Li, P. Wonka, and W. Heidrich. IntraTomo: Self-supervised Learning-based
Tomography via Sinogram Synthesis and Prediction. In 2021 IEEE/CVF International Conference
on Computer Vision (ICCV), pages 1940–1950, 2021. doi: 10.1109/ICCV48922.2021.00197.
Refining Image Segmentation on the BRATS Challenge 2023
Spatiotemporal Denoising in Time Resolved Computed Tomography
X-ray image synthesis via an open source framework
Use a new open source framework generating X-ray images for deep learning model training.
Requirement: Python, CT reconstruction
Please attach your CV and transcripts to fuxin.fan@fau.de
Data-Driven Discovery of Killer Whale Vocalization Sub-Units using Deep Learning
Cross-validation of Network Architectures and Training Data for Learning-based X-ray Scatter Estimation
Pretraining Transformers For Predictive Maintenance In Manufacturing
In the domain of predictive maintenance within manufacturing industries, the introduction of transformer-based machine learning models marks a significant leap towards more sophisticated anomaly detection mechanisms. These models can discern complex patterns in time series data, predicting potential equipment malfunctions before they lead to costly downtimes. The methodology of this thesis builds on two promising approaches: the first leverages a masked autoencoder framework designed for transformers to predict obscured parts of the input data, thereby learning the normal operational patterns of the machinery [1]. The second approach utilizes the reconstruction loss of a Variational Autoencoder (VAE) to signal deviations from the norm, which may indicate anomalies [2]. Both approaches are integral to this research’s objective of enhancing predictive maintenance strategies.
Furthering the innovation in this field, the thesis will incorporate cutting-edge transformer models such as TranAD and AnomalyBERT, which have shown exceptional results in quickly and accurately identifying anomalies in multivariate time series data [3][4]. TranAD’s focus score-based self-conditioning and adversarial training, along with AnomalyBERT’s data degradation scheme for self-supervised learning, position these models at the forefront of the predictive maintenance revolution.
The anticipated outcomes of this research encompass the development and validation of a robust framework for industrial anomaly detection. This will be achieved by adapting and optimizing transformer networks that are proficient in handling the high volatility and label scarcity characteristic of industrial datasets.
The implications of this study are profound, offering not only a technological edge to predictive maintenance but also a significant academic contribution to the application of AI in manufacturing. The models and methodologies derived from this thesis could serve as benchmarks for future research and applications in the AI and industrial maintenance landscape.
References
[1] Tang, Peiwang & Zhang, Xianchao. (2022). MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting.
[2] Niu, Zijian, Ke Yu, and Xiaofei Wu. (2020). “LSTM-Based VAE-GAN for Time-Series Anomaly Detection.” Sensors 20, no. 13: 3738. https://doi.org/10.3390/s20133738.
[3] Shreshth Tuli, Giuliano Casale, Nicholas R. Jennings. “TranAD: deep transformer networks for anomaly detection in multivariate time series data.” Proceedings of the VLDB Endowment 15.6 (2022): 1201-1214. https://doi.org/10.14778/3514061.3514067.
[4] Yungi Jeong, Eunseok Yang, Jung Hyun Ryu, Imseong Park, Myungjoo Kang. “ANOMALYBERT: SELF-SUPERVISED TRANSFORMER FOR TIME SERIES ANOMALY DETECTION USING DATA DEGRADATION SCHEME.” Presented at the ICLR 2023 workshop on Machine Learning for IoT.