Future medical scanners will become more autonomous and situation-aware (scene understanding). Intelligent algorithms for scene understanding need data to be trained and if possible, this data needs to be available early in the development process. Synthetic data can be valuable for this purpose.
The data used in this thesis is generated by making use of Augmented Reality (AR) glasses and RGB-D sensors. The user of the AR glasses runs a virtual MR examination while being recorded by a system of RGB-D sensors. The system that the user operates in AR is a digital twin of a real MR scanner [1]. The user manipulates coils, cushions, headphones, and any other required accessory and interacts with an avatar of a patient. Virtual 3D point clouds are generated from the AR scene and real 3D point clouds are recorded with the RGB-D sensors. These point cloud data sets are later fused resulting in a synthetic 3D depth data set of the whole MR examination scene.
The aim of this thesis is to develop and train algorithms for scene understanding and analysis based on these synthetic data sets. Firstly, semantic segmentation of the synthetic 3D point clouds using deep learning techniques shall be applied, and scene descriptors shall be designed and developed. Goal is to detect the elements of the provided scenes (e.g. operator, patient, magnet, coils, etc.) in synthetic and
real world data. The network will be trained using synthetic 3D data (3D point clouds). Moreover, synthetic 3D data as well as real 3D data from a real system will be used for testing the approach. As a reference, the work of Nie et al [2] will be used.
This thesis shall furthermore discuss how workflow state detectors can be designed to detect the different states of an MR examination (e.g. idle, patient preparation, coil fixation) based on the results of the semantic segmentation, scene description.
Finally, a discussion about the usefulness, advantages, and challenges of synthetic data will be provided.
[1] MAGNETOM Free.Max. Siemens Healthineers; https://www.siemens-healthineers.com/magnetic-resonance-imaging/high-v-mri/magnetom-free-max
[2] Nie, Yinyu & Hou, Ji & Han, Xiaoguang & Niesner, Matthias. (2021). RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction. 4606-4616. 10.1109/CVPR46437.2021.00458.