Index

End-to-End SMPL Parameter Estimation and Template-to-Scan Registration Tailored Towards Automatic X-ray to CT Initialization

Abstract: In the field of interventional image fusion, an automatic initialization process is highly desired to reduce reliance on manual user inputs. Recently, several techniques have been proposed which utilize landmarks to provide an efficient initialization for 2D/3D registration [1, 2]. Alternatively, another method involves directly regressing the 6D orientation of the CT volume based on the input X-ray image. Each of these approaches, however, encounters difficulties in gathering adequate ground truth data to train a sufficiently robust system [3].
In addition to these methods, there’s a growing line of research investigating the utilization of statistical shape models to provide an effective initialization for 2D/3D registration algorithms. When given X-ray images paired with ground truth statistical shape model parameters, multiple approaches can be employed to solve the problem of estimating these parameters. The challenge, however, lies in establishing a ground truth pairing between the CT volumes and the statistical shape model, with the goal of learning a network which can be directly applied for 2D/3D initialization. In essence, we aim to consolidate the two-step process: first establishing pseudo ground truth parameters, then learning a network to predict the parameters in accordance with the pseudo ground truth. Our approach has the potential to reduce systematic errors incurred during the process of establishing pseudo ground truth, providing a more robust initialization that is directly suitable for use in 2D/3D registration tasks.
Moving forward, we plan to construct our proposed method using a linear inverse kinematic solver [4], in conjunction with a self-supervised template-to-scan registration process [5]. This approach will enable us to directly learn end-to-end estimation of SMPL parameters, as well as parameters to register to the input scan (CT volume). In doing so, we can directly predict the 3D CT volume-specific initialization for a given X-ray image without the necessity of an additional explicit step to register the CT volume with the statistical shape model.

References:
[1] Bier, B., Unberath, M., Zaech, J. N., Fotouhi, J., Armand, M., Osgood, G., … & Maier, A. (2018, September). X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 55-63). Springer, Cham.
[2] Grupp, R. B., Unberath, M., Gao, C., Hegeman, R. A., Murphy, R. J., Alexander, C. P., … & Taylor, R. H. (2020). Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. International journal of computer assisted radiology and surgery, 15(5), 759-769.
[3] Grimm, M., Esteban, J., Unberath, M., & Navab, N. (2021). Pose-dependent weights and Domain Randomization for fully automatic X-ray to CT Registration. IEEE Transactions on Medical Imaging.
[4] Shetty, Karthik, et al. “PLIKS: A Pseudo-Linear Inverse Kinematic Solver for 3D Human Body Estimation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[5] Bhatnagar, Bharat Lal, et al. “Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration.” Advances in Neural Information Processing Systems 33 (2020): 12909-12922.

Contrastive Learning for Glacier Segmentation

Deep Learning-based Detection of Detector Artifacts

Learning Reconstruction Filters for CBCT Geometry

This project focuses on advancing computer tomography (CT) image reconstruction by utilizing neural network technology. The project is divided into three main sections.

In the first part, a versatile iterative reconstruction algorithm is introduced for Cone Beam CT (CBCT) reconstruction. This algorithm gradually converges to the actual values through backpropagation, using the disparity between reconstructed results and ground truth. The TV-Norm regularization method is also employed to reduce noise while preserving image edge clarity.

Recognizing the limitations of iterative reconstruction methods, the second part employs a data-driven CT reconstruction approach. A trainable filter-backprojection (FBP) reconstruction neural network is designed to enhance filter design, resulting in a filter with high-frequency noise suppression capabilities. This improves the quality of reconstruction.

In the third part, the data-driven FBCT approach is extended to the Feldkamp-Davis-Kress (FDK) reconstruction algorithm for CBCT. Through neural network training, a latent mapping is learned, progressively converging towards the Ram-lak filter. This sets the groundwork for learning complex filters tailored to non-circular trajectories.

Application of projection-based metrics to the optimisation of arbitrary CT scanning trajectory

This work focuses on the optimization of CT trajectory through the use of projection-based metrics. The objective is to improve reconstruction accuracy, reduce the number of required projections, and minimize scan time. Existing metrics will be implemented and evaluated, while an optimization approach will be explored and refined. Additionally, a new metric will be developed by combining existing metrics using different weights or a learned result from a neural network. The optimized trajectory will be evaluated using RMSE and SSIM indexes to assess the quality of the reconstruction. The study aims to efficiently optimize CT trajectories by considering multiple metrics and potentially introducing a new metric for trajectory optimization.

Tackling Travelling Salesman Problem with Graph Neural Network and Reinforcement Learning

This project focuses on solving the Travelling Salesman Problem using Graph Neural Network combined with Reinforcement Learning algorithms. Two variants of Graph Neural Network are tested, including Graph Pointer Network and Hybrid Pointer Network, both trained in Actor-Critic algorithm and double Q-learning algorithm separately. Double Q-learning is tried carefully as it is rarely applied in the training of Graph Neural Network compared with Actor-Critic. The models are tested on various types of TSP instances, showing that double Q-learning algorithm is a potential competitor in the improvement of
Graph Neural Networks.

Synthetic Projection Generation with Angle Conditioning

Computed Tomography (CT) plays a vital role in medical imaging, offering cross-sectional views of internal structures. Yet, radiation exposure during CT scans poses health risks. This study explores the application of existing deep learning models to synthesize CT projections at unknown arbitrary angles. Multiple input images from varying angles, along with their corresponding ground truth data, train different network architectures to reproduce target images from different view angles.  This approach potentially reduces radiation exposure and addresses challenges in obtaining specific missing angular views. Experimental results confirm the effectiveness and feasibility of the methodology, establishing it as a valuable tool in CT imaging.

On how to learn and use the Detectability Index efficiently for CT trajectory optimisation

Optimizing the CT scan trajectory is crucial for industrial computed tomography as it can enhance the quality
of image reconstruction and reduce scanning time. However, determining the optimal trajectory is challenging
due to the large solution space of the nondeterministic polynomial time-hard optimization problem.
The objective of this study is to propose a suitable architecture for optimizing the trajectory of a robot-based
computed tomography (CT) system. This architecture aims to improve the quality of reconstructed images,
effectively representing the detectability index for a given task.
The goal of this optimization is to reduce artefacts in the CT images and potentially decrease the scanning time.
To achieve this objective, the proposed method requires a CAD model of the test specimen, simulates possible
X-ray projections and predicts the detectability index using a suitable regression neural network architecture.

Context-Assisted Killer Whale Photo Identification using Deep Learning

Improving the Robustness of In-Circuit Test in Electronic Production using Deep Neural Network and Synthetic Tabular Data

Introduction

Since the fourth Industrial revolution, named ‘Industry 4.0’, the manufacturing
production process has become more flexible, traceable, and less complicated.
The worldwide demand for electronic products, particularly Printed Circuit
Boards (PCBs), is continuously increasing [1]. The manufacturing of PCBs in
the electronic industries needs high precision to ensure product optimality and
reliability because any defect in PCBs can lead to fatal flaws at the product
level. Even if only one component on the PCB does not work properly, there
is a possibility that the whole board is defective. Thus, accurate production
of PCB is critical. Machine learning techniques are widely used in electronic
industries to improve the manufacturing process and for detecting faults in order
to deliver high-quality PCBs to customers.

The thesis is conducted at Siemens AG in Erlangen, an independent man-
ufacturer of electronic products in a production environment driven by the
requirement of customers. The main technologies involved in the production
are Surface-Mount Technology (SMT) and Through-Hole Technology (THT)
[2]. During the SMT assembly line, the circuit boards are mounted with SMT
components using soldering paste and a reflow-oven system to ensure that the
component adheres to the PCB and has an electrical connection. Then PCBs
are passed through the THT assembly line to mount the THT components by
wires using drilling holes on the board and THT soldering to connect the com-
ponents onto the board. Once SMT and THT components are placed on the
circuit, multiple optical inspection tests like the Automatic Optical Inspection
(AOI) or Automatic X-ray Inspection (AXI) are applied during the components
mounting procedure. At the end of the process, the Printed Circuit Board As-
sembly (PCBAs) are tested using In-Circuit Test (ICT) and subsequently, a
Functional test to confirm that the complete PCB is functioning well [3].

Using ICT, the measurement values of all passive components on the PCBAs
are measured in order to check for defective components or a defective soldering
connection. Each component is defined with upper and lower threshold values.
The measured values are compared to the default values. The test fails when
the measured value does not fall within the default range. This raises an error
flag in In-Circuit Testing. This error could also be a Pseudo error (false alarm)
due to measurement errors. So, an additional process of manual inception is
done for every error flag from In-Circuit Testing of PCBAs at the production
line. Checking all the error calls of PCBA manually during production, was
heavily time-consuming and cost-intensive.

Other factors such as contaminated or misaligned ICT needles, can also
influence a failure besides out-of-range measured values [4]. The measured values
vary substantially due to external effects such as changing temperatures caused
by soldering. Therefore, these thresholds are required to be adjusted. Thus, test
engineers are bound to tune the upper and lower thresholds. This leads to a
high degree of misclassification of Real error from Pseudo error (false alarm) [4].
Moreover, high Pseudo error percentages during production increase the chance
of missing a real defect due to negligence, potentially leading to an increase in
the error slip ratio.

In an unautomated setting, the PCBAs pass through multiple ICTs if the
first ICT test fails. This additional testing ensures that the error is not due to a
temporary fault caused by a non-robust measurement. So, the dataset contains
the boards with the same serial number multiple times because of the ICT
reruns. If this rerun also fails, manual inspection is carried out. To reduce labor-
intensive manual Inspections, especially for non-critical cases, namely Pseudo
error, Cognitive Analytics for Test Engineering (CATE) is used [5]. CATE is an
automated machine learning model implemented to support In-Circuit Testing
of PCBAs in order to eliminate the ICT reruns and also to distinguish Pseudo-
errors from Real Errors [6]. Using neural network models like CATE in PCB
production can help test engineers avoid false positive error flags on the shop
floor, which will help them to accelerate and optimize their work.

Problem Statement

Machine Learning techniques can provide remarkable solutions when using huge
uncorrupted data. However, such solutions are ineffective if the data is limited
and noisy, which is typical for a manufacturing environment. CATE only sends
the actual Real errors to the inspection, which reduces the manual effort to
the mandatory minimum [7]. This thesis aims to overcome the shortcoming
of CATE such as highly imbalanced and noisy datasets on Real errors and
Pseudo errors, and the One-sided decision (upper limit) threshold of ICTs on
measurement values of the components [5]. Besides, the current approach is not
robust against production process variations and known error patterns at new
components.

To overcome the current challenges in CATE, this thesis presents the approaches to using synthetic tabular data for model training and optimizing the
model parameters to include a two-sided test evaluation. The noisy nature of
the dataset and imbalanced target class will be addressed using synthetic tab-
ular data. Thus, the real error patterns like open solder joints or solder bridges
can be stimulated [5]. Moreover, I want to analyze the impact of different
approaches on model structure and parameters.
Based on these improvements the following five hypotheses shall be examined:
• The synthetic dataset will be similar to the error patterns of the original dataset [12].
• Applying CTGAN will improve the model’s robustness and generalization to learn the distribution of neural network model features [8].
• Adding the GAN approach will produce better outcomes compared to traditional sampling techniques for class imbalance [9] [10].
• Applying synthetic data to XGBoost will enhance the performance metrics of the model, which reduces the Pseudo errors to improve production  efficiency [11] [12].
• Adding uncertainty measure to the neural network model will reduce the misclassfication rate by using the probability cut-off value [13].

Data
The data for CATE is constructed of the output produced by the ICT, ground
truth from inspections and error database. The data consists of the following:
• Sartikelnr: The type of board that is in the production line.
• Sid: Serial number of the boards of the same type. E.g.: T-P56079964
• ICT: If a board passes the ICT test, the result is TRUE. otherwise, FALSE
• Functional test: If a board passes the subsequent the functional test, then
TRUE. otherwise, FALSE
• Component ID: The type of components that are mounted onto the boards
such as R1000 for a Resistor with ID 1000)
• Measurement Value: The measurement value is the actual value captured
by the ICT for a respective component. E.g.: the measurement of 100
Ohm for a resistor.
• Unit: The unit of measurement is logged to normalize the measurement
values

• Lower limit: If the measurement value is lower than this threshold, the
result is set to FAIL.
• Upper limit: If the measurement value is higher than this threshold, the
result is set to FAIL.
• Result: The actual test result is represented as either PASS or FAIL.
When the upper or lower limits are not surpassed the result is PASS.
• Ground Truth: The Ground truth shows the values from inspection data,
if a board passes the test in first time, then the label is FIRST PASS.
SECOND PASS or MULTI PASS if a board passes the test in reruns. if
it fails, then the label is ONLY FAIL. ERROR and PSEUDO ERROR
represents the error flag of the board. INSPECTION label tells us if the
board was in manual inspection.

Methodology

Building a model

The model will be built to generate a synthetic tabular dataset. One of the
approaches that are considered for synthetic tabular data generation is the Con-
ditional Tabular Generative Adversarial Network (CTGAN), a specialization of
the GAN architecture for synthesizing tabular data was presented in 2019 by
Lei Xu et al. in ’Modeling Tabular Data Using Conditional GAN’ [8]. CTGAN
is used to handle the imbalanced and noisy dataset and believes the approach
has the potential to enhance the prediction accuracy [14]. The similarity of cat-
egorical correlations between synthetic and original datasets will be measured.
The eXtreme Gradient Boosting (XGBoost) algorithm is used in the CATE
use case for the model classification task of distinguishing the Real errors from
the Pseudo errors. XGBoost modeling method will be tested with CTGAN and
other traditional sampling techniques. Then, it will be validated on a specific
validation dataset to verify the robustness against the data drift and known
error patterns at new components.
In order to avoid the misclassification rate, the uncertainty measure will be
added to the model. This will tell us how uncertain the model prediction is.
The uncertainty rate in the model aims to be as confident as possible about
a prediction but also to limit the number of uncertain predictions [13]. The
probability cut-off value of uncertainty on the ICT model will alert the test
engineers that the error requires a manual check.

Implementation

For implementing this thesis several toolkits can be used. The most prominent
being sdv for CTGAN [8], Scikit-learn [15], and Keras [16]. The training of the
model will take place centrally with the help of AWS Sagemaker.

Evaluation

The model evaluation will be partitioned into three areas. First, a comparison
between the synthetic dataset and the original dataset is needed to have a sim-
ilarity measure. Here, The evaluation metric presented by Melle Mendikowski
can be used [17]. Second, the performance of the model using the CTGAN,
other sampling approaches and also the original dataset as input to XGBoost
will be estimated by the model’s F1-score. Lastly, to estimate more informed
and precise decisions on error flags, uncertainty estimation will be examined
[13].

References

[1] Tingting Chen et al., “Machine Learning in Manufacturing towards Indus-
try 4.0: From ‘For Now’ to ‘Four-Know,’” In: Applied Sciences, vol. 13,
no. 3, p. 1903, Feb. 2023. doi: 10.3390/app13031903.
[2] Eva Jabbar et al. “Conditional Anomaly Detection for Quality
and Productivity Improvement of Electronics Manufacturing Sys-
tems”. In: Lecture Notes in Computer Science. Vol. 11943 LNCS.
Springer, 2019, pp. 711–724, doi: 10.1007/978-3-030-37599-7 59. url:
http://link.springer.com/10.1007/978-3-030-37599-7 59.
[3] S S Zakaria et al. “Automated Detection of Printed Circuit Boards (PCB)
Defects by Using Machine Learning in Electronic Manufacturing: Cur-
rent Approaches”. In: IOP Conference Series: Materials Science and
Engineering, (Feb. 2020), p. 012064.issn: 1757-8981, doi: 10.1088/1757-
899X/767/1/012064. url: https://iopscience.iop.org/article/10.1088/1757-
899X/767/1/012064.
[4] Nabil El Belghiti Alaoui et al. “Upgrading In-Circuit Test of High
Density PCBAs Using Electromagnetic Measurement and Principal
Component Analysis”. In: Journal of Electronic Testing34.6 (Dec.
2018), pp. 749–762 isbn :0923-8174. doi:10.1007/s10836-018-5763 4. url:
http://link.springer.com/10.1007/s10836-018-5763-4.
[5] Erik Schwulera and Michael Plendl. CATE (Cognitive Analytics for Test
En- gineering) or Use of Artificial Intelligence (AI) to Optimize Test Pa-
rameters in Real time at the Edge. Tech. rep. Siemens AG, 2020.
[6] Siemens AG. Prescriptive Analytics for Test Engineering. Tech. rep.
Siemens AG, 2018.
[7] Ziqiu Kang et al. ”Machine learning applications in production lines: A
systematic literature review”. In: Computers & Industrial Engineering (Jul.
2020). url: https://doi.org/10.1016/j.cie.2020.106773
[8] Lei Xu et al. ”Modeling Tabular Data using Conditional GAN”. In: 33rd
Conference on Neural Information Processing Systems (NeurIPS 2019),
arXiv:1907.00503. url: https://doi.org/10.48550/arXiv.1907.00503
[9] Fajardo, Val Andrei et al. “On oversampling imbalanced data with deep
conditional generative models.” In: Expert Syst. Appl. 169 (2021): 114463.
[10] Wentao Mao et al. ”Imbalanced Fault Diagnosis of Rolling Bearing Based
on Generative Adversarial Network: A Comparative Study,” In: IEEE
Access, vol. 7, pp. 9515-9530, 2019. doi: 10.1109/ACCESS.2018.2890693.
[11] Vadim Borisov et al. ”Deep Neural Networks and Tabular Data: A Sur-
vey”, In: IEEE Transactions on Neural Networks and Learning Systems,
pp(99):1-21. doi: 10.1109/TNNLS.2022.3229161
[12] Aziira, A. H. et al. ”Generation of Synthetic Continuous Numerical Data
Using Generative Adversarial Networks”, In: Journal of Physics: Con-
ference Series, Volume 1577, Issue 1, article id. 012027 (Jul. 2020). doi:
10.1088/1742-6596/1577/1/012027
[13] Gawlikowski et al. “A Survey of Uncertainty in Deep Neural Networks”,
ArXiv abs/2107.03342, 2021.
[14] Subhajit Chatterjee et al. ”A Synthetic Data Generation Technique for
Enhancement of Prediction Accuracy of Electric Vehicles Demand”. In:
Sensors. 2023; 23(2):594. url: https://doi.org/10.3390/s23020594
[15] F. Pedregosa et al. “Scikit-Learn: Machine Learning in Python”. In: Jour-
nal of Machine Learning Research 12 (2011), pp. 2825–2830.
[16] Chollet et al, ”Keras”, 2015. url: https://github.com/fchollet/keras.
[17] Melle Mendikowski et al. ”Creating Customers That Never Existed – Syn-
thesis of E-commerce Data Using CTGAN”. In: Machine Learning and
Data Mining in Pattern Recognition. International Conference on Machine
Learning and Data Mining in Pattern Recognition (MLDM-2022), pp. 91–
105, volume: 284, isbn: 978-3-942952-93-4