Evaluation of a Pixel-wise Regression Model Solving a Segmentation Task and a Deep Learning Model with the Matthew’s Correlation Coefficient as an Early Stopping Criterion
With global sea level rising and mass loss of polar ice sheets as the main cause, it becomes increasingly important to enchance ice dynamics modeling. A very fundamental information for this is the calving front position (CFP) of glaciers. Traditionally the delineating of the CFP has been done manually, which is a very subjective, tedious and expensive task. Since then, there has been a lot of development in automating this process. Gourmelon et al. [1] introduce the first publicly available benchmark dataset for calving front delineation on synthetic aperture radar (SAR) imagery dubbed CaFFe. The dataset consists of the SAR imagery and two corresponding labels: one showing the calving front vs the background and the other showing different landscape regions. However, for this paper we will only look at methods using the former. As there are many different approaches to calving front delineation the question of what method provides the best performance arises. Subsequently, the aim of this thesis is to evaluate the codes of the following two papers [2],[3] on the CaFFe benchmark dataset and compare their performance with the baselines provided by Gourmelon et al. [1].
- paper 1: Davari et al. [2] reformulates the segmentation problem into a pixel-wise regression task by using a Convolutional Neural Network (CNN) that gets optimized to predict a distance map containing a distance value for each pixel of the image to extract the glacier calving front line with the help of a second U-net.
- paper 2: Davari et al. [3] proposes a deep learning model with the Mathew Correlation Coefficient as an early stopping criterion to counter the extreme class imbalance of this problem. Moreover, a distance map based binary cross-entropy (BCE) loss function gets introduced to add context about the important regions for segmentation. To make a fair and reasonable comparison, the hyperparameters of each model will be optimized on the CaFFe benchmark dataset and the model weights will be re-trained on CaFFe’s train set. The evaluation will be conducted on the provided test set and the metrics introduced in Gourmelon et al. [1] will be used for the comparison.
References
[1] Gourmelon, N.; Seehaus, T.; Braun, M.; Maier, A.; and Christlein, V.: Calving Fronts and Where to Find Them: A Benchmark Dataset and Methodology for Automatic Glacier Calving Front Extraction from SAR Imagery, Earth Syst. Sci. Data Discuss. [preprint]. 2022, https://doi.org/10.5194/essd-2022-139, in review.
[2] A. Davari, C. Baller, T. Seehaus, M. Braun, A. Maier and V. Christlein, “Pixelwise Distance Regression for Glacier Calving Front Detection and Segmentation,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-10, 2022, Art no. 5224610, doi: 10.1109/TGRS.2022.3158591.
[3] A. Davari et al., “On Mathews Correlation Coefficient and Improved Distance Map Loss for Automatic Glacier Calving Front Segmentation in SAR Imagery,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-12, 2022, Art no. 5213212, doi: 10.1109/TGRS.2021.3115883.