Скачать или смотреть Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations

by Jonas Frey, Hermann Blum, Francesco Milano, Roland Siegwart and Cesar Cadena.
A previous version of this work was awarded the NeurIPS Robot Learning Workshop Best Paper Runner-up award.

arxiv Link: https://arxiv.org/abs/2111.02156
IEEE Link: https://ieeexplore.ieee.org/document/...
Accepted for IEEE Robotics and Automation Letters (R-AL 2022)

@inproceedings{frey2022continual,
author={Jonas Frey and Hermann Blum and Francesco Milano and Roland Siegwart and Cesar Cadena},
journal={IEEE Robotics and Automation Letters (R-AL 2022)},
title={Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations},
year={2022}
}
Video by Jonas Frey

Abstract:
Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adapting the neural network to the robot's environment during deployment, without any need for external supervision. Leveraging complementary data representations, we generate a supervision signal, by probabilistically accumulating consecutive 2D semantic predictions in a volumetric 3D map. We then train the network on renderings of the accumulated semantic map, effectively resolving ambiguities and enforcing multi-view consistency through the 3D representation. In contrast to scene adaptation methods, we aim to retain the previously-learned knowledge, and therefore employ a continual learning experience replay strategy to adapt the network. Through extensive experimental evaluation, we show successful adaptation to real-world indoor scenes both on the ScanNet dataset and on in-house data recorded with an RGB-D sensor. Our method increases the segmentation accuracy on average by 9.9% compared to the fixed pre-trained neural network, while retaining knowledge from the pre-training dataset.