We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation.
- Authors
Zhang, Sha; Deng, Jiajun; Bai, Lei; Li, Houqiang; Ouyang, Wanli; Zhang, Yanyong
- Abstract
We present a hybrid-view-based knowledge distillation framework, termed HVDistill, to guide the feature learning of a point cloud neural network with a pre-trained image network in an unsupervised manner. By exploiting the geometric relationship between RGB cameras and LiDAR sensors, the correspondence between the two modalities based on both image-plane view and bird-eye view can be established, which facilitates representation learning. Specifically, the image-plane correspondences can be simply obtained by projecting the point clouds, while the bird-eye-view correspondences can be achieved by lifting pixels to the 3D space with the predicted depths under the supervision of projected point clouds. The image teacher networks provide rich semantics from the image-plane view and meanwhile acquire geometric information from the bird-eye view. Indeed, image features from the two views naturally complement each other and together can ameliorate the learned feature representation of the point cloud student networks. Moreover, with a self-supervised pre-trained 2D network, HVDistill requires neither 2D nor 3D annotations. We pre-train our model on nuScenes dataset and transfer it to several downstream tasks on nuScenes, SemanticKITTI, and KITTI datasets for evaluation. Extensive experimental results show that our method achieves consistent improvements over the baseline trained from scratch and significantly outperforms the existing schemes. The source code is available at https://github.com/zhangsha1024/HVDistill.
- Subjects
POINT cloud; KNOWLEDGE transfer; DISTILLATION; EYE tracking; SOURCE code; PERVAPORATION
- Publication
International Journal of Computer Vision, 2024, Vol 132, Issue 7, p2585
- ISSN
0920-5691
- Publication type
Article
- DOI
10.1007/s11263-023-01981-w