We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Micro-object pose estimation with sim-to-real transfer learning using small dataset.
- Authors
Zhang, Dandan; Barbot, Antoine; Seichepine, Florent; Lo, Frank P.-W.; Bai, Wenjia; Yang, Guang-Zhong; Lo, Benny
- Abstract
Three-dimensional (3D) pose estimation of micro/nano-objects is essential for the implementation of automatic manipulation in micro/nano-robotic systems. However, out-of-plane pose estimation of a micro/nano-object is challenging, since the images are typically obtained in 2D using a scanning electron microscope (SEM) or an optical microscope (OM). Traditional deep learning based methods require the collection of a large amount of labeled data for model training to estimate the 3D pose of an object from a monocular image. Here we present a sim-to-real learning-to-match approach for 3D pose estimation of micro/nano-objects. Instead of collecting large training datasets, simulated data is generated to enlarge the limited experimental data obtained in practice, while the domain gap between the generated and experimental data is minimized via image translation based on a generative adversarial network (GAN) model. A learning-to-match approach is used to map the generated data and the experimental data to a low-dimensional space with the same data distribution for different pose labels, which ensures effective feature embedding. Combining the labeled data obtained from experiments and simulations, a new training dataset is constructed for robust pose estimation. The proposed method is validated with images from both SEM and OM, facilitating the development of closed-loop control of micro/nano-objects with complex shapes in micro/nano-robotic systems. High-resolution scanning tunnelling microscopy is a state-of-the-art imaging technique at the nanometer scale. This work presents a novel deep learning approach for 3D pose estimation of micro/nano-objects, particularly useful in regimes of limited experimental data.
- Subjects
SCANNING electron microscopes; GENERATIVE adversarial networks; SCANNING tunneling microscopy; POSE estimation (Computer vision); OPTICAL microscopes; DEEP learning; DATA distribution
- Publication
Communications Physics, 2022, Vol 5, Issue 1, p1
- ISSN
2399-3650
- Publication type
Article
- DOI
10.1038/s42005-022-00844-z