Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Images
TL;DR
An annotation pipeline consisting of a diffusion model to lift 2D hand keypoint sequences to 4D hand motion and a diffusion loss to account for the multimodality in hand motion distribution is designed.
Abstract
We tackle the problem of forecasting bimanual 3D hand motion and articulation from a single image in everyday settings. To address the lack of 3D hand annotations in diverse settings, we design an annotation pipeline consisting of a diffusion model to lift 2D hand keypoint sequences to 4D hand motion. For the forecasting model, we adopt a diffusion loss to account for the multimodality in hand motion distribution. Extensive experiments on 6 datasets show the benefits of training with our imputed labels (14% improvement) and the effectiveness of our lifting (45% better)&forecasting (16.4% gain) models, over the best baselines, especially in zero-shot generalization to everyday images.
