It is found that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and the faithful reproduction of individual animal locomotion is identified as an open challenge for the community.
Abstract
We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/.
Learning highly dynamic motions from video provides a scalable alternative to expensive data collection for humanoid robots, yet it is challenged by height floating, missing kinematic and force information, and limited robustness. We present KungfuAthleteBot (KAB), a unified framework that systematically addresses thes...
Zhong-Xiang Lei, Lu-Lu Cao, Xu-Yang Wang et al.· 5 citations
Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical...
This work presents an end-to-end framework for converting simulation-grounded videos into physically executable dexterous robot demonstrations and demonstrates diverse rigid and articulated manipulation tasks in simulation and successful open-loop transfer of the optimized trajectories to a real dexterous robot.
Xiao-Wen Qiu, Yian Wang, Hao-Xiang You et al.· 0 citations
Quantitative measurements of insect flight behaviour are essential for understanding the biome-chanics, control, and ecology of flight, yet obtaining such measurements under free-flight conditions remains challenging. Small body size, rapid wing motion, visual symmetry, and frequent occlusions complicate three-dimensio...
Antoine Cribellier, A.-J. Buchner· bioRxiv· 0 citations
Underwater robots support a growing range of work, from inspecting offshore structures and pipelines to surveying reefs and accompanying divers. Most of these tasks, however, are still flown by hand by a pilot, which raises the question of whether they can be carried out autonomously instead. On land, vision-language-a...
Zhen-Qi Wu, T. Dong, Aditya Penumarti et al.· 0 citations
TrackFish3D is presented, a geometry-driven self-supervised framework for dense multi-camera 3D tracking of schooling fish, which introduces a self-supervised contrastive objective that separates co-visible individuals in the embedding space, together with a temporal predictor that preserves identities and bridges shor...
P. Phurtivilai, Zhiyang Dou, Yi-Fan Wu et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.