Skip to content
Preprint

Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos

Sep 2026 · 0 citations · 62 references
Computer Science

TL;DR

It is found that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and the faithful reproduction of individual animal locomotion is identified as an open challenge for the community.

Abstract

We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/.

View source

Similar papers

Preprint Feb 2026

KungfuAthleteBot: Learning High-Dynamic Motion from Videos with Unified Robust Recovery

Learning highly dynamic motions from video provides a scalable alternative to expensive data collection for humanoid robots, yet it is challenged by height floating, missing kinematic and force information, and limited robustness. We present KungfuAthleteBot (KAB), a unified framework that systematically addresses thes...

Zhong-Xiang Lei, Lu-Lu Cao, Xu-Yang Wang et al. · 5 citations
Preprint Sep 2026

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical...

Zihan Wang, Zhen Wu, Pieter Abbeel et al. · 3 citations · ⚡1
Preprint Oct 2026

Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations

This work presents an end-to-end framework for converting simulation-grounded videos into physically executable dexterous robot demonstrations and demonstrates diverse rigid and articulated manipulation tasks in simulation and successful open-loop transfer of the optimized trajectories to a real dexterous robot.

Xiao-Wen Qiu, Yian Wang, Hao-Xiang You et al. · 0 citations
Open access Sep 2026

FLiTrak3D: Improved deep-learning-based 3D insect flight kinematics tracking using spatial and temporal encoding

Quantitative measurements of insect flight behaviour are essential for understanding the biome-chanics, control, and ecology of flight, yet obtaining such measurements under free-flight conditions remains challenging. Small body size, rapid wing motion, visual symmetry, and frequent occlusions complicate three-dimensio...

Antoine Cribellier, A.-J. Buchner · 0 citations
Review Oct 2026

AQUAACT: A Joystick-Level Underwater Teleoperation Dataset and Closed-Loop Benchmark

Underwater robots support a growing range of work, from inspecting offshore structures and pipelines to surveying reefs and accompanying divers. Most of these tasks, however, are still flown by hand by a pilot, which raises the question of whether they can be carried out autonomously instead. On land, vision-language-a...

Zhen-Qi Wu, T. Dong, Aditya Penumarti et al. · 0 citations
Sep 2026

TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos

TrackFish3D is presented, a geometry-driven self-supervised framework for dense multi-camera 3D tracking of schooling fish, which introduces a self-supervised contrastive objective that separates co-visible individuals in the embedding space, together with a temporal predictor that preserves identities and bridges shor...

P. Phurtivilai, Zhiyang Dou, Yi-Fan Wu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.