Finetuning Vision-Language Models for Grid-Based Navigation

Abstract

We present a method for teaching vision-language models (VLMs) to perform spatial navigation in grid-based environments using structured supervision derived from distance fields. Our approach combines binary space partitioning for maze generation, Dijkstra-based distance field computation, and A* pathfinding to create supervised finetuning (SFT) datasets. We finetune two 4B parameter models—Gemma-3-4B-IT and Qwen-3-VL-4B-Instruct—and evaluate them against larger thinking models including GLM-4.6V (106B) and GPT-5. Our best finetuned model achieves 9/10 maze completions, matching or exceeding the performance of models 25 × larger while fitting on a single GPU. These results suggest that two dimensional navigation and planning can be achieved with relatively small parameter counts, and hence task-specific finetuning of small VLMs can yield deployable navigation systems suitable for resource-constrained environments such as edge robotics platforms.