Vision-Language-Action Model

A vision-language-action model (VLA) is a robot control policy built by fine-tuning a vision-language model to output actions instead of, or in addition to, text. The action space is represented as tokens in the same vocabulary the model already uses for language, so a single transformer pretrained on internet-scale image and text data can be adapted to map camera observations and an instruction directly to motor commands. Google DeepMind introduced the term with RT-2 in 2023 (Brohan et al., arXiv:2307.15818), building on the earlier RT-1 robot transformer and on vision-language models such as PaLI-X and PaLM-E.

Because a VLA inherits its backbone from a model trained on broad web data, it can transfer some of that model's semantic knowledge to robotic tasks it was never explicitly trained on. This includes recognizing novel objects, following instructions phrased in unfamiliar ways, and chaining simple steps of reasoning before acting, such as picking the object best suited to a described purpose. OpenVLA (Kim et al., 2024, arXiv:2406.09246) showed this recipe could be reproduced openly: fine-tuning a 7-billion-parameter open backbone on about a million robot demonstrations from the Open X-Embodiment dataset, and matching or beating closed models many times its size.

VLAs are now the dominant framing for generalist robot manipulation research. Variants explore diffusion-based action decoders, multi-embodiment training across different robot bodies, and larger action vocabularies for finer motor control. A 2025 survey (arXiv:2505.04769) tracks the resulting spread of architectures, training recipes, and evaluation benchmarks across the field.