How should real, augmented, and simulated trajectories be mixed when fine-tuning a Vision-Language-Action (VLA) model on a data-scarce manipulator?

In my previous work with discrete-action environments (Minecraft), agents operated under the assumption of “code-as-action” where the environment engine guaranteed physical consistency. That paradigm exposes two critical bottlenecks when applied to high-fidelity, real-world robotics:

1. Autoregressive Latency: Token-by-token generation is computationally prohibitive for high-frequency control loops (e.g., <10Hz).
2. Action Fidelity: Discrete tokens fail to capture the continuous, high-dimensional manifolds of joint-space trajectories.

I attempted to bridge this gap by fine-tuning a VLA on a 1,200-trajectory corpus (50 real, 150 augmented, 1,000 sim). However, instead of generalizing, the policy developed “Synthetic Amnesia”, catastrophic forgetting where simulation artifacts overwrite real-world physical priors.

This post documents the counter-intuitive reality of VLA fine-tuning: unanchored synthetic data scales visual robustness but degrades physical fidelity. I introduce a Golden Ratio mixing strategy that anchors the model in reality, improving success rates from 8% to 84%.

Architecture: The SmolVLA Approach

To address the latency and fidelity requirements of high-frequency control, SmolVLA decouples visual-semantic reasoning from motor-control execution:

1. Visual-Semantic Backbone: SmolVLM (SigLIP-Vision + SmolLM2-Language) as a frozen feature extractor.
2. Intermediate Feature Tapping: To bypass the autoregressive bottleneck, activations are extracted from intermediate transformer blocks (Layer 15 of 30). This assumes that the later layers, optimized for linguistic syntax and abstract reasoning, are redundant for low-level spatial grounding.
3. Flow-Matching Action Head: These features are projected into a lightweight MLP-Transformer that predicts continuous joint trajectories via Conditional Flow Matching (CFM).

Model Selection: The Latency Bottleneck

Before settling on the final architecture, I benchmarked heavier contenders: OpenVLA (7B) ^[10] and quantized variants of Pi-Zero ^[1].

Local GPU constraints (8GB of VRAM) initially struggled to fit these larger models at acceptable fidelity, so I offloaded inference to an A100 cloud instance on Lambda Labs and developed a ROS 2 Humble ^[11] server to handle communication between the local robot and the cloud backbone.

The network round-trip time (serialization + inference + deserialization) pushed the control loop below the 10Hz required for stability. The robot moved in “jerky,” reactive bursts rather than smooth flows.

This constraint led me to SmolVLA ^[12]. At ~450M parameters, it runs comfortably on my local hardware, eliminating network complexity entirely and allowing for a tight, high-frequency control loop.

Although local inference is pragmatic for rapid experimentation, my future work will explore hybrid cloud-edge inference, leveraging methodologies from my previous inference optimization research.

Action Policy: CFM & Receding Horizon Control (RHC)

To generate smooth actions without the computational cost of traditional diffusion policies, SmolVLA uses Conditional Flow Matching (CFM) ^[13].

Unlike diffusion policies, which iteratively denoise an action over many inference steps, CFM learns a deterministic velocity field that transports random noise to a trajectory in a small number of integration steps. This captures the distribution of valid movements while remaining cheap enough for a high-frequency control loop.

The combination of the lightweight architecture and CFM enables Receding Horizon Control (RHC):

• Trajectory Prediction: The action head predicts a continuous chunk of H = 50 steps (1.67s of control).

• Asynchronous Inference: While the SO-101 arm executes the current chunk at 30Hz (33.3ms/step), the model asynchronously infers the next chunk in the background.

• Latency Masking: Because of layer skipping, inference takes only ~100ms on an NVIDIA 3060 Ti, well within the 1.67s execution window. This allows the robot to move seamlessly without the “stutter” typical of synchronous VLM policies.

A Note on Action Space: Joint Space vs. Cartesian Poses

Ideally, a robotic foundation model should predict End-Effector (EE) Poses. Predicting “where the hand should be” (Cartesian space) generalizes across embodiments better than predicting “how the motors should angle” (Joint space).

However, my early attempts at EE control hit a wall. Mapping EE predictions to joint angles requires an Inverse Kinematics (IK) solver. I experimented with and began building a differentiable Jacobian-based solver, but the VLA lacked proprioceptive grounding, frequently predicting “valid” Cartesian coordinates that were physically impossible for the arm to reach, forcing the IK solver into singularities or self-collisions.

Sholto Douglas noted in his work on orientation control that “Simplicity wins” - using Euler angles often beats elegant but fragile Quaternion math. Similarly, I found that predicting Joint Positions directly was significantly more robust for this iteration. While this locks the model to the SO-101’s specific kinematics, it guarantees that every predicted action is physically executable.

Developing a robust IK stack (integrating Pinocchio ^[7] with Null-space Projection ^[6]) and translating the existing dataset into Cartesian space is a priority for the next iteration. This will allow the model to learn “Implicit Kinematics” while maintaining the cross-embodiment benefits of EE control.

The Filament Problem: Manifold Support in Action Space

Flow matching models learn a vector field that transports a noise distribution to the target action distribution. With only 50 real-world trajectories, the data represents thin “filaments” floating in the high-dimensional action space of the policy.

• Extrapolation Failure: If the robot is perturbed even slightly off these demonstrated filaments, the vector field becomes undefined or highly stochastic. This results in the “jittery” behavior (Exp. A) where the model fails to converge to a stable sink.
• The Simulation Solution: Simulation data provides the necessary topological support. By filling the volume between real-world filaments, simulation allows the model to learn a continuous, smooth vector field across the entire workspace, providing the “recovery” behavior needed for robustness.

However, this volume comes at a cost. In naive mixing scenarios, the sheer quantity of simulation data (1000 vs. 50) drowned out the real-world signal. The model learned the approximate physics of the sim engines rather than the real robot’s dynamics.

Data: Mixing Real and Synthetic Trajectories

To address the “synthetic amnesia” problem, I adopted a Tri-Partitioned Strategy that treats augmented data as a distinct bridge domain.

The final dataset consisted of three distinct buckets:

• Real Teleoperation Data (N=50): Collected on the SO-101 instructor arm setup ^[9] to anchor the policy in true physical dynamics.
• Augmented Real Data (N=150): Generated via segmentation-based background replacement ^[2] and Kornia ^[3] visual transforms. This forces the vision encoder to decouple kinematic signatures from visual framing.

• Simulation Data (N=1000): Generated primarily in ManiSkill ^[15] (SAPIEN ^[16]) with a small fraction from MuJoCo. I matched the visual assets (block color, table texture) to the physical lab setup (Digital Twin). This provides the “volume” required for stable CFM vector fields.

Following established best practices for VLA fine-tuning, the real data collection focused on diversity rather than sheer volume. I collected N=50 high-quality “anchor” episodes, ensuring varied robot joint configurations by starting and ending trajectories in five distinct zones around the workspace.

Technical Note on Simulation: I initially invested significant time porting the SO-100 to MuJoCo; however, I ultimately pivoted to ManiSkill (SAPIEN) for the bulk of data generation. ManiSkill’s GPU-parallelized experts allowed for 10k trajectories in minutes, significantly accelerating the research pipeline.

Training: The Golden Ratio Strategy

The hypothesis was that oversampling real data prevents drift, but excessive synthetic volume without explicit anchoring would cause catastrophic forgetting of real-world physics. I conducted an ablation study to find the inflection point where simulation supports the policy without overwriting reality.

Ablation: Batch Composition Analysis

I drew inspiration from “Golden Ratio Mixing” ^[4], which posits that maintaining a high proportion of real-world data in minibatches is necessary to anchor the model, even when the underlying dataset is predominantly synthetic. I adopted this principle; not their literal φ ratio. The 15/35/50 optimum emerged from ablation, not from closed-form derivation.

For these benchmarks, Success is defined as grasping the object and placing it within 3cm of the target within 15 seconds, averaged over N=25 eval rollouts across held-out initial conditions (background, lighting, and object coordinates) not seen during teleop collection.

• Baseline (zero-shot SmolVLA): ~5% success due to domain shift in my custom setup.
• Fine-tuned variants:

Fine-Tuning Strategy

Training was performed for 20,000 steps using a rented NVIDIA A100 (80GB). Starting from the lerobot/smolvla_base checkpoint, I froze the VLM backbone and fully fine-tuned the flow-matching Action Expert. This focused the gradients on the robot’s control policy rather than overwriting the semantic knowledge of the vision encoder.

Hyperparameters: AdamW optimizer (β=[0.9, 0.95], weight decay=1e-10), learning rate 1e-4 with 1,000-step warmup, batch size 64. Training was stopped at 20k steps (~4 hours) following the recommended fine-tuning schedule.

Why the 15 / 35 / 50 Split Works

Although the underlying dataset is 96% synthetic, the tri-partitioned strategy ensures that 50% of every gradient update is derived from real-world kinematics (Real + Augmented). This prevents the model from overwriting the delicate friction and contact dynamics of the real arm with simulation artifacts.

By combining diverse teleoperation data, visual augmentations, and simulation data via Weighted Stratified Sampling ^[14], I fine-tuned SmolVLA to execute smooth, confident motions. This approach successfully eliminated the jitter seen in earlier runs and enabled the model to generalize to new object locations.

Limitations

This study is limited to a single manipulation task (pick-and-place) on a single robot morphology (SO-101). Training used a single seed, and evaluation did not include hardware randomization (camera extrinsics, payload, or friction). I leave further ablations on simulation fidelity, augmentation composition, and the scaling laws of real-world trajectory counts to future work. Extending these findings to multi-task settings and diverse embodiments remains an open research question.

Future Research Directions

Cross-Embodiment & End-Effector (EE) Control

The current policy predicts Joint Positions (see Architecture for the IK-failure rationale). The next iteration targets End-Effector (EE) Poses via Pinocchio ^[7] and null-space IK ^[6], so the VLA predicts “what the hand should do” rather than “how the motors should move.”

• Open Research Question: Can a VLA learn “Implicit Kinematics” from Cartesian supervision? Switching to EE prediction hides joint-limit constraints. Will the VLA learn to output only those poses that are solvable, or will it hallucinate “valid” 3D coordinates that force the IK solver into singularities or self-collisions?

Scalability and Multi-Task Generalization

SmolVLA (450M parameters) is sufficient for 30Hz RHC on a 3060 Ti, but it likely lacks the world-knowledge priors required for zero-shot generalization to novel objects. Scaling to larger backbones (7B+) and training on more heterogeneous task data (e.g., OpenVLA ^[10], Pi-Zero ^[1]) is the natural progression for this research.

References

1. Pi-Zero — Physical Intelligence. physicalintelligence.company
2. Augmented Datasocks — Pravsels. github.com/pravsels/augmented_datasocks
3. Kornia — Riba et al. kornia.readthedocs.io
4. Golden Ratio Mixing — He et al. arXiv:2502.18049
5. MimicGen — Mandlekar et al. mimicgen.github.io
6. Robot Control III — Yan-Bin Jia. Iowa State notes (PDF)
7. Pinocchio — Carpentier et al. github.com/stack-of-tasks/pinocchio
8. MoveIt — Coleman et al. moveit.ai
9. SO-ARM101 — The Robot Studio. github.com/TheRobotStudio/SO-ARM100
10. OpenVLA — Kim et al. openvla.github.io
11. ROS 2 Humble — Open Source Robotics Foundation. docs.ros.org/humble
12. SmolVLA — Hugging Face. lerobot/smolvla_base, fine-tune docs
13. Conditional Flow Matching — Lipman et al. arXiv:2210.02747
14. RT-1 — Brohan et al. arXiv:2206.02077
15. ManiSkill — Gu et al. github.com/haosulab/ManiSkill
16. SAPIEN — Xiang et al. sapien.ucsd.edu

NVIDIA’s Voyager laid the foundation for autonomous curriculum learning and tool‑use, but long-horizon, multi-stage objectives still tend to break down due to forgetting, shallow planning, and brittle tool selection.

My goal in this project was to push those boundaries by:

• optimizing long-horizon reliability via memory, planning persistence, and tool robustness.
• and automating improvement of the agent’s scaffolding via verifier-driven optimization (DSPy ^[1]).

On a suite of multi-stage objectives, these changes improved success rates an average of ~20% while reducing prompting iterations needed by >40% in several cases (relative to NVIDIA’s baseline Voyager framework ^[9]).

• Evaluation: 5 random seeds × 4 episodes per seed = 20 episodes/task (no initial inventory).
• Prompting iterations: number of planner/tool-selector regeneration attempts, capped at 50 as in Voyager.
• Success criteria: Task dependent - typically an inventory check, or structure built.

What I built

This project improves upon the Voyager framework with:

1) Structured memory

• Compressed episodic memory: key events, failures, and environmental discoveries.
• Task memory: current high-level objective and sub-tasks, with progress markers.
• Skill/tool memory: what tools exist and when they’re useful.

This makes the agent’s behavior more consistent across long rollouts and reduces repeated mistakes. This also introduces an optimization point, allowing DSPy to learn optimal selective memory strategies.

This mirrors a broader pattern in language-agent research: maintaining an episodic trace, compressing it into higher-level reflections, and retrieving it to steer future plans. ^[2]

2) Hierarchical planning

The agent plans at multiple time scales:

• High-level plan: stages/subgoals (“collect wood → craft pickaxe → collect stone → build house”).
• Mid-level steps: concrete actions per stage (tool calls / environment API usage [i.e walk, mine, craft]).
• Execution loop: act → observe → update memory → re-plan only when needed.

The key change is plan persistence: the agent doesn’t re-derive its intent every step, which reduces thrashing.

3) Tool use that actually holds up over long horizons

• explicit tool “contracts” (inputs/outputs + failure handling),
• validation of tool results before moving on,
• recovery strategies when a tool call fails or returns unexpected state.

Treating actions/tools as first-class outputs with validation aligns with the ReAct view of interleaving reasoning with environment actions to reduce cascading errors. ^[3]

Prompt + policy improvement via DSPy

A major part of the gains came from automating improvement upon the agentic scaffolding itself.

VLMs as reward models

For long-horizon tasks, sparse environment rewards are often unusable; instead, I use a vision-language evaluator to score progress from observable evidence (images + trace). This mirrors a growing line of work that treats pretrained VLMs as zero-shot reward models for language-conditioned goals ^[4][5], turning “is the goal satisfied?” into a learned scoring function over visual states.

Concretely, the evaluator consumes (goal, plan, tool calls, observations) and outputs a scalar score plus failure tags (e.g., missed grasp, wrong inventory precondition). Those scores become the optimization target for improving the agent’s scaffolding.

DSPy turns scaffolding design into optimization

Rather than hand-tuning prompts, I treat the planner/tool-selector/memory-updater as an optimizable program and use DSPy-style compilation/teleprompting to search for module settings that maximize verifier score. This is closer to black-box policy search over scaffolding than weight-updating RL, useful when you want rapid iteration without finetuning.

DSPy frames prompt-and-module design as an optimizable program (compile/teleprompt), replacing ad-hoc prompt tinkering with systematic search over module behaviors.

This turned “prompt engineering” into something closer to search + optimization, using an internal (synthetic) reward function that is grounded in environmental evidence. As mentioned earlier, the majority of performance gains came from this iterative, evolution-like approach.

Why this matters

Long-horizon embodied autonomy can’t be solved solely by “a bigger model” and “more data”. In these environments, moving beyond impressive demos and into reliable behavior requires:

• memory that persists and stays relevant,
• plans that evolve with the environment,
• tool use that is robust to errors and infinite loops,
• evaluation signals that are aligned with success.

Related work: Odyssey (open-world skills)

Recent work like Odyssey ^[6][7] extends the Voyager line by expanding what the agent can do: it equips Minecraft agents with a large open-world skill library (primitive + compositional skills), augments the base model with domain-specific Minecraft knowledge via a fine-tuned LLaMA-3 variant trained on a large Minecraft-Wiki Q&A corpus, and introduces a capability-oriented benchmark spanning long-term planning, dynamic-immediate planning, and autonomous exploration tasks.

This project is intentionally orthogonal. Rather than primarily increasing skill/action coverage or adding domain knowledge through finetuning, I focus on making long-horizon behavior reliably hold up given a set of tools, hardening execution and optimizing the agent’s scaffolding for consistency over long horizons.

References

1. DSPy — Khattab et al. arXiv:2310.03714
2. Reflexion — Shinn et al. arXiv:2303.11366
3. ReAct — Yao et al. arXiv:2210.03629
4. VLMs as Zero-Shot Reward Models — Rocamonde et al., ICLR 2024. OpenReview
5. VLMs as a Source of Rewards — Baumli et al. arXiv:2312.09187
6. Odyssey — Liu et al. arXiv:2407.15325
7. Odyssey (code) — github.com/zju-vipa/Odyssey
8. Voyager — Wang et al. arXiv:2305.16291
9. Voyager (code) — github.com/MineDojo/Voyager