← All articles

Anchoring Simulation Physics To Reality - VLA Fine-Tuning

When does simulation data stop helping and start erasing real physics? Rebalancing real, augmented, and simulated data when fine-tuning a VLA on my SO-101 raised pick-and-place success from 8% to 84%, without collecting more data. This post covers a six-way ablation of real / aug / sim batch composition.

8% → 84%
Pick-and-place success
15% / 35% / 50%
Real / aug / sim mix
N = 25
Eval rollouts
Nicolas Basile · · 11 min read
embodied ai robotics
Exp. E: confident pick-and-place after the 15/35/50 mix.

Introduction

How should real, augmented, and simulated trajectories be mixed when fine-tuning a Vision-Language-Action (VLA) model on a data-scarce manipulator?

In my previous work with discrete-action environments (Minecraft), agents operated under the assumption of “code-as-action” where the environment engine guaranteed physical consistency. That paradigm exposes two critical bottlenecks when applied to high-fidelity, real-world robotics:

  1. Autoregressive Latency: Token-by-token generation is computationally prohibitive for high-frequency control loops (e.g., <10Hz).
  2. Action Fidelity: Discrete tokens fail to capture the continuous, high-dimensional manifolds of joint-space trajectories.

I attempted to bridge this gap by fine-tuning a VLA on a 1,200-trajectory corpus (50 real, 150 augmented, 1,000 sim). However, instead of generalizing, the policy developed “Synthetic Amnesia”, catastrophic forgetting where simulation artifacts overwrite real-world physical priors.

This post documents the counter-intuitive reality of VLA fine-tuning: unanchored synthetic data scales visual robustness but degrades physical fidelity. I introduce a Golden Ratio mixing strategy that anchors the model in reality, improving success rates from 8% to 84%.

Architecture: The SmolVLA Approach

To address the latency and fidelity requirements of high-frequency control, SmolVLA decouples visual-semantic reasoning from motor-control execution:

  1. Visual-Semantic Backbone: SmolVLM (SigLIP-Vision + SmolLM2-Language) as a frozen feature extractor.
  2. Intermediate Feature Tapping: To bypass the autoregressive bottleneck, activations are extracted from intermediate transformer blocks (Layer 15 of 30). This assumes that the later layers, optimized for linguistic syntax and abstract reasoning, are redundant for low-level spatial grounding.
  3. Flow-Matching Action Head: These features are projected into a lightweight MLP-Transformer that predicts continuous joint trajectories via Conditional Flow Matching (CFM).
SmolVLA Architecture
SmolVLA consumes multi-view RGB and language via SigLIP and SmolLM2, fusing them with a projected state token to predict continuous actions via a flow-matching head.

Model Selection: The Latency Bottleneck

Before settling on the final architecture, I benchmarked heavier contenders: OpenVLA (7B) [10] and quantized variants of Pi-Zero [1].

Local GPU constraints (8GB of VRAM) initially struggled to fit these larger models at acceptable fidelity, so I offloaded inference to an A100 cloud instance on Lambda Labs and developed a ROS 2 Humble [11] server to handle communication between the local robot and the cloud backbone.

The network round-trip time (serialization + inference + deserialization) pushed the control loop below the 10Hz required for stability. The robot moved in “jerky,” reactive bursts rather than smooth flows.

This constraint led me to SmolVLA [12]. At ~450M parameters, it runs comfortably on my local hardware, eliminating network complexity entirely and allowing for a tight, high-frequency control loop.

Although local inference is pragmatic for rapid experimentation, my future work will explore hybrid cloud-edge inference, leveraging methodologies from my previous inference optimization research.

Action Policy: CFM & Receding Horizon Control (RHC)

To generate smooth actions without the computational cost of traditional diffusion policies, SmolVLA uses Conditional Flow Matching (CFM) [13].

Unlike diffusion policies, which iteratively denoise an action over many inference steps, CFM learns a deterministic velocity field that transports random noise to a trajectory in a small number of integration steps. This captures the distribution of valid movements while remaining cheap enough for a high-frequency control loop.

The combination of the lightweight architecture and CFM enables Receding Horizon Control (RHC):

  • Trajectory Prediction: The action head predicts a continuous chunk of H = 50 steps (1.67s of control).

  • Asynchronous Inference: While the SO-101 arm executes the current chunk at 30Hz (33.3ms/step), the model asynchronously infers the next chunk in the background.

  • Latency Masking: Because of layer skipping, inference takes only ~100ms on an NVIDIA 3060 Ti, well within the 1.67s execution window. This allows the robot to move seamlessly without the “stutter” typical of synchronous VLM policies.

Receding-Horizon Control

A Note on Action Space: Joint Space vs. Cartesian Poses

Ideally, a robotic foundation model should predict End-Effector (EE) Poses. Predicting “where the hand should be” (Cartesian space) generalizes across embodiments better than predicting “how the motors should angle” (Joint space).

However, my early attempts at EE control hit a wall. Mapping EE predictions to joint angles requires an Inverse Kinematics (IK) solver. I experimented with and began building a differentiable Jacobian-based solver, but the VLA lacked proprioceptive grounding, frequently predicting “valid” Cartesian coordinates that were physically impossible for the arm to reach, forcing the IK solver into singularities or self-collisions.

Sholto Douglas noted in his work on orientation control that “Simplicity wins” - using Euler angles often beats elegant but fragile Quaternion math. Similarly, I found that predicting Joint Positions directly was significantly more robust for this iteration. While this locks the model to the SO-101’s specific kinematics, it guarantees that every predicted action is physically executable.

Developing a robust IK stack (integrating Pinocchio [7] with Null-space Projection [6]) and translating the existing dataset into Cartesian space is a priority for the next iteration. This will allow the model to learn “Implicit Kinematics” while maintaining the cross-embodiment benefits of EE control.

The Filament Problem: Manifold Support in Action Space

Flow matching models learn a vector field that transports a noise distribution to the target action distribution. With only 50 real-world trajectories, the data represents thin “filaments” floating in the high-dimensional action space of the policy.

  • Extrapolation Failure: If the robot is perturbed even slightly off these demonstrated filaments, the vector field becomes undefined or highly stochastic. This results in the “jittery” behavior (Exp. A) where the model fails to converge to a stable sink.
  • The Simulation Solution: Simulation data provides the necessary topological support. By filling the volume between real-world filaments, simulation allows the model to learn a continuous, smooth vector field across the entire workspace, providing the “recovery” behavior needed for robustness.

However, this volume comes at a cost. In naive mixing scenarios, the sheer quantity of simulation data (1000 vs. 50) drowned out the real-world signal. The model learned the approximate physics of the sim engines rather than the real robot’s dynamics.

Data: Mixing Real and Synthetic Trajectories

To address the “synthetic amnesia” problem, I adopted a Tri-Partitioned Strategy that treats augmented data as a distinct bridge domain.

The final dataset consisted of three distinct buckets:

  • Real Teleoperation Data (N=50): Collected on the SO-101 instructor arm setup [9] to anchor the policy in true physical dynamics.
  • Augmented Real Data (N=150): Generated via segmentation-based background replacement [2] and Kornia [3] visual transforms. This forces the vision encoder to decouple kinematic signatures from visual framing.
Example of the augmented real data. Credit: DataSocks.
  • Simulation Data (N=1000): Generated primarily in ManiSkill [15] (SAPIEN [16]) with a small fraction from MuJoCo. I matched the visual assets (block color, table texture) to the physical lab setup (Digital Twin). This provides the “volume” required for stable CFM vector fields.

Following established best practices for VLA fine-tuning, the real data collection focused on diversity rather than sheer volume. I collected N=50 high-quality “anchor” episodes, ensuring varied robot joint configurations by starting and ending trajectories in five distinct zones around the workspace.

Technical Note on Simulation: I initially invested significant time porting the SO-100 to MuJoCo; however, I ultimately pivoted to ManiSkill (SAPIEN) for the bulk of data generation. ManiSkill’s GPU-parallelized experts allowed for 10k trajectories in minutes, significantly accelerating the research pipeline.

Training: The Golden Ratio Strategy

The hypothesis was that oversampling real data prevents drift, but excessive synthetic volume without explicit anchoring would cause catastrophic forgetting of real-world physics. I conducted an ablation study to find the inflection point where simulation supports the policy without overwriting reality.

Ablation: Batch Composition Analysis

I drew inspiration from “Golden Ratio Mixing” [4], which posits that maintaining a high proportion of real-world data in minibatches is necessary to anchor the model, even when the underlying dataset is predominantly synthetic. I adopted this principle; not their literal φ ratio. The 15/35/50 optimum emerged from ablation, not from closed-form derivation.

For these benchmarks, Success is defined as grasping the object and placing it within 3cm of the target within 15 seconds, averaged over N=25 eval rollouts across held-out initial conditions (background, lighting, and object coordinates) not seen during teleop collection.

  • Baseline (zero-shot SmolVLA): ~5% success due to domain shift in my custom setup.
  • Fine-tuned variants:
Exp. Batch composition
(% real / aug / sim)		 Success Diagnosis
A 100 / 0 / 0
Pure real		 8%
CI 2–25%		 Filament collapse: undefined vector field off-demonstration.
B Uniform
Naive		 36%
CI 20–55%		 Synthetic amnesia: overfit to sim physics (perfect friction).
C 35 / 15 / 50 52%
CI 33–70%		 Rigid generalization: stable but brittle to visual noise.
D 25 / 25 / 50 64%
CI 45–80%		 Visual overfitting: brittle to lighting/viewpoint changes.
E 15 / 35 / 50
Optimal		 84%
CI 65–94%		 Optimal mix: balanced anchoring vs. topological support.
F 10 / 40 / 50 72%
CI 52–86%		 Weak anchoring: visual robustness with physical drift.

Table 1. Batch composition vs. pick-and-place success. 95% Wilson CIs. N=25 eval rollouts.

Result Chart

Fine-Tuning Strategy

Training was performed for 20,000 steps using a rented NVIDIA A100 (80GB). Starting from the lerobot/smolvla_base checkpoint, I froze the VLM backbone and fully fine-tuned the flow-matching Action Expert. This focused the gradients on the robot’s control policy rather than overwriting the semantic knowledge of the vision encoder.

Hyperparameters: AdamW optimizer (β=[0.9, 0.95], weight decay=1e-10), learning rate 1e-4 with 1,000-step warmup, batch size 64. Training was stopped at 20k steps (~4 hours) following the recommended fine-tuning schedule.

Why the 15 / 35 / 50 Split Works

Although the underlying dataset is 96% synthetic, the tri-partitioned strategy ensures that 50% of every gradient update is derived from real-world kinematics (Real + Augmented). This prevents the model from overwriting the delicate friction and contact dynamics of the real arm with simulation artifacts.

By combining diverse teleoperation data, visual augmentations, and simulation data via Weighted Stratified Sampling [14], I fine-tuned SmolVLA to execute smooth, confident motions. This approach successfully eliminated the jitter seen in earlier runs and enabled the model to generalize to new object locations.

Limitations

This study is limited to a single manipulation task (pick-and-place) on a single robot morphology (SO-101). Training used a single seed, and evaluation did not include hardware randomization (camera extrinsics, payload, or friction). I leave further ablations on simulation fidelity, augmentation composition, and the scaling laws of real-world trajectory counts to future work. Extending these findings to multi-task settings and diverse embodiments remains an open research question.

Future Research Directions

Cross-Embodiment & End-Effector (EE) Control

The current policy predicts Joint Positions (see Architecture for the IK-failure rationale). The next iteration targets End-Effector (EE) Poses via Pinocchio [7] and null-space IK [6], so the VLA predicts “what the hand should do” rather than “how the motors should move.”

  • Open Research Question: Can a VLA learn “Implicit Kinematics” from Cartesian supervision? Switching to EE prediction hides joint-limit constraints. Will the VLA learn to output only those poses that are solvable, or will it hallucinate “valid” 3D coordinates that force the IK solver into singularities or self-collisions?
End-effector poses versus joint-space actions

Scalability and Multi-Task Generalization

SmolVLA (450M parameters) is sufficient for 30Hz RHC on a 3060 Ti, but it likely lacks the world-knowledge priors required for zero-shot generalization to novel objects. Scaling to larger backbones (7B+) and training on more heterogeneous task data (e.g., OpenVLA [10], Pi-Zero [1]) is the natural progression for this research.

References

  1. Pi-Zero — Physical Intelligence. physicalintelligence.company
  2. Augmented Datasocks — Pravsels. github.com/pravsels/augmented_datasocks
  3. Kornia — Riba et al. kornia.readthedocs.io
  4. Golden Ratio Mixing — He et al. arXiv:2502.18049
  5. MimicGen — Mandlekar et al. mimicgen.github.io
  6. Robot Control III — Yan-Bin Jia. Iowa State notes (PDF)
  7. Pinocchio — Carpentier et al. github.com/stack-of-tasks/pinocchio
  8. MoveIt — Coleman et al. moveit.ai
  9. SO-ARM101 — The Robot Studio. github.com/TheRobotStudio/SO-ARM100
  10. OpenVLA — Kim et al. openvla.github.io
  11. ROS 2 Humble — Open Source Robotics Foundation. docs.ros.org/humble
  12. SmolVLA — Hugging Face. lerobot/smolvla_base, fine-tune docs
  13. Conditional Flow Matching — Lipman et al. arXiv:2210.02747
  14. RT-1 — Brohan et al. arXiv:2206.02077
  15. ManiSkill — Gu et al. github.com/haosulab/ManiSkill
  16. SAPIEN — Xiang et al. sapien.ucsd.edu