MolmoMotion Explained: Open-Source 3D Motion Forecasting for Robotics

MolmoMotion is a 4-billion-parameter open-source AI model released June 17, 2026, by the Allen Institute for AI (Ai2) that forecasts where 3D points on objects will move in the future based on video frames, text action descriptions, and query point locations. Given an RGB video observation, a set of 3D points on an object, and a plain-language instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will travel over the next few seconds in real-world 3D space. The model is built on the Molmo 2 vision-language backbone and comes with 1.16 million action-annotated training videos, a benchmark dataset, and Apache 2.0 open-source weights.

The core problem MolmoMotion solves is that robots need to predict object motion to plan safe, accurate manipulation. Previous approaches use video models that predict pixel-level frames—useful for humans to watch, but not useful for a robot arm that needs to know where a specific point on an object will be in 3D world coordinates. MolmoMotion predicts trajectories directly in metric 3D space, which robots can use for trajectory planning. This single capability cut robot task failure rates by more than 20 percentage points on standard benchmarks and sped up learning 6× faster than baseline vision models.

What Is MolmoMotion?

MolmoMotion is a language-guided 3D motion forecasting model that takes video frames, query points on objects, and text action descriptions as input and outputs predicted 3D point trajectories in world coordinates. The model was developed by researchers at the Allen Institute for AI (Ai2) and released as open-source on June 17, 2026. It is not an API service—it is open weights that you download, install, and run on your own hardware or cloud infrastructure. The model is licensed under Apache 2.0, which permits commercial use, modification, and distribution as long as you include the license and attribution.

• 4-billion parameter model built on the Molmo 2 vision-language backbone, fine-tuned for 3D trajectory prediction.
• Predicts 3D point trajectories in metric world coordinates (not pixel-level video frames) using autoregressive or flow-matching variants.
• Open-source weights, code, and supporting datasets released under Apache 2.0. No subscription required.
• Trained on 1.16 million action-annotated videos from seven datasets: EgoDex, YouTube VIS, HD-EPIC, Xperience, MolmoSpaces, DROID, and Stereo4D.
• Comes with PointMotionBench, a human-validated evaluation benchmark containing 2,700 video clips across 111 object categories and 61 motion types.

How MolmoMotion Works and What It Outputs

MolmoMotion takes three inputs and outputs a 3D point trajectory. Understanding this input/output structure is critical for knowing where it fits in your robotics pipeline.

• Input 1: RGB video history (typically 3 or 1 frames at 15 fps). This gives the model context about the scene, lighting, and current object state.
• Input 2: 2D query points marked on the object in the video frame. You specify 8 points on the object you want to track (e.g., the corners of a gripper target, the center of mass).
• Input 3: Natural language action instruction, like "Pick up the mug" or "Slide the block to the right." The model reads this text to condition its prediction.
• Output: 3D point trajectories in world coordinates over 30 future frames (approximately 2 seconds at 15 fps). Each query point gets its own predicted path.
• Two variants available: Autoregressive (predicts coordinates as quantized text, faster) and Flow-Matching (predicts in continuous 3D space, smoother).

Benchmarks and Performance Numbers

MolmoMotion was evaluated on three benchmarks: a simulated robot manipulation benchmark, PointMotionBench (the new 3D motion benchmark), and real-robot transfer tests. Here are the specific numbers from the arxiv paper:

• PointMotionBench 3D motion prediction (ADE = Average Displacement Error in meters): HOT3D dataset achieved 0.109m error, WorldTrack 0.143m, DAVIS 1.227m. Lower is better.
• Robot manipulation success rate on simulated pick-and-place tasks: MolmoMotion-initialized policy reached 76.3% success vs. 56.0% baseline (20.3 percentage point improvement).
• Learning efficiency on MolmoSpaces benchmark: MolmoMotion reached 51% success at 10,000 training steps vs. 19% for the Molmo 2 baseline.
• Real-robot transfer (DROID dataset): MolmoMotion-initialized policies reached equivalent error levels 6× faster than baseline vision-only models.
• Video generation quality: When used to guide trajectory-conditioned video synthesis, MolmoMotion achieved temporal consistency of 0.968 and motion smoothness of 0.990, demonstrating physical plausibility in trajectory-guided generation.

Pricing and Cost Model

MolmoMotion is open-source and free to download. There is no API pricing, subscription fee, or licensing cost. You pay only for the hardware to run it.

• Model weights and code: $0. Apache 2.0 open-source.
• Training data (MolmoMotion-1M): $0. Released openly on HuggingFace.
• Self-hosted inference hardware: GPU with at least 8GB VRAM for the 4B model (e.g., NVIDIA A100, H100, or consumer RTX 4090). Estimated cloud cost: $0.50–2.00 per prediction depending on batch size and cloud provider.
• No per-token pricing. You pay once for the inference compute.
• Fine-tuning on your robots: $0 in software, but requires GPU time (1–5 hours on a single A100 for basic fine-tuning).

Where MolmoMotion Excels: Real Use Cases

MolmoMotion is strongest in scenarios where robots need to predict object motion and interact with diverse objects. Here are concrete applications:

• Robot pick-and-place with diverse objects: Predicting where objects will be when the gripper arrives, improving success on novel objects the robot has never seen. Tested on MolmoSpaces (100+ different objects).
• In-hand object reorientation: Predicting how an object will move when rotated or grasped, useful for dexterous manipulation with multi-finger hands.
• Egocentric manipulation prediction: Forecasting hand-object interaction from first-person video, useful for learning from human demonstration data.
• Trajectory-conditioned video generation: Using predicted motion as guidance for AI video models, producing more physically plausible motion in generated videos.
• Imitation learning initialization: Using MolmoMotion as a pretraining task before fine-tuning on downstream manipulation policies, dramatically faster convergence than random initialization.
• Outdoor dynamic scene prediction: Tested on real-world videos including people, vehicles, and weather. Generalizes beyond indoor manipulation.

Honest Limitations and When It Falls Short

MolmoMotion is strong for 3D motion prediction on diverse objects, but it has real constraints you should know before committing to it:

• 8-point-per-object limit: The model predicts trajectories for at most 8 points per object. This is sufficient for gripper planning but insufficient to capture dense surface motion or articulated finger poses. Complex deformable objects (cloth, liquids, hands) are harder to forecast with only 8 points.
• Fine-tuning required for high performance: The 76.3% benchmark result is impressive, but production success usually requires fine-tuning on 100–500 real robot videos from your specific setup. This is achievable but takes engineering effort.
• Limited to 30-frame predictions (2 seconds): The model predicts 30 frames (at 15 fps), which is roughly 2 seconds into the future. Long-horizon planning (10+ seconds) requires chaining multiple predictions or using a different approach.
• No closed-loop feedback: The model predicts based on the initial video frame. If the scene changes unexpectedly, the prediction becomes obsolete. It does not replan in response to actual motion.
• Requires clean query point annotations: You must label which 3D points on the object you want to track. Automating this labeling is possible but adds complexity.
• Generalization to new gripper types: Transfer to a gripper shape the model has never seen is possible but degrades performance. Fine-tuning on your gripper improves generalization significantly.

How MolmoMotion Compares to Alternatives

Several classes of models compete for the robotics prediction task. Here is how MolmoMotion stacks up:

• vs. Diffusion Policy: Diffusion Policy achieves strong single-task success rates on manipulation benchmarks. Trade-off: Diffusion is task-specific (train one policy per task), MolmoMotion is a general motion prior you can layer onto any downstream policy. Diffusion requires task-specific training, MolmoMotion provides broader object generalization.
• vs. Vision-Language-Action (VLA) models like OpenVLA or RT-2: These models predict robot actions directly (e.g., joint angles). MolmoMotion predicts object motion, not actions. You can use MolmoMotion as a front-end to VLA models for better grounding. VLA models are more end-to-end, MolmoMotion is more modular.
• vs. Generic video prediction models: Video models predict pixel-level frames (useful for visualization, not useful for robots). MolmoMotion predicts 3D world coordinates, which robots can directly use for trajectory planning. This is the key advantage.
• vs. Proprietary robot learning platforms: Google, OpenAI, and Meta have proprietary robot models trained on millions of hours of robot videos. MolmoMotion is trained on internet video plus robot data, so it generalizes broadly but may not match task-specific proprietary models on narrow domains. Trade-off: you get the weights, they don't give you theirs.

How to Access and Get Started with MolmoMotion

Getting started with MolmoMotion involves downloading the weights, installing the code, and running inference. Here is the practical path:

• 1. Download model weights: Visit the HuggingFace model page (allenai/MolmoMotion-4B-H3-F30 for 3-frame history variant, or H1-F32 for single-frame). Download takes 10–15 minutes on a typical internet connection.
• 2. Clone the GitHub repository: git clone https://github.com/allenai/molmo-motion.git. Install dependencies: pip install -e ".[viz]". Requires Python 3.11, PyTorch 2.x, and CUDA 11.8+.
• 3. Set environment variables: MOLMO_MOTION_1M_ROOT (path to training data, optional) and POINTMOTIONBENCH_ROOT (path to evaluation data, optional).
• 4. Run inference: Prepare video frames + query points + action text. Use the provided inference script to generate trajectory predictions. Output is 3D coordinates per frame.
• 5. Integrate into your robot stack: Use predicted trajectories as input to your trajectory planning or control module (e.g., MoveIt, CHOMP, or custom planners).
• 6. Fine-tune (optional but recommended): If the generic model under-performs, collect 100–500 real robot videos, label query points, and fine-tune for 1–5 hours on your GPU.

Compliance, Safety, and Production Deployment

If you are deploying MolmoMotion for commercial robotics, understand the licensing, compliance, and safety implications:

• Apache 2.0 license: Permits commercial use, modification, and redistribution. You must include the original license and attribution. No additional licensing fee required.
• Dependency compliance: MolmoMotion vendors SAM 3 (Meta) and ViPE (CC BY-NC 4.0). The CC BY-NC dependency may restrict commercial use in some jurisdictions. Review dependencies before production deployment.
• Data privacy: MolmoMotion predictions run locally on your hardware. No video is sent to Ai2 or any third party. Your robot data stays private.
• Safety considerations: MolmoMotion predicts where objects will move, but does not enforce physical constraints (collisions, dynamics, joint limits). You are responsible for validating predictions before sending commands to real robots.
• No SLA or support: This is community open-source. Ai2 publishes the model but does not offer SLAs, customer support, or guaranteed compatibility with new hardware. Support comes from the community (GitHub issues, forums).