dorsal/arxiv
View SchemaMolmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
| Authors | Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, Ranjay Krishna |
|---|---|
| Categories | |
| ArXiv ID | 2601.10611vv1 |
| URL | https://arxiv.org/abs/2601.10611 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).
{
"annotation_id": "2bda7130-e3e8-4eb4-b131-4c335ea68611",
"date_created": "2026-02-17T05:53:26.454000Z",
"date_modified": "2026-02-17T05:53:26.454000Z",
"file_hash": "d9d0448e5313a12c7a690aed6ed282b9aeec69e1e6e6e27a34c8c54b126b71b9",
"private": false,
"record": {
"abstract": "Today\u0027s strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q\u0026A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J\u0026F on video tracking).",
"arxiv_id": "2601.10611",
"authors": [
"Christopher Clark",
"Jieyu Zhang",
"Zixian Ma",
"Jae Sung Park",
"Mohammadreza Salehi",
"Rohun Tripathi",
"Sangho Lee",
"Zhongzheng Ren",
"Chris Dongjoo Kim",
"Yinuo Yang",
"Vincent Shao",
"Yue Yang",
"Weikai Huang",
"Ziqi Gao",
"Taira Anderson",
"Jianrui Zhang",
"Jitesh Jain",
"George Stoica",
"Winson Han",
"Ali Farhadi",
"Ranjay Krishna"
],
"categories": [
"cs.CV",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding",
"url": "https://arxiv.org/abs/2601.10611",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "f23f91f6-c11e-456c-899f-104bb057216a",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}