dorsal/arxiv
View SchemaSpeak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
| Authors | Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, Xiaoyu Shen |
|---|---|
| Categories | |
| ArXiv ID | 2601.06843vv1 |
| URL | https://arxiv.org/abs/2601.06843 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception-generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input-output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency-performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2x acceleration under balanced perception-generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching.
{
"annotation_id": "fbaccc69-b080-45fb-be88-b6e198d549a0",
"date_created": "2026-02-17T05:53:08.860000Z",
"date_modified": "2026-02-17T05:53:08.860000Z",
"file_hash": "53bdd9f7537ab8db3b9b266a4e0b9a1d52d7dc9293d6846f04e217b9bbf8e5cc",
"private": false,
"record": {
"abstract": "Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception-generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input-output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency-performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2x acceleration under balanced perception-generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching.",
"arxiv_id": "2601.06843",
"authors": [
"Junyan Lin",
"Junlong Tong",
"Hao Wu",
"Jialiang Zhang",
"Jinming Liu",
"Xin Jin",
"Xiaoyu Shen"
],
"categories": [
"cs.CV",
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models",
"url": "https://arxiv.org/abs/2601.06843",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "1e451fc8-79f9-41eb-85cc-33ab0f86923e",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}