dorsal/arxiv
View SchemaColorBrowserAgent: An Intelligent GUI Agent for Complex Long-Horizon Web Automation
| Authors | Jiamu Zhou, Jihong Wang, Weiming Zhang, Weiwen Liu, Zhuosheng Zhang, Xingyu Lou, Weinan Zhang, Huarong Deng, Jun Wang |
|---|---|
| Categories | |
| ArXiv ID | 2601.07262vv1 |
| URL | https://arxiv.org/abs/2601.07262 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
The web browser serves as a primary interface for daily human activities, making its automation a critical frontier for Human-Centred AI. While Large Language Models (LLMs) have enabled autonomous agents to interact with web GUIs, their reliability in real-world scenarios is hampered by long-horizon instability and the vast heterogeneity of site designs. In this paper, we introduce ColorBrowserAgent, a framework designed for Collaborative Autonomy in complex web tasks. Our approach integrates two human-centred mechanisms: (1) Progressive Progress Summarization, which mimics human short-term memory to maintain coherence over extended interactions; and (2) Human-in-the-Loop Knowledge Adaptation, which bridges the knowledge gap in diverse environments by soliciting expert intervention only when necessary. This symbiotic design allows the agent to learn from human tips without extensive retraining, effectively combining the scalability of AI with the adaptability of human cognition. Evaluated on the WebArena benchmark using GPT-5, ColorBrowserAgent achieves a state-of-the-art success rate of 71.2\%, demonstrating the efficacy of interactive human assistance in robust web automation.
{
"annotation_id": "6ed4d36b-cfde-44af-b7c3-d8a5b6423c05",
"date_created": "2026-02-17T05:53:11.334000Z",
"date_modified": "2026-02-17T05:53:11.334000Z",
"file_hash": "37591928b142c41eb49601a291be057809cfbe88e20866ffe15a90c3cc605852",
"private": false,
"record": {
"abstract": "The web browser serves as a primary interface for daily human activities, making its automation a critical frontier for Human-Centred AI. While Large Language Models (LLMs) have enabled autonomous agents to interact with web GUIs, their reliability in real-world scenarios is hampered by long-horizon instability and the vast heterogeneity of site designs. In this paper, we introduce ColorBrowserAgent, a framework designed for Collaborative Autonomy in complex web tasks. Our approach integrates two human-centred mechanisms: (1) Progressive Progress Summarization, which mimics human short-term memory to maintain coherence over extended interactions; and (2) Human-in-the-Loop Knowledge Adaptation, which bridges the knowledge gap in diverse environments by soliciting expert intervention only when necessary. This symbiotic design allows the agent to learn from human tips without extensive retraining, effectively combining the scalability of AI with the adaptability of human cognition. Evaluated on the WebArena benchmark using GPT-5, ColorBrowserAgent achieves a state-of-the-art success rate of 71.2\\%, demonstrating the efficacy of interactive human assistance in robust web automation.",
"arxiv_id": "2601.07262",
"authors": [
"Jiamu Zhou",
"Jihong Wang",
"Weiming Zhang",
"Weiwen Liu",
"Zhuosheng Zhang",
"Xingyu Lou",
"Weinan Zhang",
"Huarong Deng",
"Jun Wang"
],
"categories": [
"cs.HC"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "ColorBrowserAgent: An Intelligent GUI Agent for Complex Long-Horizon Web Automation",
"url": "https://arxiv.org/abs/2601.07262",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "da3f0916-6780-4fa2-889c-1cbf2d8b187f",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}