dorsal/arxiv
View SchemaVirtualEnv: A Platform for Embodied AI Research
| Authors | Kabir Swain, Sijie Han, Ayush Raina, Jin Zhang, Shuang Li, Michael Stopa, Antonio Torralba |
|---|---|
| Categories | |
| ArXiv ID | 2601.07553vv1 |
| URL | https://arxiv.org/abs/2601.07553 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.
{
"annotation_id": "353ecfa4-5b7f-4bbc-9d23-be596a1d9ebb",
"date_created": "2026-02-17T05:53:12.606000Z",
"date_modified": "2026-02-17T05:53:12.606000Z",
"file_hash": "12e19859f0e246e0632307291d1369c80ff46942191cdbeb989a59b565f0859c",
"private": false,
"record": {
"abstract": "As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.",
"arxiv_id": "2601.07553",
"authors": [
"Kabir Swain",
"Sijie Han",
"Ayush Raina",
"Jin Zhang",
"Shuang Li",
"Michael Stopa",
"Antonio Torralba"
],
"categories": [
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "VirtualEnv: A Platform for Embodied AI Research",
"url": "https://arxiv.org/abs/2601.07553",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "dfce568f-675e-4915-b982-96a3b9b2cffd",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}