dorsal/arxiv
View SchemaVoiceSculptor: Your Voice, Designed By You
| Authors | Jingbin Hu, Huakang Chen, Linhan Ma, Dake Guo, Qirui Zhan, Wenhao Li, Haoyu Zhang, Kangxiang Xia, Ziyu Zhang, Wenjie Tian, Chengyou Wang, Jinrui Liang, Shuhan Guo, Zihang Yang, Bengu Wu, Binbin Zhang, Pengcheng Zhu, Pengyuan Xie, Chuan Xie, Qiang Zhang, Jie Liu, Lei Xie |
|---|---|
| Categories | |
| ArXiv ID | 2601.10629vv2 |
| URL | https://arxiv.org/abs/2601.10629 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.
{
"annotation_id": "890b4e30-36e0-4258-8288-f2fb57411e95",
"date_created": "2026-02-17T05:53:26.426000Z",
"date_modified": "2026-02-17T05:53:26.426000Z",
"file_hash": "8f54d901331dac215b4da5e8a62631db1b3d87fad1549ccbda32d9d498539831",
"private": false,
"record": {
"abstract": "Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.",
"arxiv_id": "2601.10629",
"authors": [
"Jingbin Hu",
"Huakang Chen",
"Linhan Ma",
"Dake Guo",
"Qirui Zhan",
"Wenhao Li",
"Haoyu Zhang",
"Kangxiang Xia",
"Ziyu Zhang",
"Wenjie Tian",
"Chengyou Wang",
"Jinrui Liang",
"Shuhan Guo",
"Zihang Yang",
"Bengu Wu",
"Binbin Zhang",
"Pengcheng Zhu",
"Pengyuan Xie",
"Chuan Xie",
"Qiang Zhang",
"Jie Liu",
"Lei Xie"
],
"categories": [
"eess.AS"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "VoiceSculptor: Your Voice, Designed By You",
"url": "https://arxiv.org/abs/2601.10629",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "efe571ad-b14b-4f60-98b9-8449ed50a604",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}