{"id":16512,"date":"2024-08-20T14:20:27","date_gmt":"2024-08-20T06:20:27","guid":{"rendered":"http:\/\/139.9.1.231\/?p=16512"},"modified":"2024-09-28T18:41:39","modified_gmt":"2024-09-28T10:41:39","slug":"speechgpt","status":"publish","type":"post","link":"http:\/\/139.9.1.231\/index.php\/2024\/08\/20\/speechgpt\/","title":{"rendered":"SpeechGPT\u7cfb\u5217\uff1a\u8bed\u97f3\u5927\u8bed\u8a00\u6a21\u578b"},"content":{"rendered":"\n
\u5f00\u6e90\uff1ahttps:\/\/github.com\/0nutation\/SpeechGPT<\/a>\n\nSpeechGPT (2023\/05) - Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities\nSpeechGPT\uff082023\/05\uff09-\u4e3a\u5927\u578b\u8bed\u8a00\u6a21\u578b\u63d0\u4f9b\u5185\u5728\u7684\u8de8\u6a21\u6001\u4f1a\u8bdd\u80fd\u529b\n\nSpeechGPT-Gen (2024\/01) - Scaling Chain-of-Information Speech Generation\nSpeechGPT-Gen\uff082024\/01\uff09-\u7f29\u653e\u4fe1\u606f\u94fe\u8bed\u97f3\u751f\u6210\n\nSpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems\nSpeechAgents\uff1a\u591a\u6a21\u6001\u591a\u667a\u80fd\u4f53\u7cfb\u7edf\u7684\u4eba\u673a\u4ea4\u4e92\u4eff\u771f\n\nSpeechAlign:  \nAligning Speech Generation to Human Preferences\nSpeechAlign: \u4f7f\u8bed\u97f3\u751f\u6210\u4e0e\u4eba\u7c7b\u504f\u597d\u4fdd\u6301\u4e00\u81f4\n\npaper:\n\n   https:\/\/arxiv.org\/abs\/2401.03945<\/a> [SpeechAgents]\n   https:\/\/arxiv.org\/abs\/2305.11000<\/a> [SpeechGPT]\n   https:\/\/arxiv.org\/abs\/2401.13527<\/a> [SpeechGPT-Gen\/SpeechGPT2]\n   https:\/\/arxiv.org\/abs\/2404.05600<\/a> [SpeechAlign]\n<\/pre>\n\n\n\n\n\n
\u8fd9\u7b97\u662f\u590d\u65e6\u5927\u5b66\u90b1\u9521\u9e4f\u7ec4\u5728\u8fd9\u4e2a\u9886\u57df\u4e00\u4e2a\u6210\u7cfb\u5217\u7684\u5de5\u4f5c\uff0c\u6211\u4eec\u4e00\u4e2a\u4e00\u4e2a\u6765\u770b\u3002SpeechGPT\u662f\u4e00\u79cd\u57fa\u4e8e\u6df1\u5ea6\u5b66\u4e60\u7684\u8bed\u8a00\u6a21\u578b\uff0c\u5b83\u80fd\u591f\u7406\u89e3\u548c\u751f\u6210\u81ea\u7136\u8bed\u8a00\u3002SpeechGPT\u662f\u4e00\u79cd\u65b0\u578b\u7684\u5927\u578b\u8bed\u8a00\u6a21\u578b\uff0c\u5b83\u4e0d\u4ec5\u80fd\u591f\u7406\u89e3\u8bed\u97f3\u548c\u6587\u672c\uff0c\u8fd8\u80fd\u591f\u5728\u8fd9\u4e24\u8005\u4e4b\u95f4\u81ea\u5982\u8f6c\u6362\u3002\u7b80\u5355\u6765\u8bf4\uff0c\u5b83\u80fd\u591f\u611f\u77e5\u548c\u8868\u8fbe\u60c5\u611f\uff0c\u5e76\u6839\u636e\u4e0a\u4e0b\u6587\u548c\u4eba\u7c7b\u6307\u4ee4\u63d0\u4f9b\u591a\u79cd\u98ce\u683c\u7684\u8bed\u97f3\u54cd\u5e94\u3002\u65e0\u8bba\u662f\u8bf4\u5531\u3001\u620f\u5267\u3001\u673a\u5668\u4eba\u3001\u641e\u7b11\u8fd8\u662f\u4f4e\u8bed\uff0cSpeechGPT\u90fd\u80fd\u591f\u6839\u636e\u9700\u8981\u751f\u6210\u76f8\u5e94\u98ce\u683c\u7684\u8bed\u97f3\u3002<\/p>\n\n\n\n
SpeechGPT<\/strong><\/h2>\n\n\n\n
SpeechGPT\u505a\u7684\u662f\u517c\u5177\u8bed\u97f3\u7406\u89e3\u80fd\u529b\u548c\u8bed\u97f3\u751f\u6210\u80fd\u529b\u7684\u591a\u6a21\u6001\u6a21\u578b\u3002\u5728\u6a21\u578b\u7684\u8bad\u7ec3\u4e0a\uff0cSpeechGPT\u5927\u5e45\u5ea6\u5411LLM\u770b\u9f50\uff0c\u4f7f\u7528\u4e86\u4e09\u6bb5\u5f0f\u7684\u8bad\u7ec3\u65b9\u6cd5\uff1a\u7b2c\u4e00\u9636\u6bb5\u5148\u505a\u6a21\u6001\u9002\u5e94\u7684\u9884\u8bad\u7ec3\uff0c\u5176\u5b9e\u5c31\u662f\u62ffASR\u7684\u8bed\u97f3\u6570\u636e\u6765\u505a\u9884\u8bad\u7ec3\uff1b\u7b2c\u4e8c\u9636\u6bb5\u548c\u7b2c\u4e09\u9636\u6bb5\u90fd\u662f\u6307\u4ee4\u5fae\u8c03\uff0c\u4e0d\u8fc7\u6839\u636e\u6307\u4ee4\u6a21\u6001\u7684\u4e0d\u540c\uff0c\u7ec6\u5206\u4e3a\u4e86\u8de8\u6a21\u6001\u7684\u6307\u4ee4\u5fae\u8c03\u548c\u6a21\u6001\u94fe\u6307\u4ee4\u5fae\u8c03\u3002\u6307\u4ee4\u5fae\u8c03\u7684\u6570\u636e\u96c6\u90fd\u662f\u6765\u81eaASR\u6570\u636e\u96c6\u3002\u63cf\u8ff0\u4efb\u52a1\u9700\u6c42\u7684\u6307\u4ee4\u7531GPT-4\u751f\u6210\u3002<\/p>\n\n\n\n
\u5728\u6211\u770b\u6765\uff0c\u8fd9\u4e2a\u5de5\u4f5c\u8fd8\u662f\u76f8\u5f53\u504f\u5b66\u672f\u5316\u7684\u4f5c\u54c1\uff0c\u6587\u4e2d\u6709\u4e0d\u5c11\u70b9\u90fd\u6709\u503c\u5f97\u5546\u69b7\u7684\u5730\u65b9\uff1a\u7b2c\u4e00\uff0c\u8bed\u97f3\u7684\u79bb\u6563\u5316\u4ec5\u4ec5\u7528\u4e86HuBERT\uff0c\u6a21\u578b\u53ea\u80fd\u770b\u5230\u8bed\u97f3\u7684\u8bed\u4e49\u7279\u5f81\uff0c\u8fd9\u5bf9\u6a21\u578b\u5408\u6210\u8bed\u97f3\u7684\u97f3\u8d28\u548c\u8868\u73b0\u529b\u6709\u975e\u5e38\u5927\u7684\u5f71\u54cd\uff0cdemo\u7684\u8bed\u97f3\u4e5f\u9a8c\u8bc1\u4e86\u6211\u7684\u5224\u65ad\uff1b\u7b2c\u4e8c\uff0c\u6307\u4ee4\u5fae\u8c03\u6570\u636e\u96c6\u7684\u6784\u9020\u4e0a\u6709\u95ee\u9898\u3002\u4ed6\u4eec\u7528\u7684\u662fASR\u6570\u636e\u96c6\uff0c\u5176\u5b9e\u66f4\u597d\u7684\u9009\u62e9\u5e94\u8be5\u662fTTS\u6570\u636e\u96c6\uff0c\u53ef\u60dc\u9ad8\u8d28\u91cf\u7684TTS\u6570\u636e\u96c6\u5b9e\u5728\u662f\u592a\u5c11\u4e86\u3002ASR\u6570\u636e\u96c6\u4e2d\u7684\u6587\u672c\u548c\u8bed\u97f3\u53ef\u80fd\u5e76\u4e0d\u662f\u4e25\u683c\u5bf9\u9f50\u7684\uff0cGPT-4\u4ea7\u751f\u7684meta-prompt\u548c\u8bed\u97f3\u672c\u8eab\u7684\u7279\u5f81\u4e5f\u6709\u53ef\u80fd\u662f\u5bf9\u4e0d\u4e0a\u7684\uff0c\u6bd4\u5982prompt\u8981\u6c42\u5927\u58f0\u6717\u8bfb\uff0c\u4f46\u8bed\u97f3\u672c\u8eab\u53ef\u80fd\u662f\u7279\u5b9a\u4f4e\u6c89\u7684\u3002meta-prompt\u672c\u8eab\u5c31\u65e0\u6cd5\u505a\u5230\u8db3\u591f\u590d\u6742\u4e30\u5bcc\uff0c\u4e0d\u80fd\u63cf\u8ff0\u5230\u8bed\u97f3\u7684\u4e00\u4e9b\u7ec6\u7c92\u5ea6\u4fe1\u606f\u3002<\/p>\n\n\n\n
\u8fd9\u4e00\u90e8\u5206\uff0c\u6700\u597d\u8981\u6709\u50cf\u8bf8\u5982SALMONN\u8fd9\u6837\u7684\u591a\u6a21\u6001\u8bed\u97f3\u7406\u89e3\u6a21\u578b\u7684\u4ecb\u5165\uff0c\u50cfDALLE3\u4e00\u6837\u4e30\u5bcc\u6307\u4ee4\u7684\u591a\u6837\u6027\u3002\u81f3\u4e8e\u8bed\u97f3\u65b9\u9762\uff0c\u53ef\u4ee5\u8003\u8651\u5f15\u5165zero-shot\u7684\u8bed\u97f3\u5408\u6210\u6a21\u578b\u6216\u8005\u53d8\u58f0\u6a21\u578b\u6765\u505a\u5408\u6210\u6570\u636e\u3002\u7b2c\u4e09\uff0c\u6587\u4e2d\u7684\u8bad\u7ec3\u65b9\u6cd5\u4e5f\u6ca1\u6709\u4e0e\u4eba\u7c7b\u504f\u597d\u505a\u5bf9\u9f50\u3002<\/p>\n\n\n\n
\u8fd9\u9879\u6280\u672f\u7684\u6838\u5fc3\u5728\u4e8e\u5c06\u8fde\u7eed\u7684\u8bed\u97f3\u4fe1\u53f7\u79bb\u6563\u5316\uff0c\u4f7f\u5176\u80fd\u591f\u4e0e\u6587\u672c\u6a21\u6001\u7edf\u4e00\uff0c\u4ece\u800c\u8ba9\u6a21\u578b\u5177\u5907\u611f\u77e5\u548c\u751f\u6210\u8bed\u97f3\u7684\u80fd\u529b\u3002<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
SpeechGPT\u662f\u4e00\u79cd\u5927\u578b\u8bed\u8a00\u6a21\u578b\uff0c\u5177\u6709\u5185\u5728\u7684\u8de8\u6a21\u6001\u4f1a\u8bdd\u80fd\u529b<\/strong>\uff0c\u80fd\u591f\u6309\u7167\u4eba\u7c7b\u6307\u4ee4\u611f\u77e5\u548c\u751f\u6210\u591a\u6a21\u578b\u5185\u5bb9\u3002\u672c\u6587\u9996\u5148\u5229\u7528\u79bb\u6563\u8bed\u97f3\u8868\u793a\u6784\u9020\u4e86\u4e00\u4e2a\u5927\u89c4\u6a21\u7684\u8de8\u6a21\u6001\u8bed\u97f3\u6307\u4ee4\u96c6SpeechInstruct<\/strong>\u3002\u6b64\u5916\uff0c\u6211\u4eec\u91c7\u7528\u4e86\u4e00\u4e2a\u4e09\u9636\u6bb5\u7684\u8bad\u7ec3\u7b56\u7565\uff0c\u5305\u62ec\u6a21\u6001\u9002\u5e94\u9884\u8bad\u7ec3<\/strong>\uff0c\u8de8\u6a21\u6001\u6307\u4ee4\u5fae\u8c03<\/strong>\uff0c\u548c\u94fe\u7684\u6a21\u6001\u6307\u4ee4\u5fae\u8c03<\/strong>\u3002\u5b9e\u9a8c\u7ed3\u679c\u8868\u660e\uff0cSpeechGPT\u5177\u6709\u826f\u597d\u7684\u591a\u6a21\u6001\u4eba\u673a\u4ea4\u4e92\u80fd\u529b\uff0c\u5e76\u80fd\u5728\u4e00\u4e2a\u6a21\u578b\u4e2d\u5904\u7406\u591a\u79cd\u6a21\u6001\u3002<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\u6a21\u578b\u8bad\u7ec3\u8fc7\u7a0b\uff1a<\/p>\n\n\n\n
        \u7b2c\u4e00\u9636\u6bb5\uff0c\u4f7f\u7528LLaMA-7 B\u521d\u59cb\u5316\uff0c\u5e76\u5728LibriLight\u8bed\u97f3\u5355\u5143\u4e0a\u8fdb\u4e00\u6b65\u9884\u8bad\u7ec3\u3002\u7b2c\u4e8c\u9636\u6bb5\uff0c\u4f7f\u7528\u4e0a\u4e00\u6b65\u9884\u8bad\u7ec3\u7684\u6a21\u578b\u8fdb\u884c\u521d\u59cb\u5316\uff0c\u5e76\u5728SpeechInstruct\u8de8\u6a21\u6001\u6307\u4ee4\u96c6\u4e0a\u8fdb\u4e00\u6b65\u5fae\u8c03\u3002\u8fd9\u662f\u4e00\u4e2a\u5f3a\u5927\u7684\u57fa\u7840\u6a21\u578b\uff0c\u53ef\u4ee5\u5bf9\u9f50\u8bed\u97f3\u548c\u6587\u672c\u3002\u7b2c\u4e09\u9636\u6bb5\uff0c\u5728SpeechInstruct\u6a21\u6001\u94fe\u6307\u4ee4\u96c6\u4e0a\u8fdb\u4e00\u6b65\u8fdb\u884clora-finetuning\u3002<\/p>\n\n\n\n
\u6570\u636e\u96c6\u683c\u5f0f:<\/p>\n\n\n\n
SpeechInstruct-cross-modal<\/strong><\/a>\uff1a\u8de8\u6a21\u6001\u6307\u4ee4\u96c6\uff0cmHuBERT\u4ece\u5927\u89c4\u6a21\u82f1\u8bedASR\u6570\u636e\u96c6\u4e2d\u6807\u8bb0\u4e86\u5927\u7ea6900\u4e07\u4e2a\u5355\u5143\u6587\u672c\u6570\u636e\u5bf9\u3002\u6570\u636e\u683c\u5f0f\uff1a<\/p>\n\n\n\n
[\n    {\n        \"prefix\": \"You are an AI assistant whose name is SpeechGPT.\\n- SpeechGPT is a intrinsic cross-modal conversational language model that is developed by Fudan University.  SpeechGPT can understand and communicate fluently with human through speech or text chosen by the user.\\n- It can perceive cross-modal inputs and generate cross-modal outputs.\\n\",\n        \"plain_text\": \"[Human]: Try to speak out this sentence, please. This is input: The alchemist rode in front, with the falcon on his shoulder.<eoh> [SpeechGPT]: <sosp><661><588><604><157><596><499><596><106><596><189><63><189><665><991><162><202><393><946><327><905><907><597><660><351><557><794><788><59><754><12><977><877><333><873><835><67><940><118><686><613><169><72><644><553><535><935><101><741><384><173><894><787><380><787><196><555><721><944><250><56><812><222><915><143><390><479><330><435><647><246><650><816><325><506><686><208><613><417><755><193><411><452><111><735><6><735><63><665><644><991><535><271><333><196><918><29><202><393><946><734><390><479><330><776><167><761><907><597><660><351><557><794><75><788><15><366><896><627><168><654><659><177><183><609><710><187><493><361><470><821><59><56><198><912><742><840><431><531><76><668><576><803><791><380><660><325><801><549><366><377><164><309><584><605><193><71><39><eosp><eoa> \"\n    },\n]<\/code><\/pre>\n\n\n\n
SpeechInstruction-chain-of-modality<\/strong><\/a>\uff1a\u56db\u79cd\u8f93\u5165\u8f93\u51fa\u683c\u5f0f\u7684\u601d\u60f3\u94fe\u6837\u5f0f\u6307\u4ee4\uff0c\u5373Speech Instruction-Speech Response\u3001Speech Instruction-Text Response\u3001Text Instruction-Speech Response\u548cText Instruction-Text Response\u3002<\/p>\n\n\n\n
[\n    {\n        \"prefix\": \"You are an AI assistant whose name is SpeechGPT.\\n- SpeechGPT is a intrinsic cross-modal conversational language model that is developed by Fudan University.  SpeechGPT can understand and communicate fluently with human through speech or text chosen by the user.\\n- It can perceive cross-modal inputs and generate cross-modal outputs.\\n\",\n        \"plain_text\": \"[Human]: <sosp><661><987><511><732><951><997><111><982><189><63><665><991><535><101><741><173><945><944><503><641><124><565><734><870><290><978><833><238><761><907><430><901><185><403><557><244><583><788><663><969><896><627><143><515><663><969><660><691><251><412><260><41><740><677><253><380><382><268><506><876><417><755><16><819><80><651><80><651><80><987><588><eosp><eoh>. [SpeechGPT]: What is a bad term for poop?; [ta] A bad term for poop is excrement. It is usually used as a polite way to refer to fecal waste.; [ua] <sosp><497><63><264><644><710><823><565><577><154><331><384><173><945><29><244><326><583><728><576><663><969><896><627><143><38><515><663><24><382><251><676><412><260><41><740><677><253><382><268><876><233><878><609><389><771><865><641><124><878><609><423><384><879><487><219><522><589><337><126><119><663><748><12><671><877><377><385><902><819><619><842><419><997><829><111><666><42><277><63><665><644><389><771><685><437><641><124><258><436><139><340><11><59><518><56><948><86><258><436><139><340><347><376><940><118><944><878><173><641><124><362><734><179><961><931><878><609><423><384><879><219><522><866><337><243><935><101><741><822><89><194><630><86><555><105><79><868><220><156><824><998><870><390><422><330><776><663><969><523><105><79><799><220><357><390><479><422><330><776><485><165><86><501><119><716><205><521><787><935><101><741><89><194><664><835><67><940><118><613><417><755><902><415><772><497><eosp><eoa>.\"\n    },\n]<\/code><\/pre>\n\n\n\n
Introduction<\/h3>\n\n\n\n
        \u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u63d0\u51fa\u4e86SpeechGPT\uff0c\u4e00\u4e2a\u5927\u578b\u7684\u8bed\u8a00\u6a21\u578b\uff0c\u5177\u6709\u5185\u5728\u7684\u8de8\u6a21\u6001\u4f1a\u8bdd\u80fd\u529b\uff0c\u80fd\u591f\u611f\u77e5\u548c\u751f\u6210\u591a\u6a21\u578b\u5185\u5bb9\u3002\u6211\u4eec\u4f7f\u7528\u81ea\u76d1\u7763\u8bad\u7ec3\u7684\u8bed\u97f3\u6a21\u578b\u8fdb\u884c\u8bed\u97f3\u79bb\u6563\u5316\uff0c\u4ee5\u7edf\u4e00\u8bed\u97f3\u548c\u6587\u672c\u4e4b\u95f4\u7684\u6a21\u6001\u3002\u7136\u540e\uff0c\u79bb\u6563\u7684\u8bed\u97f3token\u6269\u5c55\u5230LLM\u7684\u8bcd\u6c47\u8868\u4e2d\uff0c\u4ece\u800c\u8d4b\u4e88\u6a21\u578b\u56fa\u6709\u7684\u611f\u77e5\u548c\u751f\u6210\u8bed\u97f3\u7684\u80fd\u529b\u3002<\/p>\n\n\n\n
        \u4e3a\u4e86\u4f7f\u6a21\u578b\u5177\u6709\u5904\u7406\u591a\u6a21\u6001\u6307\u4ee4\u7684\u80fd\u529b\uff0c\u6211\u4eec\u6784\u5efa\u4e86\u7b2c\u4e00\u4e2a\u8bed\u97f3-\u6587\u672c\u8de8\u6a21\u6001\u6307\u4ee4\u9075\u5faa\u6570\u636e\u96c6SpeechInstruct\u3002\u5177\u4f53\u5730\uff0c\u6211\u4eec\u5c06\u8bed\u97f3\u79bb\u6563\u5316\u4e3a\u79bb\u6563\u5355\u5143\uff08Hsu \u7b49\u4eba\uff0c2021<\/a>\uff09\uff0c\u5e76\u57fa\u4e8e\u73b0\u6709\u7684ASR\u6570\u636e\u96c6\u6784\u5efa\u8de8\u6a21\u6001\u5355\u5143\u6587\u672c\u5bf9\u3002\u540c\u65f6\uff0c\u6211\u4eec\u4f7f\u7528GPT-4\u4e3a\u4e0d\u540c\u7684\u4efb\u52a1\u6784\u5efa\u4e86\u6570\u767e\u6761\u6307\u4ee4\uff0c\u4ee5\u6a21\u62df\u5b9e\u9645\u7684\u7528\u6237\u6307\u4ee4\uff0c\u5982\u9644\u5f55B<\/a>\u6240\u793a<\/strong>\u3002\u6b64\u5916\uff0c\u4e3a\u4e86\u8fdb\u4e00\u6b65\u589e\u5f3a\u6a21\u578b\u7684\u8de8\u6a21\u6001\u80fd\u529b\uff0c\u6211\u4eec\u8bbe\u8ba1\u4e86\u6a21\u6001\u94fe\u6307\u4ee4\u6570\u636e\uff0c\u5373\uff0c\u8be5\u6a21\u578b\u63a5\u6536\u8bed\u97f3\u547d\u4ee4\uff0c\u4ee5\u6587\u672c\u5f62\u5f0f\u601d\u8003\u8be5\u8fc7\u7a0b\uff0c\u7136\u540e\u4ee5\u8bed\u97f3\u5f62\u5f0f\u8f93\u51fa\u54cd\u5e94\u3002<\/strong><\/p>\n\n\n\n
        \u4e3a\u4e86\u66f4\u597d\u5730\u8fdb\u884c\u8de8\u6a21\u6001\u8fc1\u79fb\u548c\u6709\u6548\u7684\u8bad\u7ec3\uff0cSpeechGPT\u7ecf\u5386\u4e86\u4e09\u4e2a\u9636\u6bb5\u7684\u8bad\u7ec3\u8fc7\u7a0b\uff1a\u6a21\u6001\u9002\u5e94\u9884\u8bad\u7ec3\uff0c\u8de8\u6a21\u6001\u6307\u4ee4\u5fae\u8c03\u548c\u6a21\u6001\u94fe\u6307\u4ee4\u5fae\u8c03\u3002\u7b2c\u4e00\u9636\u6bb5\u4f7f\u8bed\u97f3\u7406\u89e3\u4e0e\u79bb\u6563\u8bed\u97f3\u5355\u5143\u8fde\u7eed\u4efb\u52a1\u7684SpeechGPT\u3002\u7b2c\u4e8c\u9636\u6bb5\u91c7\u7528SpeechInstruct\u6765\u63d0\u9ad8\u6a21\u578b\u7684\u8de8\u6a21\u6001\u80fd\u529b\u3002\u7b2c\u4e09\u9636\u6bb5\u5229\u7528\u53c2\u6570\u6709\u6548\u7684LoRA\uff08Hu \u7b49\u4eba\uff0c2021<\/a>\u5e74\uff09\u8fdb\u884c\u5fae\u8c03\uff0c\u4ee5\u8fdb\u4e00\u6b65\u8c03\u6574\u6a21\u5f0f\u3002<\/p>\n\n\n\n
SpeechInstruct Construction:<\/h3>\n\n\n\n
SpeechInstruct\uff0c\u4e00\u4e2a\u8bed\u97f3\u6587\u672c\u8de8\u6a21\u6001\u89e3\u91ca\u8ddf\u968f\u6570\u636e\u96c6\u3002\u8be5\u6570\u636e\u96c6\u7531\u4e24\u90e8\u5206\u7ec4\u6210\uff0c\u7b2c\u4e00\u90e8\u5206\u79f0\u4e3a\u8de8\u6a21\u6001\u6307\u4ee4[Cross-modal Instruction]\uff0c\u7b2c\u4e8c\u90e8\u5206\u79f0\u4e3a\u6a21\u6001\u94fe\u6307\u4ee4[Chain-of-Modality Instruction]\u3002<\/p>\n\n\n\n
 Cross-modal Instruction\u6570\u636e\u96c6\u6784\u9020:<\/strong><\/p>\n\n\n\n
 step1 \u6570\u636e\u6536\u96c6\uff1a<\/strong> \u6536\u96c6\u4e86\u51e0\u4e2a\u5927\u89c4\u6a21\u7684\u82f1\u8bedASR\u6570\u636e\u96c6\u6765\u6784\u5efa\u8de8\u6a21\u6001\u6559\u5b66\uff0c\u5305\u62ecGigaspeech\uff08Chen et  al.\uff0c2021<\/a>\uff09\u3001Common Voice\uff08Ardila \u7b49\u4eba\uff0c2020<\/a>\uff09\u548cLibriSpeech\uff08Panayotov \u7b49\u4eba\uff0c2015\u5e74<\/a>\uff09\u3002\u6211\u4eec\u91c7\u7528mHuBERT\u4f5c\u4e3a\u8bed\u97f3\u5206\u8bcd\u5668\uff0c\u4ee5\u5c06\u8bed\u97f3\u6570\u636e\u79bb\u6563\u5316\u4e3a\u79bb\u6563\u5355\u5143\uff0c\u5e76\u53bb\u9664\u76f8\u90bb\u5e27\u7684\u91cd\u590d\u5355\u5143\u4ee5\u83b7\u5f97\u7f29\u51cf\u7684\u5355\u5143\u3002\u6700\u7ec8\uff0c\u6211\u4eec\u83b7\u5f97\u4e86900\u4e07\u4e2a\u5355\u5143\u6587\u672c\u6570\u636e\u5bf9\u3002<\/p>\n\n\n\n
step2 \u4efb\u52a1\u63cf\u8ff0\u751f\u6210<\/strong>\uff1a\u6211\u4eec\u751f\u6210\u4e0e\u8bed\u97f3-\u6587\u672c\u6570\u636e\u5bf9\u517c\u5bb9\u7684ASR\u548cTTS\u4efb\u52a1\u63cf\u8ff0\u3002\u4e0e\u81ea\u6211\u6307\u5bfc\u65b9\u6cd5\u4e0d\u540c\uff08Wang \u7b49\u4eba\uff0c2022<\/a>\uff09\uff0c\u6211\u4eec\u901a\u8fc7\u96f6zero-shot \u65b9\u6cd5\u751f\u6210\u63cf\u8ff0\u3002\u5177\u4f53\u6765\u8bf4\uff0c\u6211\u4eec\u76f4\u63a5\u5c06\u9644\u5f55A<\/a>\u4e2d\u6240\u793a\u7684\u63d0\u793a\u8f93\u5165OpenAI GPT-4\u4ee5\u751f\u6210\u4efb\u52a1\u63cf\u8ff0\u3002\u6211\u4eec\u7684\u751f\u6210\u65b9\u6cd5\u4e3a\u6bcf\u4e2a\u4efb\u52a1\u751f\u6210100\u6761\u6307\u4ee4\uff0c\u9644\u5f55B<\/a>\u4e2d\u663e\u793a\u4e86\u4e00\u4e9b\u793a\u4f8b\u3002<\/p>\n\n\n\n
step3 \u6570\u636e\u6784\u9020Instruction Formatting<\/strong> \uff1a\u5bf9\u4e8e\u79bb\u6563\u5355\u5143\u5e8f\u5217 U \u53ca\u5176\u76f8\u5173\u8054\u7684\u8f6c\u5f55 T \uff0c\u6211\u4eec\u57fa\u4e8e\u6982\u7387 p \u6765\u786e\u5b9a\u5b83\u5c06\u88ab\u7528\u4e8e\u6784\u9020ASR\u4efb\u52a1\u8fd8\u662fTTS\u4efb\u52a1\u3002\u968f\u540e\uff0c\u6211\u4eec\u4ecestep2\u76f8\u5e94\u7684\u4efb\u52a1\u63cf\u8ff0\u4e2d\u968f\u673a\u9009\u62e9\u4e00\u4e2a\u63cf\u8ff0 D \u3002\u8868\u793a\u4e3a (D,U,T) \u3002\u5728\u6b64\u4e4b\u540e\uff0c\u4f7f\u7528\u6a21\u677f[Human]\uff1a{D}\u3002\u8f93\u5165\uff1a{U}<eoh>\u3002[SpeechGPT]\uff1a{T}<eos>\u3002<\/strong>. \u4e3a\u4e86\u652f\u6301\u591a\u8f6e\u5bf9\u8bdd\uff0c\u7ec4\u88c5\u7684\u6307\u4ee4\u4ee5\u591a\u8f6e\u5bf9\u8bdd\u7684\u5f62\u5f0f\u8fde\u63a5\uff0c\u9075\u5b88\u6a21\u578b\u7684\u6700\u5927\u8f93\u5165\u957f\u5ea6\u3002<\/p>\n\n\n\n
Chain-of-Modality Instruction<\/strong>\u6570\u636e\u96c6\u6784\u9020\uff1a<\/p>\n\n\n\n
\u8bed\u97f3\u6307\u4ee4\u751f\u6210<\/strong>\uff1a \u7531\u4e8e\u7f3a\u4e4f\u8bed\u97f3\u8f93\u5165\u548c\u8bed\u97f3\u8f93\u51fa\u7684\u6307\u4ee4\u6570\u636e\uff0c\u6211\u4eec\u8bad\u7ec3\u4e86\u4e00\u4e2a\u6587\u672c\u5230\u5355\u5143\u3010text-to-unit\u3011\u751f\u6210\u5668\u6765\u5c06\u6587\u672c\u6307\u4ee4\u6570\u636e\u8f6c\u6362\u4e3a\u8bed\u97f3\u6307\u4ee4\u6570\u636e\u3002\u5177\u4f53\u5730\uff0c\u6587\u672c\u5230\u5355\u5143\u751f\u6210\u5668\u91c7\u7528Transformer\u7f16\u7801\u5668-\u89e3\u7801\u5668\u67b6\u6784\u3002\u6211\u4eec\u5728\u8de8\u6a21\u6001\u6559\u5b66\u4e2d\u5bf9LibriSpeech unit-text pairs\u8fdb\u884c\u4e86\u8bad\u7ec3\u3002 \u6211\u4eec\u4ecemoss-002-sft-data\u6570\u636e\u96c6\u4e2d\u9009\u62e9\u4e8637\uff0c969\u4e2a\u54cd\u5e94\u957f\u5ea6\u5c0f\u4e8e35\u4e2a\u5355\u8bcd\u7684\u6837\u672c,\u901a\u8fc7\u6587\u672c\u5230\u5355\u5143\u751f\u6210\u5668\u5c06\u5b83\u4eec\u7684\u6307\u4ee4\u548c\u54cd\u5e94\u8f6c\u6362\u4e3a\u5355\u5143\u5e8f\u5217\u3002\u7ed3\u679c\uff0c\u6211\u4eec\u83b7\u5f97\u4e86\u7531\u8bed\u97f3\u6307\u4ee4\u3001\u6587\u672c\u6307\u4ee4\u3001\u6587\u672c\u54cd\u5e94\u548c\u8bed\u97f3\u54cd\u5e94\u7ec4\u6210\u768437\uff0c969\u4e2a\u56db\u5143\u7ec4\uff0c\u8868\u793a\u4e3a (S\u200bp\u200be\u200be\u200bc\u200bh\u200bI,T\u200be\u200bx\u200bt\u200bI,T\u200be\u200bx\u200bt\u200bR,S\u200bp\u200be\u200be\u200bc\u200bh\u200bR) \u3002<\/p>\n\n\n\n
\u4f7f\u7528\u4e0a\u8ff0\u56db\u5143\u7ec4\uff0c\u6211\u4eec\u53ef\u4ee5\u4e3a\u56db\u79cd\u8f93\u5165\u8f93\u51fa\u683c\u5f0f\u6784\u5efa\u601d\u7ef4\u94fe\u98ce\u683c\u7684\u6307\u4ee4\uff0c\u5373\u8bed\u97f3\u6307\u4ee4-\u8bed\u97f3\u54cd\u5e94\uff0c\u8bed\u97f3\u6307\u4ee4-\u6587\u672c\u54cd\u5e94\uff0c\u6587\u672c\u6307\u4ee4-\u8bed\u97f3\u54cd\u5e94\u548c\u6587\u672c\u6307\u4ee4-\u6587\u672c\u54cd\u5e94\u3002\u5176\u76f8\u5e94\u6a21\u677f\u53ef\u53c2\u89c1\u9644\u5f55C<\/a>\u3002<\/p>\n\n\n\n
Model Structure: <\/h3>\n\n\n\n
\u6a21\u578b\u7531\u4e09\u4e2a\u4e3b\u8981\u7ec4\u4ef6\u7ec4\u6210\uff1a\u79bb\u6563\u5355\u5143\u63d0\u53d6\u5668<\/em>\uff0c\u5927\u8bed\u8a00\u6a21\u578b<\/em>\u548c\u5355\u5143\u58f0\u7801\u5668<\/em>\u3002\u5728\u8fd9\u79cd\u67b6\u6784\u4e0b\uff0cLLM\u53ef\u4ee5\u611f\u77e5\u591a\u6a21\u6001\u8f93\u5165\u5e76\u751f\u6210\u591a\u6a21\u6001\u8f93\u51fa\u3002<\/p>\n\n\n\n
\u79bb\u6563\u5355\u5143\u63d0\u53d6\u5668<\/strong> \u79bb\u6563\u5355\u5143\u63d0\u53d6\u5668\u5229\u7528\u9690\u85cf\u5355\u5143BERT\uff08HuBERT\uff09\u6a21\u578b\uff08Hsu \u7b49\u4eba\uff0c2021<\/a>\uff09\u4ee5\u5c06\u8fde\u7eed\u8bed\u97f3\u4fe1\u53f7\u53d8\u6362\u6210\u79bb\u6563\u5355\u5143\u7684\u5e8f\u5217\u3002HuBERT\u662f\u4e00\u79cd\u81ea\u76d1\u7763\u6a21\u578b\uff0c\u5b83\u901a\u8fc7\u57fa\u4e8e\u5e94\u7528\u4e8e\u6a21\u578b\u4e2d\u95f4\u8868\u793a\u7684k\u5747\u503c\u805a\u7c7b\u6765\u9884\u6d4b\u63a9\u853d\u97f3\u9891\u7247\u6bb5\u7684\u79bb\u6563\u6807\u7b7e\u6765\u5b66\u4e60\u3002\u5b83\u5177\u67091-D\u5377\u79ef\u5c42\u548cTransformer\u7f16\u7801\u5668\u7684\u7ec4\u5408\uff0c\u5c06\u8bed\u97f3\u7f16\u7801\u4e3a\u8fde\u7eed\u7684\u4e2d\u95f4\u8868\u793a\uff0ck-means\u6a21\u578b\u8fdb\u4e00\u6b65\u5c06\u8fd9\u4e9b\u8868\u793a\u8f6c\u6362\u4e3a\u805a\u7c7b\u7d22\u5f15\u5e8f\u5217\u3002\u968f\u540e\uff0c\u53bb\u9664\u76f8\u90bb\u7684\u91cd\u590d\u7d22\u5f15\uff0c\u5f97\u5230\u8868\u793a\u4e3a U=(u1,u2,\u2026,uT) \u3001 ui\u22080,1,\u2026,K\u22121 \u3001 \u22001\u2264i\u2264T \u7684\u79bb\u6563\u5355\u5143\u5e8f\u5217\uff0c\u5176\u4e2d K \u8868\u793a\u7c07\u7684\u603b\u6570\u3002<\/p>\n\n\n\n
LLM:<\/strong>\u91c7\u7528Meta AI LLaMA\uff08Touvron \u7b49\u4eba\uff0c2023<\/a>\uff09\u6a21\u578b\u4f5c\u4e3a\u6211\u4eec\u7684\u5927\u578b\u8bed\u8a00\u6a21\u578b\u3002LLaMA\u5305\u62ec\u5d4c\u5165\u5c42\u3001\u591a\u4e2aTransformer\u5757\u548cLM\u5934\u5c42\u3002LLaMA\u4e2d\u7684\u53c2\u6570\u603b\u6570\u8303\u56f4\u4ece7 B\u523065 B\u3002\u4ece1.0\u4e07\u4ebf\u4e2a\u4ee4\u724c\u7684\u5e7f\u6cdb\u8bad\u7ec3\u6570\u636e\u96c6\u4e2d\uff0cLLaMA\u5728\u5404\u79cdNLP\u57fa\u51c6\u6d4b\u8bd5\u4e2d\u4e0e\u66f4\u5927\u7684175 B GPT-3\u76f8\u6bd4\u8868\u73b0\u51fa\u4e86\u7ade\u4e89\u529b\u3002<\/p>\n\n\n\n
Unit Vocoder<\/strong>:\u8bad\u7ec3\u4e86\u4e00\u4e2a\u591a\u8bf4\u8bdd\u4eba unit  HiFi-GAN\u6765\u89e3\u7801\u8bed\u97f3\u4fe1\u53f7\uff0c \u79bb\u6563\u8868\u793a\u3002 HiFi-GAN\u67b6\u6784\u7531\u751f\u6210\u5668 \ud835\udc06 \u548c\u591a\u4e2a\u9274\u522b\u5668 \ud835\udc03 \u7ec4\u6210\u3002\u8be5\u751f\u6210\u5668\u4f7f\u7528\u67e5\u627e\u8868\uff08LUT\uff09\u6765\u5d4c\u5165\u79bb\u6563\u8868\u793a\uff0c\u5e76\u4e14\u5d4c\u5165\u5e8f\u5217\u7531\u4e00\u7cfb\u5217\u7531\u8f6c\u7f6e\u5377\u79ef\u548c\u5177\u6709\u6269\u5f20\u5c42\u7684\u6b8b\u5dee\u5757\u7ec4\u6210\u7684\u5757\u8fdb\u884c\u4e0a\u91c7\u6837\u3002 \u626c\u58f0\u5668\u5d4c\u5165\u88ab\u8fde\u63a5\u5230\u4e0a\u91c7\u6837\u5e8f\u5217\u4e2d\u7684\u6bcf\u4e2a\u5e27\u3002 \u8be5\u6ee4\u6ce2\u5668\u5177\u6709\u591a\u5468\u671f\u9274\u522b\u5668\uff08MPD\uff09\u548c\u591a\u5c3a\u5ea6\u9274\u522b\u5668\uff08MSD\uff09\u3002<\/p>\n\n\n\n
Training\uff1a<\/h3>\n\n\n\n
\u4e3a\u4e86\u5c06\u8bed\u97f3\u79bb\u6563\u8868\u793a\u7eb3\u5165LLM\uff0c\u6211\u4eec\u9996\u5148\u6269\u5c55\u8bcd\u6c47\u8868\u548c\u76f8\u5e94\u7684\u5d4c\u5165\u77e9\u9635\u3002\u6211\u4eec\u5c06\u57f9\u8bad\u8fc7\u7a0b\u5206\u4e3a\u4e09\u4e2a\u9636\u6bb5\u3002\u7b2c\u4e00\u9636\u6bb5\u662f\u5bf9\u672a\u914d\u5bf9\u8bed\u97f3\u6570\u636e\u8fdb\u884c\u6a21\u6001\u81ea\u9002\u5e94\u9884\u8bad\u7ec3\u3002\u7b2c\u4e8c\u9636\u6bb5\u662f\u8de8\u6a21\u6001\u6559\u5b66\u5fae\u8c03\u3002\u7b2c\u4e09\u9636\u6bb5\u662f\u6a21\u6001\u94fe\u6559\u5b66\u5fae\u8c03\u3002<\/p>\n\n\n\n
\u6269\u5927\u8bcd\u6c47<\/strong> \u7ed9\u5b9a\u5927\u5c0f\u4e3a |V| \u7684\u539f\u59cbLLM\u8bcd\u6c47\u8868 V \uff0c\u4e3a\u4e86\u5c06\u8bed\u97f3\u79bb\u6563\u8868\u793a\u96c6\u6210\u5230LLM\u4e2d\uff0c\u6211\u4eec\u7528\u5927\u5c0f\u4e3a |V\u2032|=K \u7684\u5355\u5143\u6807\u8bb0 V\u2032 \u7684\u9644\u52a0\u96c6\u5408\u6765\u6269\u5c55\u8bcd\u6c47\u8868\u3002\u6269\u5c55\u8bcd\u6c47\u8868 V\u2032\u2032 \u662f\u539f\u59cb\u8bcd\u6c47\u8868 V \u548c\u65b0\u8bcd V\u2032 \u7684\u8054\u5408\uff1aV\u2032\u2032=V\u222aV\u2032<\/p>\n\n\n\n
\u6211\u4eec\u5c06\u539f\u59cb\u8bcd\u5d4c\u5165\u77e9\u9635\u8868\u793a\u4e3a E\u2208\u211d|V|\u00d7d \uff0c\u5176\u4e2d d \u662f\u8bcd\u5d4c\u5165\u7684\u7ef4\u6570\u3002\u4e3a\u4e86\u9002\u5e94\u6269\u5c55\u7684\u8bcd\u6c47\u8868\uff0c\u6211\u4eec\u9700\u8981\u521b\u5efa\u4e00\u4e2a\u968f\u673a\u521d\u59cb\u5316\u7684\u5355\u8bcd\u5d4c\u5165\u77e9\u9635 <\/strong>E\u2032\u2208\u211d|V\u2032\u2032|\u00d7d \u3002 \u6211\u4eec\u901a\u8fc7\u5c06 E \u7684\u503c\u590d\u5236\u5230 E\u2032 \u7684\u524d |V| \u884c\u6765\u4fdd\u7559\u539f\u59cb\u7684\u5355\u8bcd\u5d4c\u5165\uff1aE\u2032[0:|V|,:]=E<\/p>\n\n\n\n
Stage 1: Modality-Adaptation Pre-training<\/strong><\/p>\n\n\n\n
        \u4e3a\u4e86\u4f7fLLM\u80fd\u591f\u5904\u7406\u79bb\u6563\u5355\u5143\u6a21\u6001\uff0c\u6211\u4eec\u5229\u7528\u672a\u6807\u8bb0\u7684\u8bed\u97f3\u8bed\u6599\u5e93\u5728\u4e0b\u4e00\u4e2a\u4ee4\u724c\u9884\u6d4b\u4efb\u52a1\u4e2d\u8bad\u7ec3LLM\u3002\u8fd9\u79cd\u65b9\u6cd5\u4e0eLLM\u7684\u6587\u672c\u9884\u8bad\u7ec3\u76ee\u6807\u4e00\u81f4\u3002 \u7ed9\u5b9a\u7531\u8bed\u97f3 U1,U2,\u2026,Um \u548c\u8868\u793a\u4e3a L1 \u7684LLM\u7ec4\u6210\u7684\u672a\u6807\u8bb0\u8bed\u97f3\u8bed\u6599\u5e93 C \uff0c\u8d1f\u5bf9\u6570\u4f3c\u7136\u635f\u5931\u53ef\u4ee5\u516c\u5f0f\u5316\u4e3a\uff1a<\/p>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
\u5176\u4e2d\uff0c m \u662f\u6570\u636e\u96c6 C \u4e2d\u7684\u8bed\u97f3\u7684\u6570\u91cf\uff0c nj \u662f\u8bed\u97f3 Uj \u4e2d\u7684\u79bb\u6563\u5355\u5143\u6807\u8bb0\u7684\u6570\u91cf\uff0c\u5e76\u4e14 ui,j \u8868\u793a\u7b2cj\u4e2a\u8bed\u97f3\u4e2d\u7684\u7b2ci\u4e2a\u5355\u5143\u6807\u8bb0<\/p>\n\n\n\n
Stage 2: Cross-modal Instruction Fine-Tuning<\/strong> <\/p>\n\n\n\n
 <\/strong>        \u8de8\u6a21\u6001\u6559\u5b66\u5fae\u8c03<\/strong> \u5728\u8fd9\u4e2a\u9636\u6bb5\uff0c\u6211\u4eec\u5229\u7528\u914d\u5bf9\u6570\u636e\u5bf9\u9f50\u8bed\u97f3\u548c\u6587\u672c\u6a21\u6001\u3002\u6211\u4eec\u5c06SpeechInstruct\u4e2d\u7684\u8de8\u6a21\u6001\u6307\u4ee4\u4e0emoss-002-sft\u6570\u636e\u96c6\u6df7\u5408\u4ee5\u5bfc\u51fa\u6df7\u5408\u6570\u636e\u96c6 I \uff0c\u5176\u7531\u6837\u672c T1,T2,\u2026,Tx \u7ec4\u6210\u3002\u6211\u4eec\u5728 I \u4e0a\u5bf9\u4ece\u7b2c\u4e00\u9636\u6bb5\u83b7\u5f97\u7684\u6a21\u578b L \u8fdb\u884c\u5fae\u8c03\u3002<\/p>\n\n\n\n
\u7531 t1,t2,\u2026,tnj \u7ec4\u6210\u7684\u6bcf\u4e2a\u6837\u672c Tj \u901a\u8fc7\u8fde\u63a5\u524d\u7f00\u548c\u6587\u672c\u6765\u5f62\u6210\u3002\u8bad\u7ec3\u76ee\u6807\u662f\u6700\u5c0f\u5316\u8d1f\u5bf9\u6570\u4f3c\u7136\uff0c\u635f\u5931\u8ba1\u7b97\u53ea\u8003\u8651\u6587\u672c\u90e8\u5206\uff0c\u5ffd\u7565\u524d\u7f00\uff0c\u53ef\u4ee5\u683c\u5f0f\u5316\u4e3a\uff1a<\/p>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
\u5176\u4e2d x \u662f\u8bed\u6599\u5e93 I \u4e2d\u7684\u6837\u672c\u6570\uff0c yj \u662f\u6837\u672c Tj \u4e2d\u7684\u6807\u8bb0\u603b\u6570\uff0c pj \u662f Tj \u7684\u524d\u7f00\u90e8\u5206\u4e2d\u7684\u6807\u8bb0\u6570\uff0c\u5e76\u4e14 ti,j \u8868\u793a Tj \u4e2d\u7684\u7b2ci\u4e2a\u5355\u8bcd<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Stage 3: Chain-of-Modality Instruction Fine-Tuning<\/strong><\/p>\n\n\n\n
      \u5728<\/strong>\u83b7\u5f97\u9636\u6bb52\u4e2d\u7684\u6a21\u578b\u4e4b\u540e\uff0c\u6211\u4eec\u5229\u7528\u53c2\u6570\u6709\u6548\u7684\u4f4e\u79e9\u81ea\u9002\u5e94\uff08LoRA\uff09\uff08Hu \u7b49\u4eba\uff0c2021<\/a>\u5e74\uff09\uff0c\u4ee5\u5fae\u8c03\u5b83\u7684\u94fe\u7684\u6a21\u6001\u6559\u5b66SpeechInstruct\u3002\u6211\u4eec\u5c06LoRA\u6743\u91cd\uff08\u9002\u914d\u5668\uff09\u6dfb\u52a0\u5230\u6ce8\u610f\u529b\u673a\u5236\u4e2d\uff0c\u5e76\u8bad\u7ec3\u65b0\u6dfb\u52a0\u7684LoRA\u53c2\u6570\u3002\u6211\u4eec\u91c7\u7528\u4e0e\u9636\u6bb52\u76f8\u540c\u7684\u635f\u5931\u51fd\u6570\u3002<\/p>\n\n\n\n
\u9644\u5f55A\uff1aPrompts to Generate Task Description<\/h3>\n\n\n\n
ASR:\nYou are asked to come up with a set of 100 diverse task instructions about automatic speech\nrecognition, which is about recognizing speech.\nHere are the requirements:\n1. These instructions should be to instruct someone to recognize the content of the following\nspeech.\n2. Try not to repeat the verb for each instruction to maximize diversity.\n3. The language used for instruction also should be diverse. For example, you should\ncombine questions with imperative instructions.\n4. The type of instructions should be diverse.\n5. The instructions should be in English.\n6. The instructions should be 1 to 2 sentences long. Either an imperative sentence or a\nquestion is permitted.\nList of 100 tasks:\nTTS:\nYou are asked to come up with a set of 100 diverse task instructions about text to speech,\nwhich is about recognizing speech .\nHere are the requirements:\n1. These instructions should be to instruct someone to recognize the content of the following\nspeech.\n2. Try not to repeat the verb for each instruction to maximize diversity.\n3. The language used for instruction also should be diverse. For example, you should\ncombine questions with imperative instructions.\n4. The type of instructions should be diverse.\n5. The instructions should be in English.\n6. The instructions should be 1 to 2 sentences long. Either an imperative sentence or a\nquestion is permitted.\nList of 100 tasks:\n\n<\/code><\/pre>\n\n\n\n
 \u9644\u5f55B\uff1aExamples of Task Description\u3010gpt\u751f\u6210\u7684\u4e00\u4e9b\u4f8b\u5b50\u3011<\/h3>\n\n\n\n
ASR:\nBegin by converting the spoken words into written text.\nCan you transcribe the speech into a written format?\nFocus on translating the audible content into text.\nTranscribe the speech by carefully listening to it.\nWould you kindly write down the content of the speech?\nAnalyze the speech and create a written transcription.\nEngage with the speech to produce a text-based version.\nCan you document the speech in written form?\nTransform the spoken words into text accurately.\nHow about putting the speech\u2019s content into writing?\nTTS:\nCan you please read this sentence out loud?\nRecite the following words as if you were speaking normally.\nProject your voice to clearly articulate this statement.\nWould you mind speaking these words as naturally as possible?\nWhisper the given sentence softly.\nEnunciate each word in this sentence with precision. How would you express this sentence in\n a conversational tone?\nCould you please relay the message below verbally?\nEmphasize the key points while reading the sentence.\nSing the text provided in a melodic voice.\n\n<\/code><\/pre>\n\n\n\n
  \u9644\u5f55C\uff1a <\/strong>Chain-of-Modality Instructions Templates<\/h3>\n\n\n\n
Speech Instruction-Speech Response:\n[Human]: This is a speech instruction: {SpeechI}. And your response should be speech.\n You can do it step by step. You can first transcribe the instruction and get the text Instruction.\nThen you can think about the instruction and get the text response. Last, you should speak the\n response aloud <eoh>. [SpeechGPT]: [tq] {TextI}; [ta] {TextR}; [ua] {SpeechR}<eoa>.\nSpeech Instruction-Text Response:\n[Human]: This is a speech instruction: {SpeechI}. And your response should be text. You\n can do it step by step. You can first transcribe the instruction and get the text instruction.\nThen you can think about the instruction and get the text response. <eoh>. [SpeechGPT]:\n[tq] {TextI}; [ta] {TextR}<eoa>.\nText Instruction-Speech Response:\n[Human]: This is a text instruction: {TextI}. And your response should be speech. You can\n do it step by step. You can think about the instruction and get the text response. Then you\nshould speak the response aloud <eoh>. [SpeechGPT]: [ta] {TextR}; [ua] {SpeechR}<eoa>.\nText Instruction-Text Response:\n[Human]: This is a text instruction: {TextI}. And your response should be text. You can\n think about the instruction and get the text response. [SpeechGPT]: [ta] {TextR}<eoa>.<\/code><\/pre>\n\n\n\n
SpeechAgents:\u591a\u6a21\u6001\u591a\u667a\u80fd\u4f53\u7cfb\u7edf\u7684\u4eba\u673a\u4ea4\u4e92\u4eff\u771f<\/h2>\n\n\n\n
SpeechAgents\u662f\u4e00\u4e2a\u57fa\u4e8e\u591a\u6a21\u6001LLM\u7684\u591a\u667a\u80fd\u4f53\u7cfb\u7edf<\/strong>\uff0c\u65e8\u5728\u6a21\u62df\u4eba\u7c7b\u7684\u4ea4\u6d41\u3002\u4e0e\u73b0\u6709\u7684\u57fa\u4e8eLLM\u7684\u591aAgent\u7cfb\u7edf\u4e0d\u540c\uff0cSpeechAgents\u4f7f\u7528\u591a\u6a21\u6001LLM\u4f5c\u4e3a\u5355\u4e2aAgent\u7684\u4e2d\u592e\u63a7\u5236<\/strong>\uff0c\u5e76\u4f7f\u7528\u591a\u6a21\u6001\u4fe1\u53f7\u4f5c\u4e3aAgent\u4e4b\u95f4\u4ea4\u6362\u6d88\u606f\u7684\u5a92\u4ecb<\/strong>\u3002 \u6b64\u5916\uff0c\u6211\u4eec\u63d0\u51fa\u4e86\u591a\u4ee3\u7406\u8c03\u6574<\/strong>\uff0c\u4ee5\u63d0\u9ad8LLM\u7684\u591a\u4ee3\u7406\u529f\u80fd\uff0c\u800c\u4e0d\u5f71\u54cd\u4e00\u822c\u7684\u80fd\u529b\u3002\u4e3a\u4e86\u52a0\u5f3a\u548c\u8bc4\u4f30\u4eba\u7c7b\u901a\u4fe1\u4eff\u771f\u7684\u6709\u6548\u6027\uff0c\u6211\u4eec\u5efa\u7acb\u4e86\u4eba\u7c7b\u901a\u4fe1\u4eff\u771f\u57fa\u51c6<\/strong>\u3002\u5b9e\u9a8c\u7ed3\u679c\u8868\u660e\uff0cSpeechAgents\u53ef\u4ee5\u6a21\u62df\u4eba\u7c7b\u4ea4\u6d41\u5bf9\u8bdd\u4e00\u81f4\u7684\u5185\u5bb9\uff0c\u771f\u5b9e\u7684\u8282\u594f\uff0c\u4e30\u5bcc\u7684\u60c5\u611f\u548c\u8868\u73b0\u51fa\u826f\u597d\u7684\u53ef\u6269\u5c55\u6027\uff0c\u5373\u4f7f\u591a\u8fbe25\u4e2a\u4ee3\u7406\uff0c\u8fd9\u53ef\u4ee5\u9002\u7528\u4e8e\u4efb\u52a1\uff0c\u5982\u620f\u5267\u521b\u4f5c\u548c\u97f3\u9891\u5c0f\u8bf4\u751f\u6210\u3002<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\u56fe3\uff1aSpeechAgents\u4e2d\u5355\u4e2a\u4ee3\u7406\u7684\u8bad\u7ec3\u548c\u63a8\u7406\u8fc7\u7a0b\u7684\u56fe\u793a\u3002\u5b9e\u5fc3\u7bad\u5934\u8868\u793a\u63a8\u7406\u8fc7\u7a0b\u4e2d\u7684\u6570\u636e\u6d41\u3002\u5728\u4e00\u4e2a\u4ee3\u7406\u7684\u56de\u5408\u4e2d\uff0c\u5b83\u63a5\u6536\u5305\u62ec\u573a\u666f\u3001\u80cc\u666f\u3001\u89d2\u8272\u3001\u7b80\u6863\u548c\u6765\u81ea\u8bed\u97f3\u6d88\u606f\u6d41\u5e93\u7684\u6d88\u606f\u6d41\u7684\u8f93\u5165\u3002\u667a\u80fd\u4f53\u7684\u8f93\u51fa\u5305\u62ec\u5176\u5185\u90e8\u601d\u60f3\u3001\u751f\u6210\u7684\u8bed\u97f3\u54cd\u5e94\u548c\u76f8\u5e94\u7684\u98ce\u683c\u3002\u7136\u540e\u5c06\u5177\u6709\u6837\u5f0f\u7684\u54cd\u5e94\u5199\u5165\u8bed\u97f3\u6d88\u606f\u6d41\u5e93\u3002\u865a\u7ebf\u7bad\u5934\u8868\u793a\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u6570\u636e\u6d41\u3002\u4ece\u4eba\u7c7b\u901a\u4fe1\u6a21\u62df\u57fa\u51c6\u6d4b\u8bd5\u4e2d\u7684\u811a\u672c\u89e3\u6790\u7684\u4ee3\u7406\u8f68\u8ff9\u6307\u4ee4\u4ee5\u56fe\u4e2d\u4ee3\u7406\u8f93\u5165\u548c\u8f93\u51fa\u7684\u4e32\u8054\u7684\u5f62\u5f0f\u53ef\u89c6\u5316\u5730\u8868\u793a\uff0c\u5e76\u7528\u4e8e\u591a\u6a21\u6001LLM\u7684\u591a\u4ee3\u7406\u8c03\u6574\u3002<\/p>\n\n\n\n
\"\"
Hmuan-Communication Simulation Benchmark\u6784\u5efa\u8fc7\u7a0b\u6982\u8ff0\u3002\u6211\u4eec\u901a\u8fc7\u521b\u5efa\u6a21\u62df\u4eba\u7c7b\u4ea4\u6d41\u7684\u5404\u79cd\u573a\u666f\u6765\u542f\u52a8\u8fd9\u4e00\u8fc7\u7a0b\u3002\u968f\u540e\uff0c\u4e3a\u6bcf\u4e2a\u573a\u666f\u751f\u6210\u5305\u542b\u5404\u79cd\u89d2\u8272\u7684\u89d2\u8272\u6c60\u3002\u7136\u540e\u4ece\u6c60\u4e2d\u9009\u62e9\u89d2\u8272\uff0c\u5e76\u6839\u636e\u6240\u6d89\u53ca\u7684\u7279\u5b9a\u573a\u666f\u548c\u89d2\u8272\u751f\u6210\u901a\u4fe1\u811a\u672c\u3002\u6700\u7ec8\uff0c\u901a\u8fc7\u6587\u672c\u5230\u8bed\u97f3\u8f6c\u6362\u6765\u5236\u4f5c\u591a\u6a21\u6001\u4eba\u7c7b\u6c9f\u901a\u811a\u672c\u3002<\/figcaption><\/figure>\n\n\n\n
 SpeechGPT-Gen[SpeechGPT2]: \u7f29\u653e\u4fe1\u606f\u94fe\u8bed\u97f3\u751f\u6210<\/h2>\n\n\n\n
\u5bf9\u4e8e\u4e0a\u9762\u7684\u7b2c\u4e00\u4e2a\u95ee\u9898\uff0c\u4f5c\u8005\u5728\u5176\u540e\u7684SpeechGPT-Gen\u4e2d\u505a\u4e86\u89e3\u51b3\u3002\u89e3\u51b3\u601d\u8def\u7684\u6838\u5fc3\u70b9\u5c31\u662f\uff1a\u8ba9\u6a21\u578b\u4e0d\u4ec5\u770b\u5230\u8bed\u97f3\u7684\u8bed\u4e49token\uff0c\u4e5f\u8981\u770b\u5230\u8bed\u97f3\u7684\u58f0\u5b66token\u3002<\/strong>\u5177\u4f53\u505a\u6cd5\u662f\uff1aSpeechGPT\u7684HuBERT\u7279\u5f81\u66ff\u6362\u6210\u4e86SpeechTokenizer\u4e2d\u7684\u8bed\u4e49\u7279\u5f81\uff0c\u7528SpeechGPT\u8fd9\u4e00LLM\u6765\u81ea\u56de\u5f52\u5730\u5efa\u6a21\u8bed\u4e49\u7279\u5f81\uff0c\u6709\u4e86\u8bed\u4e49\u7279\u5f81\u4e4b\u540e\uff0c\u518d\u4f7f\u7528Flow-Matching\u8fd9\u6837\u7684\u6269\u6563\u6a21\u578b\u6765\u5efa\u6a21\u58f0\u5b66\u7279\u5f81\u3002\u8fd9\u91cc\u9009\u7528Flow-Matching\u6269\u6563\u6a21\u578b\uff0c\u53ef\u80fd\u662f\u53d7\u4e86SD3\u548cVoicebox\/Audiobox\u7684\u5f71\u54cd\u3002\u4e3a\u4e86\u589e\u5f3a\u4e24\u9636\u6bb5\u5efa\u6a21\u7684\u4f9d\u8d56\u5173\u7cfb\uff0c\u4f5c\u8005\u5c06\u8bed\u4e49\u7279\u5f81\u7684\u5148\u9a8c\u4fe1\u606f\u6ce8\u5165\u5230\u7b2c\u4e8c\u9636\u6bb5\u6269\u6563\u6a21\u578b\u7684\u5148\u9a8c\u5206\u5e03\u4e2d\u3002\u53ef\u4ee5\u770b\u5230\uff0c\u8fd9\u91cc\u8bed\u97f3\u7684\u89e3\u7801\u5176\u5b9e\u4e5f\u662f\u4e00\u79cd\u5c42\u6b21\u5316\u6e10\u8fdb\u5f0f\u89e3\u7801\u3002<\/p>\n\n\n\n
      SpeechGPT 2\u662f\u4e00\u79cd\u7aef\u5230\u7aef\u7684\u8bed\u97f3\u5bf9\u8bdd\u8bed\u8a00\u6a21\u578b\uff0c\u7c7b\u4f3c\u4e8eGPT-4 o\u3002\u5b83\u53ef\u4ee5\u611f\u77e5\u548c\u8868\u8fbe\u60c5\u611f\uff0c\u5e76\u6839\u636e\u4e0a\u4e0b\u6587\u548c\u4eba\u7c7b\u6307\u4ee4\u63d0\u4f9b\u5404\u79cd\u98ce\u683c\u7684\u9002\u5f53\u8bed\u97f3\u54cd\u5e94\uff0c\u5982\u8bf4\u5531\uff0c\u620f\u5267\uff0c\u673a\u5668\u4eba\uff0c\u641e\u7b11\u548c\u8033\u8bed\u3002\u4e3a\u4e86\u89e3\u51b3\u5197\u957f\u8bed\u97f3\u5e8f\u5217\u7684\u95ee\u9898\uff0cSpeechGPT 2\u91c7\u7528\u4e86\u8d85\u4f4e\u6bd4\u7279\u7387\u8bed\u97f3\u7f16\u89e3\u7801\u5668\uff08750 bps\uff09\uff0c\u8be5\u7f16\u89e3\u7801\u5668\u5bf9\u8bed\u4e49\u548c\u58f0\u5b66\u4fe1\u606f\u8fdb\u884c\u5efa\u6a21\u3002\u5b83\u4f7f\u7528\u591a\u8f93\u5165\u591a\u8f93\u51fa\u8bed\u8a00\u6a21\u578b\uff08MIMO-LM\uff09\u3002\u76ee\u524d\uff0cSpeechGPT 2\u4ecd\u7136\u662f\u4e00\u4e2a\u57fa\u4e8e\u56de\u5408\u7684\u5bf9\u8bdd\u7cfb\u7edf\u3002\u6211\u4eec\u6b63\u5728\u5f00\u53d1\u5b9e\u65f6SpeechGPT 2\u7684\u5168\u53cc\u5de5\u7248\u672c\uff0c\u5e76\u4e14\u5df2\u7ecf\u53d6\u5f97\u4e86\u4e00\u4e9b\u6709\u5e0c\u671b\u7684\u8fdb\u5c55\u3002<\/p>\n\n\n\n
        \u5f97\u76ca\u4e8e\u6709\u6548\u7684\u8bed\u97f3\u5efa\u6a21\uff0c\u5f53\u524d\u7684\u8bed\u97f3\u5927\u8bed\u8a00\u6a21\u578b\uff08SLLM\uff09\u5728\u4e0a\u4e0b\u6587\u8bed\u97f3\u751f\u6210\u548c\u5bf9\u672a\u77e5\u8bf4\u8bdd\u4eba\u7684\u6709\u6548\u6cdb\u5316\u65b9\u9762\u8868\u73b0\u51fa\u4e86\u5353\u8d8a\u7684\u80fd\u529b\u3002\u7136\u800c\uff0c\u6d41\u884c\u7684\u4fe1\u606f\u5efa\u6a21\u8fc7\u7a0b\u662f\u7531\u67d0\u4e9b\u5197\u4f59\uff0c\u5bfc\u81f4\u8bed\u97f3\u751f\u6210\u6548\u7387\u4f4e\u4e0b\u3002\u6211\u4eec\u63d0\u51fa\u4e86\u4fe1\u606f\u94fe\u751f\u6210\uff08CoIG\uff09\uff0c\u5728\u5927\u89c4\u6a21\u8bed\u97f3\u751f\u6210\u7684\u8bed\u4e49\u548c\u611f\u77e5\u4fe1\u606f\u89e3\u8026\u7684\u65b9\u6cd5\u3002\u5728\u6b64\u57fa\u7840\u4e0a\uff0c\u6211\u4eec\u5f00\u53d1\u4e86SpeechGPT-Gen\uff0c\u8fd9\u662f\u4e00\u4e2a\u5728\u8bed\u4e49\u548c\u611f\u77e5\u4fe1\u606f\u5efa\u6a21\u65b9\u9762\u5177\u670980\u4ebf\u53c2\u6570\u7684SLLM\u3002\u5b83\u5305\u62ec\u4e00\u4e2a\u57fa\u4e8eLLM\u7684\u81ea\u56de\u5f52\u6a21\u578b\u7528\u4e8e\u8bed\u4e49\u4fe1\u606f\u5efa\u6a21<\/strong>\u548c\u4e00\u4e2a\u91c7\u7528\u6d41\u5339\u914d\u7684\u975e\u81ea\u56de\u5f52\u6a21\u578b\u7528\u4e8e\u611f\u77e5\u4fe1\u606f\u5efa\u6a21 [\u611f\u77e5\u4fe1\u606f\u5efa\u6a21\u4e3b\u8981\u5173\u6ce8\u7684\u662f\u4e0e\u611f\u5b98\u8f93\u5165\uff08\u5982\u89c6\u89c9\u3001\u542c\u89c9\u7b49\uff09\u76f4\u63a5\u76f8\u5173\u7684\u4fe1\u606f\u5904\u7406,\u975e\u81ea\u56de\u5f52\u6a21\u578b\u901a\u5e38\u7528\u4e8e\u611f\u77e5\u4fe1\u606f\u5efa\u6a21\u4e2d\uff0c\u56e0\u4e3a\u8fd9\u4e9b\u6a21\u578b\u53ef\u4ee5\u5e76\u884c\u5904\u7406\u6570\u636e\uff0c\u66f4\u52a0\u9002\u5408\u9700\u8981\u5b9e\u65f6\u5904\u7406\u6216\u4f4e\u5ef6\u8fdf\u7684\u5e94\u7528\u573a\u666f]<\/strong>\u3002\u6b64\u5916\uff0c\u6211\u4eec\u5f15\u5165\u4e86\u65b0\u7684\u65b9\u6cd5\uff0c\u6ce8\u5165\u8bed\u4e49\u4fe1\u606f\u7684\u5148\u9a8c\u5206\u5e03\uff0c\u4ee5\u63d0\u9ad8\u6d41\u5339\u914d\u7684\u6548\u7387\u3002 \u5927\u91cf\u7684\u5b9e\u9a8c\u7ed3\u679c\u8868\u660e\uff0cSpeechGPT-Gen\u5728zero-shot\u6587\u672c\u5230\u8bed\u97f3\uff0czero-shot\u8bed\u97f3\u8f6c\u6362\u548c\u8bed\u97f3\u5230\u8bed\u97f3\u5bf9\u8bdd\u65b9\u9762\u8868\u73b0\u51fa\u8272\uff0c\u5f3a\u8c03\u4e86CoIG\u5728\u6355\u6349\u548c\u5efa\u6a21\u8bed\u97f3\u7684\u8bed\u4e49\u548c\u611f\u77e5\u7ef4\u5ea6\u65b9\u9762\u7684\u5353\u8d8a\u80fd\u529b\u3002<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
SpeechGPT-Gen\u57fa\u4e8e\u4fe1\u606f\u94fe\u751f\u6210\u65b9\u6cd5\uff0c\u4f9d\u6b21\u8fdb\u884c\u8bed\u4e49\u5efa\u6a21\u548c\u611f\u77e5\u5efa\u6a21\u3002SpeechTokenizer\uff08Zhang \u7b49\u4eba\uff0c2023 b<\/a>\uff09\u7528\u4e8e\u63d0\u53d6\u8bed\u4e49\u8868\u793a\u548c\u611f\u77e5\u8868\u793a\u3002SpeechGPT-Gen\u7531\u7528\u4e8e\u8bed\u4e49\u5efa\u6a21\u7684\u57fa\u4e8eLLM\u7684\u81ea\u56de\u5f52\u6a21\u578b\u548c\u7528\u4e8e\u611f\u77e5\u5efa\u6a21\u7684\u57fa\u4e8e\u6d41\u5339\u914d\u7684\u975e\u81ea\u56de\u5f52\u6a21\u578b\u7ec4\u6210\u3002<\/p>\n\n\n\n
Speech Tokenization:<\/strong><\/p>\n\n\n\n
SpeechTokenizer\uff08Zhang \u7b49\u4eba\uff0c2023 b<\/a>\uff09\u662f\u57fa\u4e8e\u6b8b\u5dee\u77e2\u91cf\u91cf\u5316\uff08RVQ\uff09\u7684\u8bed\u97f3\u6807\u8bb0\u5316\u65b9\u6cd5\uff0c\u5e76\u4e14\u8de8\u4e0d\u540cRVQ\u5c42\u5206\u5c42\u5730\u89e3\u5f00\u8bed\u97f3\u4fe1\u606f\u7684\u4e0d\u540c\u65b9\u9762\u3002\u5177\u4f53\u5730\uff0c\u7b2c\u4e00RVQ\u91cf\u5316\u5668\u751f\u6210\u5305\u542b\u8bed\u4e49\u4fe1\u606f\u7684\u4ee4\u724c\uff0c\u800c\u540e\u7eed\u91cf\u5316\u5668\u8865\u5145\u5269\u4f59\u7684\u611f\u77e5\u4fe1\u606f\u3002<\/strong> SpeechTokenizer\u5c06\u5355\u58f0\u9053\u97f3\u9891\u4fe1\u53f7\u8868\u793a\u4e3a \u4f5c\u4e3a\u8f93\u5165\uff0c\u5176\u4e2d d \u8868\u793a\u6301\u7eed\u65f6\u95f4\uff0c fs\u2062r \u8868\u793a\u91c7\u6837\u7387\u3002 SpeechTokenizer\u7684\u8f93\u51fa\u5305\u62ec\u7528\u4e8e\u5bf9\u5e94RVQ\u5c42\u7684 Q=8 \u5206\u5c42RVQ\u4ee4\u724c (q1,\u2026,qQ) \u548c Q \u7801\u672c (C1,\u2026,CQ) \u3002 qi\u2208\u211dT \u8868\u793a\u957f\u5ea6\u4e3a T \u7684\u4e00\u7ef4\u5e8f\u5217\u3002 RVQ\u4ee4\u724c qi \u53ef\u4ee5\u7531\u7801\u672c Ci\u2208\u211dK\u00d7H \u5d4c\u5165\uff0c\u4ece\u800c\u4ea7\u751f\u8fde\u7eed\u77e2\u91cf\u5e8f\u5217 vi\u2208\u211dT\u00d7H \uff0c\u5176\u4e2d vij=Ci\u2062(qij) \u7528\u4e8e i\u22081,\u2026,Q \u3001 j\u22081,\u2026,T \u3002 \u8868\u793a\u4e3a v1:8=\u2211i=1Qvi \u7684\u6765\u81ea\u6240\u6709RVQ\u5c42\u7684\u8fde\u7eed\u8868\u793a\u7684\u603b\u548c\u5305\u542b\u8bed\u97f3\u5185\u7684\u6240\u6709\u4fe1\u606f\u3002 \u6211\u4eec\u5229\u7528\u6765\u81ea\u7b2c\u4e00RVQ\u5c42 q1 \u7684\u6807\u8bb0\u4f5c\u4e3a\u5305\u542b\u8bed\u4e49\u4fe1\u606f\u7684\u8868\u793a\uff0c\u5e76\u4e14\u5229\u7528\u4ece\u7b2c\u4e8c\u5c42\u5230\u6700\u540e\u4e00\u5c42 v2:7=\u2211i=2Qvi \u6c42\u548c\u7684\u8fde\u7eed\u5411\u91cf\u4f5c\u4e3a\u5305\u542b\u611f\u77e5\u4fe1\u606f\u7684\u8868\u793a\u3002<\/p>\n\n\n\n
LLM\u7528\u4e8e\u8bed\u4e49\u5efa\u6a21:<\/strong><\/p>\n\n\n\n
SpeechGPT\u662f\u4e00\u4e2a\u5927\u578b\u7684\u8de8\u6a21\u6001\u8bed\u8a00\u6a21\u578b\uff0c\u53ef\u4ee5\u6267\u884c\u8de8\u6a21\u6001\u6307\u4ee4\u8ddf\u968f\u548c\u8bed\u97f3\u5230\u8bed\u97f3\u5bf9\u8bdd\uff0c\u8868\u73b0\u51fa\u51fa\u8272\u7684\u8bed\u97f3\u8bed\u4e49\u5efa\u6a21\u80fd\u529b\u3002 \u5229\u7528HuBERT\uff08Hsu \u7b49\u4eba\uff0c2021<\/a>\uff09\u5c06\u8bed\u97f3\u79bb\u6563\u5316\u4e3a\u5355\u5143\uff0cSpeechGPT\u6267\u884c\u6a21\u6001\u9002\u5e94\u9884\u8bad\u7ec3\uff0c\u8de8\u6a21\u6001\u6307\u4ee4\u5fae\u8c03\u548c\u6a21\u6001\u94fe\u6307\u4ee4\u5fae\u8c03\u3002\u6211\u4eec\u901a\u8fc7\u7528SpeechTokenizer RVQ-1\u4ee4\u724c q1 \u66ff\u6362HuBERT\u79bb\u6563\u5355\u5143\u6765\u4f7f\u7528SpeechGPT\u8fdb\u884c\u8bed\u97f3\u8bed\u4e49\u5efa\u6a21<\/strong>\u3002 \u5bf9\u4e8e\u6a21\u578b\u7ed3\u6784\uff0c\u6211\u4eec\u91c7\u7528LLaMA 2 – 7 B-Chat\u4f5c\u4e3a\u9884\u8bad\u7ec3\u7684LLM\u3002 \u5bf9\u4e8e\u8bad\u7ec3\u6570\u636e\uff0c\u6211\u4eec\u91c7\u7528SpeechTokenizer RVQ-1\u4ee4\u724c\u6765\u8868\u793a\u8bed\u97f3\uff0c\u5e76\u9075\u5faaSpeechInstruct\u7684\u8fc7\u7a0b\u548c\u8bbe\u7f6e\u6765\u6784\u9020\u8bad\u7ec3\u6570\u636e\u96c6 D \uff0c\u5305\u62ec\u8de8\u6a21\u6001\u6307\u4ee4\u96c6\u548c\u6a21\u6001\u94fe\u6307\u4ee4\u96c6\u3002\u6211\u4eec\u4f7f\u7528Multilingual  Librispeech \u3001Gigaspeech\uff08Chen \u7b49\u4eba\uff0c2021<\/a>\uff09\u3001Commonvoice\uff08Ardila \u7b49\u4eba\uff0c2020<\/a>\uff09\u548cLibrispeech\uff08Panayotov \u7b49\u4eba\uff0c2015\u5e74<\/a>\uff09\u6784\u5efa\u8de8\u6a21\u6001\u6307\u4ee4\u96c6\u3002 \u5bf9\u4e8e\u8bad\u7ec3\uff0c\u6211\u4eec\u8df3\u8fc7\u6a21\u6001\u9002\u5e94\u9884\u8bad\u7ec3\uff0c\u5e76\u4f7f\u7528\u8de8\u6a21\u6001\u6307\u4ee4\u96c6 I \u5bf9LLaMA 2 – 7 B-Chat\u6267\u884c\u8de8\u6a21\u6001\u6307\u4ee4\u5fae\u8c03\u3002 \u5728\u6a21\u6001\u94fe\u6307\u4ee4\u5fae\u8c03\u4e2d\uff0c\u6211\u4eec\u9009\u62e9\u5168\u53c2\u6570\u5fae\u8c03\u800c\u4e0d\u662f\u6a21\u6001\u94fe\u6307\u4ee4\u96c6\u4e0a\u7684LoRA\u5fae\u8c03\u3002 \u8fd9\u4e24\u4e2a\u57f9\u8bad\u9636\u6bb5\u7684\u57f9\u8bad\u76ee\u6807\u53ef\u4ee5\u683c\u5f0f\u4e3a\uff1a<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\u611f\u77e5\u5efa\u6a21\u7684\u6d41\u7a0b\u5339\u914d<\/strong>:<\/p>\n\n\n\n
\u7ed9\u5b9a\u8bed\u97f3 s \u3001\u8bed\u4e49\u8868\u793a v1 \u3001\u611f\u77e5\u8868\u793a v2:8 \u548c\u7531SpeechTokenizer\u63d0\u53d6\u7684\u5b8c\u6574\u4fe1\u606f\u8868\u793a v1:8=v1+v2:8 \uff0c\u611f\u77e5\u5efa\u6a21\u662f\u6307\u5728\u7ed9\u5b9a\u63d0\u793a\u8bed\u97f3 a \u548c\u8bed\u4e49\u8868\u793a v1 \u7684\u60c5\u51b5\u4e0b\u9884\u6d4b\u5b8c\u6574\u8868\u793a v1:8 \u3002 \u6211\u4eec\u63d0\u51fa\u4e86\u4e24\u79cd\u611f\u77e5\u5efa\u6a21\u7684\u6d41\u5339\u914d\u6a21\u5f0f\uff1a\u663e\u5f0f\u94fe\u548c\u9690\u5f0f\u94fe<\/strong>\u3002<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
 SpeechAlign :\u4f7f\u8bed\u97f3\u751f\u6210\u4e0e\u4eba\u7c7b\u504f\u597d\u4fdd\u6301\u4e00\u81f4 <\/h2>\n\n\n\n
SpeechAlign\u505a\u7684\u5219\u662fSLM\u4e0e\u4eba\u7c7b\u504f\u597d\u7684\u5bf9\u9f50\uff0c\u5f7b\u5e95\u5730\u5411LLM\u7684\u8bad\u7ec3\u65b9\u6cd5\u770b\u9f50\u3002\u8be5\u5de5\u4f5c\u6784\u5efa\u4e86\u5bf9\u6bd4gold token\u548c\u5408\u6210token\u7684encodec\u6570\u636e\u96c6\uff0c\u7136\u540e\u8fdb\u884c\u504f\u597d\u4f18\u5316\u6765\u8fdb\u884c\u6539\u8fdb\u3002\u4f7f\u7528\u7684\u504f\u597d\u4f18\u5316\u65b9\u6cd5\u5305\u62ecRLHF\u548cChain of Hindsight\u3002<\/strong><\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\u672c\u6587\u9996\u5148\u5206\u6790\u7f16\u89e3\u7801\u5668\u8bed\u8a00\u6a21\u578b\u4e2d\u7684\u5206\u5e03\u5dee\u8ddd\uff0c\u5f3a\u8c03\u5b83\u5982\u4f55\u5bfc\u81f4\u8bad\u7ec3\u9636\u6bb5\u548c\u63a8\u7406\u9636\u6bb5\u4e4b\u95f4\u7684\u5dee\u5f02\uff0c\u4ece\u800c\u5bf9\u6027\u80fd\u4ea7\u751f\u8d1f\u9762\u5f71\u54cd\uff0c\u4ece\u800c\u89e3\u51b3\u8fd9\u4e00\u5dee\u8ddd\u3002\u7136\u540e\uff0c\u6211\u4eec\u63a2\u7d22\u5229\u7528\u4ece\u4eba\u7c7b\u53cd\u9988\u4e2d\u5b66\u5230\u7684\u77e5\u8bc6\u6765\u5f25\u5408\u5206\u5e03\u5dee\u8ddd\u3002\u6211\u4eec\u4ecb\u7ecd\u4e86 SpeechAlign\uff0c\u8fd9\u662f\u4e00\u79cd\u8fed\u4ee3\u5f0f\u81ea\u6211\u6539\u8fdb\u7b56\u7565\uff0c\u53ef\u5c06\u8bed\u97f3\u8bed\u8a00\u6a21\u578b\u4e0e\u4eba\u7c7b\u504f\u597d\u4fdd\u6301\u4e00\u81f4\u3002SpeechAlign \u6d89\u53ca\u6784\u5efa\u4e00\u4e2a\u504f\u597d\u7f16\u89e3\u7801\u5668\u6570\u636e\u96c6\uff0c\u5c06\u9ec4\u91d1\u7f16\u89e3\u7801\u5668\u4ee4\u724c\u4e0e\u5408\u6210\u4ee4\u724c\u8fdb\u884c\u5bf9\u6bd4\uff0c\u7136\u540e\u8fdb\u884c\u9996\u9009\u9879\u4f18\u5316\u4ee5\u6539\u8fdb\u7f16\u89e3\u7801\u5668\u8bed\u8a00\u6a21\u578b\u3002\u8fd9\u4e2a\u6539\u8fdb\u5468\u671f\u662f\u8fed\u4ee3\u8fdb\u884c\u7684\uff0c\u4ee5\u7a33\u6b65\u5c06\u5f31\u6a21\u578b\u8f6c\u6362\u4e3a\u5f3a\u6a21\u578b\u3002\u901a\u8fc7\u4e3b\u89c2\u548c\u5ba2\u89c2\u8bc4\u4f30\uff0c\u6211\u4eec\u8868\u660e SpeechAlign \u53ef\u4ee5\u5f25\u5408\u5206\u5e03\u5dee\u8ddd\u5e76\u4fc3\u8fdb\u8bed\u97f3\u8bed\u8a00\u6a21\u578b\u7684\u6301\u7eed\u81ea\u6211\u6539\u8fdb\u3002\u6b64\u5916\uff0cSpeechAlign \u5177\u6709\u5f3a\u5927\u7684\u6cdb\u5316\u529f\u80fd\uff0c\u9002\u7528\u4e8e\u8f83\u5c0f\u7684\u6a21\u578b\u3002<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
\u5f00\u6e90\uff1ahttps:\/\/github.com\/0nutation\/SpeechGPT SpeechGPT (20 … \u7ee7\u7eed\u9605\u8bfbSpeechGPT\u7cfb\u5217\uff1a\u8bed\u97f3\u5927\u8bed\u8a00\u6a21\u578b<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[21,4,38],"tags":[],"_links":{"self":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/16512"}],"collection":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/comments?post=16512"}],"version-history":[{"count":234,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/16512\/revisions"}],"predecessor-version":[{"id":19871,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/16512\/revisions\/19871"}],"wp:attachment":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/media?parent=16512"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/categories?post=16512"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/tags?post=16512"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}