{"id":31040,"date":"2026-07-11T15:04:31","date_gmt":"2026-07-11T07:04:31","guid":{"rendered":"http:\/\/139.9.1.231\/?p=31040"},"modified":"2026-07-11T15:05:17","modified_gmt":"2026-07-11T07:05:17","slug":"langflow-continuous-diffusion-rivals-discrete-in-language-modeling","status":"publish","type":"post","link":"http:\/\/139.9.1.231\/index.php\/2026\/07\/11\/langflow-continuous-diffusion-rivals-discrete-in-language-modeling\/","title":{"rendered":"LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling"},"content":{"rendered":"\n<ul><li>\u8bba\u6587\u6807\u9898\uff1aLangFlow: Continuous Diffusion Rivals Discrete in Language Modeling<\/li><\/ul>\n\n\n\n<ul><li>\u8bba\u6587\u94fe\u63a5\uff1ahttps:\/\/arxiv.org\/abs\/2604.11748<\/li><li>github\uff1ahttps:\/\/github.com\/nealchen2003\/LangFlow<\/li><li>huggingface\uff1ahttps:\/\/huggingface.co\/papers\/2604.11748<\/li><\/ul>\n\n\n\n<p>LangFlow \u5173\u6ce8\u4e00\u4e2a\u957f\u671f\u6ca1\u6709\u88ab\u5145\u5206\u89e3\u51b3\u7684\u95ee\u9898\uff1a\u8fde\u7eed\u6269\u6563\u6a21\u578b\u5728\u56fe\u50cf\u3001\u89c6\u9891\u7b49\u8fde\u7eed\u6a21\u6001\u4e0a\u5f88\u5f3a\uff0c\u4f46\u5728\u8bed\u8a00\u5efa\u6a21\u4e2d\u4e00\u76f4\u843d\u540e\u4e8e\u79bb\u6563\u6269\u6563\u3002\u4f5c\u8005\u8ba4\u4e3a\u95ee\u9898\u4e0d\u5728\u4e8e\u8fde\u7eed\u6269\u6563\u672c\u8eab\u4e0d\u53ef\u884c\uff0c\u800c\u5728\u4e8e embedding-space diffusion \u7684\u8bad\u7ec3\u76ee\u6807\u3001\u4f3c\u7136\u8bc4\u4f30\u548c\u566a\u58f0\u8c03\u5ea6\u8bbe\u8ba1\u8fd8\u4e0d\u591f\u6e05\u695a\u3002<\/p>\n\n\n\n<p>\u8fd9\u7bc7\u8bba\u6587\u7684\u6838\u5fc3\u7ed3\u8bba\u662f\uff1a\u5982\u679c\u628a embedding-space diffusion \u91cd\u65b0\u8868\u8ff0\u4e3a Flow Matching\uff0c\u5e76\u7528 Bregman divergence \u89e3\u91ca\u4ea4\u53c9\u71b5\u8bad\u7ec3\u76ee\u6807\uff0c\u518d\u914d\u5408 ODE-based NLL \u4e0a\u754c\u3001Gumbel \u566a\u58f0\u8c03\u5ea6\u548c self-conditioning\uff0c\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u6a21\u578b\u53ef\u4ee5\u5728 LM1B \u548c OpenWebText \u4e0a\u63a5\u8fd1\u751a\u81f3\u8ffd\u5e73\u4e3b\u6d41\u79bb\u6563\u6269\u6563\u8bed\u8a00\u6a21\u578b\u3002LangFlow \u5728 LM1B \u4e0a\u8fbe\u5230 PPL 30.0\uff0c\u5728 OpenWebText \u4e0a\u8fbe\u5230 PPL 24.6\uff0c\u5e76\u4e14\u5728 7 \u4e2a zero-shot \u8fc1\u79fb\u8bc4\u6d4b\u4e2d\u6709 4 \u4e2a\u8d85\u8fc7\u81ea\u56de\u5f52 Transformer\u3002<\/p>\n\n\n\n<h2>1. \u80cc\u666f\uff1a\u4e3a\u4ec0\u4e48\u8bed\u8a00\u91cc\u7684\u8fde\u7eed\u6269\u6563\u4e00\u76f4\u96be\u505a\uff1f<\/h2>\n\n\n\n<p>\u6269\u6563\u6a21\u578b\u5929\u7136\u9002\u5408\u8fde\u7eed\u7a7a\u95f4\uff0c\u56e0\u6b64\u5728\u56fe\u50cf\u548c\u89c6\u9891\u751f\u6210\u4e2d\u975e\u5e38\u6210\u529f\u3002\u4f46\u8bed\u8a00\u662f\u79bb\u6563 token \u5e8f\u5217\uff0c\u6269\u6563\u8bed\u8a00\u6a21\u578b\u901a\u5e38\u6709\u4e24\u6761\u8def\u7ebf\uff1a\u4e00\u7c7b\u662f\u76f4\u63a5\u5728\u79bb\u6563\u72b6\u6001\u4e0a\u505a\u6269\u6563\uff0c\u4f8b\u5982 absorbing-state \u6216 uniform-state discrete diffusion\uff1b\u53e6\u4e00\u7c7b\u662f\u5728 token embedding \u7a7a\u95f4\u4e2d\u505a\u8fde\u7eed\u6269\u6563\u3002\u540e\u8005\u7406\u8bba\u4e0a\u4fdd\u7559\u4e86\u8fde\u7eed\u6269\u6563\u7684\u4f18\u70b9\uff0c\u6bd4\u5982\u53ef\u7f16\u8f91\u8f68\u8ff9\u3001ODE\/SDE \u91c7\u6837\u3001\u672a\u6765\u53ef\u505a\u5c11\u6b65\u84b8\u998f\uff0c\u4f46\u8fc7\u53bb\u5728 PPL \u548c\u751f\u6210\u8d28\u91cf\u4e0a\u6ca1\u6709\u771f\u6b63\u8ffd\u4e0a\u79bb\u6563\u6269\u6563\u3002<\/p>\n\n\n\n<p>LangFlow \u7684\u5207\u5165\u70b9\u662f embedding-space diffusion\u3002\u7ed9\u5b9a\u8bcd\u8868\u5d4c\u5165\u77e9\u9635 \\(E \\in \\mathbb{R}^{V \\times d}\\)\uff0c\u4e00\u4e2a token \u5e8f\u5217 \\(y=(y_1,\\ldots,y_L)\\) \u4f1a\u5148\u88ab\u6620\u5c04\u6210\u8fde\u7eed\u5d4c\u5165\u5e8f\u5217 \\(x_1 = E[y]\\)\u3002\u6a21\u578b\u4e0d\u662f\u5728 one-hot simplex \u4e0a\u6269\u6563\uff0c\u800c\u662f\u5728\u8fde\u7eed embedding \u7a7a\u95f4\u4e2d\u4ece\u9ad8\u65af\u566a\u58f0\u9010\u6b65\u79fb\u52a8\u5230 clean embedding\u3002<\/p>\n\n\n\n<p>\u53ef\u4ee5\u628a LangFlow \u7684\u751f\u6210\u8fc7\u7a0b\u62bd\u8c61\u4e3a\u4e00\u4e2a ODE\uff1a<\/p>\n\n\n\n\\(\n\\frac{d x_t}{d t}=v_\\theta(x_t,t), \\quad x_0 \\sim \\mathcal{N}(0,I), \\quad x_1 \\sim p_{\\mathrm{data}}\n\\)\n\n\n\n<p>\u5176\u4e2d \\(v_\\theta\\) \u662f\u6a21\u578b\u5b66\u4e60\u5230\u7684 velocity field\u3002\u8bad\u7ec3\u548c\u91c7\u6837\u7684\u5173\u952e\uff0c\u4e0d\u662f\u76f4\u63a5\u56de\u5f52\u67d0\u4e2a embedding\uff0c\u800c\u662f\u8ba9\u6a21\u578b\u5728\u566a\u58f0\u72b6\u6001\u4e0b\u9884\u6d4b clean token \u7684\u6982\u7387\u5206\u5e03\u3002<\/p>\n\n\n\n<h2>2. LangFlow \u7684\u6a21\u578b\u8bbe\u8ba1<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"689\" src=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-24-1024x689.png\" alt=\"\" class=\"wp-image-31372\" srcset=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-24-1024x689.png 1024w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-24-300x202.png 300w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-24-768x517.png 768w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-24.png 1123w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>LangFlow \u7684\u4e3b\u5e72\u7ed3\u6784\u4f7f\u7528\u4e0e\u5f3a\u79bb\u6563\u6269\u6563\u57fa\u7ebf\u76f8\u540c\u7684 modified DiT-style Transformer\uff0c\u5e76\u52a0\u5165 RoPE \u4f4d\u7f6e\u7f16\u7801\u3002\u6b63\u6587\u5b9e\u9a8c\u914d\u7f6e\u4e3a\u7ea6 130M \u53c2\u6570\u300112 \u5c42\u3001hidden size 768\u300112 \u4e2a attention heads\u3002\u8fd9\u6837\u8bbe\u8ba1\u7684\u597d\u5904\u662f\uff1a\u5b9e\u9a8c\u5bf9\u6bd4\u65f6\uff0cLangFlow \u4e0e SEDD\u3001MDLM\u3001Duo \u7b49\u57fa\u7ebf\u7684\u7f51\u7edc\u5bb9\u91cf\u57fa\u672c\u5bf9\u9f50\uff0c\u6027\u80fd\u63d0\u5347\u66f4\u80fd\u5f52\u56e0\u4e8e\u8fde\u7eed\u6269\u6563\u6846\u67b6\u548c\u8bad\u7ec3\u7b56\u7565\uff0c\u800c\u4e0d\u662f\u6a21\u578b\u89c4\u6a21\u3002<\/p>\n\n\n\n<p>\u6a21\u578b\u8f93\u5165\u662f noisy embedding \\(x_\\gamma\\)\uff0c\u65f6\u95f4\u6761\u4ef6\u4e0d\u76f4\u63a5\u4f7f\u7528\u666e\u901a\u65f6\u95f4 \\(t\\)\uff0c\u800c\u4f7f\u7528 log noise-to-signal ratio\uff1a<\/p>\n\n\n\n\\(\n\\gamma = \\log \\frac{\\sigma^2}{\\alpha^2}\n\\)\n\n\n\n<p>\u5728 variance-preserving \u8def\u5f84\u4e0b\uff0c\u566a\u58f0\u72b6\u6001\u53ef\u4ee5\u5199\u6210\uff1a<\/p>\n\n\n\n\\(\nx_\\gamma = \\alpha_\\gamma x_1 + \\sigma_\\gamma \\epsilon,\\quad\n\\alpha_\\gamma = \\frac{1}{\\sqrt{1+e^\\gamma}},\\quad\n\\sigma_\\gamma = \\sqrt{\\frac{e^\\gamma}{1+e^\\gamma}}\n\\)\n\n\n\n<p>\u5f53 \\(\\gamma\\) \u5f88\u5927\u65f6\uff0c\u72b6\u6001\u63a5\u8fd1\u7eaf\u566a\u58f0\uff1b\u5f53 \\(\\gamma\\) \u5f88\u5c0f\u65f6\uff0c\u72b6\u6001\u63a5\u8fd1 clean embedding\u3002\u8fd9\u6837\u505a\u7684\u76f4\u89c9\u662f\uff1a\u8bed\u8a00 denoising \u7684\u96be\u5ea6\u4e3b\u8981\u7531\u566a\u58f0\u5f3a\u5ea6\u63a7\u5236\uff0c\u800c\u4e0d\u662f\u7531\u4efb\u610f\u5b9a\u4e49\u7684\u65f6\u95f4\u53d8\u91cf\u63a7\u5236\u3002<\/p>\n\n\n\n<p>LangFlow \u8fd8\u505a\u4e86\u4e09\u4e2a\u5c0f\u4f46\u91cd\u8981\u7684\u5de5\u7a0b\u4fee\u6539\uff1a\u7b2c\u4e00\uff0c\u5c06 self-conditioning \u7684\u8f93\u5165\u5e76\u5165\u4e3b\u8f93\u5165\uff1b\u7b2c\u4e8c\uff0c\u628a token embedding \u5f52\u4e00\u5316\u5230\u5355\u4f4d\u7403\u9762\u540e\u518d\u7f29\u653e\uff0c\u4f7f\u6570\u636e\u65b9\u5dee\u4e0e\u566a\u58f0\u65b9\u5dee\u66f4\u5339\u914d\uff1b\u7b2c\u4e09\uff0c\u5728 logits \u4e0a\u52a0\u5165 tokenwise bias\uff0c\u6539\u5584\u8bad\u7ec3\u521d\u671f\u7684\u6982\u7387\u9884\u6d4b\u3002\u8fd9\u4e9b\u4fee\u6539\u6ca1\u6709\u663e\u8457\u6539\u53d8\u53c2\u6570\u91cf\uff0c\u4f46\u4f1a\u5f71\u54cd\u8bad\u7ec3\u7a33\u5b9a\u6027\u3002<\/p>\n\n\n\n<h2>3. \u8bad\u7ec3\u76ee\u6807\uff1a\u7528 Bregman divergence \u89e3\u91ca\u4ea4\u53c9\u71b5<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"345\" src=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-25-1024x345.png\" alt=\"\" class=\"wp-image-31373\" srcset=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-25-1024x345.png 1024w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-25-300x101.png 300w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-25-768x259.png 768w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-25.png 1062w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>\u8fc7\u53bb\u4e00\u4e9b\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u6a21\u578b\u4f1a\u76f4\u63a5\u7528 MSE \u56de\u5f52 clean embedding\uff0c\u4f46\u8bba\u6587\u6307\u51fa\u8fd9\u79cd\u505a\u6cd5\u53ef\u80fd\u5bfc\u81f4 embedding collapse\uff1a\u4e0d\u540c token \u7684 embedding \u88ab\u9519\u8bef\u5730\u62c9\u8fd1\uff0c\u524a\u5f31\u8bed\u8a00\u8868\u793a\u7684\u53ef\u5206\u6027\u3002LangFlow \u6539\u4e3a\u9884\u6d4b clean token \u7684\u7c7b\u522b\u5206\u5e03\uff0c\u5e76\u4f7f\u7528\u4ea4\u53c9\u71b5\u8bad\u7ec3\u3002<\/p>\n\n\n\n<p>\u4f5c\u8005\u7684\u7406\u8bba\u8d21\u732e\u662f\u8bf4\u660e\uff1a\u4ea4\u53c9\u71b5\u4e0d\u662f\u4e00\u4e2a\u4e34\u65f6\u6280\u5de7\uff0c\u800c\u53ef\u4ee5\u770b\u6210 Bregman-divergence Flow Matching \u5728 categorical data \u4e0a\u7684\u4e00\u4e2a\u7279\u6b8a\u5f62\u5f0f\u3002Bregman divergence \u5b9a\u4e49\u4e3a\uff1a<\/p>\n\n\n\n\\(\nD_F(q,p)=F(q)-F(p)-\\langle \\nabla F(p), q-p\\rangle\n\\)\n\n\n\n<p>\u5f53\u9009\u62e9\u4e0e\u8d1f\u71b5\u76f8\u5173\u7684\u51f8\u51fd\u6570\u65f6\uff0ctoken-level \u4ea4\u53c9\u71b5\u53ef\u4ee5\u81ea\u7136\u6062\u590d\u51fa\u6765\u3002LangFlow \u7684\u8bad\u7ec3\u76ee\u6807\u53ef\u4ee5\u7b80\u5316\u5199\u4e3a\uff1a<\/p>\n\n\n\n\\(\n\\mathcal{L}_{\\mathrm{CE}}(\\theta)\n=\n\\mathbb{E}_{\\gamma,\\,y,\\,\\epsilon}\n\\left[-\\log p_\\theta(y \\mid x_\\gamma,\\gamma)\\right]\n\\)\n\n\n\n<p>\u6a21\u578b\u8f93\u51fa\u7684\u662f \\(p_\\theta(\\cdot \\mid x_\\gamma,\\gamma)\\)\uff0c\u5373 clean token \u7684\u6982\u7387\u5206\u5e03\u3002\u91c7\u6837\u65f6\uff0c\u518d\u628a\u8fd9\u4e2a\u6982\u7387\u5206\u5e03\u6620\u5c04\u56de\u8fde\u7eed denoised embedding\uff1a<\/p>\n\n\n\n\\(\n\\hat{x}_1\n=\n\\sum_{i=1}^{V} p_\\theta(i \\mid x_\\gamma,\\gamma) E_i\n\\)\n\n\n\n<p>\u8fd9\u6837\u5c31\u628a\u4e24\u4e2a\u4e16\u754c\u8fde\u8d77\u6765\u4e86\uff1a\u8bad\u7ec3\u5728 token space \u4e2d\u7528\u4ea4\u53c9\u71b5\u4f18\u5316\uff0c\u91c7\u6837\u5728 embedding space \u4e2d\u6cbf ODE \u505a\u8fde\u7eed\u79fb\u52a8\u3002<\/p>\n\n\n\n<h2>4. ODE-based NLL\uff1a\u8ba9\u8fde\u7eed\u6269\u6563\u4e5f\u80fd\u8ba4\u771f\u8bc4\u4f30 PPL<\/h2>\n\n\n\n<p>\u8bed\u8a00\u6a21\u578b\u7684\u6838\u5fc3\u6307\u6807\u662f perplexity\uff0c\u4f46 embedding-space diffusion \u8fc7\u53bb\u4e3b\u8981\u4f9d\u8d56 SDE-based bound\uff0c\u548c\u5b9e\u9645 ODE \u91c7\u6837\u5e76\u4e0d\u5b8c\u5168\u4e00\u81f4\u3002LangFlow \u9009\u62e9\u53ea\u7528 deterministic ODE \u91c7\u6837\uff0c\u56e0\u4e3a ODE \u4fdd\u7559\u4ece\u566a\u58f0\u5230\u6570\u636e\u7684\u786e\u5b9a\u6027\u6620\u5c04\uff0c\u4e5f\u66f4\u9002\u5408\u672a\u6765\u505a flow-based distillation \u548c few-step generation\u3002<\/p>\n\n\n\n<p>\u8bba\u6587\u63a8\u5bfc\u4e86\u4e00\u4e2a ODE-based NLL \u4e0a\u754c\u3002\u535a\u5ba2\u91cc\u53ef\u4ee5\u628a\u5b83\u7406\u89e3\u4e3a\uff1a\u6cbf\u7740\u53cd\u5411 ODE \u8f68\u8ff9\u79ef\u5206\u6982\u7387\u5bc6\u5ea6\u53d8\u5316\uff0c\u518d\u52a0\u4e0a\u672b\u7aef token \u89e3\u7801\u6982\u7387\uff0c\u4ece\u800c\u5f97\u5230\u53ef\u7528\u4e8e PPL \u8bc4\u4f30\u7684\u4e0a\u754c\uff1a<\/p>\n\n\n\n\\(\n-\\log p_\\theta(y)\n\\le\n\\mathcal{L}_{\\mathrm{ODE}}(y)\n\\)\n\n\n\n<p>\u5176\u4e2d \\(\\mathcal{L}_{\\mathrm{ODE}}\\) \u5305\u542b ODE trajectory \u4e0a\u7684 divergence term\u3002\u8bba\u6587\u5b9e\u9a8c\u4e2d\uff0cPPL \u8bc4\u4f30\u4f7f\u7528 128-step Heun-2 solver\uff0c\u5e76\u7528 Hutchinson trace estimator \u4f30\u8ba1 divergence\u3002\u8fd9\u4e00\u70b9\u5f88\u5173\u952e\uff0c\u56e0\u4e3a\u5b83\u8ba9\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u6a21\u578b\u4e0d\u518d\u53ea\u80fd\u62a5\u544a\u751f\u6210\u6837\u672c\u7684 Gen. PPL\uff0c\u800c\u53ef\u4ee5\u548c\u79bb\u6563\u6269\u6563\u5728 PPL \u4e0a\u66f4\u516c\u5e73\u5730\u6bd4\u8f83\u3002<\/p>\n\n\n\n<h2>5. Gumbel \u566a\u58f0\u8c03\u5ea6\uff1a\u8bed\u8a00\u4e0d\u662f\u56fe\u50cf<\/h2>\n\n\n\n<p>\u8bba\u6587\u6700\u6709\u542f\u53d1\u6027\u7684\u7ecf\u9a8c\u53d1\u73b0\u662f\uff1a\u56fe\u50cf\u6269\u6563\u91cc\u5e38\u7528\u7684\u5747\u5300\u566a\u58f0\u8c03\u5ea6\uff0c\u76f4\u63a5\u642c\u5230\u8bed\u8a00\u4e0a\u4f1a\u6d6a\u8d39\u5927\u91cf\u8bad\u7ec3\u4e0e\u91c7\u6837\u6b65\u9aa4\u3002\u4f5c\u8005\u89c2\u5bdf\u5230\uff0c\u5728\u67d0\u4e9b\u566a\u58f0\u533a\u95f4\uff0c\u6a21\u578b\u7684 CE loss \u51e0\u4e4e\u4e3a 0\uff0c\u8bf4\u660e\u6a21\u578b\u5df2\u7ecf\u80fd\u8f7b\u677e\u9884\u6d4b\u6b63\u786e token\uff0c\u8fd9\u4e9b\u533a\u95f4\u7ee7\u7eed\u5206\u914d\u5927\u91cf step \u6ca1\u6709\u592a\u591a\u4fe1\u606f\u589e\u76ca\u3002<\/p>\n\n\n\n<p>LangFlow \u63d0\u51fa information-uniform principle\uff1a\u566a\u58f0\u91c7\u6837\u5bc6\u5ea6\u5e94\u8be5\u5339\u914d\u6bcf\u5355\u4f4d\u566a\u58f0\u6c34\u5e73\u5e26\u6765\u7684\u4fe1\u606f\u589e\u76ca\u3002\u76f4\u89c2\u5199\u6cd5\u662f\uff1a<\/p>\n\n\n\n\\(\np(\\gamma) \\propto \\left|\\frac{d H(y \\mid x_\\gamma)}{d\\gamma}\\right|\n\\)\n\n\n\n<p>\u8fd9\u91cc \\(H(y \\mid x_\\gamma)\\) \u53ef\u4ee5\u7406\u89e3\u4e3a\u5728\u566a\u58f0\u72b6\u6001 \\(x_\\gamma\\) \u4e0b clean token \u7684\u540e\u9a8c\u71b5\u3002\u4f5c\u8005\u53d1\u73b0\u8fd9\u4e2a\u4fe1\u606f\u589e\u76ca\u66f2\u7ebf\u5f88\u9002\u5408\u7528 Gumbel \u5206\u5e03\u62df\u5408\uff1a<\/p>\n\n\n\n\\(\np(\\gamma;\\mu,\\beta)\n=\n\\frac{1}{\\beta}\n\\exp\\left(\n-\\frac{\\gamma-\\mu}{\\beta}\n-\\exp\\left(-\\frac{\\gamma-\\mu}{\\beta}\\right)\n\\right)\n\\)\n\n\n\n<p>\u5b9e\u8df5\u4e2d\uff0cLangFlow \u8ba9 Gumbel scheduler \u7684\u53c2\u6570\u53ef\u5b66\u4e60\u3002\u8bad\u7ec3\u65f6\u4ece\u8be5\u5206\u5e03\u91c7\u6837 \\(\\gamma\\)\uff0c\u91c7\u6837\u65f6\u6309 Gumbel \u5206\u5e03\u5206\u4f4d\u70b9\u5b89\u6392 ODE step\u3002\u8bba\u6587\u62a5\u544a\uff0c\u8fd9\u4e00\u8bbe\u8ba1\u80fd\u628a LangFlow \u7684 Gen. PPL \u4ece 1000 \u7ea7\u522b\u663e\u8457\u964d\u5230 154.2\uff0c\u8bf4\u660e\u566a\u58f0\u8c03\u5ea6\u4e0d\u662f\u7ec6\u679d\u672b\u8282\uff0c\u800c\u662f\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u5efa\u6a21\u80fd\u5426\u5de5\u4f5c\u7684\u5173\u952e\u3002<\/p>\n\n\n\n<h2>6. Self-conditioning\uff1a\u8fde\u7eed\u6269\u6563\u548c\u79bb\u6563\u6269\u6563\u7684\u6548\u679c\u4e0d\u540c<\/h2>\n\n\n\n<p>Self-conditioning \u7684\u505a\u6cd5\u662f\u628a\u4e0a\u4e00\u6b65\u9884\u6d4b\u7ed3\u679c\u4f5c\u4e3a\u989d\u5916\u8f93\u5165\u5582\u56de\u6a21\u578b\u3002\u8bad\u7ec3\u65f6\u968f\u673a\u5f00\u542f\uff0c\u91c7\u6837\u65f6\u59cb\u7ec8\u5f00\u542f\u3002LangFlow \u8bad\u7ec3\u4e2d self-conditioning \u6982\u7387\u4e3a 0.25\u3002<\/p>\n\n\n\n<p>\u6709\u610f\u601d\u7684\u662f\uff0c\u8bba\u6587\u53d1\u73b0 self-conditioning \u5bf9\u79bb\u6563\u6269\u6563\u548c\u8fde\u7eed\u6269\u6563\u7684\u4f5c\u7528\u4e0d\u4e00\u6837\u3002\u5728 LM1B \u6d88\u878d\u4e2d\uff0cMDLM \u52a0\u5165 self-conditioning \u540e Gen. PPL \u4ece 103.9 \u964d\u5230 94.9\uff0c\u4f46 PPL \u4ece 31.0 \u53d8\u5dee\u5230 32.7\uff1bLangFlow \u5219\u4ece Gen. PPL 154.2\u3001PPL 49.0 \u6539\u5584\u5230 Gen. PPL 81.5\u3001PPL 30.0\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u5bf9 LangFlow \u6765\u8bf4\uff0cself-conditioning \u540c\u65f6\u63d0\u5347\u751f\u6210\u8d28\u91cf\u548c\u4f3c\u7136\u4e0a\u754c\uff0c\u662f\u628a\u8fde\u7eed\u6269\u6563\u8ffd\u5230\u79bb\u6563\u6269\u6563\u6c34\u5e73\u7684\u5173\u952e\u7ec4\u4ef6\u3002<\/p>\n\n\n\n<h2>7. \u5b9e\u9a8c\u8bbe\u7f6e\u4e0e\u5173\u952e\u7ed3\u679c<\/h2>\n\n\n\n<p>\u8bba\u6587\u4e3b\u8981\u5728 LM1B \u548c OpenWebText\uff08OWT\uff09\u4e0a\u8bc4\u6d4b\u3002LM1B \u4f7f\u7528 context length 128 \u548c bert-base-uncased tokenizer\uff1bOWT \u4f7f\u7528 context length 1024 \u548c gpt2-large tokenizer\u3002\u6a21\u578b\u8bad\u7ec3 1M steps\uff0cbatch size 512\u3002Gen. PPL \u901a\u8fc7\u751f\u6210 1024 \u4e2a\u6837\u672c\u5e76\u7528 GPT2-Large \u8ba1\u7b97\u5e73\u5747 perplexity \u5f97\u5230\uff1bPPL \u5219\u62a5\u544a\u5404\u6269\u6563\u6a21\u578b\u7684\u4e0a\u754c\u3002<\/p>\n\n\n\n<p><strong>LM1B\uff1a<\/strong>LangFlow \u7684 PPL \u4e3a 30.0\uff0c\u662f\u8868\u4e2d\u6269\u6563\u8bed\u8a00\u6a21\u578b\u91cc\u6700\u597d\u7684\u7ed3\u679c\uff1bGen. PPL \u4e3a 92.2\uff0c\u4f4e\u4e8e MDLM \u7684 103.9\u3001SEDD Absorb \u7684 115.9\u3001UDLM \u7684 99.8 \u548c Duo \u7684 97.6\uff0c\u4ec5\u7565\u5f31\u4e8e Plaid \u7684 77.3\u3002\u76f8\u6bd4\u65e9\u671f\u8fde\u7eed\u65b9\u6cd5 Diffusion-LM \u7684 PPL 118.6\uff0cLangFlow \u7684\u63d0\u5347\u975e\u5e38\u660e\u663e\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"643\" height=\"750\" src=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-26.png\" alt=\"\" class=\"wp-image-31374\" srcset=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-26.png 643w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-26-257x300.png 257w\" sizes=\"(max-width: 643px) 100vw, 643px\" \/><\/figure>\n\n\n\n<p><strong>OpenWebText\uff1a<\/strong>LangFlow \u7684 Gen. PPL \u4e3a 36.5\uff0c\u662f\u8868\u4e2d\u6700\u4f18\uff1bPPL \u4e3a 24.6\uff0c\u63a5\u8fd1 MDLM \u7684 23.2 \u548c SEDD Absorb \u7684 24.1\uff0c\u5e76\u4f18\u4e8e SEDD Uniform \u7684 29.7\u3001UDLM \u7684 27.4 \u548c Duo \u7684 25.2\u3002\u8fd9\u8bf4\u660e LangFlow \u4e0d\u53ea\u662f\u5c0f\u6570\u636e\u96c6\u4e0a\u6709\u6548\uff0c\u5728\u66f4\u63a5\u8fd1\u771f\u5b9e\u7f51\u9875\u8bed\u6599\u7684 OWT \u4e0a\u4e5f\u6709\u7ade\u4e89\u529b\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"608\" src=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-27-1024x608.png\" alt=\"\" class=\"wp-image-31375\" srcset=\"http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-27-1024x608.png 1024w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-27-300x178.png 300w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-27-768x456.png 768w, http:\/\/139.9.1.231\/wp-content\/uploads\/2026\/07\/image-27.png 1027w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><strong>Zero-shot \u8fc1\u79fb\uff1a<\/strong>\u7528 OWT \u8bad\u7ec3\u540e\u7684\u6a21\u578b\u5728 PTB\u3001Wikitext\u3001LM1B\u3001Lambada\u3001AG News\u3001PubMed\u3001Arxiv \u4e0a\u8bc4\u6d4b\u3002LangFlow \u5728 PTB \u4e3a 81.20\u3001Wikitext \u4e3a 32.28\u3001Lambada \u4e3a 46.93\uff0c\u5747\u4e3a\u6269\u6563\u6a21\u578b\u4e2d\u7684\u7b2c\u4e00\uff1bArxiv \u4e3a 38.47\uff0c\u4ec5\u7565\u5f31\u4e8e MDLM \u7684 37.37\u3002\u8bba\u6587\u603b\u7ed3\u4e3a\uff1aLangFlow \u5728 7 \u4e2a\u4efb\u52a1\u4e2d\u6709 4 \u4e2a\u8d85\u8fc7\u81ea\u56de\u5f52 Transformer\uff0c\u5e76\u5728 3 \u4e2a\u4efb\u52a1\u4e2d\u8d85\u8fc7 MDLM\u3002<\/p>\n\n\n\n<p><strong>\u91c7\u6837\u6b65\u6570\uff1a<\/strong>\u5728 LM1B \u4e0a\uff0cLangFlow \u7684 NFE \u4ece 128 \u964d\u5230 64\u300132\u300116 \u65f6\uff0cGen. PPL \u5206\u522b\u4e3a 92.24\u3001104.83\u3001127.32\u3001179.60\uff0c\u8d28\u91cf\u968f\u6b65\u6570\u51cf\u5c11\u800c\u4e0b\u964d\uff0c\u4f46\u6ca1\u6709\u7ecf\u8fc7\u4e13\u95e8 few-step \u84b8\u998f\u3002OWT \u4e0a\uff0c\u5728 1024 NFE \u65f6 LangFlow Gen. PPL \u4e3a 36.53\uff0c\u660e\u663e\u4f18\u4e8e Duo 77.69\u3001SEDD Uniform 99.90\u3001MDLM 104.85 \u548c SEDD Absorb 105.03\uff1b\u5373\u4f7f 128 NFE\uff0cLangFlow \u4ecd\u6709 60.09\u3002<\/p>\n\n\n\n<h2>8. \u5173\u952e\u521b\u65b0\u70b9\u603b\u7ed3<\/h2>\n\n\n\n<ul><li><strong>\u628a embedding-space diffusion \u63a5\u5230 Flow Matching\uff1a<\/strong>LangFlow \u7528\u8fde\u7eed ODE \u89c6\u89d2\u91cd\u65b0\u7ec4\u7ec7\u8bed\u8a00\u6269\u6563\uff0c\u800c\u4e0d\u662f\u628a\u8fde\u7eed\u6269\u6563\u5f53\u4f5c\u7b80\u5355\u7684 embedding \u56de\u5f52\u3002<\/li><li><strong>\u4ea4\u53c9\u71b5\u76ee\u6807\u6709\u7406\u8bba\u89e3\u91ca\uff1a<\/strong>\u901a\u8fc7 Bregman divergence\uff0c\u4f5c\u8005\u8bf4\u660e token-level CE \u662f categorical Flow Matching \u7684\u5408\u7406\u76ee\u6807\uff0c\u907f\u514d\u4e86 MSE \u5e26\u6765\u7684 embedding collapse \u98ce\u9669\u3002<\/li><li><strong>ODE-based NLL \u4e0a\u754c\uff1a<\/strong>\u8ba9\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u6a21\u578b\u53ef\u4ee5\u7528\u66f4\u8d34\u8fd1 ODE \u91c7\u6837\u7684\u65b9\u5f0f\u8bc4\u4f30 PPL\uff0c\u8fd9\u662f\u8bba\u6587\u7684\u6838\u5fc3\u7406\u8bba\u8d21\u732e\u4e4b\u4e00\u3002<\/li><li><strong>information-uniform \u566a\u58f0\u8c03\u5ea6\uff1a<\/strong>\u6839\u636e\u540e\u9a8c\u71b5\u53d8\u5316\u5206\u914d\u566a\u58f0\u5bc6\u5ea6\uff0c\u5e76\u7528\u53ef\u5b66\u4e60 Gumbel \u5206\u5e03\u5b9e\u73b0\uff0c\u663e\u8457\u6539\u5584\u751f\u6210\u8d28\u91cf\u3002<\/li><li><strong>self-conditioning \u8bad\u7ec3\u534f\u8bae\u4fee\u6b63\uff1a<\/strong>\u8bba\u6587\u8bc1\u660e continuous DLM \u4e2d self-conditioning \u4e0d\u53ea\u662f\u6539\u5584 Gen. PPL\uff0c\u4e5f\u80fd\u5927\u5e45\u6539\u5584 PPL\uff0c\u8fd9\u548c\u79bb\u6563\u6269\u6563\u4e2d\u7684\u73b0\u8c61\u4e0d\u540c\u3002<\/li><li><strong>\u516c\u5e73\u5bf9\u6bd4\u79bb\u6563\u6269\u6563\uff1a<\/strong>\u6a21\u578b\u89c4\u6a21\u3001\u8bad\u7ec3\u6b65\u6570\u548c\u4e3b\u5e72\u7ed3\u6784\u5c3d\u91cf\u5bf9\u9f50\uff0c\u4f7f LangFlow \u4e0e SEDD\u3001MDLM\u3001Duo \u7b49\u65b9\u6cd5\u7684\u6bd4\u8f83\u66f4\u6709\u8bf4\u670d\u529b\u3002<\/li><\/ul>\n\n\n\n<h2>9. \u5c40\u9650<\/h2>\n\n\n\n<p>LangFlow \u8bc1\u660e\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u6a21\u578b\u6709\u673a\u4f1a\u8ffd\u4e0a\u79bb\u6563\u6269\u6563\uff0c\u4f46\u5b83\u8fd8\u4e0d\u662f\u5bf9\u81ea\u56de\u5f52\u8bed\u8a00\u6a21\u578b\u7684\u5168\u9762\u66ff\u4ee3\u3002\u9996\u5148\uff0cAR Transformer \u5728 LM1B \u548c OWT \u7684 PPL \u4ecd\u66f4\u4f4e\uff0c\u4f8b\u5982 LM1B \u4e3a 22.8\u3001OWT \u4e3a 17.5\u3002\u5176\u6b21\uff0cLangFlow \u7684\u9ad8\u8d28\u91cf\u91c7\u6837\u4ecd\u9700\u8981\u8f83\u591a ODE steps\uff0c\u5c11\u6b65\u751f\u6210\u8fd8\u4f9d\u8d56\u672a\u6765\u7684 distillation\u3002\u7b2c\u4e09\uff0cOWT \u751f\u6210\u6837\u672c\u7684 entropy \u504f\u4f4e\uff0c\u4f5c\u8005\u4e5f\u627f\u8ba4\u8fd9\u53ef\u80fd\u53cd\u6620\u5168\u5c40\u8bcd\u9891\u504f\u7f6e\uff0c\u4ecd\u9700\u8981\u66f4\u7ec6\u7684\u8d28\u91cf\u5206\u6790\u3002<\/p>\n\n\n\n<p>\u8fd9\u7bc7\u8bba\u6587\u6700\u503c\u5f97\u5b66\u4e60\u7684\u5730\u65b9\uff0c\u4e0d\u662f\u67d0\u4e00\u4e2a\u6307\u6807\u5237\u65b0\uff0c\u800c\u662f\u5b83\u628a\u8fde\u7eed\u6269\u6563\u8bed\u8a00\u5efa\u6a21\u4e2d\u51e0\u4e2a\u539f\u672c\u5206\u6563\u7684\u95ee\u9898\u8fde\u6210\u4e86\u95ed\u73af\uff1a\u5982\u4f55\u8bad\u7ec3\u3001\u5982\u4f55\u8bc4\u4f30\u3001\u5982\u4f55\u8c03\u5ea6\u566a\u58f0\u3001\u5982\u4f55\u91c7\u6837\u3001\u5982\u4f55\u907f\u514d embedding collapse\u3002\u5bf9\u4e8e\u5173\u6ce8 diffusion LLM\u3001\u975e\u81ea\u56de\u5f52\u751f\u6210\u3001\u53ef\u7f16\u8f91\u6587\u672c\u751f\u6210\u548c\u5c11\u6b65\u751f\u6210\u7684\u4eba\u6765\u8bf4\uff0cLangFlow \u662f\u4e00\u7bc7\u503c\u5f97\u91cd\u70b9\u770b\u7684\u57fa\u7840\u8bba\u6587\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u8bba\u6587\u6807\u9898\uff1aLangFlow: Continuous Diffusion Rivals Discrete in  &hellip; <a href=\"http:\/\/139.9.1.231\/index.php\/2026\/07\/11\/langflow-continuous-diffusion-rivals-discrete-in-language-modeling\/\" class=\"more-link\">\u7ee7\u7eed\u9605\u8bfb<span class=\"screen-reader-text\">LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[29,4,9,34],"tags":[],"_links":{"self":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/31040"}],"collection":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/comments?post=31040"}],"version-history":[{"count":4,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/31040\/revisions"}],"predecessor-version":[{"id":31377,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/31040\/revisions\/31377"}],"wp:attachment":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/media?parent=31040"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/categories?post=31040"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/tags?post=31040"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}