{"id":8906,"date":"2022-10-08T21:17:59","date_gmt":"2022-10-08T13:17:59","guid":{"rendered":"http:\/\/139.9.1.231\/?p=8906"},"modified":"2022-10-11T19:58:28","modified_gmt":"2022-10-11T11:58:28","slug":"cyclemlp","status":"publish","type":"post","link":"http:\/\/139.9.1.231\/index.php\/2022\/10\/08\/cyclemlp\/","title":{"rendered":"CycleMLP\uff1a\u7528\u4e8e\u5bc6\u96c6\u9884\u6d4b\u7684\u7c7b\u4f3c MLP \u7684\u67b6\u6784"},"content":{"rendered":"\n

paper\uff1ahttps:\/\/arxiv.org\/abs\/2107.10224<\/a><\/p>\n\n\n\n

\u4f5c\u8005\u5355\u4f4d\uff1a\u9999\u6e2f\u5927\u5b66, \u5546\u6c64\u79d1\u6280
\u4ee3\u7801\uff1ahttps:\/\/github.com\/ShoufaChen\/CycleMLP<\/a><\/p>\n\n\n\n

\u6838\u5fc3\uff1a\u7528 Cycle-FC\u6765\u66ff\u6362Spatial FC\uff08\u8ba1\u7b97\u91cf\u5927\u4e14\u7f51\u7edc\u5bf9\u4e8e\u4e0d\u540c\u56fe\u50cf\u5206\u8fa8\u7387\u7684\u8f93\u5165\u4e0d\u53ef\u63a5\u53d7\uff0c\u4e14\u4e0d\u80fd\u7528\u4e8e\u4e0b\u6e38\u4efb\u52a1\uff09<\/strong><\/p>\n\n\n\n

\u672c\u6587\u63d0\u51fa\u4e86\u4e00\u4e2a\u7b80\u5355\u7684 MLP-like \u7684\u67b6\u6784 CycleMLP\uff0c\u5b83\u662f\u89c6\u89c9\u8bc6\u522b\u548c\u5bc6\u96c6\u9884\u6d4b\u7684\u901a\u7528\u4e3b\u5e72\uff0c\u4e0d\u540c\u4e8e\u73b0\u4ee3 MLP \u67b6\u6784\uff0c\u4f8b\u5982 MLP-Mixer\u3001ResMLP \u548c gMLP\uff0c\u5176\u67b6\u6784\u4e0e\u56fe\u50cf\u5927\u5c0f\u76f8\u5173\uff0c\u56e0\u6b64\u662f\u5728\u76ee\u6807\u68c0\u6d4b\u548c\u5206\u5272\u4e2d\u4e0d\u53ef\u884c\u3002<\/p>\n\n\n\n

\u4e0e\u73b0\u4ee3\u65b9\u6cd5\u76f8\u6bd4\uff0cCycleMLP \u6709\u4e24\u4e2a\u4f18\u52bf\u3002<\/p>\n\n\n\n

(1) \u53ef\u4ee5\u5e94\u5bf9\u5404\u79cd\u56fe\u50cf\u5c3a\u5bf8\u3002<\/p>\n\n\n\n

(2) \u5229\u7528\u5c40\u90e8\u7a97\u53e3\u5b9e\u73b0\u5bf9\u56fe\u50cf\u5927\u5c0f\u7684\u7ebf\u6027\u8ba1\u7b97\u590d\u6742\u5ea6\u3002<\/p>\n\n\n\n

\u76f8\u6bd4\u4e4b\u4e0b\uff0c\u4ee5\u524d\u7684 MLP \u5177\u6709\u4e8c\u6b21\u8ba1\u7b97\uff0c\u56e0\u4e3a\u5b83\u4eec\u5177\u6709\u5b8c\u5168\u7684\u7a7a\u95f4\u8fde\u63a5\u3002<\/p>\n\n\n\n

\u5355\u4e2a CycleMLP Block \u4f9d\u7136\u662f\u5206\u4e3a Token-mixing MLP \u548c Channel mixing MLP\uff0c\u5176\u4e2d\u4f5c\u8005\u4e3b\u8981\u7684\u8d21\u732e\u70b9\u5728\u4e8e\u66ff\u6362 MLP-mixer \u7684 Token-mixing MLP \u4e3a Cycle-FC\u3002\u6240\u4ee5\u6574\u4e2a CycleMLP Block \u53ef\u4ee5\u63cf\u8ff0\u4e3a\uff1a<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

\u4f55\u4e3a Cycle-FC<\/strong> \uff1f\u8981\u56de\u7b54\u8fd9\u4e2a\u95ee\u9898\uff0c\u6211\u4eec\u9996\u5148\u6765\u56de\u987e\u4e00\u4e0b Channel FC \u4ee5\u53ca Spatial FC.<\/p>\n\n\n\n

Channel FC \u5373\u901a\u9053\u65b9\u5411\u7684\u6620\u5c04\uff0c\u7b49\u6548\u4e0e1\u00d71 \u5377\u79ef\uff0c\u5176\u53c2\u6570\u91cf\u4e0e\u56fe\u50cf\u5c3a\u5bf8\u65e0\u5173\uff0c\u800c\u4e0e\u901a\u9053\u6570\uff08token \u7ef4\u5ea6\uff09\u6709\u5173\u3002\u5047\u8bbe\u8f93\u5165\u8f93\u51fa\u7279\u5f81\u56fe\u5c3a\u5bf8\u4e00\u81f4\uff0c\u5219\u53c2\u6570\u91cf\u4e3a C^2\uff0c\u5176\u4e2d C \u4e3a\u901a\u9053\u6570\u3002\u800c\u8ba1\u7b97\u91cf\u5219\u4e3a HWC^2\uff0c\u5176\u4e2d H W \u5206\u522b\u4e3a\u7279\u5f81\u56fe\u7684\u9ad8\u548c\u5bbd\u3002\u5982\u679c\u53ea\u8003\u8651\u8ba1\u7b97\u91cf\u4e0e\u56fe\u50cf\u5c3a\u5bf8\u7684\u5f71\u54cd\u7684\u8bdd\uff0c\u5219\u4e3a O ( H W ) \u3002<\/p>\n\n\n\n

Spatial FC \u5373 MLP-Mixer \u4f7f\u7528\u7684 Token-mixing \u5168\u8fde\u63a5\u5c42\uff0c\u5728\u8fd9\u91cc\u6211\u4eec\u90fd\u662f\u53ea\u8003\u8651\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\uff0c\u5219\u5176\u5b9e\u73b0\u7684\u662f H W \u2212 > H W \u7684\u6620\u5c04\uff0c\u53c2\u6570\u91cf\u4e3a H^2W^2\uff0c\u8ba1\u7b97\u91cf\u4e5f\u4e3a H 2 W 2 C H^2W^2C\uff0c\u5982\u679c\u53ea\u8003\u8651\u8ba1\u7b97\u91cf\u4e0e\u56fe\u50cf\u5c3a\u5bf8\u7684\u5f71\u54cd\u7684\u8bdd\uff0c\u5219\u4e3a O(H^2W^2)\u3002\u5e76\u4e14HW \u5927\u5c0f\u56fa\u5b9a\uff0c\u7f51\u7edc\u5bf9\u4e8e\u4e0d\u540c\u56fe\u50cf\u5206\u8fa8\u7387\u7684\u8f93\u5165\u4e0d\u53ef\u63a5\u53d7\uff0c\u4e14\u4e0d\u80fd\u7528\u4e8e\u4e0b\u6e38\u4efb\u52a1\u4ee5\u53ca\u4f7f\u7528\u7c7b\u4f3c EfficientNetV2 \u7b49\u7684\u591a\u5206\u8fa8\u7387\u8bad\u7ec3\u7b56\u7565\u3002<\/p>\n\n\n\n

\u4e3a\u4ec0\u4e48\u6211\u4eec\u53ef\u4ee5\u5728\u590d\u6742\u5ea6\u5206\u6790\u65f6\u53ea\u8003\u8651 H W \u7684\u5f71\u54cd\u5462<\/strong>\uff1f\u56e0\u4e3a\u5728\u91d1\u5b57\u5854\u7ed3\u6784\u7684 MLP \u4e2d\uff0c\u901a\u5e38\u4e00\u5f00\u59cb\u7684 patch size \u4e3a 4\uff0c\u7136\u540e\u8f93\u5165\u5c3a\u5bf8\u4e3a 224\u00d7224\uff0c\u5219\u4e00\u5f00\u59cb\u7684 H = W = 56 = 224 \/ 4 \uff0c\u800c C = 64 \u6216\u8005 96 \uff0c\u6240\u4ee5C\u226aHW\u3002\u5982\u679c\u5bf9\u4e8e\u4e0b\u6e38\u4efb\u52a1\u800c\u8a00\uff0c\u4f8b\u5982\u8f93\u5165\u53d8\u4e3a\u4e86512\u00d7512\uff0c\u5219\u5b83\u4eec\u4e4b\u95f4\u7684\u5dee\u8ddd\u66f4\u5927\u4e86\u3002\u4e3a\u6b64\u5728\u8fd9\u91cc\u6211\u4eec\u53ef\u4ee5\u5728\u590d\u6742\u5ea6\u5206\u6790\u4e2d\u6682\u65f6\u53ea\u8003\u8651 H W  \u800c\u5ffd\u7565 C \u3002<\/p>\n\n\n\n

\u4e3a\u4e86\u540c\u65f6\u514b\u670d Spatial \u5bf9\u4e8e\u56fe\u50cf\u8f93\u5165\u5c3a\u5bf8\u654f\u611f\u4ee5\u53ca\u8ba1\u7b97\u91cf\u5927\u7684\u95ee\u9898\uff0c\u4f5c\u8005\u63d0\u51fa\u4e86 Cycle-FC\u3002\u5176\u53ea\u662f\u7528\u901a\u9053\u65b9\u5411\u7684\u6620\u5c04\u5e76\u4e14\u8ba1\u7b97\u91cf\u548c Channel FC \u4fdd\u6301\u4e00\u81f4\u3002\u5176\u8bf4\u767d\u4e86\u5c31\u662f\u4e0d\u65ad\u5730\u4ee5 [+1 0 -1 0 +1 0 -1 0 +1 \u2026] \u7684\u65b9\u5f0f\u79fb\u52a8\u7279\u5f81\u56fe\uff0c\u5c06\u4e0d\u540c\u7a7a\u95f4\u4f4d\u7f6e\u7684\u7279\u5f81\u5bf9\u9f50\u5230\u540c\u4e00\u4e2a\u901a\u9053\u4e0a\uff0c\u7136\u540e\u4f7f\u75281\u00d71 \u5377\u79ef\u3002<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
\"\"<\/figure>\n\n\n\n

\u56de\u5fc6 AS-MLP\uff0c\u5176\u91c7\u7528\u7684\u7279\u5f81\u56fe\u79fb\u52a8\u65b9\u5f0f\u5219\u4e3a [+1 0 -1 +1 0 -1 +1 0 -1] \u8fd9\u6837\u7684\u6210\u7ec4\u65b9\u5f0f\uff0cCycleMLP \u5219\u662f\u4f7f\u7528\u201c\u697c\u68af\u578b\u201d\u65b9\u5f0f\uff0c\u4f46\u662f\u5176\u601d\u60f3\u6ca1\u6709\u672c\u8d28\u4e0d\u540c\u3002\u6b64\u5916\uff0cAS-MLP \u786e\u5b9e\u5bf9\u7279\u5f81\u56fe\u8fdb\u884c\u4e86 Shift\uff0c\u5e76\u4e14\u91c7\u7528\u4e86 zero-padding\uff0c\u800c CycleMLP \u5728\u5177\u4f53\u5b9e\u73b0\u8fc7\u7a0b\u4e2d\u5219\u662f\u4f7f\u7528\u53ef\u53d8\u5f62\u5377\u79ef<\/strong>\u52a0\u4ee5\u5b9e\u73b0\u7684\u3002\u6211\u4e2a\u4eba\u5bf9\u4e8e AS-MLP \u4e0e CycleMLP \u7684\u7406\u89e3\u5982\u4e0b\u56fe\u6240\u793a\uff0c\u53ef\u89c1\u4ed6\u4eec\u5176\u5b9e\u6838\u5fc3\u601d\u60f3\u662f\u4e00\u81f4\u7684\u3002<\/p>\n\n\n\n

from torchvision.ops.deform_conv import deform_conv2d<\/code><\/p>\n\n\n\n

\"img3\"\/<\/figure>\n\n\n\n

CycleMLP \u4e0e AS-MLP \u53ea\u5e76\u884c H \u548c W \u65b9\u5411\u7684\u79fb\u52a8\u4e0d\u540c\uff0cCycleMLP \u5176\u5b9e\u662f\u4e09\u6761\u652f\u8def\u5e76\u884c\uff1aH \u65b9\u5411\uff0cW \u65b9\u5411\uff0c\u4ee5\u53ca\u4e0d\u79fb\u52a8\u7279\u5f81\u56fe\u505a\u901a\u9053\u65b9\u5411\u6620\u5c04\u3002\u6b64\u5916\uff0cAS-MLP \u5728\u4e00\u5f00\u59cb\u8fd8\u505a\u4e86\u4e00\u6b21 Channel Projection \u8fdb\u884c\u964d\u7ef4\u3002<\/p>\n\n\n\n

\"img5\"\/<\/figure>\n\n\n\n

CycleMLP \u6700\u7ec8\u4f7f\u7528\u7684\u548c ViP \u4e00\u6837\uff0c\u4f7f\u7528 Split Attention \u6765\u878d\u5408\u4e09\u6761\u652f\u8def<\/strong>\u3002<\/p>\n\n\n\n

class<\/strong> CycleMLP<\/strong>(nn.Module):\n    def<\/strong> __init__<\/strong>(self, dim, qkv_bias=<\/strong>False, qk_scale=<\/strong>None, attn_drop=<\/strong>0., proj_drop=<\/strong>0.):\n        super().__init__()\n        self.mlp_c =<\/strong> nn.Linear(dim, dim, bias=<\/strong>qkv_bias)\n\n        self.sfc_h =<\/strong> CycleFC(dim, dim, (1, 3), 1, 0)\n        self.sfc_w =<\/strong> CycleFC(dim, dim, (3, 1), 1, 0)\n\n        self.reweight =<\/strong> Mlp(dim, dim \/\/<\/strong> 4, dim *<\/strong> 3)\n\n        self.proj =<\/strong> nn.Linear(dim, dim)\n        self.proj_drop =<\/strong> nn.Dropout(proj_drop)\n\n    def<\/strong> forward<\/strong>(self, x):\n        B, H, W, C =<\/strong> x.shape\n        h =<\/strong> self.sfc_h(x.permute(0, 3, 1, 2)).permute(0, 2, 3, 1)\n        w =<\/strong> self.sfc_w(x.permute(0, 3, 1, 2)).permute(0, 2, 3, 1)\n        c =<\/strong> self.mlp_c(x)\n\n        a =<\/strong> (h +<\/strong> w +<\/strong> c).permute(0, 3, 1, 2).flatten(2).mean(2)\n        a =<\/strong> self.reweight(a).reshape(B, C, 3).permute(2, 0, 1).softmax(dim=<\/strong>0).unsqueeze(2).unsqueeze(2)\n\n        x =<\/strong> h *<\/strong> a[0] +<\/strong> w *<\/strong> a[1] +<\/strong> c *<\/strong> a[2]\n\n        x =<\/strong> self.proj(x)\n        x =<\/strong> self.proj_drop(x)\n\n        return<\/strong> x<\/code><\/pre>\n\n\n\n
\u6700\u540e\u63d0\u4e00\u53e5\uff0c\u4f5c\u8005\u5c06\u6295\u5f71\u533a\u95f4\u5b9a\u4e49\u4e3a\u662f Pseudo-Kernel<\/strong>\uff0c\u8fd9\u5176\u5b9e\u4e5f\u662f\u6211\u4eec\u5e38\u8bf4\u7684 \u611f\u53d7\u91ce<\/strong> \u4e00\u8bcd\u3002<\/p>\n\n\n\n
\"img4\"\/<\/figure><\/div>\n\n\n\n
2.2 \u6574\u4f53\u7f51\u7edc\u7ed3\u6784<\/h4>\n\n\n\n
CycleMLP \u7684 Patch Embedding \u4e5f\u5f88\u6709\u7279\u8272\uff0c\u4f7f\u7528\u5377\u79ef\u6838\u5927\u5c0f\u4e3a\u00a07\u00d77\u00a0\uff0c\u6b65\u957f\u4e3a 4 \u7684\u5377\u79ef\u3002\u540e\u7eed Hire-MLP \u5176\u5b9e\u4e5f\u662f\u8fd9\u6837\u8fdb\u884c\u7684 Patch Embedding\u3002\u76f8\u6bd4\u800c\u8a00 Swin \u4f7f\u7528\u5377\u79ef\u6838\u5927\u5c0f\u4e3a\u00a04\u00d74\uff0c\u6b65\u957f\u4e3a 4 \u7684\u5377\u79ef\u3002\u5728\u8fd1\u671f\u7684\u6211\u81ea\u5df1\u7684\u5c0f\u5b9e\u9a8c\u4e2d\u4e5f\u53d1\u73b0\uff1aPatch Embedding \u65f6\u5177\u6709\u91cd\u53e0\u4f1a\u66f4\u597d\uff0c\u8fd9\u6837\u53ef\u4ee5\u907f\u514d\u8fb9\u754c\u6548\u5e94\u5e76\u5728\u5c0f\u6570\u636e\u96c6\u4e0a\u63d0\u5347\u6027\u80fd<\/strong>\u3002CycleMLP \u4e2d\u95f4\u91c7\u7528\u591a\u9636\u6bb5\u91d1\u5b57\u5854\u6a21\u578b\uff0c\u603b\u5171\u5206\u4e3a 4 \u4e2a\u9636\u6bb5\uff0c\u6bcf\u4e2a\u9636\u6bb5\u4ea4\u66ff\u91cd\u590d\u4f7f\u7528 CycleMLP Block\u3002\u4e0b\u91c7\u6837\u4f7f\u7528\u5377\u79ef\u6838\u5927\u5c0f\u4e3a3\u00d73\uff0c\u6b65\u957f\u4e3a 2 \u7684\u5377\u79ef\uff0c\u8fd9\u6837\u505a\u4e5f\u6709\u91cd\u53e0<\/strong>\uff0cHire-MLP \u4e5f\u662f\u8fd9\u6837\u5b50\u54c8\u3002\u6700\u540e\u7ecf\u8fc7\u5168\u5c40\u6c60\u5316\u540e\u8fde\u63a5\u4e00\u4e2a\u5168\u8fde\u63a5\u5206\u7c7b\u5668\u5373\u53ef\u3002\u4f5c\u8005\u4e00\u5171\u63d0\u51fa\u6765\u4e86\u56db\u79cd\u914d\u7f6e\uff1a<\/p>\n\n\n\n
\"\u8bf7\u6dfb\u52a0\u56fe\u7247\u63cf\u8ff0\"\/<\/figure>\n\n\n\n
\u5728\u8fd9\u56db\u79cd\u914d\u7f6e\uff0cSi\u200b\u00a0\u6307 Patch Embedding \u4e2d\u7684 Patch size\uff0cCi\u200b\u00a0\u6307 Patch Embedding \u7684\u8f93\u51fa\u7f16\u7801\u7279\u5f81\u7ef4\u5ea6\uff0cE i \u200b\u00a0\u4e3a Channel-mixing MLP \u4e2d\u4e24\u4e2a\u5168\u8fde\u63a5\u5c42\u4e2d\u7b2c\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\u7684 expand radio\uff0cLi\u200b\u00a0\u5219\u662f\u4e0d\u540c Stage \u4e2d Block \u7684\u91cd\u590d\u6b21\u6570\u3002<\/p>\n\n\n\n
<\/a><\/a>3. \u4e0b\u6e38\u4efb\u52a1\u5b9e\u9a8c<\/h3>\n\n\n\n
CycleMLP \u65e8\u5728\u4e3a MLP \u6a21\u578b\u7684\u76ee\u6807\u68c0\u6d4b\u3001\u5b9e\u4f8b\u5206\u5272\u548c\u8bed\u4e49\u5206\u5272\u63d0\u4f9b\u4e00\u4e2a\u6709\u7ade\u4e89\u529b\u7684\u57fa\u7ebf\u3002\u4e0e AS-MLP \u4e0d\u540c\u4e4b\u5904\u5728\u4e8e\uff0cCycleMLP \u5728 ADE20K \u4e0a\u8fdb\u884c\u5b9e\u9a8c\uff0c\u800c AS-MLP \u5728 COCO \u4e0a\u8fdb\u884c\u7684\u5b9e\u9a8c\u3002\u8fd9\u771f\u7684\u662f\u5de7\u5408\uff0c\u8fd8\u662f\u6545\u610f\u907f\u5f00<\/strong>\uff1f\u4e0d\u6562\u95ee\u4e5f\u4e0d\u6562\u8bf4\u3002<\/p>\n\n\n\n
\u76ee\u6807\u68c0\u6d4b\u6027\u80fd\u8868\u73b0<\/strong>\uff1a\u76f8\u6bd4 PVT\uff0cCycleMLP \u90fd\u66f4\u5177\u6709\u4f18\u52bf\u3002<\/p>\n\n\n\n
\"\u8bf7\u6dfb\u52a0\u56fe\u7247\u63cf\u8ff0\"\/<\/figure>\n\n\n\n
\u8bed\u4e49\u5206\u5272\u6027\u80fd\u8868\u73b0<\/strong>\uff1a\u7279\u522b\u662f\uff0cCycleMLP \u5728 ADE20K val \u4e0a\u8fbe\u5230\u4e86 45.1 mIoU\uff0c\u4e0e Swin (45.2 mIOU) \u76f8\u5f53\u3002<\/p>\n\n\n\n
\"\u8bf7\u6dfb\u52a0\u56fe\u7247\u63cf\u8ff0\"\/<\/figure>\n\n\n\n
<\/a><\/a>4. \u6d88\u878d\u5b9e\u9a8c<\/h3>\n\n\n\n
\u4f5c\u8005\u4e00\u5171\u8fdb\u884c\u4e86\u4e09\u7ec4\u6d88\u878d\u5b9e\u9a8c\uff1a<\/p>\n\n\n\n