{"id":7258,"date":"2022-09-07T11:17:00","date_gmt":"2022-09-07T03:17:00","guid":{"rendered":"http:\/\/139.9.1.231\/?p=7258"},"modified":"2022-09-03T11:18:12","modified_gmt":"2022-09-03T03:18:12","slug":"python_curlimg","status":"publish","type":"post","link":"http:\/\/139.9.1.231\/index.php\/2022\/09\/07\/python_curlimg\/","title":{"rendered":"Python\u722c\u866b\uff1a\u5e38\u7528\u7684\u722c\u866b\u5de5\u5177\u6c47\u603b"},"content":{"rendered":"\n

\u6700\u8fd1\u9700\u8981\u8dd1\u4e00\u4e2a\u98ce\u683c\u8fc1\u79fbcyclegan\u9879\u76ee\uff0c\u8fd9\u4e2a\u5e76\u4e0d\u96be\uff0cgithub\u4e0a\u968f\u4fbfsearch\u4e00\u4e2a\u5c31\u53ef\u4ee5\uff0c\u4f46\u662f\u6570\u636e\u96c6\u5f88\u662f\u5934\u75bc\uff0c\u6ca1\u6709\u6bd4\u8f83\u5408\u9002\u7684\u6570\u636e\u96c6\uff0c\u56e0\u6b64\u9700\u8981\u81ea\u5df1\u5728\u7f51\u4e0a\u5bfb\u627e\u4e00\u4e9b\u56fe\u7247\uff0c\u4f46\u5982\u679c\u4e0d\u4f7f\u7528\u722c\u866b\u722c\u6570\u636e\uff0c\u4e0d\u77e5\u9053\u8981\u5230\u7334\u5e74\u9a6c\u6708\uff0c\u56e0\u6b64\u9700\u8981\u4f7f\u7528\u722c\u866b\u722c\u53d6\u8c37\u6b4c\u3001\u767e\u5ea6\u3001\u4ee5\u53ca\u4e00\u4e9b\u56fe\u7247\u7f51\u7ad9\u7684\u56fe\u7247\uff0c\u4e4b\u524d\u5012\u662f\u5b66\u8fc7request\u5e93\uff0c\u4f46\u6ca1\u600e\u4e48\u7528\u8fc7\uff0c\u56e0\u6b64\u5148\u5f00\u4e2a\u5e16\u5b50\uff0c\u8bb0\u5f55\u4e0b\u76f8\u5173\u77e5\u8bc6\u3002<\/p>\n\n\n\n\n\n

\u722c\u866b\u6574\u4f53\u601d\u8def\uff1a\u9875\u9762\u4e0b\u8f7d –> \u9875\u9762\u89e3\u6790 –> \u6570\u636e\u5b58\u50a8<\/strong><\/p>\n\n\n\n

\u4e00\u3001\u9875\u9762\u4e0b\u8f7d\u5668<\/h2>\n\n\n\n
 requests(\u5fc5\u5b66)\n      \n     Requests: HTTP for Humans\u2122<\/a><\/strong><\/code><\/pre>\n\n\n\n
  1. python\u722c\u866b\u5165\u95e8requests\u6a21\u5757<\/a><\/li>
  2. Python\u722c\u866b\uff1arequests\u5e93\u57fa\u672c\u4f7f\u7528<\/a><\/li>
  3. Python\u722c\u866b\uff1a\u4f7f\u7528requests\u5e93\u4e0b\u8f7d\u5927\u6587\u4ef6<\/a><\/li>
  4. Python\u722c\u866b\uff1arequests\u591a\u8fdb\u7a0b\u722c\u53d6\u732b\u773c\u7535\u5f71\u699c\u5355<\/a><\/li>
  5. requests InsecureRequestWarning: Unverified HTTPS request is being made.<\/a><\/li><\/ol>\n\n\n\n
    1. scrapy
      1. Python\u7f51\u7edc\u722c\u866b\u4e4bscrapy\u6846\u67b6<\/a><\/li>
      2. scrapy\u5b66\u4e60<\/a><\/li>
      3. Python\u722c\u866b\uff1a\u5173\u4e8escrapy\u6a21\u5757\u7684\u8bf7\u6c42\u5934<\/a><\/li>
      4. Python\u722c\u866b\uff1ascrapy\u6846\u67b6\u8bf7\u6c42\u53c2\u6570meta\u3001headers\u3001cookies\u4e00\u63a2\u7a76\u7adf<\/a><\/li>
      5. Python\u722c\u866b\uff1ascrapy\u8f85\u52a9\u529f\u80fd\u5b9e\u7528\u51fd\u6570<\/a><\/li><\/ol><\/li>
      6. selenium+chrome + PhantomJS(\u6293\u53d6\u52a8\u6001\u7f51\u9875\uff0c\u4e0d\u63a8\u8350)
        1. mac\u4e0b\u5b89\u88c5selenium+phantomjs+chromedriver<\/a><\/li>
        2. Python\u722c\u866b\uff1aselenium\u6a21\u5757\u57fa\u672c\u4f7f\u7528<\/a><\/li>
        3. Python\u722c\u866bselenium\u6a21\u5757<\/a><\/li>
        4. Python\u722c\u866b\uff1aselenium\u548cChrome\u65e0\u5934\u6d4f\u89c8\u5668\u6293\u53d6\u70ef\u725b\u6570\u636e\u52a8\u6001\u7f51\u9875<\/a><\/li>
        5. Python\u722c\u866b\uff1a\u5229\u7528selenium\u722c\u53d6\u6dd8\u5b9d\u5546\u54c1\u4fe1\u606f<\/a><\/li>
        6. Python\u722c\u866b\uff1aselenium\u4f7f\u7528chrome\u548cPhantomJS\u5b9e\u7528\u53c2\u6570<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
          1. Splash(\u6293\u53d6\u52a8\u6001\u7f51\u9875\uff0c\u63a8\u8350)
            1. Python\u722c\u866b\uff1asplash\u7684\u5b89\u88c5\u4e0e\u7b80\u5355\u793a\u4f8b<\/a><\/li>
            2. Python\u722c\u866b\uff1asplash+requests\u7b80\u5355\u793a\u4f8b<\/a><\/li>
            3. Python\u722c\u866b\uff1ascrapy\u5229\u7528splash\u722c\u53d6\u52a8\u6001\u7f51\u9875<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
              \u603b\u7ed3\uff1a \u5bf9\u4e8e\u4e0b\u8f7d\u5668\u800c\u8a00\uff0cpython\u81ea\u5e26\u7684urllib\u5c31\u4e0d\u8981\u82b1\u65f6\u95f4\u53bb\u5b66\u4e86\uff0c\u5b66\u4e86\u5c31\u5fd8\uff0c\u76f4\u63a5requests<\/code>\u80fd\u6ee1\u8db3\u5927\u90e8\u5206\u6d4b\u8bd5+\u6293\u53d6\u9700\u6c42\uff0c\u8fdb\u9636\u5de5\u7a0b\u5316scrapy<\/code>\uff0c\u52a8\u6001\u7f51\u9875\u4f18\u5148\u627eAPI\u63a5\u53e3<\/code>\uff0c\u5982\u679c\u6709\u7b80\u5355\u52a0\u5bc6\u5c31\u7834\u89e3\uff0c\u5b9e\u5728\u56f0\u96be\u5c31\u4f7f\u7528splash\u6e32\u67d3<\/code><\/p>\n\n\n\n
              \u4e8c\u3001\u9875\u9762\u89e3\u6790\u5668<\/h2>\n\n\n\n
              1. BeautifulSoup(\u5165\u95e8\u7ea7)<\/strong>
                1. Python\u722c\u866b\u5165\u95e8BeautifulSoup\u6a21\u5757<\/a><\/strong><\/li>
                2. Beautiful Soup 4.4.0 \u6587\u6863\u00b6<\/a><\/strong><\/li><\/ol><\/li><\/ol>\n\n\n\n
                  1. pyquery \uff08\u7c7b\u4f3cjQuery\uff09
                    1. Python\u722c\u866b\uff1apyquery\u6a21\u5757\u89e3\u6790\u7f51\u9875<\/a><\/li><\/ol><\/li>
                    2. lxml
                      1. Python\u722c\u866b\uff1a\u4f7f\u7528lxml\u89e3\u6790\u7f51\u9875\u5185\u5bb9<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
                        1. parsel
                          1. Extract text using CSS or XPath selectors<\/a><\/li><\/ol><\/li>
                          2. scrapy\u7684Selector (\u5f3a\u70c8\u63a8\u8350, \u6bd4\u8f83\u9ad8\u7ea7\u7684\u5c01\u88c5\uff0c\u57fa\u4e8eparsel)
                            1. \u9009\u62e9\u5668(Selectors)<\/a><\/li>
                            2. python\u722c\u866b\uff1ascrapy\u6846\u67b6xpath\u548ccss\u9009\u62e9\u5668\u8bed\u6cd5<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
                              \u603b\u7ed3\uff1a \u5176\u5b9e\u89e3\u6790\u5668\u5b66\u4e60\u4e00\u4e2a\u5c31\u591f\u4e86\uff0c\u5176\u4ed6\u90fd\u4e0d\u7528\u5b66\uff0c\u5f88\u591a\u57f9\u8bad\u4f1a\u6559\u4f60\u4ece\u4e0a\u5230\u4e0b\u7684\u5b66\u4e60\uff0c\u6211\u4e0d\u662f\u5f88\u63a8\u8350\uff0c\u76f4\u63a5\u5b66\u4e60scrapy\u7684Selector<\/code> \u5c31\u884c\uff0c\u7b80\u5355\u3001\u76f4\u63a5\u3001\u9ad8\u6548<\/p>\n\n\n\n
                              \u4e09\u3001\u6570\u636e\u5b58\u50a8<\/h2>\n\n\n\n
                              1. txt\u6587\u672c
                                1. Python\u5168\u6808\u4e4b\u8def\uff1a\u6587\u4ef6file\u5e38\u7528\u64cd\u4f5c<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
                                  1. csv\u6587\u4ef6
                                    1. python\u8bfb\u53d6\u5199\u5165csv\u6587\u4ef6<\/a><\/li><\/ol><\/li>
                                    2. sqlite3 \uff08python\u81ea\u5e26\uff09
                                      1. Python\u7f16\u7a0b\uff1a\u4f7f\u7528\u6570\u636e\u5e93sqlite3<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
                                        1. MySQL
                                          1. SQL\uff1apymysql\u6a21\u5757\u8bfb\u5199mysql\u6570\u636e<\/a><\/li><\/ol><\/li>
                                          2. MongoDB
                                            1. Python\u7f16\u7a0b\uff1amongodb\u7684\u57fa\u672c\u589e\u5220\u6539\u67e5\u64cd\u4f5c<\/a><\/li><\/ol><\/li><\/ol>\n\n\n\n
                                              \u603b\u7ed3\uff1a \u6570\u636e\u5b58\u50a8\u6ca1\u6709\u4ec0\u4e48\u53ef\u6df1\u7a76\u7684\uff0c\u6309\u7167\u4e1a\u52a1\u9700\u6c42\u6765\u5c31\u884c\uff0c\u4e00\u822c\u5feb\u901f\u6d4b\u8bd5\u4f7f\u7528MongoDB<\/code>\uff0c\u4e1a\u52a1\u4f7f\u7528MySQL<\/code><\/p>\n\n\n\n
                                              \u56db\u3001\u5176\u4ed6\u5de5\u5177<\/h2>\n\n\n\n
                                              1. execjs \uff1a\u6267\u884cjs Python\u722c\u866b\uff1aexecjs\u5728python\u4e2d\u8fd0\u884cjavascript\u4ee3\u7801<\/a><\/li>
                                              2. pyv8: \u6267\u884cjs mac\u5b89\u88c5pyv8\u6a21\u5757-JavaScript\u7ffb\u8bd1\u6210python<\/a><\/li>
                                              3. html5lib 1. Python\u722c\u866b\uff1ascrapy\u5229\u7528html5lib\u89e3\u6790\u4e0d\u89c4\u8303\u7684html\u6587\u672c<\/a><\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"
                                                \u6700\u8fd1\u9700\u8981\u8dd1\u4e00\u4e2a\u98ce\u683c\u8fc1\u79fbcyclegan\u9879\u76ee\uff0c\u8fd9\u4e2a\u5e76\u4e0d\u96be\uff0cgithub\u4e0a\u968f\u4fbfsearch\u4e00\u4e2a\u5c31\u53ef\u4ee5\uff0c\u4f46\u662f\u6570\u636e\u96c6\u5f88 … \u7ee7\u7eed\u9605\u8bfbPython\u722c\u866b\uff1a\u5e38\u7528\u7684\u722c\u866b\u5de5\u5177\u6c47\u603b<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8],"tags":[],"_links":{"self":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/7258"}],"collection":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/comments?post=7258"}],"version-history":[{"count":13,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/7258\/revisions"}],"predecessor-version":[{"id":7271,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/posts\/7258\/revisions\/7271"}],"wp:attachment":[{"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/media?parent=7258"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/categories?post=7258"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/139.9.1.231\/index.php\/wp-json\/wp\/v2\/tags?post=7258"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}