目錄
項(xiàng)目一 Python 基礎(chǔ)認(rèn)知 ····················································································.1
任務(wù)一 Python 概述 ·······································································································.1
一、Python 簡介 ······································································································.1
二、安裝Python ······································································································.2
三、安裝PyCharm ···································································································.6
四、Python 語法規(guī)范 ·······························································································.11
任務(wù)二 Python 命令的組成 ·····························································································.13
一、基本符號(hào) ·········································································································.14
二、常量與變量 ······································································································.16
三、數(shù)據(jù)類型 ·········································································································.19
四、功能符號(hào) ·········································································································.24
任務(wù)三 程序結(jié)構(gòu) ·········································································································.26
一、表達(dá)式語句 ······································································································.26
二、順序結(jié)構(gòu) ·········································································································.27
三、選擇結(jié)構(gòu) ·········································································································.28
四、循環(huán)結(jié)構(gòu) ·········································································································.30
五、條件表達(dá)式 ······································································································.31
六、程序的流程控制 ································································································.32
項(xiàng)目實(shí)戰(zhàn) ·····················································································································.33
實(shí)戰(zhàn) 輸出百度網(wǎng)址 ································································································.33
項(xiàng)目二 網(wǎng)絡(luò)爬蟲基礎(chǔ)認(rèn)知 ················································································.35
任務(wù)一 網(wǎng)絡(luò)爬蟲概述 ···································································································.35
一、網(wǎng)絡(luò)爬蟲的基本原理 ··························································································.36
二、網(wǎng)絡(luò)爬蟲系統(tǒng)框架 ·····························································································.37
三、爬行策略 ·········································································································.37
四、網(wǎng)絡(luò)爬蟲的分類 ································································································.38
五、開源網(wǎng)絡(luò)爬蟲框架/項(xiàng)目 ······················································································.39
任務(wù)二 HTTP ·············································································································.41
一、HTTP 的工作原理 ·····························································································.41
二、Urllib 模塊庫 ···································································································.42
三、URL 定義 ·······································································································.43
四、URL 編碼設(shè)置 ·································································································.47
任務(wù)三 網(wǎng)頁請(qǐng)求過程 ···································································································.50
一、發(fā)送請(qǐng)求報(bào)文 ··································································································.51
二、返回響應(yīng) ········································································································.52
三、HTTP 消息 ······································································································.53
項(xiàng)目實(shí)戰(zhàn) ·····················································································································.54
實(shí)戰(zhàn)一 搜索商品網(wǎng)址 ····························································································.54
實(shí)戰(zhàn)二 搜索食品價(jià)格網(wǎng)址 ······················································································.56
項(xiàng)目三 Urllib 請(qǐng)求模塊庫的應(yīng)用 ········································································.58
任務(wù)一 發(fā)送網(wǎng)頁請(qǐng)求 ···································································································.58
一、基本HTTP 請(qǐng)求 ·······························································································.58
二、Request 網(wǎng)絡(luò)請(qǐng)求 ·····························································································.66
三、設(shè)置請(qǐng)求頭 ·····································································································.67
四、Handler 方法發(fā)送請(qǐng)求 ·······················································································.69
五、設(shè)置代理IP ····································································································.71
六、身份驗(yàn)證 ········································································································.73
任務(wù)二 網(wǎng)頁下載 ·········································································································.77
一、網(wǎng)頁結(jié)構(gòu) ········································································································.77
二、寫入網(wǎng)頁文件 ··································································································.77
三、網(wǎng)頁文件下載 ··································································································.79
項(xiàng)目實(shí)戰(zhàn) ·····················································································································.82
實(shí)戰(zhàn)一 下載Python 學(xué)習(xí)網(wǎng)址 ··················································································.82
實(shí)戰(zhàn)二 下載公司網(wǎng)頁HTML 文件 ············································································.85
項(xiàng)目四 安裝Urllib3 請(qǐng)求模塊庫并發(fā)送請(qǐng)求 ··························································.87
任務(wù)一 安裝Urllib3 請(qǐng)求模塊庫 ······················································································.87
一、安裝Anaconda ·································································································.87
二、安裝Urllib3 模塊庫 ···························································································.92
任務(wù)二 發(fā)送請(qǐng)求 ·········································································································.95
一、創(chuàng)建代理對(duì)象 ··································································································.96
二、請(qǐng)求方法 ········································································································.98
三、定義請(qǐng)求頭 ·····································································································.99
四、設(shè)置代理IP ···································································································.101
五、自動(dòng)重試 ·······································································································.102
六、重定向 ··········································································································.103
項(xiàng)目實(shí)戰(zhàn) ····················································································································.104
實(shí)戰(zhàn) 發(fā)送請(qǐng)求訪問淘寶 ························································································.104
項(xiàng)目五 Requests 請(qǐng)求模塊庫的應(yīng)用 ·································································.106
任務(wù)一 網(wǎng)頁請(qǐng)求 ·······································································································.106
一、標(biāo)準(zhǔn)的HTTP 請(qǐng)求 ···························································································.107
二、返回響應(yīng)消息 ·································································································.109
三、JSON 格式數(shù)據(jù) ·······························································································.114
任務(wù)二 發(fā)送請(qǐng)求方法 ·································································································.117
一、發(fā)送GET 請(qǐng)求方法 ·························································································.118
二、發(fā)送POST 請(qǐng)求方法 ························································································.120
三、其他請(qǐng)求方法 ·································································································.125
任務(wù)三 復(fù)雜網(wǎng)絡(luò)請(qǐng)求 ·································································································.126
一、復(fù)雜請(qǐng)求頭 ····································································································.126
二、上傳文件 ·······································································································.129
三、Cookies 驗(yàn)證 ··································································································.131
四、會(huì)話保持 ·······································································································.131
任務(wù)四 異常處理 ·······································································································.133
一、try-except 語句 ································································································.133
二、Urllib 異常處理模塊 ·························································································.134
三、Urllib3 異常處理模塊 ·······················································································.135
四、request 異常處理模塊 ·······················································································.135
項(xiàng)目實(shí)戰(zhàn) ···················································································································.138
實(shí)戰(zhàn) 爬取豆瓣最受歡迎的影評(píng)網(wǎng)址 ·········································································.138
項(xiàng)目六 解析網(wǎng)頁 ···························································································.141
任務(wù)一 正則表達(dá)式解析網(wǎng)頁 ························································································.141
一、正則表達(dá)式模式 ······························································································.142
二、使用re 模塊實(shí)現(xiàn)正則表達(dá)式 ··············································································.143
三、字符串查找 ····································································································.144
四、字符串替換 ····································································································.148
五、字符串分割 ····································································································.149
任務(wù)二 XPath 解析網(wǎng)頁 ·······························································································.150
一、XPath 概述 ····································································································.150
二、XPath 網(wǎng)頁解析 ······························································································.152
三、獲取節(jié)點(diǎn)信息 ·································································································.154
四、節(jié)點(diǎn)關(guān)系 ·······································································································.160
五、查找節(jié)點(diǎn)信息 ·································································································.162
六、屬性節(jié)點(diǎn) ·······································································································.163
七、XPath 運(yùn)算符 ·································································································.165
八、XML 節(jié)點(diǎn)軸 ··································································································.168
任務(wù)三 BeautifulSoup 解析網(wǎng)頁 ······················································································.170
一、安裝BeautifulSoup ···························································································.171
二、創(chuàng)建BeautifulSoup 對(duì)象 ····················································································.171
三、通過屬性獲取節(jié)點(diǎn)內(nèi)容 ·····················································································.173
四、根據(jù)節(jié)點(diǎn)關(guān)系獲取節(jié)點(diǎn) ·····················································································.176
五、查找節(jié)點(diǎn)內(nèi)容 ·································································································.178
六、通過CSS 選擇器查找節(jié)點(diǎn)內(nèi)容 ···········································································.182
項(xiàng)目實(shí)戰(zhàn) ····················································································································.183
實(shí)戰(zhàn)一 獲取查詢網(wǎng)中河北省石家莊市的郵編區(qū)號(hào) ·······················································.183
實(shí)戰(zhàn)二 爬取銷售熱門圖書名稱 ···············································································.186
實(shí)戰(zhàn)三 下載銷售熱門圖書的圖片 ············································································.188
項(xiàng)目七 Scrapy 網(wǎng)絡(luò)爬蟲框架 ···········································································.190
任務(wù)一 Scrapy 網(wǎng)絡(luò)爬蟲框架基礎(chǔ)認(rèn)知 ·············································································.190
一、Scrapy 網(wǎng)絡(luò)爬蟲框架基礎(chǔ) ··················································································.190
二、Scrapy 常用命令 ······························································································.192
三、創(chuàng)建Scrapy 項(xiàng)目 ·····························································································.193
任務(wù)二 使用模板創(chuàng)建Spider 文件 ··················································································.194
一、創(chuàng)建網(wǎng)絡(luò)爬蟲文件命令 ·····················································································.195
二、創(chuàng)建basic 模板文件 ·························································································.196
三、創(chuàng)建crawl 模板文件 ·························································································.197
四、創(chuàng)建csvfeed 模板文件 ······················································································.198
五、創(chuàng)建xmlfeed 模板文件 ······················································································.198
任務(wù)三 Scrapy 網(wǎng)絡(luò)爬蟲文件 ·························································································.199
一、Spider 類 ·······································································································.199
二、配置網(wǎng)絡(luò)爬蟲 ·································································································.201
三、啟動(dòng)網(wǎng)絡(luò)爬蟲 ·································································································.202
四、提取數(shù)據(jù) ·······································································································.207
項(xiàng)目實(shí)戰(zhàn) ····················································································································.209
實(shí)戰(zhàn) 提取景區(qū)名稱 ······························································································.209