Python中scrapy下载保存图片的方法(python,scrapy,移动开发)

在日常爬虫练习中，我们爬取到的数据需要进行保存操作，在scrapy中我们可以使用ImagesPipeline这个类来进行相关操作，这个类是scrapy已经封装好的了，我们直接拿来用即可。

在使用ImagesPipeline下载图片数据时，我们需要对其中的三个管道类方法进行重写，其中 ― get_media_request 是对图片地址发起请求

― file path 是返回图片名称

― item_completed 返回item,将其返回给下一个即将被执行的管道类

那具体代码是什么样的呢，首先我们需要在pipelines.py文件中，导入ImagesPipeline类，然后重写上述所说的3个方法：

fromscrapy.pipelines.imagesimportImagesPipelineimportscrapyimportosclassImgsPipLine(ImagesPipeline):defget_media_requests(self,item,info):yieldscrapy.Request(url=item["img_src"],meta={"item":item})#返回图片名称即可deffile_path(self,request,response=None,info=None):item=request.meta["item"]print("########",item)filePath=item["img_name"]returnfilePathdefitem_completed(self,results,item,info):returnitem

方法定义好后，我们需要在settings.py配置文件中进行设置，一个是指定图片保存的位置IMAGES_STORE = "D:ImgPro"，然后就是启用“ImgsPipLine”管道，

ITEM_PIPELINES={"imgPro.pipelines.ImgsPipLine":300,#300代表优先级，数字越小优先级越高}

设置完成后，我们运行程序后就可以看到“D:ImgPro”下保存成功的图片。

完整代码如下：

spider文件代码：

#-*-coding:utf-8-*-importscrapyfromimgPro.itemsimportImgproItemclassImgSpider(scrapy.Spider):name="img"allowed_domains=["www.521609.com"]start_urls=["http://www.521609.com/daxuemeinv/"]defparse(self,response):#解析图片地址和图片名称li_list=response.xpath("//div[@class="index_imglist_center"]/ul/li")forliinli_list:item=ImgproItem()item["img_src"]="http://www.521609.com/"+li.xpath("./a[1]/img/@src").extract_first()item["img_name"]=li.xpath("./a[1]/img/@alt").extract_first()+".jpg"#print("***********")#print(item)yielditem

items.py文件

importscrapyclassImgproItem(scrapy.Item):#definethefieldsforyouritemherelike:#name=scrapy.Field()img_src=scrapy.Field()img_name=scrapy.Field()

pipelines.py文件

fromscrapy.pipelines.imagesimportImagesPipelineimportscrapyimportosfromimgPro.settingsimportIMAGES_STOREasIMGSclassImgsPipLine(ImagesPipeline):defget_media_requests(self,item,info):yieldscrapy.Request(url=item["img_src"],meta={"item":item})#返回图片名称即可deffile_path(self,request,response=None,info=None):item=request.meta["item"]print("########",item)filePath=item["img_name"]returnfilePathdefitem_completed(self,results,item,info):returnitem

settings.py文件

importrandomBOT_NAME="imgPro"SPIDER_MODULES=["imgPro.spiders"]NEWSPIDER_MODULE="imgPro.spiders"IMAGES_STORE="D:ImgPro"#文件保存路径LOG_LEVEL="WARNING"ROBOTSTXT_OBEY=False#设置user-agentUSER_AGENTS_LIST=["Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.1(KHTML,likeGecko)Chrome/22.0.1207.1Safari/537.1","Mozilla/5.0(X11;CrOSi6862268.111.0)AppleWebKit/536.11(KHTML,likeGecko)Chrome/20.0.1132.57Safari/536.11","Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/536.6(KHTML,likeGecko)Chrome/20.0.1092.0Safari/536.6","Mozilla/5.0(WindowsNT6.2)AppleWebKit/536.6(KHTML,likeGecko)Chrome/20.0.1090.0Safari/536.6","Mozilla/5.0(WindowsNT6.2;WOW64)AppleWebKit/537.1(KHTML,likeGecko)Chrome/19.77.34.5Safari/537.1","Mozilla/5.0(X11;Linuxx86_64)AppleWebKit/536.5(KHTML,likeGecko)Chrome/19.0.1084.9Safari/536.5","Mozilla/5.0(WindowsNT6.0)AppleWebKit/536.5(KHTML,likeGecko)Chrome/19.0.1084.36Safari/536.5","Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1063.0Safari/536.3","Mozilla/5.0(WindowsNT5.1)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1063.0Safari/536.3","Mozilla/5.0(Macintosh;IntelMacOSX10_8_0)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1063.0Safari/536.3","Mozilla/5.0(WindowsNT6.2)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1062.0Safari/536.3","Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1062.0Safari/536.3","Mozilla/5.0(WindowsNT6.2)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1061.1Safari/536.3","Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1061.1Safari/536.3","Mozilla/5.0(WindowsNT6.1)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1061.1Safari/536.3","Mozilla/5.0(WindowsNT6.2)AppleWebKit/536.3(KHTML,likeGecko)Chrome/19.0.1061.0Safari/536.3","Mozilla/5.0(X11;Linuxx86_64)AppleWebKit/535.24(KHTML,likeGecko)Chrome/19.0.1055.1Safari/535.24","Mozilla/5.0(WindowsNT6.2;WOW64)AppleWebKit/535.24(KHTML,likeGecko)Chrome/19.0.1055.1Safari/535.24"]USER_AGENT=random.choice(USER_AGENTS_LIST)DEFAULT_REQUEST_HEADERS={"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language":"en",#"User-Agent":"Mozilla/5.0(WindowsNT10.0;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/58.0.3029.110Safari/537.36","User-Agent":USER_AGENT}#启动pipeline管道ITEM_PIPELINES={"imgPro.pipelines.ImgsPipLine":300,}