problem: when collecting a page, it may return empty content due to network reasons, but this collection record is recorded in the DupeFilter of redis, so that it cannot be collected again.
excuse me: how to manually remove the failed url from the xx:DupeFilter of redis during the writing process of redis.
finally got it.
introduces
from scrapy.utils.request import request_fingerprint
in spiders, manually determine whether the response meets the crawling requirements, and if not, delete the fingerprint.
from scrapy.utils.request import request_fingerprint
def parse(self,response):
ajaxT = json.loads(response.text)
if ajaxT['status'] == 'success':
-sharp
else:
-sharpredis
fp = request_fingerprint(response.request, include_headers=None)
self.server.srem(self.name + ':dupefilter', fp)scrapyapi5 because I originally wanted to get 100 ip, at a time and put it in the agent pool, but because the agent is unstable, it can not provide support for a long time. So I gave up the idea of getting 100 ip at once. 5request ...
when scrapy crawls a picture of a web page, the class that inherits ImagesPipelines is customized in the pipelines file. but the custom pipelines cannot be executed after running the program. Item cannot pass the following is a custom pipelines clas...
I set the crawler to run every 6 hours, and it did. The problem with is that it runs immediately after each point starts, and then executes every 6 hours. how do you stop it from running at the start of the point? ! @web Oh, it s all right. Jus...
when I crawl a page with scrapy, I find that I can only request one page at a time, but the posts on the official website and Baidu say that the concurrency can be controlled through CONCURRENT_REQUESTS , but I tried it or it didn t work? CONCURRENT_...
execute after entering the project, the error shows scrapy command not found , but I-sharpscrapy can be run, the scrapy crawl test crawler command can also be executed alone, only the scheduled command will appear scrapy:command not found ...
after crawling the navigation, the URL crawl that you want to continue in-depth navigation, and then the unified return value is written to xlsx < H1 >--coding: utf-8--< H1 > from lagou.items import LagouItem; import scrapy class LaGouSpider (...
scrapy.Request cannot enter callback code is as follows: def isIdentifyingCode(self, response): -sharp pass def get_identifying_code(self, headers): -sharp -sharp return scrapy.Req...
A nasty piece of html code that writes js in div. It s a keyboard paging code xpath found that the tagged content in is gone, like this I am China person what I get is: I am human. China does not have , and then some people say that my xpath ...
when scrapy saves data through Pipeline (in txt format), some data gbk codec can t encode character appears as follows. class TxtPipeline(object): def process_item(self,item,spider): path=os.getcwd() filename = path + dat...
scrapy tutorial: http: scrapy-chs.readthedocs. Environment: python3.6 + windows7 Project structure Directory: mySpider:scrapy crawl domz: there is no [dmoz] output as mentioned in the tutorial, is there any new file, is there something I don t...
shouldn t text () extract the text information inside? I m a little confused ...
I use CrawlSpider combined with the following Rules to automatically turn the page and climb the movie information of Douban top250: rules = ( Rule(LinkExtractor(restrict_xpaths= span[@class="next"] a ), callback= parse_...
Why do these url jump back to the selenium of middleware via selenium jump to the url request crawled down the page in scrapy, instead of calling back to the following def def parse(self, response): contents = response.xpath( *[@id="...
as shown in the figure below, when the page is the food section of the whole city, for example, the URL of Xi an food is "http: www.dianping.com xian ch10 ", you can crawl the data normally (figure 1). 50 "http: www.dianping.com xian ... " Please ...
The params parameter of requests can be easily set: requests.get (url, headers=Header, params=Param) but scrapy s Request: class Request(object_ref): def __init__(self, url, callback=None, method= GET , headers=None, body=None, ...
uses the scrapy.Request method to collect pages, but nothing is done. import scrapy def ret(response): print( start print ) print(response.body) url = https: doc.scrapy.org en latest intro tutorial.html v = scrapy.http.Request(url=url,...
goal: you want to launch the current request repeatedly when the request ip fails, or when the CAPTCHA is encountered, until the request succeeds, so as to reduce the data omission of crawling. question: I don t know if my thinking is correct. At pres...
def gen_media_requests(self, item, info): for image_url in item[ cimage_urls ]: yield scrapy.Request(image_url, meta={ item : item}) def file_path(self, request, response=None, info=None): item = request.meta.get(...
it is normal for the same proxy ip, to request with requests, but the request with scrapy.FormRequest will time out . related codes In [11]: r = requests.post( http: httpbin.org post , proxies={ http : proxy_server, https : proxy_server}) 2018...
operating system cetnos7 python3.7 scrapy crawl my crawler 2018-07-12 08:49:04 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: mm) 2018-07-12 08:49:05 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w...