what if the url starts from the list page, and after the list also grabs the title and other information, the traversed list enters the detail page according to the title, and there are multiple pages of url in this detail page that need to be followed (for example, an article has 1 page, 2 pages, 3 pages)? What should item do with it?
for example, Baidu Tieba, from the home page list!
, grab the relevant information here and then enter the post to grab all the buildings
this is very simple. After start_request, scrapy will use the parse method by default. Of course, you can also define other methods to resolve, and then in the parse method, parse to get the link to the specific details page, and then again yield Request (url=' details page link', callback=self.parse) this form of cycle to take the next page, but there will be a hole, but I think your two layers should be almost enough. I don't bother to mention this pit.
Hello, I have encountered the same obstacle as you, but I can help you with the item save. The
format is as follows:
use the
yield Request (url=' detail page link in the list interface, and callback=self.detail_parse (detail_parse is the detail page parsing), mata {'item':item})
Previous: The problem of transaction blocking caused by database foreign key constraints
Next: How do you do this font effect on Apple's official website?
scrapyapi5 because I originally wanted to get 100 ip, at a time and put it in the agent pool, but because the agent is unstable, it can not provide support for a long time. So I gave up the idea of getting 100 ip at once. 5request ...
when scrapy crawls a picture of a web page, the class that inherits ImagesPipelines is customized in the pipelines file. but the custom pipelines cannot be executed after running the program. Item cannot pass the following is a custom pipelines clas...
I set the crawler to run every 6 hours, and it did. The problem with is that it runs immediately after each point starts, and then executes every 6 hours. how do you stop it from running at the start of the point? ! @web Oh, it s all right. Jus...
when I crawl a page with scrapy, I find that I can only request one page at a time, but the posts on the official website and Baidu say that the concurrency can be controlled through CONCURRENT_REQUESTS , but I tried it or it didn t work? CONCURRENT_...
execute after entering the project, the error shows scrapy command not found , but I-sharpscrapy can be run, the scrapy crawl test crawler command can also be executed alone, only the scheduled command will appear scrapy:command not found ...
after crawling the navigation, the URL crawl that you want to continue in-depth navigation, and then the unified return value is written to xlsx < H1 >--coding: utf-8--< H1 > from lagou.items import LagouItem; import scrapy class LaGouSpider (...
scrapy.Request cannot enter callback code is as follows: def isIdentifyingCode(self, response): -sharp pass def get_identifying_code(self, headers): -sharp -sharp return scrapy.Req...
A nasty piece of html code that writes js in div. It s a keyboard paging code xpath found that the tagged content in is gone, like this I am China person what I get is: I am human. China does not have , and then some people say that my xpath ...
when scrapy saves data through Pipeline (in txt format), some data gbk codec can t encode character appears as follows. class TxtPipeline(object): def process_item(self,item,spider): path=os.getcwd() filename = path + dat...
scrapy tutorial: http: scrapy-chs.readthedocs. Environment: python3.6 + windows7 Project structure Directory: mySpider:scrapy crawl domz: there is no [dmoz] output as mentioned in the tutorial, is there any new file, is there something I don t...
shouldn t text () extract the text information inside? I m a little confused ...
I use CrawlSpider combined with the following Rules to automatically turn the page and climb the movie information of Douban top250: rules = ( Rule(LinkExtractor(restrict_xpaths= span[@class="next"] a ), callback= parse_...
Why do these url jump back to the selenium of middleware via selenium jump to the url request crawled down the page in scrapy, instead of calling back to the following def def parse(self, response): contents = response.xpath( *[@id="...
as shown in the figure below, when the page is the food section of the whole city, for example, the URL of Xi an food is "http: www.dianping.com xian ch10 ", you can crawl the data normally (figure 1). 50 "http: www.dianping.com xian ... " Please ...
The params parameter of requests can be easily set: requests.get (url, headers=Header, params=Param) but scrapy s Request: class Request(object_ref): def __init__(self, url, callback=None, method= GET , headers=None, body=None, ...
uses the scrapy.Request method to collect pages, but nothing is done. import scrapy def ret(response): print( start print ) print(response.body) url = https: doc.scrapy.org en latest intro tutorial.html v = scrapy.http.Request(url=url,...
goal: you want to launch the current request repeatedly when the request ip fails, or when the CAPTCHA is encountered, until the request succeeds, so as to reduce the data omission of crawling. question: I don t know if my thinking is correct. At pres...
def gen_media_requests(self, item, info): for image_url in item[ cimage_urls ]: yield scrapy.Request(image_url, meta={ item : item}) def file_path(self, request, response=None, info=None): item = request.meta.get(...
it is normal for the same proxy ip, to request with requests, but the request with scrapy.FormRequest will time out . related codes In [11]: r = requests.post( http: httpbin.org post , proxies={ http : proxy_server, https : proxy_server}) 2018...
operating system cetnos7 python3.7 scrapy crawl my crawler 2018-07-12 08:49:04 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: mm) 2018-07-12 08:49:05 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w...