recently, I am doing a web crawler, first grab the list page, then get the url, of all the content pages of the list page, and then call the url loop of all the list pages to the crawling method, which leads to the uncontrollable crawling order. I want to know how to control the crawling order.
for example: page An is being crawled, page An is finished, page B is being crawled, page B is finished. In this order.
grab function:
reduce and then keep it in then
https://codeshelper.com/a/11.
" take a look at the app and request that this piece be processed in order.
can be put in then. You can think that this piece of logic has become Synchronize, and it will be very inefficient. You can put each result in a fixed position in the array. Use the promise.all method to determine that the returned array to be output is processed after all has been loaded.
here are my personal thoughts. The subject can try
.promise-queue-plus</a>;
:
var Queue = require('promise-queue-plus');
var q1 = new Queue(1); //1
//
doRequest('list.html')
.then(content => {
return this.parseList(content); //
})
//
.then(links => {
return q1.addLikeArray(links,doRequest,{
"retry":1, //
"timeout":10000 //
},true);
})
.then (allContent => {
console.log(allContent);
})