I want to climb some e-commerce websites, which have a lot of pictures. Now I"m using cheerio,. I find that it can"t get the images loaded lazily on the page, that is, the images generated by js processing. Is there any way or other library to do this?
it is recommended that puppeteer
can only be obtained by analyzing the js of the web page developed by the Google team, or run a real browser to climb
the url that lazily loads images should be saved in img tags, such as data-src, using chrome's headless mode. Check the properties of those pictures.
if it doesn't work, use phantomjs.
the best way to climb a page is to climb the interface, analyze the calling mode of the interface, input parameters, and cookie.
print and see that it has been 5 seconds since the whole request. I don t know why ...
there is an error downloading pdf files in batches using download module. In the process of downloading, it always stops when downloading more than 20 or 40 files var arr = [{ url: "http: pdf.dfcfw.com pdf H2_AN201803271111860450_1.pdf&qu...
phantomjs crawl NetEase Yun music playlist, the code is as follows var webpage = require( webpage ); var page = webpage.create(); page.open( https: img.codeshelper.com upload img 2021 04 11 4sxkk4v4vs016110.png ); console.log(page.cont...
if you use superagent to request a web page, not all ssr page data can be obtained through the interface (I know it s good to climb the interface directly, but I have a special need to do so). I hope to get the data in the form of cheerio analysis page...
the node service is opened at the front end. When calling interface an of the test environment, you can access it with localhost:3000 a, and then use the request of node to request the address of the test environment. Equivalent to being an agent but ...
serer.js I would like to use the following method to act as a proxy. I can get the data of the test environment locally and debug it locally. options-related configuration let request = http.request(options, function(response){ respon...
problem description has written about simple crawlers. According to the web page opened by url , the content in the web page is the information to be crawled, which is easier to do. now the page defaults to no data or not the data you want, so you n...