write a service interface with node. The front end sends a request and crawls the web page after receiving the request. If there are four or five concurrency, the puppeteer seems to be queued to open, causing the request to get stuck and the response is particularly slow. Is there any solution for this?
you can crawl a web page into a task system. The front end submits the task by request (after the task is saved, it returns immediately), and the back end processes the crawling task in turn.
do you use one page object or four or five page objects?
copied a crawler from the website to crawl some product information, but I don t know why the single page of the product just can t climb down. but tested that all other pages are crawable. Why? found the next returned error information, which seems...
there is currently an API interface program using express. With PM2 deployment and cluster mode enabled, 3 instances are required to count the overall qps (number of queries per second). now the interface puts the content into the kafka message queue a...
how does superagent get url after redirection my previous idea was to set .redirects (0) , and then get the redirected url, by Location in the response header, but this failed ask the great god what I should do ...
vue+node * * < H2 > 1. Problem found < H2 > * * * * < H2 > 2. Find the problem < H2 > * * pm2 list pm2 node nodekill -9 [pid] ...
the business you want to achieve is to render the page first, and then add content to the page through ctx.body the core code is as follows await ctx.render( crawler , { title: , content: `<h2>< h2> <h4>...
ask for help, boss! Superagent post submits the form data to the database with garbled code as follows: ...
now there is a CAPTCHA sent, and then the client requests to come back, compare it, and then determine whether it is successful or not. But I don t know where to store this CAPTCHA! Please give me some advice ...
...
I want to climb some e-commerce websites, which have a lot of pictures. Now I m using cheerio,. I find that it can t get the images loaded lazily on the page, that is, the images generated by js processing. Is there any way or other library to do this?...
what s wrong with my node code? browsers can return data by posting the address in it. api ...
wrote a simple program that reported an error at run time, and Baidu checked that it was caused by the separation of express and body-parser, so I installed body-parser. But then the same error is reported. The code is as follows: ...
1. Description problem: I made a small crawler, want to crawl some pictures of a website, now climb down, the image path can be printed out. however, I need to download these pictures to a local images folder, write the code is no problem, I have tr...
two node.js services are started. Service A forwards the request to B, and the An agent is configured as follows: var proxy = require ( http-proxy-middleware ); app.use ( api ,proxy ({) target: http:192.168.10.111:3000 , secure:false, change...
print and see that it has been 5 seconds since the whole request. I don t know why ...
there is an error downloading pdf files in batches using download module. In the process of downloading, it always stops when downloading more than 20 or 40 files var arr = [{ url: "http: pdf.dfcfw.com pdf H2_AN201803271111860450_1.pdf&qu...
how should I write such a question? use the node package superagent, to save the picture to the local disk. Then return the address to the front end. ...
the old project. Today, we found that the cpu is unusually high. Through Aliyun s node performance platform, we can get the flame diagram when it is soaring. What is the longest program in the figure? is it the main program? New to node, I don t know ...
I am using Phantomjs for automatic login <script type="text html" id="js_table_tpl"> {if data.length} {each data as item i} <div class="user_item"> <div class="user_item_inner"> <...
phantomjs crawl NetEase Yun music playlist, the code is as follows var webpage = require( webpage ); var page = webpage.create(); page.open( https: img.codeshelper.com upload img 2021 04 11 4sxkk4v4vs016110.png ); console.log(page.cont...
the small crawler written by node reported an error when it was written to parse the crawled data with cheerio, saying that it was a problem of circular invocation. Paste code: $( -sharplive-list-contentbox>li ).each((i, ele) => { l...