Node crawler, a lot of data, does not move in the middle of climbing, is there any solution

the crawler stops in the middle of the climb, does not report an error, and does not continue to execute.

there is no time to continue from the current implementation of the method. The node solution, or the compiler has a solution in this regard.

Mar.02,2021

get a file to record the information you have climbed.
kill re-read file record crawl when not moving.
but it is recommended that you solve the immobile problem first

  1. there may be an endless loop
  2. it is possible to wait asynchronously without setting a timeout

suggest breaking point or console.log debugging


Don't move is a very vague concept
you should see what the log is outputting and what problems you encounter

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1e8db7d-45ec2.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1e8db7d-45ec2.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?