Boss, I have a question that has puzzled me for a long time:
I made a crawler with nodejs, crawled data regularly every day, node version v9.8.0, regularly tried the system command crontab, now uses node-schedule, to use pm2 manager.
after running for a period of time, sometimes three or four days, sometimes a week or more, the crawler gets stuck. There is no error log, pm2 does not have any log information, and the program does not exit, which seems to be stuck in the background.
the dependencies used by the program are as follows:
"axer": "0.0.5",
"log4js": "^ 1.1.1",
"moment": "^ 2.18.1",
"mongodb": "^ 3.0.5",
"node-schedule": "^ 1.2.4",
"bluebird": "^ 3.5.1",
"util": "^ 0.10.3",
"xml2js": "^ 0.4.17"
Boss, have you ever encountered similar problems? Can you give me some hints? Thank you very much!