The crawler didn't turn off regularly when scrapy started the crawler.

I used crontab to start a crawler, but the crawler didn"t shut down in 2 minutes.
now there is a solution that the while loop
runs normally within 2 minutes. If the running time exceeds 1 hour, turn off the spider. How to do this?

Mar.15,2022

it is recommended to search for extended middleware. Excetions, has instructions that spider did not receive a signal.


you can set CLOSESPIDER_TIMEOUT parameters

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1be34b1-30ebc.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1be34b1-30ebc.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?