after you have climbed with scrapy-redis spiders, you will not be able to climb again. If you change the name of the spider, you can crawl again, and if you change back to the original name, you will start the deduplication mechanism again. Although you need dupefilter, how to solve this problem if you generate some data during debugging and want to crawl again.
how to delete crawled url data previously recorded by dupefilter.
solved
it's hard to wait several days for no one to reply
I also encountered the same problem, but what you said above gave me an idea. I changed the name of the spider and I can use it. Then it is found that the key, with the name of the crawler is saved in redis and the key can be deleted. 
SCHEDULER_FLUSH_ON_START = True
you can add this sentence in the settings to automatically clean up the key in redis
after the scrapy-redis distributed crawler starts, can it run scrapy runspdier xx.py on a new machine to add slaves while it is crawling? Will you crawl the same url? A running project has configured scrapy-redis-related settings (REDIS_HOST, etc.) in...
I want to climb a website with about 1 billion data. Url is http: xxx.com id=xx accesses and extracts the data and stores it in the database . where the id parameter in url is predictable, ranging from 0 to 1000000000 so I can generate these 1 bill...
I ran a redis container with the following command: docker run --name redis_env --hostname redis -p 6379:6379 -v $PWD DBVOL redis data: data:rw --privileged=true -d redis redis-server I succ...