- this exception occurs during a distributed crawler using
scrapy-redis
, not from the beginning, but from the crawler. Five machines are used to crawl at the same time.
- exception information
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 58, in run
self.crawler_process.start()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 291, in start
reactor.run(installSignalHandlers=False) -sharp blocking call
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1267, in run
self.mainLoop()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1276, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 902, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/utils/reactor.py", line 41, in __call__
return self._func(*self._a, **self._kw)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 122, in _next_request
if not self._next_request_from_scheduler(spider):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 149, in _next_request_from_scheduler
request = slot.scheduler.next_request()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis_bloomfilter/scheduler.py", line 175, in next_request
request = self.queue.pop(block_pop_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis/queue.py", line 115, in pop
results, count = pipe.execute()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3443, in execute
return execute(conn, stack, raise_on_error)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3339, in _execute_transaction
self.immediate_execute_command("DISCARD")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3275, in immediate_execute_command
return self.parse_response(conn, command_name, **options)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3402, in parse_response
self, connection, command_name, **options)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 768, in parse_response
response = connection.read_response()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/connection.py", line 638, in read_response
raise response
redis.exceptions.ResponseError: DISCARD without MULTI