def process_item(self, item, spider):
print("")
print(item["file_url"], item["name"])
key_word = {"file_url": item["file_url"], "name": item["name"]}
res = self.db.find(key_word)
if res:
print("")
raise DropItem("Duplicate item found: %s" % item)
else:
print("*******************************************************************************")
self.db.insert({"file_url": item["file_url"], "name": item["name"]})
return item
this is found in the database. The judgment of the joint group key above me seems to be wrong, because there is no url and name in the program at all. Dropitme asks for advice
db.XiaoMiQuan.find ()
{"_ id": ObjectId ("5bbf14dbc96b5b3f5627d11d"), "file_url": "https://baogaocos.seedsufe.com/2018/07/19/doc_1532004923556.pdf"," name ":" AMCHAM- China"s "Belt and Road Initiative": impact on American Enterprises (English)-2018.6-8 pages .pdf "}