Python uses threading and Queue, to implement fast crawling requirements, but what is the reason for finding that some data written to the file is repeatedly written? - Codes Helper

there are a large number of repetitive results in the running results. I think what is the reason that the novice does not understand very well? ask the great god to let me know?

this problem did not occur until it was changed to threading+Queue.

unrelated code does not show

import requests
import json,time,random

from collections import defaultdict

import threading
import Queue

from RotateUserAgentMiddleware import user_agent_list

class Aiqiyispider (threading.Thread):

def __init__(self,in_queue):
    threading.Thread.__init__(self)
    self.in_queue = in_queue

    -sharp
    
.
.
.
.
.

def new_videourl(self,fristurl):
    """
    :
    :return:
    """
    

def b_Spider(self,biaoq):
    """
    new_videourl
    :return:
    """
    

def run(self):
    """
    
    :return:
    """
    while True:
        try:
            line = self.in_queue.get(timeout =2)
            self.new_videourl(line)
        except Exception,err:
            break

if _ _ name__=="__main__":

peacial = []
queue = Queue.Queue()

for i in xrange(4):
    t = Aiqiyispider(queue)
    -sharp t.setDaemon(True)
    -sharp t.start()
    peacial.append(t)

for tt in peacial:
    tt.start()

file_qis = 

for qis in file_qis:
    all_file_id = qis.strip().split("\t")[0].split("|")[0]
    queue.put(all_file_id)

for tt in peacial:
    tt.join()

print "END"

I just want to know the reason for this result. Why did you show up?

below is an intercept of some of the results. You can see that 331844400 appears several times, and my result shows 22 times this line of data

331844400 331844400 {"genre": "pop", "music type": "single MV", "language": "Korean", "region": "Japan and South Korea"} Music EXO-ice cream cake CF tidbits
357314200 357314200 {"genre": "pop", "music type": "single MV", "language": "Korean", "region": "Japan and South Korea"} Music SUPER JUNIOR-D&E "Touch (Can You Feel It?)" Touch Dance (everyone"s Touch Dance)
218013800 218013800 {"genre": "Pop", "Music Type": "single MV", "language": "Korean", "region": "Japan and Korea"} Music Bigbang-Oh Yeah > 812884100 812884100 {"genre": "Rock", "Music type": "single MV", "language": "Japanese", "region": "Japan and South Korea"} Music Sakai Xiangtai-zero
331844400 331844400 {"genre": "pop", "music type": "single MV", "language": "Korean", "region": "Japan and South Korea"} Music EXO-Ice Cream Cake CF tidbits
628752500 628752500 {"genre": "Pop", "Music Type": "single MV", "language": "Korean", "region": "Japan and South Korea"} Music GFRIEND-FINGERTIP
331844400 331844400 {"genre": "Pop", "Music Type": "single MV", "language": "Korean", "region": "Japan and Korea"} Music EXO-Ice Cream Cake CF