there are hundreds of files. But none of them are big, and the largest ones are only a few megabytes
.I use pycurl to download.
I put the download address on the list.
take out the first download, wait for it, and then take the second one.
but the question is, with hundreds of download connections, how can pycurl determine that the current file has been downloaded and then move on to the next one.
but the question is, with hundreds of download connections, how can pycurl determine that the current file has been downloaded and then move on to the next one.
pycurl understands the HTTP protocol, so you don't have to worry about it.
use tomorrow to execute concurrently
from tomorrow import threads
@threads(10)
def get_data(url):
...
return res
urls = [...]
res = [get_data(url) for url in urls]