look at the library such as celery aps, but there is too little Chinese information on the Internet.
talk about the demand:
1 for example, there are 100 sports games a day, and the URL that generates these 100 games is each thrown into the task queue for a task.
2 crawls data every 20 minutes before this time according to its status, such as the end time. When the time is up, delete the task and stop crawling.
3 the task will return data to be stored in the database
is there a solution like this? what kind of technology will be used to solve the problem?