I want to use celery to make a distributed crawler
Let celery grab the data and store it in my mysql database
but the mysql database I bought only has 50 connections (database provider limit)
so I can"t start 100 worker to grab data at the same time?
is there a way to get worker to grab the data and return it to master, so that master can store it in the database?
do you have a solution?
Thank you
production data is stored in redis's queue (List). Then start a process of slow consumption.
I need to execute two sets of tasks step by step. The subtasks in each set of tasks are executed in parallel celery, but the second group of tasks needs to wait for the first group of tasks to be completed before continuing to execute . from celery impo...
@celery_app.task(name=u"abc", routing_key="xxx") def func_abc(a, b, c, d): pass @task_success.connect(sender=u"abc") def on_abc_success(sender, result, **kwargs): pass the overall logic of the code is like this...
according to the tutorial pip3 install celery, enter celery help prompt on the command line after the tutorial celery: can t find the command. How to break ...
supervisor is a process management tool under Linux, which can manage the services running in the foreground, while the services in the background need to be transformed into foreground before they can be managed through supervisor. The celery process s...
1. My project is django 1.11, python 2.7, using celery 4.1, using django_celery_beat third-party libraries. After installation and migration, there is a scheduled task table in the admin background, which can be used normally. However, I recently used d...
the database has 10w records, which may increase to 20w in half a year, but it should not exceed 100w in the end. Server configuration: python3.6 celery+rabbitMQ CVM ubuntu 16.04 1G 1 core Database postgresql 10, with a limit of 100 connections t...
there is a graphics class celery task that must be run under windows. typing celery directly on the command line can run successfully as a whole and get the correct results. while using nssm to package celery as a service to start, although the task c...
question 1: is it normal to start by printing the following information? [D 180708 22:10:53 callback:161] Added: { callback : < bound method Connection._on_connection_error of < pika.adapters.tornado_connection.TornadoConnection object at 0x11043c8d0...
problem description celery uses prefork mode (4 processes) and is used in conjunction with psycopg2. Each startup can successfully complete a celery task. InterfaceError: cursor already closed appears when the second task is carried out. What should...