Redis acts as a message queue to control the consumption interval of the queue.

has encountered a difficult problem recently.

I use redis for message queuing. Each queue contains a task id,

the simple process is as follows:

1. Read id, consumption from list

2. Read the details of this task from the database according to id,.

the data structure is as follows: for example,

ID=1 URL:abc.com
ID=2 URL:abd.com
ID=3 URL:ab3.com
.

I need to do a function now.

controls the access interval for each url"s domain name. For example, abc.com and efg.com are included in the url in the task

I need to control the consumption of one task containing abc.com every 5 seconds and one task containing efg.com every 10 seconds

what if you can accomplish this function while taking into account the performance?

< hr >

add a condition: distributed, millions of queues, dozens of servers

< hr >

come back after a short time to look at this problem. There are many articles on the redis delay queue.
the professional message queuing rabbitmq also has tutorials in this area.

-0-

it took me a year to find the right answer. It was disgusting.

answer: https://codeshelper.com/q/10.

Feb.28,2021

add a judgment if abc.com sleep (5) is included


this is how I deal with it at the moment:

1, lpop task.
2. Check whether this task has an interval of X seconds.
3. If it meets, consumption, if not, no consumption. If not, use the lpush command to push into the queue.

the challenge of this feature is performance. If there is only one server, there is no problem, but when there are millions of task queues, there is a problem.
to perform these tests, each additional step will add a lot of extra overhead, and the qps will be very high. Some queues can be consumed in time, but do not. Causing a lot of accumulation.


there is another question: why do you push the same task multiple times without detection? Didn't you pop first?


you can store it in two zset queues, sorted by time as score. When the 5s zset1, is inserted, record the time that can be consumed (current time + 5), and take the oldest record to spend each time to determine whether the current time exceeds the consumption time. Then put 10s into zset2, for the same operation.


the landlord can learn that beanstalked is a memory queue

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1b31f3e-2bdd0.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1b31f3e-2bdd0.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?