Design a background program, extract a large amount of data according to the rules to generate short messages and send them, how to design to ensure that the short messages do not repeat and the stability of the system?

requirements are as follows:

there are one or more databases that hold all kinds of user data, such as contract information, time, amount, and so on.

now we are going to develop a background system to regularly extract data from the specified database according to certain rules (provide database connection and sql), to dynamically generate SMS messages according to the SMS template (given), and then call the SMS interface to send to the user.

templates such as reminders for repayment, holiday wishes, contract activation, etc.

the demand is not complex, but the amount of data is large. There are more than a dozen templates, and it is estimated that about 100,000 SMS messages will be generated every day.

our idea is that a template is paired with a scheduled task, the data is extracted in a loop, and then a text message is generated and finally sent.

however, it should be noted that SMS messages must not be repeated. In case the service is terminated abnormally during generation or transmission, SMS messages cannot be sent repeatedly after restart.

to meet this requirement, you need to store the generated data in the database, check whether it has been sent, and then send it when you meet the conditions.

the technology selection is that springboot, background tasks use quartz. More than a dozen templates, that is to say, there will be more than a dozen threads, more than a dozen threads frequently write to the database, and frequently read the database when sending, we feel that there will be hidden dangers. All specially come up to ask everyone"s opinion, do you have any good suggestions?

Nov.29,2021

No problem using the database. More than 10 concurrent databases are fine. If you use redis faster, pure memory.

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1e498b7-44d5e.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1e498b7-44d5e.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?