excuse me,
A 4-core 8G machine with logstash, deployed on it can probably support several log input nodes of filebeat. If there is no message queue as buffer, do you have any experience in this field?
similarly, how many logstash nodes can a 4-core 16G machine support if elasticsearch, is deployed?
Thank you!
tell me about my practical experience. I built a set of ELK to collect logs from a large system of the company.
there are about 30 Filebeat nodes and 3 logstash nodes (8C16G). The data is about 1TB in a day. The problem encountered by
at the beginning is that there is a delay in sending data in logstash, and the throughput of the logstash cluster cannot be broken through after increasing the throughput of the logstash cluster to 5000e/s by adjusting the number of threads and batch size. So joined three Kafka clusters (all 8C8G), each Topic is divided into three partitions, so that three logstash consumption, now can reach 10000e 8C8G s, basically meet the demand and do not continue to optimize.
if you expand your capacity or increase the amount of data in the future, it is recommended to join the message queue as a buffer, and it is convenient to manage it later. For example, if you want to add a Logstash node, you do not need to make changes to each filebeat.
feels that ELK still eats resources very much, if real-time performance is guaranteed.
Previous: What's the difference between webpack cli and webpack package builds?
Next: You can drag and drop the project source code that generates the background management system.