demand: one of the machines in the online cluster deploys the ftp server, and other machines schedule get data every day through the ftp client. Because there are many services running on the cluster, it is necessary to limit the speed of ftp in order not to affect other services
current solution: a simple ftp server is built using python"s pyftplib, and the client uses the speed limit code snippet of Linux NetKit"s ftp,python:
-sharp limit for bandwidth
dtp_handler = ThrottledDTPHandler
dtp_handler.read_limit = 30 * 1024 * 1024 -sharp 30 MB/s
dtp_handler.write_limit = 30 * 1024 * 1024 -sharp 30 MB/s
-sharp Instantiate FTP handler class
handler = FTPHandler
handler.authorizer = authorizer
handler.dtp_handler = dtp_handler
current problem: it is not found that the Nic is full through network monitoring (CPU, memory and disk IO are also far from reaching the alarm threshold), but there are a large number of delay alarms in other services deployed on the cluster. This may be due to the fact that ftp speed limit and network monitoring are based on seconds, but service monitoring is based on milliseconds. When the ftp server reaches the speed limit threshold within 1 second, it will no longer be processed, but in the short time when it reaches the threshold, it has occupied a lot of bandwidth, thus affecting other processes. The code snippet for pyftplib to implement the speed limit:
def _throttle_bandwidth(self, len_chunk, max_speed):
"""A method which counts data transmitted so that you burst to
no more than x Kb/sec average.
"""
self._datacount += len_chunk
if self._datacount >= max_speed:
self._datacount = 0
now = timer()
sleepfor = (self._timenext - now) * 2
if sleepfor > 0:
-sharp we"ve passed bandwidth limits
def unsleep():
if self.receive:
event = self.ioloop.READ
else:
event = self.ioloop.WRITE
self.add_channel(events=event)
self.del_channel()
self._cancel_throttler()
self._throttler = self.ioloop.call_later(
sleepfor, unsleep, _errback=self.handle_error)
self._timenext = now + 1
excuse me:
1. Is the reason I analyzed correct? If not, what else could the problem be?
2. If correct, is there a simple way or a mature solution to achieve a more accurate speed limit?