Dear all,
I am trying to run a Kafka experiment on Kubernetes where I would like to control the number of messages sent/produced to the broker per unit time/interval, and the number of messages being consumed by the consumer per unit time/interval.
Controlling the number of messages sent/produced to the broker per unit time (according to a predetermined workload /trace) is easy. E.g., for a trace of 20 messages per 15 secs, and then 100 messages per next 15 secs, and finally, 45 msgs per the next 15 seconds…, the methodology is simple: send the 20 messages(in single batch), sleeping (the producer main thread) for 15 seconds, followed by sending the 100 messages, and sleeping the producer thread for 15 seconds and so on… I am sending the message in a single batch.
However, controlling the number of consumed messages per unit time (say 15 seconds) looks more complex. In fact, to emulate a consumer with maximum rate of 100 messages every 15 seconds interval, I am setting max.poll.records=100 and sleeping the poll thread for 15 seconds. The issue is that I want to ensure that during the 15 seconds interval at the producer if this last produced say 150 messages, then during the next first 15 seconds interval at the consumer 100 (and not 150) messages are consumed (recall maximul.poll.records =100).
On the other hand, say during the 15 seconds interval at the producer if this last produced say 70 messages, then during the next first corresponding 15 seconds interval at the consumer exactly 70 messages are consumed; before sleeping the poll loop for 15 seconds, and continue so on.
===
Perhaps in simpler words: divide the producer and the consumer work into s seconds interval, during the s seconds at the producer, this last produce p messages. The consumer is configured with c messages per second, and during the s seconds consumption interval (at the consumer) if p<c then exactly p messages are consumed, otherwise c messages are consumed.
Any hint please on the best way to achieve this experimental scenario? if any
Thanks so much for your time.
I am trying to run a Kafka experiment on Kubernetes where I would like to control the number of messages sent/produced to the broker per unit time/interval, and the number of messages being consumed by the consumer per unit time/interval.
Controlling the number of messages sent/produced to the broker per unit time (according to a predetermined workload /trace) is easy. E.g., for a trace of 20 messages per 15 secs, and then 100 messages per next 15 secs, and finally, 45 msgs per the next 15 seconds…, the methodology is simple: send the 20 messages(in single batch), sleeping (the producer main thread) for 15 seconds, followed by sending the 100 messages, and sleeping the producer thread for 15 seconds and so on… I am sending the message in a single batch.
However, controlling the number of consumed messages per unit time (say 15 seconds) looks more complex. In fact, to emulate a consumer with maximum rate of 100 messages every 15 seconds interval, I am setting max.poll.records=100 and sleeping the poll thread for 15 seconds. The issue is that I want to ensure that during the 15 seconds interval at the producer if this last produced say 150 messages, then during the next first 15 seconds interval at the consumer 100 (and not 150) messages are consumed (recall maximul.poll.records =100).
On the other hand, say during the 15 seconds interval at the producer if this last produced say 70 messages, then during the next first corresponding 15 seconds interval at the consumer exactly 70 messages are consumed; before sleeping the poll loop for 15 seconds, and continue so on.
===
Perhaps in simpler words: divide the producer and the consumer work into s seconds interval, during the s seconds at the producer, this last produce p messages. The consumer is configured with c messages per second, and during the s seconds consumption interval (at the consumer) if p<c then exactly p messages are consumed, otherwise c messages are consumed.
Any hint please on the best way to achieve this experimental scenario? if any
Thanks so much for your time.
Comments
Post a Comment