Yes but you have more than 1 POS terminal per location so you still don't need 20,000 partitions. Just one per location. How many locations do you have?
In doesn't matter anyway since you can build a Kafka cluster with up to 200,000 partitions if you use the latest versions of Kafka.
https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
"As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions"
-hans
> On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <akuterin@gmail.com> wrote:
>
> Thanks, Hans!
> We use location specific SKU pricing and send specific price lists to the
> specific POS terminal.
>
> пн, 1 апр. 2019 г., 3:01 Hans Jespersen <hans@confluent.io>:
>
>> Doesn't every one of the 20,000 POS terminals want to get the same price
>> list messages? If so then there is no need for 20,000 partitions.
>>
>> -hans
>>
>>> On Mar 31, 2019, at 7:24 PM, <akuterin@gmail.com> <akuterin@gmail.com>
>> wrote:
>>>
>>> Hello!
>>>
>>>
>>>
>>> I ask for your help in connection with the my recent task:
>>>
>>> - Price lists are delivered to 20,000 points of sale with a frequency of
>> <10
>>> price lists per day.
>>>
>>> - The order in which the price lists follow is important. It is also
>>> important that the price lists are delivered to the point of sale online.
>>>
>>> - At each point of sale, an agent application is deployed, which
>> processes
>>> the received price lists.
>>>
>>>
>>>
>>> This task is not particularly difficult. Help in solving the task is not
>>> required.
>>>
>>>
>>>
>>> The difficulty is that Kafka in our company is a new "silver bullet", and
>>> the project manager requires me to implement the following technical
>>> decision:
>>>
>>> deploy 20,000 Kafka consumer instances (one instance for each point of
>> sale)
>>> for one topic partitioned into 20,000 partitions - one partition per
>>> consumer.
>>>
>>> Technical problems obtained in experiments with this technical decision
>> do
>>> not convince him.
>>>
>>>
>>>
>>> Please give me references to the books/documents/blogposts. which clearly
>>> shows that Kafka not intended for this way to use (references to other
>>> anti-patterns/pitfalls will be useful).
>>>
>>> My own attempts to find such references were unsuccessful.
>>>
>>>
>>>
>>> Thank you!
>>>
>>>
>>>
>>
In doesn't matter anyway since you can build a Kafka cluster with up to 200,000 partitions if you use the latest versions of Kafka.
https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
"As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions"
-hans
> On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <akuterin@gmail.com> wrote:
>
> Thanks, Hans!
> We use location specific SKU pricing and send specific price lists to the
> specific POS terminal.
>
> пн, 1 апр. 2019 г., 3:01 Hans Jespersen <hans@confluent.io>:
>
>> Doesn't every one of the 20,000 POS terminals want to get the same price
>> list messages? If so then there is no need for 20,000 partitions.
>>
>> -hans
>>
>>> On Mar 31, 2019, at 7:24 PM, <akuterin@gmail.com> <akuterin@gmail.com>
>> wrote:
>>>
>>> Hello!
>>>
>>>
>>>
>>> I ask for your help in connection with the my recent task:
>>>
>>> - Price lists are delivered to 20,000 points of sale with a frequency of
>> <10
>>> price lists per day.
>>>
>>> - The order in which the price lists follow is important. It is also
>>> important that the price lists are delivered to the point of sale online.
>>>
>>> - At each point of sale, an agent application is deployed, which
>> processes
>>> the received price lists.
>>>
>>>
>>>
>>> This task is not particularly difficult. Help in solving the task is not
>>> required.
>>>
>>>
>>>
>>> The difficulty is that Kafka in our company is a new "silver bullet", and
>>> the project manager requires me to implement the following technical
>>> decision:
>>>
>>> deploy 20,000 Kafka consumer instances (one instance for each point of
>> sale)
>>> for one topic partitioned into 20,000 partitions - one partition per
>>> consumer.
>>>
>>> Technical problems obtained in experiments with this technical decision
>> do
>>> not convince him.
>>>
>>>
>>>
>>> Please give me references to the books/documents/blogposts. which clearly
>>> shows that Kafka not intended for this way to use (references to other
>>> anti-patterns/pitfalls will be useful).
>>>
>>> My own attempts to find such references were unsuccessful.
>>>
>>>
>>>
>>> Thank you!
>>>
>>>
>>>
>>
Comments
Post a Comment