Thanks for the response Lianet.
This problem happens only in kraft mode brokers. Works fine if we use kafka
broker in non-kraft mode. (WIth zookeeper)
From what I understand: (I may be wrong. this is what I read online)
In zookeeper mode, on getting a partitionsFor request, the broker creates a
topic if not present and then immediately returns the number of partitions.
All this happens synchronously.
In kraft mode, the broker creates a topic asynchronously. So, unless u
retry, you wont get the total number of partitions.
Like I said, this is what I read.
On Tue, Feb 17, 2026 at 3:19 AM Lianet Magrans <lianetm@apache.org> wrote:
> Hi Anand,
>
> The partitionsFor API does not guarantee that a single call will return the
> data if the topic does not exist. It issues a first metadata request to
> create the topic, and waits for the api timeout for a response (timeout
> param or default.api.timeout.ms config). Then returns whatever it gets
> from
> the broker in that single call (empty if the topic is not found within the
> timeout, described in the java docs
>
> https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415
> ).
> Also there are old integration tests with the pattern you describe: a first
> call to ensure topic is created, then a following one expected to retrieve
> the partitions for the topic (e.g, in 3.9 branch
>
> https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
> )
>
> That being said, I wonder if the difference you're mentioning is due to
> running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
> Not sure if tuning some Kraft configs on your cluster could help (I'm not
> familiar with the kraft details really), but metadata is eventually
> consistent anyways, so for sure it would be advisable to use the
> partitionsFor considering that it may require several calls if the topic is
> not found. From there, we can still consider improving for sure (e.g,
> improve the client to continue attempting to find the topic while there is
> time?). If you can file a jira with your setup (broker configs, params
> passed to the client API) and behaviour, we can better look into it.
>
> Thanks!
> Lianet
>
> On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
>
> > Hello All,
> >
> > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version
> > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor
> API,
> > I see that it works when the topic is already created in kafka. But if it
> > is not yet created, the above API call returns empty - no information.
> >
> > I noticed that just before I call partitionsFor API, the topic is not
> > created. But after I call this API, the topic gets created, but this API
> > still returns no information. Of course, if I call the same API again, I
> do
> > get valid information for this topic.
> >
> > I used to have kafka-clients-3.9.0.jar earlier. There both these cases
> were
> > working fine. Is this a new change in behavior in
> kafka-clients-4.1.1.jar?
> >
> > Thanks,
> > Anand
> >
>
This problem happens only in kraft mode brokers. Works fine if we use kafka
broker in non-kraft mode. (WIth zookeeper)
From what I understand: (I may be wrong. this is what I read online)
In zookeeper mode, on getting a partitionsFor request, the broker creates a
topic if not present and then immediately returns the number of partitions.
All this happens synchronously.
In kraft mode, the broker creates a topic asynchronously. So, unless u
retry, you wont get the total number of partitions.
Like I said, this is what I read.
On Tue, Feb 17, 2026 at 3:19 AM Lianet Magrans <lianetm@apache.org> wrote:
> Hi Anand,
>
> The partitionsFor API does not guarantee that a single call will return the
> data if the topic does not exist. It issues a first metadata request to
> create the topic, and waits for the api timeout for a response (timeout
> param or default.api.timeout.ms config). Then returns whatever it gets
> from
> the broker in that single call (empty if the topic is not found within the
> timeout, described in the java docs
>
> https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415
> ).
> Also there are old integration tests with the pattern you describe: a first
> call to ensure topic is created, then a following one expected to retrieve
> the partitions for the topic (e.g, in 3.9 branch
>
> https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
> )
>
> That being said, I wonder if the difference you're mentioning is due to
> running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
> Not sure if tuning some Kraft configs on your cluster could help (I'm not
> familiar with the kraft details really), but metadata is eventually
> consistent anyways, so for sure it would be advisable to use the
> partitionsFor considering that it may require several calls if the topic is
> not found. From there, we can still consider improving for sure (e.g,
> improve the client to continue attempting to find the topic while there is
> time?). If you can file a jira with your setup (broker configs, params
> passed to the client API) and behaviour, we can better look into it.
>
> Thanks!
> Lianet
>
> On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
>
> > Hello All,
> >
> > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version
> > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor
> API,
> > I see that it works when the topic is already created in kafka. But if it
> > is not yet created, the above API call returns empty - no information.
> >
> > I noticed that just before I call partitionsFor API, the topic is not
> > created. But after I call this API, the topic gets created, but this API
> > still returns no information. Of course, if I call the same API again, I
> do
> > get valid information for this topic.
> >
> > I used to have kafka-clients-3.9.0.jar earlier. There both these cases
> were
> > working fine. Is this a new change in behavior in
> kafka-clients-4.1.1.jar?
> >
> > Thanks,
> > Anand
> >
>
Comments
Post a Comment