Hi Subra,
It seems to me both use an async approach. The root cause is that the
returned error code is different. Please check
https://issues.apache.org/jira/browse/KAFKA-20089 for more details.
Thanks,
Chia-Ping
Subra I <iamsubra100@gmail.com> 於 2026年2月19日週四 下午9:58寫道：
> Thanks for the response Lianet.
>
> This problem happens only in kraft mode brokers. Works fine if we use kafka
> broker in non-kraft mode. (WIth zookeeper)
>
> From what I understand: (I may be wrong. this is what I read online)
>
> In zookeeper mode, on getting a partitionsFor request, the broker creates a
> topic if not present and then immediately returns the number of partitions.
> All this happens synchronously.
>
> In kraft mode, the broker creates a topic asynchronously. So, unless u
> retry, you wont get the total number of partitions.
>
> Like I said, this is what I read.
>
> On Tue, Feb 17, 2026 at 3:19 AM Lianet Magrans <lianetm@apache.org> wrote:
>
> > Hi Anand,
> >
> > The partitionsFor API does not guarantee that a single call will return
> the
> > data if the topic does not exist. It issues a first metadata request to
> > create the topic, and waits for the api timeout for a response (timeout
> > param or default.api.timeout.ms config). Then returns whatever it gets
> > from
> > the broker in that single call (empty if the topic is not found within
> the
> > timeout, described in the java docs
> >
> >
> https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415
> > ).
> > Also there are old integration tests with the pattern you describe: a
> first
> > call to ensure topic is created, then a following one expected to
> retrieve
> > the partitions for the topic (e.g, in 3.9 branch
> >
> >
> https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
> > )
> >
> > That being said, I wonder if the difference you're mentioning is due to
> > running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
> > Not sure if tuning some Kraft configs on your cluster could help (I'm not
> > familiar with the kraft details really), but metadata is eventually
> > consistent anyways, so for sure it would be advisable to use the
> > partitionsFor considering that it may require several calls if the topic
> is
> > not found. From there, we can still consider improving for sure (e.g,
> > improve the client to continue attempting to find the topic while there
> is
> > time?). If you can file a jira with your setup (broker configs, params
> > passed to the client API) and behaviour, we can better look into it.
> >
> > Thanks!
> > Lianet
> >
> > On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
> >
> > > Hello All,
> > >
> > > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on
> version
> > > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor
> > API,
> > > I see that it works when the topic is already created in kafka. But if
> it
> > > is not yet created, the above API call returns empty - no information.
> > >
> > > I noticed that just before I call partitionsFor API, the topic is not
> > > created. But after I call this API, the topic gets created, but this
> API
> > > still returns no information. Of course, if I call the same API again,
> I
> > do
> > > get valid information for this topic.
> > >
> > > I used to have kafka-clients-3.9.0.jar earlier. There both these cases
> > were
> > > working fine. Is this a new change in behavior in
> > kafka-clients-4.1.1.jar?
> > >
> > > Thanks,
> > > Anand
> > >
> >
>
It seems to me both use an async approach. The root cause is that the
returned error code is different. Please check
https://issues.apache.org/jira/browse/KAFKA-20089 for more details.
Thanks,
Chia-Ping
Subra I <iamsubra100@gmail.com> 於 2026年2月19日週四 下午9:58寫道：
> Thanks for the response Lianet.
>
> This problem happens only in kraft mode brokers. Works fine if we use kafka
> broker in non-kraft mode. (WIth zookeeper)
>
> From what I understand: (I may be wrong. this is what I read online)
>
> In zookeeper mode, on getting a partitionsFor request, the broker creates a
> topic if not present and then immediately returns the number of partitions.
> All this happens synchronously.
>
> In kraft mode, the broker creates a topic asynchronously. So, unless u
> retry, you wont get the total number of partitions.
>
> Like I said, this is what I read.
>
> On Tue, Feb 17, 2026 at 3:19 AM Lianet Magrans <lianetm@apache.org> wrote:
>
> > Hi Anand,
> >
> > The partitionsFor API does not guarantee that a single call will return
> the
> > data if the topic does not exist. It issues a first metadata request to
> > create the topic, and waits for the api timeout for a response (timeout
> > param or default.api.timeout.ms config). Then returns whatever it gets
> > from
> > the broker in that single call (empty if the topic is not found within
> the
> > timeout, described in the java docs
> >
> >
> https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415
> > ).
> > Also there are old integration tests with the pattern you describe: a
> first
> > call to ensure topic is created, then a following one expected to
> retrieve
> > the partitions for the topic (e.g, in 3.9 branch
> >
> >
> https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
> > )
> >
> > That being said, I wonder if the difference you're mentioning is due to
> > running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
> > Not sure if tuning some Kraft configs on your cluster could help (I'm not
> > familiar with the kraft details really), but metadata is eventually
> > consistent anyways, so for sure it would be advisable to use the
> > partitionsFor considering that it may require several calls if the topic
> is
> > not found. From there, we can still consider improving for sure (e.g,
> > improve the client to continue attempting to find the topic while there
> is
> > time?). If you can file a jira with your setup (broker configs, params
> > passed to the client API) and behaviour, we can better look into it.
> >
> > Thanks!
> > Lianet
> >
> > On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
> >
> > > Hello All,
> > >
> > > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on
> version
> > > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor
> > API,
> > > I see that it works when the topic is already created in kafka. But if
> it
> > > is not yet created, the above API call returns empty - no information.
> > >
> > > I noticed that just before I call partitionsFor API, the topic is not
> > > created. But after I call this API, the topic gets created, but this
> API
> > > still returns no information. Of course, if I call the same API again,
> I
> > do
> > > get valid information for this topic.
> > >
> > > I used to have kafka-clients-3.9.0.jar earlier. There both these cases
> > were
> > > working fine. Is this a new change in behavior in
> > kafka-clients-4.1.1.jar?
> > >
> > > Thanks,
> > > Anand
> > >
> >
>
Comments
Post a Comment