Hi Anand,
The partitionsFor API does not guarantee that a single call will return the
data if the topic does not exist. It issues a first metadata request to
create the topic, and waits for the api timeout for a response (timeout
param or default.api.timeout.ms config). Then returns whatever it gets from
the broker in that single call (empty if the topic is not found within the
timeout, described in the java docs
https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415).
Also there are old integration tests with the pattern you describe: a first
call to ensure topic is created, then a following one expected to retrieve
the partitions for the topic (e.g, in 3.9 branch
https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
)
That being said, I wonder if the difference you're mentioning is due to
running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
Not sure if tuning some Kraft configs on your cluster could help (I'm not
familiar with the kraft details really), but metadata is eventually
consistent anyways, so for sure it would be advisable to use the
partitionsFor considering that it may require several calls if the topic is
not found. From there, we can still consider improving for sure (e.g,
improve the client to continue attempting to find the topic while there is
time?). If you can file a jira with your setup (broker configs, params
passed to the client API) and behaviour, we can better look into it.
Thanks!
Lianet
On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
> Hello All,
>
> I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version
> 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor API,
> I see that it works when the topic is already created in kafka. But if it
> is not yet created, the above API call returns empty - no information.
>
> I noticed that just before I call partitionsFor API, the topic is not
> created. But after I call this API, the topic gets created, but this API
> still returns no information. Of course, if I call the same API again, I do
> get valid information for this topic.
>
> I used to have kafka-clients-3.9.0.jar earlier. There both these cases were
> working fine. Is this a new change in behavior in kafka-clients-4.1.1.jar?
>
> Thanks,
> Anand
>
The partitionsFor API does not guarantee that a single call will return the
data if the topic does not exist. It issues a first metadata request to
create the topic, and waits for the api timeout for a response (timeout
param or default.api.timeout.ms config). Then returns whatever it gets from
the broker in that single call (empty if the topic is not found within the
timeout, described in the java docs
https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415).
Also there are old integration tests with the pattern you describe: a first
call to ensure topic is created, then a following one expected to retrieve
the partitions for the topic (e.g, in 3.9 branch
https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176
)
That being said, I wonder if the difference you're mentioning is due to
running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case?
Not sure if tuning some Kraft configs on your cluster could help (I'm not
familiar with the kraft details really), but metadata is eventually
consistent anyways, so for sure it would be advisable to use the
partitionsFor considering that it may require several calls if the topic is
not found. From there, we can still consider improving for sure (e.g,
improve the client to continue attempting to find the topic while there is
time?). If you can file a jira with your setup (broker configs, params
passed to the client API) and behaviour, we can better look into it.
Thanks!
Lianet
On Tue, Jan 20, 2026 at 8:03 AM Subra I <iamsubra100@gmail.com> wrote:
> Hello All,
>
> I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version
> 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor API,
> I see that it works when the topic is already created in kafka. But if it
> is not yet created, the above API call returns empty - no information.
>
> I noticed that just before I call partitionsFor API, the topic is not
> created. But after I call this API, the topic gets created, but this API
> still returns no information. Of course, if I call the same API again, I do
> get valid information for this topic.
>
> I used to have kafka-clients-3.9.0.jar earlier. There both these cases were
> working fine. Is this a new change in behavior in kafka-clients-4.1.1.jar?
>
> Thanks,
> Anand
>
Comments
Post a Comment