Dear Kafka Community,
I am reaching out for guidance on how to retrieve the current user who is
attempting to create a topic within the CreateTopicPolicy implementation in
Apache Kafka. Our goal is to restrict topic creation based on a configured
maximum limit for each user.
While working on implementing the topic creation restriction based on a
user-specific configured maximum limit, I found that the CreateTopicPolicy
interface provides limited data through the RequestMetadata object,
including parameters like numPartitions and replicationFactor. However, it
does not straightforwardly provide the username or user-specific
information required for policy enforcement.
Our objective is to identify the current user creating the topic within the
CreateTopicPolicy and use this information to enforce topic creation
restrictions based on the maximum configured limit for each user.
I have developed a solution in a Java application using Kafka APIs to
retrieve user-specif...
Posts
Hi!
We're having a Kafka 3.4.1 cluster in use which we access through a Scala Library "zio-Kafka".
This lib itself uses the Java Client in Version 3.6.1
We have to authenticate with the broker through Kerberos/SAML.
It appears that the broker regularly queries LDAP to see if the token is still valid.
Sometimes this call is taking too long, and in our client we see this log message:
[Producer clientId=producer-1] Connection to node 1546333927 (xxx.acme.corp/ 10.123.45.181:6668 ) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
The broker logs this
Potential performance problem: getGroups(user=yyacme_abc_krb) took 36492 milliseconds.
I'm tasked with making our code robust against this kind of failure (inc...
+1 (non-binding). I used the staged binaries (based on Scala 2.13) and
Maven artifacts to run my tests. All seems to work fine.
Thanks & Regards
Jakub
On Wed, Jun 19, 2024 at 10:55 AM Igor Soarez < soarez@apache.org > wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.7.1.
>
> This is a bugfix release with several fixes.
>
> Release notes for the 3.7.1 release:
> https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday June 28, 11am UTC.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~soarez/kafka-3.7.1-rc2/
>
> * Docker release artifact to be voted upon:
> apache/kafka:3.7.1-rc2
>
> * Maven artifacts to b...
Hi all,
I have an issue that spans Kafka and K8s... Do you think a Kafka bug is appropriate? Is there an alternative configuration to prevent this from happening again? Would it be any different with KRaft?
Here's what happened:
* A big disruption occurs on a node running the kafka-2 broker. Lots of I/O, OCI, Docker errors in /var/log/messages.
* The Controller sees kafka-2 disappear and it moves leadership to other brokers which have their replicas. Everything is good
* The node and kafka-2 aren't actually dead. The Controller sees kafka-2 return and marks it as part of the cluster. I guess it briefly lost its ZooKeeper registration and then reregistered itself.
* However, Kubelet is not responsive and the rest of the K8s cluster has marked the node as "Unavailable", "Kubelet stopped posting node status."
* Because of this, K8s has removed the kafka-2 pod from the headless-service, so its DNS name cannot be resolved anymore
* A preferred replica...
Hello Kafka users, developers and client-developers,
This is the second candidate for release of Apache Kafka 3.7.1.
This is a bugfix release with several fixes.
Release notes for the 3.7.1 release:
https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html
*** Please download, test and vote by Friday June 28, 11am UTC.
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~soarez/kafka-3.7.1-rc2/
* Docker release artifact to be voted upon:
apache/kafka:3.7.1-rc2
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/
* Javadoc:
https://home.apache.org/~soarez/kafka-3.7.1-rc2/javadoc/
* Tag to be voted upon (off 3.7 branch) is the 3.7.1 tag:
https://github.com/apache/kafka/releases/tag/3.7.1-rc2
* Documentation:
https://kafka.apache.org/37/documentation.ht...
First off, I'd suggest reading https://kafka.apache.org/documentation/#upgrade < https://kafka.apache.org/documentation/#upgrade > pretty carefully. It has your exact question more or less defined and outlined. As I mentioned, they are putting a lot of effort into how the world should migrate from 2.x to 3.x in the last few releases.
Without digging into it too much, it looks like you can upgrade to version 3.7.0 (I'd suggest 3.6.2 however or waiting until 3.7.1 is released) from any version of 0.8.x and up without needing to upgrade to the latest of that series. Test to be sure, it was a bit painful to upgrade to the 3.3.x series when we did that a while back but was totally worth it afterwards. But we also switched at the same time to a raft thing as well and I'm unsure how much of it was the zookeeper to raft and how much was simply the broker upgrades. Just read the upgrade guide.
On Mon Jun 17, 2024, 12:58 PM GMT, Wojciech Frycz <mailto: Wojciech.Frycz@...
Hi, Thanks for the answer. However I still do not have a clear picture on how we should go from version 2.2.1 which we are on at the moment. Options Straight jump from 2.2.1 to 3.6.3 Jump from 2.2.1 to 2.6.2 and then the second jump from 2.6.2. to 3.6.3 What's you recommendation here? Thanks, Wojciech Frycz Group Head of Engineering – Sports, MrGreen, Evoke, Data Migrations Mobile: +48 790 417 534 Grand Parade part of William Hill ul. Kotlarska 11 | 31-539 Krakow From: Sejal Patel <sejal@playerzero.ai> Date: Monday, 17 June 2024 at 14:16 To: users@kafka.apache.org <users@kafka.apache.org>, kafka-clients@googlegroups.com <kafka-clients@googlegroups.com>, dev@kafka.apache.org <dev@kafka.apache.org>, Wojciech Frycz <Wojciech.Frycz@grandparade.co.uk> Cc: Cezary Gajdzinski <Cezary.Gajdzinski@williamhi...
Hey Wojciech,
It has been a long time since we did the jump but we had waited until 3.3.x release when we did the jump and there were a lot of manual steps at that time. We then had another manual step when going to either 3.4 or 3.5 don't remember which but that one was easier cause we did have the option to turn off every broker and kraft server at once to do that one safely. I would suggest going from 2.6.2 to 3.6.2 and then stopping and waiting until 3.7.1 comes out as there is a pretty serious flaw in 3.7.0. And I would suggest probably seeing what the official migration guides have to say because my understanding is that the bulk of the changes since we started the migration is around making sure it is a smooth and safe upgrade process overall.
The other option that you could always do (our fallback plan) is to spinup a brand new cluster and to copy all the topics from the old cluster to the new cluster. Tedious and takes several days depending on how much data you have...
Hi Sejal, Thanks for reaching out. What would be your recommended strategy to go from 2.2.1 into 3.6.2 version? Can we securely do it in one jump or we should use some step in between (e.g. to jump form 2.2.1 to 2.6.2 and then from 2.6.2 to 3.6.2)? Thanks, Wojciech Frycz Group Head of Engineering – Sports, MrGreen, Evoke, Data Migrations Mobile: +48 790 417 534 Grand Parade part of William Hill ul. Kotlarska 11 | 31-539 Krakow From: Sejal Patel <sejal@playerzero.ai> Date: Saturday, 15 June 2024 at 17:28 To: users@kafka.apache.org <users@kafka.apache.org>, kafka-clients@googlegroups.com <kafka-clients@googlegroups.com>, dev@kafka.apache.org <dev@kafka.apache.org> Cc: Cezary Gajdzinski <Cezary.Gajdzinski@williamhill.com>, Wojciech Frycz <Wojciech.Frycz@grandparade.co.uk>, Slawomir Kmiecik <slawom...