Kafka

David, thanks for running this behemoth of a release. With the help of my coworker Mahsa Seifikar, we have verified the following upgrade scenario: * Provision a ZK based Kafka cluster on 3.5 and create some topics * Upgrade to 3.9 * Migrate to KRaft * Upgrade to 4.0 We found a few docs issues which have since been fixed. One minor 3.9 docs issue is still pending, but that's not a blocker. I also ran through the quickstart. +1 binding -David A On Fri, Mar 14, 2025 at 7:57 AM Federico Valeri < fedevaleri@gmail.com > wrote: > Hi David, I've tested the RC4 and this is what I've done: > > - Used the binaries to create a cluster and run clients from exampled > module > - Tested the upgrade issue we found in KAFKA-18979 > - Tested the dynamic quorum feature (both scale up and down) > - Tested group.protocol=consumer with a simple Java app > - Browsed the documentation and javadoc > - Tested the Docker images ...

Hi David, I've tested the RC4 and this is what I've done: - Used the binaries to create a cluster and run clients from exampled module - Tested the upgrade issue we found in KAFKA-18979 - Tested the dynamic quorum feature (both scale up and down) - Tested group.protocol=consumer with a simple Java app - Browsed the documentation and javadoc - Tested the Docker images +1 non binding Thanks for running the release. Cheers Fede On Fri, Mar 14, 2025 at 11:11 AM David Jacot < dajac@apache.org > wrote: > > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for release of Apache Kafka 4.0.0. > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > - Drop broker a...

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 4.0.0. - This is the first release without Apache Zookeeper - The Next Generation of the Consumer Rebalance Protocol is Generally Available - The Transactions Server-Side Defense (Phase 2) is Generally Available - Queues for Kafka is in Early Access - Kafka uses log4j2 - Drop broker and tools support for Java 11 - Remove old client protocol API versions Release notes for the 4.0.0 release: https://dist.apache.org/repos/dist/dev/kafka/4.0.0-rc4/RELEASE_NOTES.html *** Please download, test and vote by Monday, March 17, 9am PT. Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.0.0-rc4/ * Docker release artifacts to be voted upon: apache/kafka:4.0.0-rc4 apache/kafka-native:4.0.0-rc4 * Maven a...

Hi Adrien, Thanks for the report! I had some questions about your observations: > Upon > waking, the thread simply continues its sleep operation without any > awareness that a significant amount of time may have passed. Were you able to take a stack trace, attach a debugger to see the state of the thread/variables, or check logging to confirm this? time.sleep() ultimately calls Thread.sleep(), which delegates to the JVM. If there is a single Thread.sleep() invocation which never returns after the system hibernating, I don't think there is much that Kafka can do to mitigate that, and it would affect many more systems than just the credential refresh. It's worth ruling this out with some more investigation. What seems more likely to me is that either: * The thread is exiting, possibly due to a thread interruption * The thread is trying to refresh but unable to complete it successfully for some reason * The thread erroneously computes the refr...

Hi Can I perform semi structuring on the fly with Apache Kafka? Let's say if I load a pdf - can it map it to the semi structured common schema before I send it to my Database? I appreciate any guidance. Christopher Sincerely, Christopher Galliano 504.325.3500

Hello Kafka community, I've encountered an issue with OAuth authentication in Kafka when running on a system that goes to sleep/hibernates. I believe I've identified a flaw in the token refresh mechanism that affects reliability in certain environments. When using OAuth authentication between brokers and controllers, the token refresh mechanism fails after system sleep/hibernation, causing all authentication to fail until the service is restarted. I observed this on my Confluent Platform setup running on a MacBook: - OAuth token was set to refresh at 18:31:29 - System went to sleep at 18:19 - System woke up at 18:53, after the tokens had expired at 18:42 - No refresh login attempt occurred after wakeup - All authentication failed with expired tokens After reviewing the ExpiringCredentialRefreshingLogin class code, I can see the issue stems from how the refresh thread sleeps until the next scheduled refresh time: log.info ("[Princ...

Hello Kafka community, I've encountered an issue with OAuth authentication in Kafka when running on a system that goes to sleep/hibernates. I believe I've identified a flaw in the token refresh mechanism that affects reliability in certain environments. When using OAuth authentication between brokers and controllers, the token refresh mechanism fails after system sleep/hibernation, causing all authentication to fail until the service is restarted. I observed this on my Confluent Platform setup running on a MacBook: OAuth token was set to refresh at 18:31:29 System went to sleep at 18:19 System woke up at 18:53, after the tokens had expired at 18:42 No refresh login attempt occurred after wakeup All authentication failed with expired tokens After reviewing the ExpiringCredentialRefreshingLogin class code, I can see the issue stems from how the refresh thread sleeps until the next scheduled refresh time: log.info ("[Principal={}]: Expiring credential re-l...