Kafka

Thanks Omer for answer. It helps to understand. Couple of follow up questions 1. Is there a way to confirm this using any kafka logs (message with encrypted info). 2. Is this error intermittent or happens always? Because I do see plaintext calls succeed for the same client and topic Regards, Nanda -----Original Message----- From: Ömer Şiar Baysal < osiarbaysal@gmail.com > Sent: Thursday, May 29, 2025 1:19 PM To: users@kafka.apache.org Cc: rradutiu@gmail.com Subject: Re: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs Hi, This is a known issue, https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4493 Basically when broker receives first few bytes of the encrypted message on its plaintext listener, it may end up trying to allocate a huge buffer for the message, because it assumes the received bytes are for plaintext listener. Since encrypted message is no longer a clear text, its first 4 byte...

Hi, This is a known issue, https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4493 Basically when broker receives first few bytes of the encrypted message on its plaintext listener, it may end up trying to allocate a huge buffer for the message, because it assumes the received bytes are for plaintext listener. Since encrypted message is no longer a clear text, its first 4 bytes can resemble anything (sometimes a huge number) thus resulting in OOM. Hope that clears it up for you. Ömer Şiar Baysal On Thu, May 29, 2025, 18:18 Nanda Naga <nandanaga@microsoft.com.invalid> wrote: > Thanks Radu for the response. > > Wondering why it is out of memory? Any config to resolve this or any > change in client to resolve this? > > Also, we use librdkafka client to connect to plaintext port. The memory > errors seems intermittent as I see produces happening to plaintext as well. > > Regards, > Nanda > > --...

Thanks Radu for the response. Wondering why it is out of memory? Any config to resolve this or any change in client to resolve this? Also, we use librdkafka client to connect to plaintext port. The memory errors seems intermittent as I see produces happening to plaintext as well. Regards, Nanda -----Original Message----- From: Radu Radutiu < rradutiu@gmail.com > Sent: Wednesday, May 28, 2025 3:08 AM To: users@kafka.apache.org Subject: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs [You don't often get email from rradutiu@gmail.com . Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] The error usually occurs when you try to connect with a client speaking TLS to the plain text port of the Kafka broker. Radu On Wed, May 28, 2025 at 12:37 AM Nanda Naga <nandanaga@microsoft.com.invalid> wrote: > I see this exception in our environment in multiple machines. What ...

The error usually occurs when you try to connect with a client speaking TLS to the plain text port of the Kafka broker. Radu On Wed, May 28, 2025 at 12:37 AM Nanda Naga <nandanaga@microsoft.com.invalid> wrote: > I see this exception in our environment in multiple machines. What config > will help to resolve this error? > > The exception I get is at > > java.lang.OutOfMemoryError: Unable to allocate 369295624 bytes > at java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632) > at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:125) > at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) > at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:228) > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:293) > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259) > at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417) > at > org.apache.kafka....

Hi there, our Kafka Streams applications occasionally show messages like the following: [AdminClient clientId=prod.index-document-aggregator-0-admin] Skipping return offset for prod.document-20 due to error UNSTABLE_OFFSET_COMMIT. As far as I understand, this originates from Kafka brokers rather than Kafka Streams, see ListConsumerOffsetsHandler#handleResponse ( https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/internals/ListConsumerGroupOffsetsHandler.java ). May someone please shed some light on this message and its implications? Or, more specifically: Does this imply possible data loss? (We're not using exactly_once semantics, duplicate processing would be not so much of an issue.) Are there broker side logs or metrics that might help identify the root cause? (Replica fetcher metrics show all zeros.) Thanks, Karsten

I see this exception in our environment in multiple machines. What config will help to resolve this error? The exception I get is at java.lang.OutOfMemoryError: Unable to allocate 369295624 bytes at java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:125) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:228) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:293) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259) at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417) at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:103) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:118) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) at org.apache.kafka.common.network.KafkaChannel.read(Kafka...

Hello. There is a weird problem where we missed almost all metrics for producers. The Kafka server version is 2.13-3.7.1. The JMX-exporter config is as follows: lowercaseOutputName: true whitelistObjectNames: - "kafka.server:*" - "kafka.log:*" - "kafka.network:*" - "kafka.controller:*" - "kafka.security:*" - "kafka.coordinator:*" - "jvm.gc:*" - "jvm.threads:*" - "jvm.memory:*" rules: - pattern: ".*" I am looking to have a metric: "kafka.server:client-id=connector-producer-my-producer,type=Produce" : {... However, when I try to dump what the Mbean has, I am getting nothing: $ java -jar jmx-dump-0.12.0-standalone.jar --dump-all -p 44544 -h 127.0.0.1 | grep -c "connector-producer" 0 I am 100% sure that I have tons of consumers. However, I don't see them in the metrics. Can we debug why the Kafka server doesn...