Kafka

Hello , Thanks in advance for any help. In our kafka cluster a couple of partitions in __consumer_offsets and 1 regular topic getting data corruption issue while replicas trying to read from leader. Similar messages for other partitions as well . [2022-02-28 21:57:29,941] ERROR [ReplicaFetcher replicaId=6, leaderId=1, fetcherId=2] Found invalid messages during fetch for partition __consumer_offsets-10 offset 108845487 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = 1524235439) in topic partition __consumer_offsets-10 another topic partitions with same errors [2022-02-28 22:17:00,235] ERROR [ReplicaFetcher replicaId=6, leaderId=1, fetcherId=0] Found invalid messages during fetch for partition px-11351-xxxxxx-a56c642-0 offset 11746872 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.CorruptRecordException: Record is corrupt (stored crc = 475179617) in topic partition px-1135...

I've used HPAs scaling on lag before by feeding lag metrics from Prometheus into the K8s metrics server as custom metrics. That said, you need to carefully control scaling frequency to avoid excessive consumer group rebalances. The cooperative sticky assignor can minimise pauses, but not remove them entirely. There's a lot of knobs you can use to tune HPAs these days: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior Good luck :) On Tue, 1 Mar 2022 at 08:49, David Ballano Fernandez < dfernandez@demonware.net > wrote: > Hello Guys, > > I was wondering how you guys do autoscaling of you consumers in kubernetes > if you do any. > > We have a mirrormaker-like app that mirrors data from cluster to cluster at > the same time does some topic routing. I would like to add hpa to the app > in order to scale up/down depending on avg cpu. but as you know a consumer > a...

We run Kafka Streams (Java) apps on Kubernetes to *consume*, *process* and *produce* real time data in our Kafka Cluster (running Confluent Community Edition v7.0/Kafka v3.0). How can we do a deployment of our apps in a way that limits downtime on consuming records? Our initial target was approx *2 sec* downtime a single time for each task. We are aiming to do continuous deployments of changes to the production environment, but deployments are too disruptive by causing downtime in record consumption in our apps, leading to latency in produced real time records. Since this question has already been described in detail on Stack Overflow ( https://stackoverflow.com/questions/71222496/how-to-achieve-high-availability-in-a-kafka-streams-app-during-deployment ), but has not been answered yet, we would like to refer to it instead of copy/pasting the content in this mailing list. Please let me know if you prefer to have the complete question in the mailing list instead.

Hello Guys, I was wondering how you guys do autoscaling of you consumers in kubernetes if you do any. We have a mirrormaker-like app that mirrors data from cluster to cluster at the same time does some topic routing. I would like to add hpa to the app in order to scale up/down depending on avg cpu. but as you know a consumer app has lots of variables being partitions of topics being consumed a pretty important one. Since kubernetes checks cpu avg, there are chances that pods/consumers won't be scaled up to the number of partitions possibly creating some hot spots. Anyways i would like to know how you deal if you do at all with this. thanks!

I see what the issue is – trying to resolve – looking at the consumer offset's, the ones that are logging this message are all not in sync for this broker. I read that setting this parameter - replica.lag.time.max.ms , to a time before the failure may allow for the consumer offset to get back in sync. I'm letting this run. Jim //////////// On 2/25/22, 7:09 PM, "Jim Langston" < jlangston@resolutebi.com > wrote: Hi all , after upgrading to 2.8.1 and restarting the brokers, one of the brokers is continually logging messages similar to this Feb 25 23:58:17 bigdata-worker2.dc.res0.local WARN: (Logging.scala:70) [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Reset fetch offset for partition __consumer_offsets-6 from 45647 to current leader's start offset 0 Feb 25 23:58:17 bigdata-worker2.dc.res0.local INFO: (Logging.scala:66) [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Current offset 45647 for partitio...

Thank you Malcolm. Will go through this. On Sat, Feb 26, 2022 at 2:22 AM Malcolm McFarland < mmcfarland@cavulus.com > wrote: > Maybe this could help? > https://github.com/dimas/kafka-reassign-tool > > Cheers, > Malcolm McFarland > Cavulus > > > On Fri, Feb 25, 2022 at 9:00 AM Kafka Life < lifekafka999@gmail.com > wrote: > > > Dear Experts > > > > do you have any solution for this please > > > > On Tue, Feb 22, 2022 at 8:31 PM Kafka Life < lifekafka999@gmail.com > > wrote: > > > > > Dear Kafka Experts > > > > > > Does anyone have a dynamically generated Json file based on the Under > > > replicated partition in the kafka cluster. > > > Everytime when the URP is increased to over 500 , it is a tedious job > to > > > manually create a Json file . > > > > > > I request you to share any such dynamic...

Hi Jim, Thanks for reporting the issue. Could you file a jire bug here < https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13542?filter=allopenissues > ? > I have found several blogs/user group references to the message but none have successfully stop the message Could you share these references link so that we can have more info about it? Thank you. Luke On Sat, Feb 26, 2022 at 8:16 AM Jim Langston < jlangston@resolutebi.com > wrote: > Hi all , after upgrading to 2.8.1 and restarting the brokers, one of the > brokers is continually logging messages similar to this > > Feb 25 23:58:17 bigdata-worker2.dc.res0.local WARN: (Logging.scala:70) > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Reset fetch offset > for partition __consumer_offsets-6 from 45647 to current leader's start > offset 0 > Feb 25 23:58:17 bigdata-worker2.dc.res0.local INFO: (Logging.scala:66) > [ReplicaFetcher replicaId=2, le...

Hi all , after upgrading to 2.8.1 and restarting the brokers, one of the brokers is continually logging messages similar to this Feb 25 23:58:17 bigdata-worker2.dc.res0.local WARN: (Logging.scala:70) [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Reset fetch offset for partition __consumer_offsets-6 from 45647 to current leader's start offset 0 Feb 25 23:58:17 bigdata-worker2.dc.res0.local INFO: (Logging.scala:66) [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Current offset 45647 for partition __consumer_offsets-6 is out of range, which typically implies a leader change. Reset fetch offset to 45647 I have let this run without interruption hoping it would clear on its own ( I have been rotating /var/log/syslog with fills up), but the messages just keep logging. Is there something I have missed during the upgrade ? Is there something that can be done to stop the messages ( I have found several blogs/user group references to the message but none have succe...

Maybe this could help? https://github.com/dimas/kafka-reassign-tool Cheers, Malcolm McFarland Cavulus On Fri, Feb 25, 2022 at 9:00 AM Kafka Life < lifekafka999@gmail.com > wrote: > Dear Experts > > do you have any solution for this please > > On Tue, Feb 22, 2022 at 8:31 PM Kafka Life < lifekafka999@gmail.com > wrote: > > > Dear Kafka Experts > > > > Does anyone have a dynamically generated Json file based on the Under > > replicated partition in the kafka cluster. > > Everytime when the URP is increased to over 500 , it is a tedious job to > > manually create a Json file . > > > > I request you to share any such dynamically generated script /json file. > > > > Thanks in advance. > > > >> >

Got it. I'm not totally sure how the spring wires the properties from the file to the actual configs in Streams, but just a general suggestion: in Kafka Streams you can specify the config overrides for the internally embedded producer and consumer respectively: https://kafka.apache.org/10/documentation/streams/developer-guide/config-streams.html#kafka-consumers-and-producer-configuration-parameters So if you can figure out how to wire the configs with the embedded producer and consumer, then they can indeed use different accounts for reading and writing. Guozhang On Thu, Feb 24, 2022 at 4:22 AM Alessandro Ernesto Mascherpa < alessandro.mascherpa@piksel.com > wrote: > Hello Guozhang, > > For authentication I'm using the following configuration: > > spring.kafka.properties.sasl.jaas.config = > org.apache.kafka.common.security.plain.PlainLoginModule required > username="<username>" password="<password...

Dear Experts do you have any solution for this please On Tue, Feb 22, 2022 at 8:31 PM Kafka Life < lifekafka999@gmail.com > wrote: > Dear Kafka Experts > > Does anyone have a dynamically generated Json file based on the Under > replicated partition in the kafka cluster. > Everytime when the URP is increased to over 500 , it is a tedious job to > manually create a Json file . > > I request you to share any such dynamically generated script /json file. > > Thanks in advance. > >>

Hello Guozhang, For authentication I'm using the following configuration: spring.kafka.properties.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password> "; spring.kafka.properties.ssl.truststore.location=/path/to/truststore.jks spring.kafka.properties.ssl.truststore.password=trustore-secret spring.kafka.properties.ssl.truststore.type=JKS spring.kafka.properties.security.protocol=SASL_SSL spring.kafka.properties.sasl.mechanism=PLAIN defined in a .properties file. With 'acocunt' I mean the pair username/password and the read/write rights the username has. If a simplified code snippet may be useful, the stream is created as: @Bean public KStream<String,Message> plain(StreamsBuilder builder) { KStream<String,Message> stream = builder.stream( "A" ); stream.map( ... ).to( "B" ); return stream; } ...

Hello Navneeth, Just to verify some behaviors, could you try 1) not using instance.id config, hence no static members, 2) upgrade to the latest version of Kafka, respectively (i.e. do not do them at the same time) and see if either one of them help with the imbalance issue? On Sun, Feb 20, 2022 at 2:17 AM Luke Chen < showuon@gmail.com > wrote: > Hi Navneeth, > > To know the reason why there's more than one partition in the same stream > task, we should know why the rebalance triggered. > That might have to look into the logs. > > > I have configured standby to be 1 which means there will be > one more copy of the state store and warm up by default is 2. What's the > difference, will there be 2 copies now? > > You should have "at most" 3 copies now, which is 1 standby + 2 warmup when > having enough Kafka stream instances. > > Thank you. > Luke > > On Sat, Feb 19, 2022 at 2:18 PM N...

Hello Alessandro, Could you elaborate a bit more on what authN methanisms you are using, and by `account` what do you mean explicitly? Guozhang On Wed, Feb 23, 2022 at 5:10 AM Alessandro Ernesto Mascherpa < alessandro.mascherpa@piksel.com > wrote: > Hi All, > I'm facing a problem with user authentication in Kafka streams in Kafka > v.3.0.0. > > A Java application reads from topic A as a stream and, in the same stream, > writes to topic B. > The two topics are configured with two different accounts, hence is it > feasible for a stream to read using an account and to write using another > account? And if so, how should I configure the stream? > > I'm available for further information and discussion > Thanks in advance > Alessandro > > This message is private and confidential. If you have received this > message in error, please notify the sender or servicedesk@piksel.com and > remove it from...

Hi All, I'm facing a problem with user authentication in Kafka streams in Kafka v.3.0.0. A Java application reads from topic A as a stream and, in the same stream, writes to topic B. The two topics are configured with two different accounts, hence is it feasible for a stream to read using an account and to write using another account? And if so, how should I configure the stream? I'm available for further information and discussion Thanks in advance Alessandro This message is private and confidential. If you have received this message in error, please notify the sender or servicedesk@piksel.com and remove it from your system. Piksel Inc is a Delaware corporation, whose registered office is 2100 Powers Ferry Road SE, Suite 400, Atlanta, GA 30339, USA

I *think* it should work - you just need to get your advertised.listeners set up correctly. I wrote a blog about this that should help you understand the config more: www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ -- Robin Moffatt | Staff Developer Advocate | robin@confluent.io | @rmoff On Tue, 22 Feb 2022 at 19:54, DESSEAUX Samuel (Gaz Réseau Distribution France) <samuel.desseaux@externe.grdf.fr.invalid> wrote: > Hello > > My need may be tricky but i've the following problem. > > I have a cluster Kafka behind a F5 Vip and my filebeat collector have to > sens data to topics. > > Actually,i can't write anything in the topics. > To give more details,the main goal of the vip is to secure connections > between servers and the vip > > So,is it really possible to do this architecture ? > If yes,how can i do this? > > Best regards > > Samuel Desseaux ...

Hello My need may be tricky but i've the following problem. I have a cluster Kafka behind a F5 Vip and my filebeat collector have to sens data to topics. Actually,i can't write anything in the topics. To give more details,the main goal of the vip is to secure connections between servers and the vip So,is it really possible to do this architecture ? If yes,how can i do this? Best regards Samuel Desseaux Télécharger Outlook pour Android< https://aka.ms/AAb9ysg > « Ce message est confidentiel et destiné à l'usage du (des) seul(s) destinataire(s) concerné(s). Il peut également contenir des informations à usage restreint, soumises à droits d'auteur ou à d'autres dispositions légales. Si vous l'avez reçu par erreur, nous vous prions de bien vouloir nous en informer par retour et de l'effacer de votre système. La copie du message et la communication de son contenu à...

Dear Kafka Experts Does anyone have a dynamically generated Json file based on the Under replicated partition in the kafka cluster. Everytime when the URP is increased to over 500 , it is a tedious job to manually create a Json file . I request you to share any such dynamically generated script /json file. Thanks in advance. >

Hi! Yeah sorry that's a typo, I meant 2.8.1 From: Luke Chen < showuon@gmail.com > Date: Thursday, 17 February 2022 at 03:28 To: Kafka Users < users@kafka.apache.org > Subject: [EXTERNAL] Re: Leader: none in __consumer_offsets topic CAUTION: This email originated from outside of Mandiant. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Miguel, Could you let us know which version of Kafka you're using? There's no v3.8.1 Kafka currently. Thanks. Luke On Wed, Feb 16, 2022 at 12:12 AM Miguel Angel Corral <miguel.corral@mandiant.com.invalid> wrote: > Hi, > > Recently, in a 3.8.1 Kafka cluster with 3 brokers, the topic > __consumer_offsets became leaderless: > > $ /kafka-topics.sh --zookeeper <zookeeper_addresses> --describe > --under-replicated-partitions > Topic: __consumer_offsets Partition: 0 > Leader: none ...

Hi! Yeah sorry that's a typo, I meant 2.8.1 From: Luke Chen < showuon@gmail.com > Date: Thursday, 17 February 2022 at 03:28 To: Kafka Users < users@kafka.apache.org > Subject: [EXTERNAL] Re: Leader: none in __consumer_offsets topic Hi Miguel, Could you let us know which version of Kafka you're using? There's no v3.8.1 Kafka currently. Thanks. Luke On Wed, Feb 16, 2022 at 12:12 AM Miguel Angel Corral <miguel.corral@mandiant.com.invalid> wrote: > Hi, > > Recently, in a 3.8.1 Kafka cluster with 3 brokers, the topic > __consumer_offsets became leaderless: > > $ /kafka-topics.sh --zookeeper <zookeeper_addresses> --describe > --under-replicated-partitions > Topic: __consumer_offsets Partition: 0 > Leader: none Replicas: 103,101,102 Isr: > Topic: __consumer_offsets Partition: 1 > Leader: none Replicas: 101,102,103 Isr: ...

Hi Navneeth, To know the reason why there's more than one partition in the same stream task, we should know why the rebalance triggered. That might have to look into the logs. > I have configured standby to be 1 which means there will be one more copy of the state store and warm up by default is 2. What's the difference, will there be 2 copies now? You should have "at most" 3 copies now, which is 1 standby + 2 warmup when having enough Kafka stream instances. Thank you. Luke On Sat, Feb 19, 2022 at 2:18 PM Navneeth Krishnan < reachnavneeth2@gmail.com > wrote: > Hi Guozhang, > > Thanks and sorry for the late reply. I'm overriding the > GROUP_INSTANCE_ID_CONFIG > & APPLICATION_SERVER_CONFIG. > Rest all are defaults. Even then I see more than one partition being > allocated to the same stream task. > > Also I have an additional question regarding the replicas. The default > values for the co...

Hi Guozhang, Thanks and sorry for the late reply. I'm overriding the GROUP_INSTANCE_ID_CONFIG & APPLICATION_SERVER_CONFIG. Rest all are defaults. Even then I see more than one partition being allocated to the same stream task. Also I have an additional question regarding the replicas. The default values for the configs num.standby.replicas & max.warmup.replicas are 0 & 2 respectively. I have configured standby to be 1 which means there will be one more copy of the state store and warm up by default is 2. What's the difference, will there be 2 copies now? Thanks On Fri, Feb 4, 2022 at 1:13 AM Guozhang Wang < wangguoz@gmail.com > wrote: > Hello Navneeth, > > Could you describe how you ended up with more than one partition > assigned to the same thread after certain rebalance(s)? Do you override any > default config values such as instance.id (for static consumer members), > etc? > > Also I'd suggest upgr...

On Dataproc package kafka-python does not exist not installed as standard sudo su - to root and install it as above as root pip list|grep kafka root@ctpcluster-m:~# pip install kafka-python Collecting kafka-python Downloading kafka_python-2.0.2-py2.py3-none-any.whl (246 kB) |████████████████████████████████| 246 kB 22.0 MB/s Installing collected packages: kafka-python Successfully installed kafka-python-2.0.2 hduser@ctpcluster-m: /home/hduser> pip list|grep kafka kafka-python 2.0.2 HTH view my Linkedin profile < https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/ > https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary dam...

Have you installed the correct package kafka-python? *pip install kafka-python* Collecting kafka-python Downloading kafka_python-2.0.2-py2.py3-none-any.whl (246 kB) |████████████████████████████████| 246 kB 1.9 MB/s Installing collected packages: kafka-python Successfully installed kafka-python-2.0.2 *pip list|grep kafka* *kafka-python 2.0.2* *python3* Python 3.7.3 (default, Apr 3 2021, 20:42:31) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux Type "help", "copyright", "credits" or "license" for more information. *>>> from kafka import KafkaProducer* *>>>* view my Linkedin profile < https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/ > https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this...

Hello All, I've a GCP Dataproc cluster, and i'm running a Spark StructuredStreaming job on this. I'm trying to use KafkaProducer to push aggregated data into a Kafka topic, however when i import KafkaProducer (from kafka import KafkaProducer), it gives error ``` Traceback (most recent call last): File "/tmp/7e27e272e64b461dbdc2e5083dc23202/StructuredStreaming_GCP_Versa_Sase_gcloud.py", line 14, in <module> from kafka.producer import KafkaProducer File "/opt/conda/default/lib/python3.8/site-packages/kafka/__init__.py", line 23, in <module> from kafka.producer import KafkaProducer File "/opt/conda/default/lib/python3.8/site-packages/kafka/producer/__init__.py", line 4, in <module> from .simple import SimpleProducer File "/opt/conda/default/lib/python3.8/site-packages/kafka/producer/simple.py", line 54 return '<SimpleProducer batch=%s>' %...

It can be done with the consumer API. However, you're just going to end up re-implementing what is already there in the streams DSL. It will be far easier to use the Stream DSL join functionality to accomplish this. I've never tried to do it with a simple consumer. On Wed, Feb 16, 2022 at 6:45 PM pradeep s < sreekumar.pradeep@gmail.com > wrote: > Thanks Chad! if we want to consume from multiple topics and persist to a > database , can i go with a consumer and lookup the record and update > .Requirement is to consume from item topic and price topic and create a > record in postgress . Both topic have item id in message which is the key > in postgress database . Can this be done with a simple consumer ? > > On Thu, Jan 13, 2022 at 11:11 AM Chad Preisler < chad.preisler@gmail.com > > wrote: > > > Yes Kafka streams can be used to do this. There are probably several ways > > to implement this. We did something l...

Hi Miguel, Could you let us know which version of Kafka you're using? There's no v3.8.1 Kafka currently. Thanks. Luke On Wed, Feb 16, 2022 at 12:12 AM Miguel Angel Corral <miguel.corral@mandiant.com.invalid> wrote: > Hi, > > Recently, in a 3.8.1 Kafka cluster with 3 brokers, the topic > __consumer_offsets became leaderless: > > $ /kafka-topics.sh --zookeeper <zookeeper_addresses> --describe > --under-replicated-partitions > Topic: __consumer_offsets Partition: 0 > Leader: none Replicas: 103,101,102 Isr: > Topic: __consumer_offsets Partition: 1 > Leader: none Replicas: 101,102,103 Isr: > Topic: __consumer_offsets Partition: 2 > Leader: none Replicas: 102,103,101 Isr: > Topic: __consumer_offsets Partition: 3 > Leader: none Replicas: 103,102,101 Isr: > ...

Thanks Chad! if we want to consume from multiple topics and persist to a database , can i go with a consumer and lookup the record and update .Requirement is to consume from item topic and price topic and create a record in postgress . Both topic have item id in message which is the key in postgress database . Can this be done with a simple consumer ? On Thu, Jan 13, 2022 at 11:11 AM Chad Preisler < chad.preisler@gmail.com > wrote: > Yes Kafka streams can be used to do this. There are probably several ways > to implement this. We did something like this in Java using a groupByKey() > and reduce() functions. The three topics we wanted to combine into one > topic had different schemas and different java class types. So to combine > them together into one aggregated object we did the following. > > - Create a class with data members of the three objects we wanted to > combine. Let's call it AggregateObj. > - Create a KStream for eac...

Hello, I did a test with these two rb scripts which take the time almost the same. do you have the further idea? $ cat async-pub.rb require 'kafka' kafka = Kafka.new("localhost:9092", client_id: "ruby-client", resolve_seed_brokers: true) producer = kafka.async_producer(required_acks: :all,max_buffer_size: 50_000,max_queue_size: 10_000) 10000.times do message = rand.to_s producer.produce(message, topic: "mytest") end producer.deliver_messages producer.shutdown $ cat sync-pub.rb require 'kafka' kafka = Kafka.new("localhost:9092", client_id: "ruby-client", resolve_seed_brokers: true) producer = kafka.producer(required_acks: :all,max_buffer_size: 50_000) 10000.times do message = rand.to_s producer.produce(message, topic: "mytest") end producer.deliver_messages Thanks On 2022/2/16 10:18, Luke Chen wrote: > Hi frakass, > > I thin...

Hi frakass, I think the most difference for sync and async send (or "publish" like you said), is the throughput. You said the performance is almost the same, and I would guess the "acks" config in your environment might be 0? Or maybe the produce rate is slow? Or "max.in.flight.requests.per.connection" is 1? Usually, when "acks=all", you have to wait for the records completely replicated into all brokers before server response in "sync" mode, which is why the throughput will be slow. Compared with async mode, the producer send will return immediately after appending the records, and wait for the response in callback function, no matter it's acks=0 or acks=all. Hope that helps. Luke On Wed, Feb 16, 2022 at 9:10 AM frakass < capitnfrakass@free.fr > wrote: > for a producer, is there a principle that when to use sync publishing, > and when to use async publishing? > > for the s...

for a producer, is there a principle that when to use sync publishing, and when to use async publishing? for the simple format messages, i have tested both, their performance are almost the same. Thank you. frakass

Hi, Recently, in a 3.8.1 Kafka cluster with 3 brokers, the topic __consumer_offsets became leaderless: $ /kafka-topics.sh --zookeeper <zookeeper_addresses> --describe --under-replicated-partitions Topic: __consumer_offsets Partition: 0 Leader: none Replicas: 103,101,102 Isr: Topic: __consumer_offsets Partition: 1 Leader: none Replicas: 101,102,103 Isr: Topic: __consumer_offsets Partition: 2 Leader: none Replicas: 102,103,101 Isr: Topic: __consumer_offsets Partition: 3 Leader: none Replicas: 103,102,101 Isr: Topic: __consumer_offsets Partition: 4 Leader: none Replicas: 101,103,102 Isr: Topic: __consumer_offsets Partition: 5 Leader: none Replicas: 102,101,103 Isr: Topic: __consumer_offsets Parti...

Hi, Can anyone update me on the solution requested to encrypt the credentials in kafka jaas conf file and in server.properties. Thanks, KSunil On Tue, Feb 8, 2022 at 10:46 AM Sunil Kumar < ksk702@gmail.com > wrote: > Hi, > > We have a requirement to encrypt the passwords defined in the kafka jaas > conf file after enabling SASL_SSL while starting the broker we are passing > the kafka jaas conf file as export argument in the kafka start script. > JIRA issue is reported is > https://issues.apache.org/jira/browse/KAFKA-13652 for the same above > problem. > > Please suggest if it is already addressed or alternative ways. > > > > Thanks, > KSunil > > >

Hi Deepak, Kafka 3.0 deprecated Java 8, but (as I understand it) build support will not be removed until Kafka 4.0. Therefore, you can upgrade to the 3.x release which has the log4j fixes and this will still be built with Java 8 support. Cheers, Tom Cooper On Fri, Feb 11, 2022 at 17:41, Deepak Jain < deepak.jain@cumulus-systems.com > wrote: > Hi Luke, > > Thanks for your prompt reply. > > Our application uses Java 8 but it seems the java 8 support is deprecated from Kafka 3.0.0 release onwards. > > Please let us know if Kafka is planning to upgrade Log4j to latest version in Kafka future release (2.8.x) which supports Java 8. > > Regards, > Deepak > > From: Luke Chen < showuon@gmail.com > > Sent: 11 February 2022 18:15 > To: Deepak Jain < deepak.jain@cumulus-systems.com > > Cc: users@kafka.apache.org ; Alap Patwardhan < alap@cumulus-systems.com > > Subject: Re: Kafka Log4j2.x upgrade plan > > Hi Deepak, > ...

Hi Luke, Thanks for your prompt reply. Our application uses Java 8 but it seems the java 8 support is deprecated from Kafka 3.0.0 release onwards. Please let us know if Kafka is planning to upgrade Log4j to latest version in Kafka future release (2.8.x) which supports Java 8. Regards, Deepak From: Luke Chen < showuon@gmail.com > Sent: 11 February 2022 18:15 To: Deepak Jain < deepak.jain@cumulus-systems.com > Cc: users@kafka.apache.org ; Alap Patwardhan < alap@cumulus-systems.com > Subject: Re: Kafka Log4j2.x upgrade plan Hi Deepak, The PR to upgrade to log4j 2 is already under review. And so far it looks good. So I think it's possible to be merged into v3.2.0. But still, it's not guaranteed. PR is here: https://github.com/apache/kafka/pull/7898 . Welcome to provide comments to make it get merged faster. Thank you. Luke On Fri, Feb 11, 2022 at 7:41 PM Deepak Jain < deepak.jain@cumulus-systems.com <mailto: deepak.jain...

Hi Deepak, The PR to upgrade to log4j 2 is already under review. And so far it looks good. So I think it's possible to be merged into v3.2.0. But still, it's not guaranteed. PR is here: https://github.com/apache/kafka/pull/7898 . Welcome to provide comments to make it get merged faster. Thank you. Luke On Fri, Feb 11, 2022 at 7:41 PM Deepak Jain < deepak.jain@cumulus-systems.com > wrote: > Hi Luke, > > > > First of all Congratulations. Thanks for all your contributions. > > > > Please let us know if Kafka is planning to upgrade Log4j to latest version > in Kafka future release. Our Customer is eagerly waiting and following with > us regarding the same. > > > > Regards, > > Deepak > > > > *From:* Luke Chen < showuon@gmail.com > > *Sent:* 21 January 2022 12:35 > *To:* Deepak Jain < deepak.jain@cumulus-systems.com > > *Cc:* users@kafka.apache.org ; ...

Hi Luke, First of all Congratulations. Thanks for all your contributions. Please let us know if Kafka is planning to upgrade Log4j to latest version in Kafka future release. Our Customer is eagerly waiting and following with us regarding the same. Regards, Deepak From: Luke Chen < showuon@gmail.com > Sent: 21 January 2022 12:35 To: Deepak Jain < deepak.jain@cumulus-systems.com > Cc: users@kafka.apache.org ; Alap Patwardhan < alap@cumulus-systems.com > Subject: Re: Kafka Log4j2.x upgrade plan Hi Deepak, So far, we don't have an ETA for log4j2. Please check this discussion: https://issues.apache.org/jira/browse/KAFKA-9366 Thank you. Luke On Fri, Jan 21, 2022 at 1:57 PM Deepak Jain < deepak.jain@cumulus-systems.com <mailto: deepak.jain@cumulus-systems.com >> wrote: Hi Luke, We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4...

Congrats, Luke! On Wed, Feb 9, 2022, at 3:22 PM, Guozhang Wang wrote: > The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer and > we are pleased to announce that he has accepted! > > Luke has been actively contributing to Kafka since early 2020. He has > made more than 120 commits on various components of Kafka, with notable > contributions to the rebalance protocol in Consumer and Streams (KIP-766, > KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few), as well > as making an impact on improving test stability of the project. Aside from > all his code contributions, Luke has been a great participant in > discussions across the board, a very active and helpful reviewer of other > contributors' works, all of which are super valuable and highly appreciated > by the community. > > > Thanks for all of your contributions Luke. Congratulations! > > -- Guozhang, on behalf of the Apa...

Congrats Luke! Thanks for all your contributions! Jakub On Thu, Feb 10, 2022 at 12:22 AM Guozhang Wang < wangguoz@gmail.com > wrote: > The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer and > we are pleased to announce that he has accepted! > > Luke has been actively contributing to Kafka since early 2020. He has > made more than 120 commits on various components of Kafka, with notable > contributions to the rebalance protocol in Consumer and Streams (KIP-766, > KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few), as well > as making an impact on improving test stability of the project. Aside from > all his code contributions, Luke has been a great participant in > discussions across the board, a very active and helpful reviewer of other > contributors' works, all of which are super valuable and highly appreciated > by the community. > > > Thanks for all of your contribution...

On Thu, Feb 10, 2022 at 8:45 AM Audrius Petrosius < a.petrosius@gmail.com > wrote: > Hello, > > We are encountering such issues on AWS EBS based system, nothing in AWS > logs. > > Is it memory or IO issue, as it states in one line > > in dir /srv/kafka/disk1 due to IOException > (kafka.server.LogDirFailureChannel) > java.io.IOException: Map failed and in second > > Caused by: java.lang.OutOfMemoryError: Map failed > From the error it looks like a case of a missing tuning for the `vm.max_map_count` kernel parameter: https://kafka.apache.org/documentation/#os Regards, -- Alex

Congratulations Luke! On Thu, 10 Feb 2022 at 06:41, Josep Prat <josep.prat@aiven.io.invalid> wrote: > Congrats Luke! > > ——— > Josep Prat > > Aiven Deutschland GmbH > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491715557497 > > w: aiven.io > > e: josep.prat@aiven.io > > On Thu, Feb 10, 2022, 07:07 Randall Hauch < rhauch@gmail.com > wrote: > > > Congratulations, Luke! > > > > On Wed, Feb 9, 2022 at 11:02 PM Matthias J. Sax < mjsax@apache.org > > wrote: > > > > > Congratulations! Glad to have you onboard, Luke! > > > > > > -Matthias > > > > > > On 2/9/22 16:37, Bill Bejeck wrote: > > > > Congrats Luke! Well deserved. > > > > > > > > -Bill ...

Hello, We are encountering such issues on AWS EBS based system, nothing in AWS logs. Is it memory or IO issue, as it states in one line in dir /srv/kafka/disk1 due to IOException (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed and in second Caused by: java.lang.OutOfMemoryError: Map failed > [2022-02-07 21:35:43,046] WARN [ReplicaManager broker=77] Unable to > describe replica dirs for /srv/kafka/disk1 (kafka.server.ReplicaManager) > org.apache.kafka.common.errors.KafkaStorageException: Log directory > /srv/kafka/disk1 is offline [2022-02-07 21:34:54,355] ERROR Failed to clean up log for > invent_points_of_interest-39 in dir /srv/kafka/disk1 due to IOException > (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed at > sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938) at kafka.log. > AbstractIndex.<init>(AbstractIndex.scala:124) at kafka.log.OffsetIndex.< > init>(OffsetI...