Skip to main content

Posts

Machine Learning Engineer application for Kafka – Wiem Ktari

Dear Kafka team, My name is Wiem Ktari, and I'm a computer science graduate specializing in data science and AI, building AI-powered applications, machine learning models, and data-driven solutions. I'm writing to express my interest in joining Kafka as a machine learning engineer, and I would be genuinely excited to bring my experience to your team. I'd love to work on building scalable systems for mission-critical applications in energy and utilities, where reliable data processing is essential. My experience with distributed systems and real-time data pipelines would let me contribute to your infrastructure quickly. I'd give this work my full commitment and focus on delivering robust, maintainable solutions. A few things I've delivered recently: - Built an AI-assisted pipeline reducing migration time by over 90% across 60 screens - Developed a RAG-based course recommendation system with semantic search and personalized learning paths - Created an auto...

Kafka for person image crops (10–100 KB): strict no-disk persistence requirement Hi everyone,

Hi everyone, We're designing an on-premises video analytics system (~200 camera views, NVIDIA GPUs) and evaluating Kafka for transporting person image crops between services. *Context* - Payloads: JPEG or raw person crops (~10–100 KB), potentially a few hundred per second per server. - Privacy requirement: person images must never be persisted to disk and must be deleted promptly when a person leaves the site. - Alternative: keep crops in GPU memory for local Re-ID inference and publish only metadata and 384-D embeddings (~1.5 KB each) through Kafka. *Questions* 1. *Memory-only operation:* Is there a supported production configuration that guarantees Kafka record payloads never reach persistent storage? Specifically, is using tmpfs for log.dirs supported, and what other components or OS mechanisms (KRaft metadata, swap, crash dumps, etc.) would need to be considered? 2. *Record deletion:* C...

Re: Kafka tracing by Otel

Sunil, hi. Check out my blog maybe - I did have some problems getting the complete end to end trace data working in OpenSearch (it needed Data Prepper) - there may be similar issues with your stack. And as Kafka broker data/spans is missing, you have to infer time spent in Kafka from client data - not ideal I know! Regards, Paul From: sunil chaudhari <sunilmchaudhari05@gmail.com> Date: Wednesday, 11 November 2026 at 4:15 pm To: users@kafka.apache.org <users@kafka.apache.org> Cc: Brebner, Paul <Paul.Brebner@netapp.com> Subject: Re: Kafka tracing by Otel EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Thanks Paul for the quick response. We have a telemetry elastic cluster. There are kafka clients both producers and consumers already instrumented. My problem is.... On the service map on kibana , when I see the graph of clients connecting to Kakfa, if i click kafka icon, i see only the service latency, throughput and failed tr...

Re: Kafka tracing by Otel

Thanks Paul for the quick response. We have a telemetry elastic cluster. There are kafka clients both producers and consumers already instrumented. My problem is.... On the service map on kibana , when I see the graph of clients connecting to Kakfa, if i click kafka icon, i see only the service latency, throughput and failed transaction rates. And those are all client metrics not the kafka one. I want to see kafka metrics in same servicemap coming via otlp protoocl. What I am going to try next: Kafka Broker- Jolokia agent- Edot efge collector( elastic distribution of open telemetry) - Straight to Telemetry Kibana. Does it make sense? will it show me kafka metrics to service map on Kibana? On Wed, 11 Nov 2026 at 10:03 AM, Brebner, Paul via users < users@kafka.apache.org> wrote: > Hi Sunil, > > I wish it was possible to introduce Otel tracing into Kafka brokers - this > would be extremely useful for a variety of purposes. > Unfortunatel...

Re: Kafka tracing by Otel

Hi Sunil, I wish it was possible to introduce Otel tracing into Kafka brokers - this would be extremely useful for a variety of purposes. Unfortunately, it’s only possible for Kafka clients at present (e.g. a blog from last year https://www.instaclustr.com/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-data-prepper-and-opentelemetry-collectors/ ) The Kafka protocol isn’t supported by Otel at present. Regards, Paul From: sunil chaudhari <sunilmchaudhari05@gmail.com> Date: Wednesday, 11 November 2026 at 3:26 pm To: users@kafka.apache.org <users@kafka.apache.org> Subject: Kafka tracing by Otel EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hello kafka experts, Is it possible to instrument apache kafka broker using otel agent to monitor traces of incoming and outgoing clients to kafka?

Kafka talks at Community over Code Sydney next week (Nov 18-19)

For anyone based in Sydney, there’s still an opportunity to attend our first ever Australian Community over Code in Sydney next week! There are three Kafka talks: Paul Brebner, "AI for Kafka, and Kafka for AI”, 19 November, Room 1. PoAn Yang, "How Apache Kafka handles CPU/Memory in New Consumer Protocol”, 19 November, Room 1. Ming-Yen Chung, “From Now On, I Want Every Single Message”, 19 November, Room 1. The tickets are cheap, and if you are student/academic they are only $20! Register here: https://communityovercode.apache.org/events/sydney-2026/ I hope to see you there, Paul Brebner -- NetApp Instaclustr Technology Evangelist Read my blogs at https://www.instaclustr.com/paul-brebner/ Follow me on https://www.linkedin.com/in/paul-brebner-0a547b4/

Re: Precision of the guide at Encryption and Authentication using SSL

I feel like that is not necessary, I tend to look at the current documentation for open source tools usually unless I am explicitly working with something that requires an older version, so I guess the ssl guide is enough for 4.4.0 On Tue, 27 Oct 2026 at 12:10, Gergely Harmadás <harmadasg@gmail.com> wrote: > Hi Chia-Ping, > > The plan sounds good to me. Since the document is really outdated, should > we also backport it to older supported releases like 4.3 and 4.2, what do > you think? > > > Thanks, > Gergely > > On Tue, 27 Oct 2026 at 03:48, Chia-Ping Tsai <chia7712@apache.org> wrote: > > > I haven’t looked at the PR yet: > https://github.com/apache/kafka/pull/23907 > > > > Doc fixes don’t need to wait for a release. Once the PR gets merged, we > > can file a patch to kafka-site to update the web site right away :) > > > > On 2026/10/21 22:01:28 Gergely Harmadás wrote: ...