Re: fluentd with kafka 3 issues

Hi,

this seems a fluentd issue not a kafka one.

maybe you can you try another log shipper to validate this assumption;
there are plenty and fluentd is already an old (and buggy and slow one);
you can try fluent-bit or vector maybe.

--
Raphael Mazelier

On 12/08/2022 04:58, pengyh wrote:
> do you konw if the current fluentd supports kafka 3 (my version is
> 3.2.1)?
>
> fluentd dry-run outputs:
>
> 2022-08-12 10:47:24 +0800 [info]: gem
> 'fluent-plugin-calyptia-monitoring' version '0.1.3'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch'
> version '5.2.3'
> 2022-08-12 10:47:24 +0800 [info]: gem
> 'fluent-plugin-flowcounter-simple' version '0.1.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version
> '0.18.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics'
> version '0.1.2'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-opensearch'
> version '1.0.7'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus'
> version '2.0.3'
> 2022-08-12 10:47:24 +0800 [info]: gem
> 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-record-modifier'
> version '2.1.0'
> 2022-08-12 10:47:24 +0800 [info]: gem
> 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-s3' version '1.7.1'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-sd-dns' version
> '0.1.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-systemd' version
> '1.0.5'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-td' version '1.2.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-utmpx' version
> '0.5.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-webhdfs' version
> '1.5.0'
> 2022-08-12 10:47:24 +0800 [info]: gem 'fluentd' version '1.15.1'
> 2022-08-12 10:47:24 +0800 [info]: starting fluentd-1.15.1 as dry run
> mode ruby="2.7.6"
> 2022-08-12 10:47:24 +0800 [info]: brokers has been set:
> ["127.0.0.1:9092"]
>
> my pipeline is: apache2 weblogs -> fluentd -> kafka -> spark
>
> But the issue I met is I can't write the output to the correct topic
> from fluentd, the messages always fall into the default topic, not the
> one I specified.
>
> The plugin I am using is:
> https://github.com/fluent/fluent-plugin-kafka
>
> which is built-in by td-agent (the latest version as well).
>
> Thanks.