Hi Anindya,
Currently, setting transaction.id in producer props won't enable transactions. Users need to set --transaction-duration-ms to enable transactions in kafka-producer-perf-test.
We recognize this can be confusing, so there's a JIRA ticket for it: https://issues.apache.org/jira/browse/KAFKA-16900
I've submitted a pull request to address this issue. With the update, users can enable transactions by either:
- Setting transaction.id=<id> via --producer-props
- Setting transaction.id=<id> in the config file via --producer.config
- Setting --transaction-id <id>
- Setting --transaction-duration-ms=<ms>
I believe this will resolve your issue. If you have any thoughts or suggestions, please feel free to share them on the PR or JIRA.
On 2019/10/18 19:18:15 Anindya Haldar wrote:
> Anyone who might have experienced this, or have a known solution? Would appreciate some insights here.
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>
> > On Oct 17, 2019, at 5:38 PM, Anindya Haldar <anindya.haldar@oracle.com> wrote:
> >
> > I am trying to test a 3 node Kafka cluster using the producer and consumer test perf scripts that ship with the binaries. When I am trying to enable transaction in the producer script it runs into errors. For example:
> >
> > # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 10000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all transactional.id=1111 enable.idempotence=true
> > Exception in thread "main" java.lang.IllegalStateException: Cannot perform a 'send' before completing a call to initTransactions when transactions are enabled.
> > at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:362)
> > at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
> > at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
> > at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
> > at org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
> >
> > A non-transactional test succeeds though:
> >
> > bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 10000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all
> > 10000 records sent, 6997.900630 records/sec (1.71 MB/sec), 112.32 ms avg latency, 499.00 ms max latency, 101 ms 50th, 206 ms 95th, 213 ms 99th, 214 ms 99.9th.
> >
> > The version I am using is: kafka_2.12-2.3.0
> >
> > Does the producer perf test script support transactions? If it does, how do we run the producer with transaction support?
> >
> > Sincerely,
> > Anindya Haldar
> > Oracle Responsys
> >
>
>
Currently, setting transaction.id in producer props won't enable transactions. Users need to set --transaction-duration-ms to enable transactions in kafka-producer-perf-test.
We recognize this can be confusing, so there's a JIRA ticket for it: https://issues.apache.org/jira/browse/KAFKA-16900
I've submitted a pull request to address this issue. With the update, users can enable transactions by either:
- Setting transaction.id=<id> via --producer-props
- Setting transaction.id=<id> in the config file via --producer.config
- Setting --transaction-id <id>
- Setting --transaction-duration-ms=<ms>
I believe this will resolve your issue. If you have any thoughts or suggestions, please feel free to share them on the PR or JIRA.
On 2019/10/18 19:18:15 Anindya Haldar wrote:
> Anyone who might have experienced this, or have a known solution? Would appreciate some insights here.
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>
> > On Oct 17, 2019, at 5:38 PM, Anindya Haldar <anindya.haldar@oracle.com> wrote:
> >
> > I am trying to test a 3 node Kafka cluster using the producer and consumer test perf scripts that ship with the binaries. When I am trying to enable transaction in the producer script it runs into errors. For example:
> >
> > # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 10000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all transactional.id=1111 enable.idempotence=true
> > Exception in thread "main" java.lang.IllegalStateException: Cannot perform a 'send' before completing a call to initTransactions when transactions are enabled.
> > at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:362)
> > at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
> > at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
> > at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
> > at org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
> >
> > A non-transactional test succeeds though:
> >
> > bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 10000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all
> > 10000 records sent, 6997.900630 records/sec (1.71 MB/sec), 112.32 ms avg latency, 499.00 ms max latency, 101 ms 50th, 206 ms 95th, 213 ms 99th, 214 ms 99.9th.
> >
> > The version I am using is: kafka_2.12-2.3.0
> >
> > Does the producer perf test script support transactions? If it does, how do we run the producer with transaction support?
> >
> > Sincerely,
> > Anindya Haldar
> > Oracle Responsys
> >
>
>
Comments
Post a Comment