Skip to main content

Kafka Streams - Producer attempted to produce with an old epoch.

Hi,
I have a kafka streams application deployed on 5 nodes and with full traffic I am getting the error message:

org.apache.kafka.common.errors.InvalidProducerEpochException: Producer attempted to produce with an old epoch.
Written offsets would not be recorded and no more records would be sent since the producer is fenced, indicating the task may be migrated out

I have 5 x 24 CPU/48 core machines with 128gb of ram. These machines are the kafka brokers with 2x1TB disks for kafka logs and also running the kafka Streams application.
2x replication factor on topic, topic is producing about 250k per second. I have 2 aggregations in the topology to 2 output topics, the final output topics are in the 10s of k range per second.

I'm assuming I have a bottleneck somewhere, I increased the broker thread counts and observed that this frequency of this error reduced, but it's still happening.
Here's the broker configuration I'm using now, but I might be overshooting some of these values.

num.network.threads=48
num.io.threads=48
socket.send.buffer.bytes=512000
socket.receive.buffer.bytes=512000
replica.socket.receive.buffer.bytes=1024000
socket.request.max.bytes=10485760
num.replica.fetchers=48
log.cleaner.threads=48
queued.max.requests=48000

I can't find good documentation on the effect of broker configuration on performance.

Comments