Hi all,
Recently we upgraded our application from the more primitive Java client APIs (kafka.javaapi.consumer.SimpleConsumer, kafka.api.FetchRequest and friends) to the more friendly poll-based org.apache.kafka.clients.consumer.KafkaConsumer using Kafka Java client libraries version 1.1.0.
The upgrade went fine and meant we could remove a LOT of custom code we had previously needed to use. This was also released into a version of the application that went into QA / staging environments of a client of ours. The upgrade (environment was not wiped) in the staging environment went fine as well without any errors. Due to an unrelated serious issue that we discovered a couple of days later, we had to advice the client a roll-back from the new application version (still had not gone into production). We didn't think this would lead to any issues.
What DID occur when the application was downgraded in the staging client environment, however, was Kafka clients starting to get a lot of offset fetch errors:
WARN [2019-04-24 01:19:43,269] custom.application.package.KafkaClient: Failed to fetch messages from Kafka. Error code: 1
Now, error code 1 means offset out of range as far as I'm aware. I would not have expected this since I was pretty sure offsets were stored in ZooKeeper with the old SimpleConsumer (I can see the offsets being stored in nodes like /consumers/<CONSUMER GROUP ID>/offsets/<TOPIC>/0) and in Kafka internal topic __consumer_offsets with the new KafkaConsumer (I can NOT see offsets in ZooKeeper for a couple of new-style consumers we already had).
The ZooKeeper-stored consumer offsets should also never have been "out of range" since there would strictly always be more events since they were last used. Could this error simply mean that the ZooKeeper offsets was TOO OLD (meaning Kafka had thrown away messages)?
I wouldn't have been surprised to see some kind of error if the offset data was shared.
This is NOT a serious issue for us at the moment, we were just surprised that the downgrade ran into errors.
Regards,
Andreas Nilsson
Recently we upgraded our application from the more primitive Java client APIs (kafka.javaapi.consumer.SimpleConsumer, kafka.api.FetchRequest and friends) to the more friendly poll-based org.apache.kafka.clients.consumer.KafkaConsumer using Kafka Java client libraries version 1.1.0.
The upgrade went fine and meant we could remove a LOT of custom code we had previously needed to use. This was also released into a version of the application that went into QA / staging environments of a client of ours. The upgrade (environment was not wiped) in the staging environment went fine as well without any errors. Due to an unrelated serious issue that we discovered a couple of days later, we had to advice the client a roll-back from the new application version (still had not gone into production). We didn't think this would lead to any issues.
What DID occur when the application was downgraded in the staging client environment, however, was Kafka clients starting to get a lot of offset fetch errors:
WARN [2019-04-24 01:19:43,269] custom.application.package.KafkaClient: Failed to fetch messages from Kafka. Error code: 1
Now, error code 1 means offset out of range as far as I'm aware. I would not have expected this since I was pretty sure offsets were stored in ZooKeeper with the old SimpleConsumer (I can see the offsets being stored in nodes like /consumers/<CONSUMER GROUP ID>/offsets/<TOPIC>/0) and in Kafka internal topic __consumer_offsets with the new KafkaConsumer (I can NOT see offsets in ZooKeeper for a couple of new-style consumers we already had).
The ZooKeeper-stored consumer offsets should also never have been "out of range" since there would strictly always be more events since they were last used. Could this error simply mean that the ZooKeeper offsets was TOO OLD (meaning Kafka had thrown away messages)?
I wouldn't have been surprised to see some kind of error if the offset data was shared.
This is NOT a serious issue for us at the moment, we were just surprised that the downgrade ran into errors.
Regards,
Andreas Nilsson
Comments
Post a Comment