Kafka

Hi Team, I am facing error Exception in thread " topic_name-0ce8394a-2394-4919-e46ce630c6ad-StreamThread-4" Java.lang.IllegalStateException :Tried to lookup lag for unknown task 1_0. Please suggest a solution. Thanks, Hima

Thank you Matthias for the clarification -----Original Message----- From: Matthias J. Sax <mjsax@mailbox.org.INVALID> Sent: Thursday, October 17, 2024 8:46 AM To: users@kafka.apache.org Subject: Re: [EXT MSG] Re: KAFKA POLL Not sure if I understand (1). For (2), you don't need to worry about it. The consumer will handle all this for you internally, and only return a full message from `poll()`. -Matthias On 10/16/24 00:16, Sathishkumar Palanisamy wrote: > Thanks for response , understood, > > 1. > Exception classes isn't available for any fetching related i believe > in javadocs TOPIC -> testing, partition count =1, offset count = 2 > 2. > Assuming the message 1st topics 1st message we are trying to read > "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut...

Not sure if I understand (1). For (2), you don't need to worry about it. The consumer will handle all this for you internally, and only return a full message from `poll()`. -Matthias On 10/16/24 00:16, Sathishkumar Palanisamy wrote: > Thanks for response , understood, > > 1. > Exception classes isn't available for any fetching related i believe in javadocs > TOPIC -> testing, partition count =1, offset count = 2 > 2. > Assuming the message 1st topics 1st message we are trying to read > "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est lab...

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.8.1. This is a bugfix release with several fixes. Release notes for the 3.8.1 release: https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc0/RELEASE_NOTES.html **** Please download, test and vote by Monday, October 21, 9am ET* Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc0/ * Docker release artifacts to be voted upon: apache/kafka:3.8.1-rc0 apache/kafka-native:3.8.1-rc0 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc0/javadoc/ * Tag to be voted upon (off 3.8 branch) is the 3.8.1 tag: https://github.com/apache/kafka/releases/tag/3.8.1-rc0 * Doc...

Thanks for response , understood, 1. Exception classes isn't available for any fetching related i believe in javadocs TOPIC -> testing, partition count =1, offset count = 2 2. Assuming the message 1st topics 1st message we are trying to read "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum." Under network latency of 1 kbps bandwidth, 1mb offset message with value of fetch.min.bytes = 1 bytes along with fetch.max.wait.ms = 0.5 seconds , it will fetch 1 kilobytes of data but that data if is in avro serialization it will be in binary right, so the fetched data in b...

1. As pointed out in the JavaDocs > @param timeoutMs The time, in milliseconds, spent waiting in poll if data is not available in the buffer. 2. If there is no data with the timeout, `poll()` return an "empty" `ConsumerRecords` object. For error/exception I cannot say, as it depends on the error/exception... The exception which could be thrown are documented in the JavaDocs, so maybe you can start there? > consumerRecords<String, String> records = consumer.poll(10000L); > * whether this records count will be 0 in this case ? Yes, the `ConsumerRecords` object will be empty. > if poll gets 500kb from topic in first poll within 10s after processing assuming no errors comes up and records count is 0 then in next poll whether next 500kb is fetched or it will attempt to again fetch full message ? If the message is 1MB in size, it will either be fetched completely not not at all. There will only be one fetch request an...

Hi Team, Does anyone know in the experimental scenario, JAVA runtime 8 or 11 1. Kafka's topic -> Testing_topic 2. Kafka's topic partition count -> 1 3. Kafka's topic partition offset -> 2 [two messages are available, each weight about 1 mb ] 4. Bandwidth of 1 kb and Kafka's topic offset's message of 1 mb is available each 5. In consumer scenario * While polling the consumer using consumer.poll(10000L) deprecated method supporting the long argument * With properties (exactly used properties) i. props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap.server.config.url); ii. props.put(ConsumerConfig.GROUP_ID_CONFIG, group-1); ...