Hi Yoda,
for certain cases, Kafka Streams allows you to specify handlers that
skip the problematic record. Those handlers are:
1. deserialization exception handler configured in
default.deserialization.exception.handler
2. time extractor set in default.timestamp.extractor and in the Consumed
object
3. production exception handler configured in
default.production.exception.handler
Kafka Streams provides implementations for handlers 1 and 2 to skip the
problematic records, that are LogAndContinueExceptionHandler and
LogAndSkipOnInvalidTimestamp, respectively.
For some more details have a look at
https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling
If problematic records cause an exception in user code, the user code
needs to provide functionality to skip the problematic record.
Best,
Bruno
On 10.08.21 13:26, Yoda Jedi Master wrote:
> Hi Bruno, thank you for your answer.
> I mean that the message that caused the exception was consumed and replaced
> thread will continue from the next message. How then does it handle
> uncaught exceptions, if it will fail again?
>
>
> On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna <cadonna@apache.org> wrote:
>
>> Hi Yoda,
>>
>> What do you mean exactly with "skipping that failed message"?
>>
>> Do you mean a record consumed from a topic that caused an exception that
>> killed the stream thread?
>>
>> If the record killed the stream thread due to an exception, for example,
>> a deserialization exception, it will probably also kill the next stream
>> thread that will read that record. Replacing a stream thread does not
>> skip records but it can result in duplicate records depending on the
>> application's processing mode determined by the
>> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
>>
>> Best,
>> Bruno
>>
>>
>>
>> On 10.08.21 11:15, Luke Chen wrote:
>>> Hi Yoda,
>>> For your question:
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>> be replaced with another thread and it will continue processing messages,
>>> skipping that failed message?
>>>
>>> --> Yes, if everything goes well after `replace thread`, you can ignore
>>> this failed message. Just one reminder that you should check the failed
>>> message to avoid this `uncaught exception` thrown again, because if this
>>> happens frequently, it'll impact application performance.
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yodajema@gmail.com>
>> wrote:
>>>
>>>> "REPLACE_THREAD - Replaces the thread receiving the exception and
>>>> processing continues with the same number of configured threads. (Note:
>>>> this can result in duplicate records depending on the application's
>>>> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
>>>>
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>>> be replaced with another thread and it will continue processing
>> messages,
>>>> skipping that failed message?
>>>>
>>>
>>
>
for certain cases, Kafka Streams allows you to specify handlers that
skip the problematic record. Those handlers are:
1. deserialization exception handler configured in
default.deserialization.exception.handler
2. time extractor set in default.timestamp.extractor and in the Consumed
object
3. production exception handler configured in
default.production.exception.handler
Kafka Streams provides implementations for handlers 1 and 2 to skip the
problematic records, that are LogAndContinueExceptionHandler and
LogAndSkipOnInvalidTimestamp, respectively.
For some more details have a look at
https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling
If problematic records cause an exception in user code, the user code
needs to provide functionality to skip the problematic record.
Best,
Bruno
On 10.08.21 13:26, Yoda Jedi Master wrote:
> Hi Bruno, thank you for your answer.
> I mean that the message that caused the exception was consumed and replaced
> thread will continue from the next message. How then does it handle
> uncaught exceptions, if it will fail again?
>
>
> On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna <cadonna@apache.org> wrote:
>
>> Hi Yoda,
>>
>> What do you mean exactly with "skipping that failed message"?
>>
>> Do you mean a record consumed from a topic that caused an exception that
>> killed the stream thread?
>>
>> If the record killed the stream thread due to an exception, for example,
>> a deserialization exception, it will probably also kill the next stream
>> thread that will read that record. Replacing a stream thread does not
>> skip records but it can result in duplicate records depending on the
>> application's processing mode determined by the
>> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
>>
>> Best,
>> Bruno
>>
>>
>>
>> On 10.08.21 11:15, Luke Chen wrote:
>>> Hi Yoda,
>>> For your question:
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>> be replaced with another thread and it will continue processing messages,
>>> skipping that failed message?
>>>
>>> --> Yes, if everything goes well after `replace thread`, you can ignore
>>> this failed message. Just one reminder that you should check the failed
>>> message to avoid this `uncaught exception` thrown again, because if this
>>> happens frequently, it'll impact application performance.
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yodajema@gmail.com>
>> wrote:
>>>
>>>> "REPLACE_THREAD - Replaces the thread receiving the exception and
>>>> processing continues with the same number of configured threads. (Note:
>>>> this can result in duplicate records depending on the application's
>>>> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
>>>>
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>>> be replaced with another thread and it will continue processing
>> messages,
>>>> skipping that failed message?
>>>>
>>>
>>
>
Comments
Post a Comment