Re: Ensuring that the message is persisted after acknowledgement

Hi Sunil

The article that you shared talks about acks. But even if the message is
received by all in-sync replicas and kafka sends response back to the
producer, it is possible that none of the replicas did not flush the
messages to disk. So, if all the replicas crash for some reason, the
messages would be lost. For our application, we require some way to
guarantee that the messages are persisted to disk.

Regards,
Kunal

On Tue, Aug 24, 2021 at 8:40 PM Vairavanathan Emalayan <
vairavanathan.emalayan@cohesity.com> wrote:

>
>
> ---------- Forwarded message ---------
> From: sunil chaudhari <sunilmchaudhari05@gmail.com>
> Date: Fri, Aug 20, 2021 at 8:00 AM
> Subject: Re: Ensuring that the message is persisted after acknowledgement
> To: <users@kafka.apache.org>
> Cc: Vairavanathan Emalayan <vairavanathan.emalayan@cohesity.com>
>
>
> Hi Kunal,
> This article may help you.
>
> https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e
>
>
> Cheers,
> Sunil.
>
> On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <kunal.goyal@cohesity.com>
> wrote:
>
>> Hello,
>>
>> We are exploring using Kafka for our application. Our requirement is that
>> once we write some messages to Kafka, it should be guaranteed that the
>> messages are persisted to disk.
>> We found this
>> <
>> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
>> >
>> article which says that a Kafka broker acknowledges a record after it has
>> written the record to the buffer of the I/O device; it does not issue an
>> explicit fsync operation nor does it wait for the OS to confirm that the
>> data has been written. Is this statement true for the current
>> implementation? If so, is there any way in which we can ensure fsync is
>> called before acknowledgement of messages?
>> Any help would be appreciated.
>>
>> --
>>
>> Thanks & Regards
>>
>> Kunal Goyal
>>
>