Hi,
It does look as index corruption... Can you post script that stops kafka?
On Wednesday, April 29, 2020, 06:38:18 PM GMT+2, JP MB <jose.brandao1994@gmail.com> wrote:
>
> Can you try using the console consumer to display messages/keys and
> timestamps ?
> --property print.key=true --property print.timestamp=true
There are a lot off messages so I'm picking an example without and with
timeindex entry. All of them have a null key:
Offset 57 CreateTime:1588074808027 Key:null - no time index
Offset 144 CreateTime:1588157145655 Key:null - has time index
Hmm, how are you doing your rolling deploys?
It's rollout deployment, we take one node down and spin up another a new
one. One at a time.
I'm wondering if the time indexes are being corrupted by unclean
> shutdowns. I've been reading code and the only path I could find that led
> to a largest
> timestamp of 0 was, as you've discovered, where there was no time index.
> WRT to the corruption - the broker being SIGKILLed (systemctl by default
> sends SIGKILL 90 seconds after SIGTERM, and our broker needed 120s to shut
> down cleanly) has caused index corruption for us in the past - although in
> our case it was recovered from automatically by the broker. Just took 2
> hours.
This would be a perfect justification for it but we use systemctl stop and
it takes around 4 seconds to shut down so I believe it ends gracefully
before SIGKILL?
Also, are you moving between versions with these deploys?
No, we have several clusters where this is happening. The information I
showed you is from a cluster with version 2.3 but with 10.2 for inter
broker protocol communication and log format. We have also experienced this
in fully updated 2.4 and 2.4.1 clusters. But to sum, the experiences are
done always deploying (again) the version already there.
Thanks all for the efforts so far.
...
It does look as index corruption... Can you post script that stops kafka?
On Wednesday, April 29, 2020, 06:38:18 PM GMT+2, JP MB <jose.brandao1994@gmail.com> wrote:
>
> Can you try using the console consumer to display messages/keys and
> timestamps ?
> --property print.key=true --property print.timestamp=true
There are a lot off messages so I'm picking an example without and with
timeindex entry. All of them have a null key:
Offset 57 CreateTime:1588074808027 Key:null - no time index
Offset 144 CreateTime:1588157145655 Key:null - has time index
Hmm, how are you doing your rolling deploys?
It's rollout deployment, we take one node down and spin up another a new
one. One at a time.
I'm wondering if the time indexes are being corrupted by unclean
> shutdowns. I've been reading code and the only path I could find that led
> to a largest
> timestamp of 0 was, as you've discovered, where there was no time index.
> WRT to the corruption - the broker being SIGKILLed (systemctl by default
> sends SIGKILL 90 seconds after SIGTERM, and our broker needed 120s to shut
> down cleanly) has caused index corruption for us in the past - although in
> our case it was recovered from automatically by the broker. Just took 2
> hours.
This would be a perfect justification for it but we use systemctl stop and
it takes around 4 seconds to shut down so I believe it ends gracefully
before SIGKILL?
Also, are you moving between versions with these deploys?
No, we have several clusters where this is happening. The information I
showed you is from a cluster with version 2.3 but with 10.2 for inter
broker protocol communication and log format. We have also experienced this
in fully updated 2.4 and 2.4.1 clusters. But to sum, the experiences are
done always deploying (again) the version already there.
Thanks all for the efforts so far.
...
Comments
Post a Comment