Hello! We are currently getting hit by https://issues.apache.org/jira/browse/KAFKA-8522. We've tried setting the delete.retention.ms to be very low, but it still doesn't allow our topic to be cleansed of deletes. Setting it to 0 appears to be the only solution.
Our suppress and window changelog topics have min.cleanable.dirty.ratio set to 0.05 (the topic can get pretty big, which makes restores painfully slow), and a segment.ms of 12 hours to allow the topic to stay as small as possible without stressing the log cleaner tooo much.
With this background, my question it: will setting delete.retention.ms to 0 cause any unforeseen side effects for our restoring windows/suppress stores? Would it be possible to consume a valid message and then miss the following tombstone with these settings? And then maybe somehow cause our suppress store to emit an old should-have-already-been-deleted record? Or is the only downside the extra work we are causing in the log cleaner?
Thank you!
Our suppress and window changelog topics have min.cleanable.dirty.ratio set to 0.05 (the topic can get pretty big, which makes restores painfully slow), and a segment.ms of 12 hours to allow the topic to stay as small as possible without stressing the log cleaner tooo much.
With this background, my question it: will setting delete.retention.ms to 0 cause any unforeseen side effects for our restoring windows/suppress stores? Would it be possible to consume a valid message and then miss the following tombstone with these settings? And then maybe somehow cause our suppress store to emit an old should-have-already-been-deleted record? Or is the only downside the extra work we are causing in the log cleaner?
Thank you!
Comments
Post a Comment