Re: Transactional markers are not deleted from log segments when policy is compact

Hello Pranavi,

it sounds like this was solved in the release candidate 2.2.1-RC1 (
https://issues.apache.org/jira/browse/KAFKA-8335)
Take a look at it.

Hope that helps.


Cheers!
--
Jonathan




On Fri, May 31, 2019 at 8:59 AM Pranavi Chandramohan <
pranavic@campaignmonitor.com> wrote:

> Hi all,
>
> We use Kafka version 2.11-1.1.1. We produce and consume transactional
> messages and recently we noticed that 2 partitions of the __consumer_offset
> topic have very high disk usage (256GB)
> When we looked at the log segments for these 2 partitions, there were files
> that were 6 months old.
> By dumping the content of an old log segment using the following command
>
> kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration
> --print-data-log --files 00000000003949894887.log | less
>
>
> we found that all the records were COMMIT transaction markers.
>
> offset: 1924582627 position: 183 CreateTime: 1548972578376 isvalid:
> true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId:
> 126015 producerEpoch: 0 sequence: -1 isTransactional: true headerKeys:
> [] endTxnMarker: COMMIT coordinatorEpoch: 28
>
>
> Why are the commit transaction markers not compacted and deleted?
>
>
> Log cleaner config
>
> max.message.bytes 10000120
> min.cleanable.dirty.ratio 0.1
> compression.type uncompressed
> cleanup.policy compact
> retention.ms 2160000000
> segment.bytes 104857600
>
> # By default the log cleaner is disabled and the log retention policy
> will default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> log.cleaner.enable=true
> # give larger heap space to log cleaner
> log.cleaner.dedupe.buffer.size=1342177280
>


--
Santilli Jonathan