Hi,
I have a KStreams app that outputs a KTable
to a topic with cleanup policy "compact,delete".
I have the Confluent S3 Connector to store this
table in S3 where I do further analysis with hive.
Now my question is, if there's a way to trigger
log compaction right before the S3 Connector
reads the data so I store less data in S3 then
when it simply copies all data from the stream?
Thanks,
Daniel
I have a KStreams app that outputs a KTable
to a topic with cleanup policy "compact,delete".
I have the Confluent S3 Connector to store this
table in S3 where I do further analysis with hive.
Now my question is, if there's a way to trigger
log compaction right before the S3 Connector
reads the data so I store less data in S3 then
when it simply copies all data from the stream?
Thanks,
Daniel
Comments
Post a Comment