Re: Leverage multiple disks for kafka streams stores

Hi Adrian,

I am afraid that you cannot set the state directory for a single state
store to a different directory than all other stores.

Maybe the following blog post can help you debug and solve your issue:

https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance

Specifically look at the section "High disk I/O and write stalls":
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/#write-stalls

Best,
Bruno


On 19.05.22 10:56, Adrian Tubio wrote:
> Hi there,
>
> My kafka streams topology has one store that is particularly busy, that
> alongside other stores in the same topology is exhausting I/O which leads
> to write stalls and increased latency.
>
> The amount of compaction that this store does with regards to others is
> about 3/4 times more, so we were wondering if, since we have more
> disks/volumes available, would it be possible to set a different path for
> this store so it falls into a different disk?
>
> I don't seem to be able to find any way to do it, ideally it should be done
> via RocksDbConfigSetter, but that doesn't seem to offer that possibility as
> it seems the state store comes from StateStoreContext which is initialized
> from the STATE_DIR_CONFIG global setting.
>
> Has anyone done something similar?
>
> Best regards,
>
> Adrian Tubio
>