-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl4zHaEACgkQu8PBaGu5
w1FDExAAq2+0EFdtrZSTQUz5uoWlyBhS/4+rb2I69DxtS3iuEzkw7/ydG/nH5oMv
EhWcFGAnoVncpoLkz1aWZiuDXd4tATVfz1a/bOzC/4XtWDIHSrY4UrimiJq88tN0
Pz4Sat7gyzqZdHnVTv4mOVgagrdWR179cuCGN3ZBJd+sOqPM6EviVshlnGz6k3Zr
e4AE2SmR8iTpufbLRABBDP3vO/WaQ1rQ5u3CXSM7nKvDFvM5exg0Jjra6ZNf3Olq
djN8apn6yB94/Z3kk5WRjrh2v/LCaMrpFV0iU5qrvucaLqs20tUuXttfXjYDHCQ1
VFgYCnkIRMl5tR3E7IPJUNDT+Ul0/SGq4dzHzRLH1Z2Yr+SNx3x1Kg7I/wVdrgMm
KkFqH8GeG0jyWdDIyHPGkUu2vJDDd+WNN8Gcd+CUKCSZLmYWdKbUPaQG9iuxX2mf
4q7kP2RKQUdGSZfhh5T+XAahMGcd7stZR7lEY/XIgexQSvEY6vNhuronoDa46mdq
QaAuCBN9dYONpz+sFX8/D34VEGqycxwoK3QefpCsDM0Ve3Buo46GexuB/59Wzwfk
jRYV7DGnmVbnEw64FpHyok2BrqN8oZEFj3t6Yy3mtjZhF/Q+UnCjlnRFpdFvvOdf
aa7ErlLFh5VY12DT8QUzAcgFkyEAPRKW2iOYFqFu3WmyN4In3QQ=
=2Dpd
-----END PGP SIGNATURE-----
What you say is correct. This is a severe bug rendering standby tasks
useless for in-memory state stores. Can you pleas open a ticket so we
can fix it asap?
-Matthias
On 1/27/20 6:05 AM, Igor Danis wrote:
> Hi all,
>
> I have question about kafka-streams, particularly in-memory state-store
> (/org.apache.kafka.streams.state.internals.InMemoryKeyValueStore/).
>
> I believe that topology is irrelevant here, but let's say I have one
> source topic with single partition
> feeding data into one statefull processor associated to single in-memory
> state store.
> This results in topology with single task.
>
> This topology is run in 2 application instances:
> - First instance (A) runs the task in active mode
> - Second instance (B) runs the task as standby
>
> Our use-case is low-latency processing, hence we need to keep rebalance
> downtime as low as possible (ideally few hundreds milliseconds).
>
> Let's say that we kill instance A, which triggers rebalance and B takes
> over the processing.
>
> We found that, when task on B transitions from STANDBY into ACTIVE mode,
> it closes in-memory state-store and effectively throws
> away any state read from changelog while it was in STANDBY. No
> checkpoints nor state is preserved.
>
> Subsequently in ACTIVE mode, it reads again changelog with
> restore-consumer. Depending on the size of the changelog this operation
> can take few minutes during which no processing is done. This happens
> despite of B having up-to-date standby-replica,
> which is really counterintuitive. What is the reason for this behavior ?
>
> Note that we initially used persistent RocksDB state-store, but we had
> simillar issues with latency (only this time it was due to RocksDB
> compaction I believe), so we prefer in-memory solution.
>
> If this question is more appropriate for developer mailing list, please
> let me know.
>
> Thanks and Regards,
> Igor
>
>
>
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl4zHaEACgkQu8PBaGu5
w1FDExAAq2+0EFdtrZSTQUz5uoWlyBhS/4+rb2I69DxtS3iuEzkw7/ydG/nH5oMv
EhWcFGAnoVncpoLkz1aWZiuDXd4tATVfz1a/bOzC/4XtWDIHSrY4UrimiJq88tN0
Pz4Sat7gyzqZdHnVTv4mOVgagrdWR179cuCGN3ZBJd+sOqPM6EviVshlnGz6k3Zr
e4AE2SmR8iTpufbLRABBDP3vO/WaQ1rQ5u3CXSM7nKvDFvM5exg0Jjra6ZNf3Olq
djN8apn6yB94/Z3kk5WRjrh2v/LCaMrpFV0iU5qrvucaLqs20tUuXttfXjYDHCQ1
VFgYCnkIRMl5tR3E7IPJUNDT+Ul0/SGq4dzHzRLH1Z2Yr+SNx3x1Kg7I/wVdrgMm
KkFqH8GeG0jyWdDIyHPGkUu2vJDDd+WNN8Gcd+CUKCSZLmYWdKbUPaQG9iuxX2mf
4q7kP2RKQUdGSZfhh5T+XAahMGcd7stZR7lEY/XIgexQSvEY6vNhuronoDa46mdq
QaAuCBN9dYONpz+sFX8/D34VEGqycxwoK3QefpCsDM0Ve3Buo46GexuB/59Wzwfk
jRYV7DGnmVbnEw64FpHyok2BrqN8oZEFj3t6Yy3mtjZhF/Q+UnCjlnRFpdFvvOdf
aa7ErlLFh5VY12DT8QUzAcgFkyEAPRKW2iOYFqFu3WmyN4In3QQ=
=2Dpd
-----END PGP SIGNATURE-----
What you say is correct. This is a severe bug rendering standby tasks
useless for in-memory state stores. Can you pleas open a ticket so we
can fix it asap?
-Matthias
On 1/27/20 6:05 AM, Igor Danis wrote:
> Hi all,
>
> I have question about kafka-streams, particularly in-memory state-store
> (/org.apache.kafka.streams.state.internals.InMemoryKeyValueStore/).
>
> I believe that topology is irrelevant here, but let's say I have one
> source topic with single partition
> feeding data into one statefull processor associated to single in-memory
> state store.
> This results in topology with single task.
>
> This topology is run in 2 application instances:
> - First instance (A) runs the task in active mode
> - Second instance (B) runs the task as standby
>
> Our use-case is low-latency processing, hence we need to keep rebalance
> downtime as low as possible (ideally few hundreds milliseconds).
>
> Let's say that we kill instance A, which triggers rebalance and B takes
> over the processing.
>
> We found that, when task on B transitions from STANDBY into ACTIVE mode,
> it closes in-memory state-store and effectively throws
> away any state read from changelog while it was in STANDBY. No
> checkpoints nor state is preserved.
>
> Subsequently in ACTIVE mode, it reads again changelog with
> restore-consumer. Depending on the size of the changelog this operation
> can take few minutes during which no processing is done. This happens
> despite of B having up-to-date standby-replica,
> which is really counterintuitive. What is the reason for this behavior ?
>
> Note that we initially used persistent RocksDB state-store, but we had
> simillar issues with latency (only this time it was due to RocksDB
> compaction I believe), so we prefer in-memory solution.
>
> If this question is more appropriate for developer mailing list, please
> let me know.
>
> Thanks and Regards,
> Igor
>
>
>
Comments
Post a Comment