-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl0Vu+IACgkQu8PBaGu5
w1GBvw/9FSeBRt2u2V9k0auWz1zUc1htbXmZmj7azO2PNJ6m8vVOEeWu6m3lClMt
A4kn9t1dvSQJ85PLJMGlI+d0TkY/U/q3UL+HmHP9FpV7dFYls9fSrWzrWAYOqo95
q62/kmcZZvE2VlsVgX3qCYjBlvPtezL8t7krw7qU23lQEv9rBQP2ITNQRVKMUewd
rM4LmZlBZDj557sohS8nDDbSr2z+K0HJeSXtUCQLp7VoutPEKQRhGxHx08bVXeHa
0oxUNelY6Wibuvif3h2flTeLaVFcaviszIjMCOrRcIhbtlKeVWmwOEnrQzW31v69
kkB/XXPoYlWeLae6lgyVSjn+dkTk4rnKhRJrA7C5iIqSiB0GCEc/yB2DQOazQd5F
1vEm9jcwd7Rq3J7+y4mlnOBNP/Zlm8Qlf8qPYgfEhpE/FoQZ3u8KAAswh2CUmNHa
/2BDMnQ1PiEVn+oK3Oxl3t85hDbTpVODZS0Xt2XTcghECVU5oIxzgKjwEMX0Rop+
HfPnsW79pMcW9lnc8Qgs77BsN3xdtwOBzb84Cwvd97fgIXbrSTmS7vGxkTlMjFd7
FCcuX6MX3heqT9xkLj2BTZwEa0asMP8WoLZLAJTB5wpXGgnzco8VZcH+1tgHB1Ss
nItdVRES0ue5zzf2Pjagn864gIEX2/zeDVbu67opAURhz7ll/YE=
=hgBr
-----END PGP SIGNATURE-----
Your expectation and understanding how GlobalKTables work is correct.
If you can reproduce the issue in an IDE, set a breakpoint in
`GlobalStreamThread#pollAndUpdate()` to see if data is fetched from the
brokers.
-Matthias
On 6/26/19 7:09 AM, Imre Gábor wrote:
> Hi,
>
> I am new to Kafka and observed a strange behavior. (I am using Kafka through the spring-kafka library)
>
> I think I misunderstand something about the "liveness" of the data in a global table.
> I thought if I define and materialize a GlobalKTable at the startup of my application, and messages arrive later to the underlying topic, the data in the store are automatically updated.
> i.e. if I have a method that queries the store of the table, it will see the most up-to-date data. (With a bit of latency of course.)
> However, it seems not to be the case:
>
>
> 1. My application defines a GlobalKTable over a topic, materialized with a name and serializers passed.
>
> 2. I have a business logic method that calls store() method with the name of this global table, and tries to get a value by key.
>
> 3. Next, I wrote a test case, that starts the application, and I send a message with a producer to the appropriate topic, and I expect I can get it by its key from the global table's store.
>
> However, it returns null. I can see the message in the topic so it is surely delivered.
>
> 4. If I iterate through the store via store.all() before calling store.get(key), the KeyValue is there with the appropriate key! However, store.get(key) still returns null.
>
> 5. I implemented hashCode and equals for my key class appropriately, so it cannot be the problem
>
> 6. Also, it is a not about a small latency, because if I wait about a minute, still cannot get the data.
>
> 7. If I modify my test case so that I do not send a test message (because there is one in the topic already), and rerun the test (which starts the application and defined the global table),
>
> the value is correctly found by key. (I guess the message from the topic is read when the table is created.)
>
> Do you have any idea what am I missing?
>
> Best regards
>
> Gabor
>
Comment: GPGTools - https://gpgtools.org
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl0Vu+IACgkQu8PBaGu5
w1GBvw/9FSeBRt2u2V9k0auWz1zUc1htbXmZmj7azO2PNJ6m8vVOEeWu6m3lClMt
A4kn9t1dvSQJ85PLJMGlI+d0TkY/U/q3UL+HmHP9FpV7dFYls9fSrWzrWAYOqo95
q62/kmcZZvE2VlsVgX3qCYjBlvPtezL8t7krw7qU23lQEv9rBQP2ITNQRVKMUewd
rM4LmZlBZDj557sohS8nDDbSr2z+K0HJeSXtUCQLp7VoutPEKQRhGxHx08bVXeHa
0oxUNelY6Wibuvif3h2flTeLaVFcaviszIjMCOrRcIhbtlKeVWmwOEnrQzW31v69
kkB/XXPoYlWeLae6lgyVSjn+dkTk4rnKhRJrA7C5iIqSiB0GCEc/yB2DQOazQd5F
1vEm9jcwd7Rq3J7+y4mlnOBNP/Zlm8Qlf8qPYgfEhpE/FoQZ3u8KAAswh2CUmNHa
/2BDMnQ1PiEVn+oK3Oxl3t85hDbTpVODZS0Xt2XTcghECVU5oIxzgKjwEMX0Rop+
HfPnsW79pMcW9lnc8Qgs77BsN3xdtwOBzb84Cwvd97fgIXbrSTmS7vGxkTlMjFd7
FCcuX6MX3heqT9xkLj2BTZwEa0asMP8WoLZLAJTB5wpXGgnzco8VZcH+1tgHB1Ss
nItdVRES0ue5zzf2Pjagn864gIEX2/zeDVbu67opAURhz7ll/YE=
=hgBr
-----END PGP SIGNATURE-----
Your expectation and understanding how GlobalKTables work is correct.
If you can reproduce the issue in an IDE, set a breakpoint in
`GlobalStreamThread#pollAndUpdate()` to see if data is fetched from the
brokers.
-Matthias
On 6/26/19 7:09 AM, Imre Gábor wrote:
> Hi,
>
> I am new to Kafka and observed a strange behavior. (I am using Kafka through the spring-kafka library)
>
> I think I misunderstand something about the "liveness" of the data in a global table.
> I thought if I define and materialize a GlobalKTable at the startup of my application, and messages arrive later to the underlying topic, the data in the store are automatically updated.
> i.e. if I have a method that queries the store of the table, it will see the most up-to-date data. (With a bit of latency of course.)
> However, it seems not to be the case:
>
>
> 1. My application defines a GlobalKTable over a topic, materialized with a name and serializers passed.
>
> 2. I have a business logic method that calls store() method with the name of this global table, and tries to get a value by key.
>
> 3. Next, I wrote a test case, that starts the application, and I send a message with a producer to the appropriate topic, and I expect I can get it by its key from the global table's store.
>
> However, it returns null. I can see the message in the topic so it is surely delivered.
>
> 4. If I iterate through the store via store.all() before calling store.get(key), the KeyValue is there with the appropriate key! However, store.get(key) still returns null.
>
> 5. I implemented hashCode and equals for my key class appropriately, so it cannot be the problem
>
> 6. Also, it is a not about a small latency, because if I wait about a minute, still cannot get the data.
>
> 7. If I modify my test case so that I do not send a test message (because there is one in the topic already), and rerun the test (which starts the application and defined the global table),
>
> the value is correctly found by key. (I guess the message from the topic is read when the table is created.)
>
> Do you have any idea what am I missing?
>
> Best regards
>
> Gabor
>
Comments
Post a Comment