Kafka

-----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl6qemQACgkQO4miYXKq /OgqHg/8DEQoBgK6u987AqifVAmf/qHrobjVD/ELdcaT4Drb33fc3dX+Ch86PEIj CoZE67Jc5sA4GBy8fIC8ocsYLrDsMgfBg/JsSPO20i7pSJSklDDlGc8Ru5onRt7N 1HoDNmx1dRPhMfXiQLDHKs+H4NcGmBVcrWek0vbs9YA/R/IVbu3mX2cp4t1a/U0c TM03HlPyFn5t7RsdvdupbwKA6fMWx2lwdxF6eiwkey6xqfGRlp4X/09pF8iVYBaK bAXa0zdHQbSjxy/vswJJE78mWST4lgC53ts7qO1SwZ6/FZVU8pS+fJUCRQdlikHY 7sUI2uUx3ukLceCMnFWKtpH9mRSKgwEK/7dObUlgpMacgIwVgu4wShuUhvZRQwpq a/UHd8Um6zogK5wa/9eg8G6HJfKNTxsi7BEaKPjhK3nOITJQX/OdcuJtrNpfbYcP fueb5lRti1Es+UjGl+lg0gG4DQJDCxO9k+rFXEma4nJGmxNNY3LWWPmqJYKCH/Ip RYTVf6zLY1KXkrQUed3a0ObKADPeMkxVz4b6KBdEDIvTGL9u2aQs+IUJkhiLN0iX lI7r4RxMuTSY+t6Nzr8v+Vzd6I3Q/uPCAyeFqprF/mb9PWBMrAnN8elkK0QdZ2Uz XuUKmdhKyhBNhEFzLUn7KikNt/dWtEVVtj2QoCQkXEPrOvUmrfU= =L9et -----END PGP SIGNATURE----- > Thanks for the information. So, does it mean that the state store modified >> locally by each application is replicated to all other applic...

Hi All, I am wondering if this is possible: i have been asked to use state stores as a general replicated cache among multiple instances of service instances however the state store is created through streambuilder but is not actually modified through stream processor topology however it is to be modified from outside the stream topology. So, essentially, the state store is just to be created from streambuilder and then to be used as an application level cache that will get replicated between application instances. Is this possible using state stores? Secondly, if possible, is this a good design approach? Appreciate your response since I don't know the internals of state stores.

Is there documentation or example for mirror maker 2.0 ? On 4/29/20, 9:04 PM, "Liam Clarke-Hutchinson" < liam.clarke@adscale.co.nz > wrote: [External] Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller < blak3mill3r@gmail.com > wrote: > Oh, and it looks like Confluent has released a newer replacement for > MirrorMaker called Replicator > > > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fmulti-dc-deployments%2Freplicator%2Fmigrate-replicator.html&amp;data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cdd5c2b32b1aa4567421e08d7ecbb9def%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637238162871927627&amp;sdata=f6zWwsX%2FEjBxwpiT%2FuaOBB5RSi6HnFcN13S9iukAIqA%3D&amp;...

Hi Clark My source cluster is on localhost:9092 My Target cluster was localhost:9091 I am not aware of something like Mirror maker 1 and 2 Because I am using mirror maker which come default from Apache Kafka 2.3.0 On Thu, Apr 30, 2020, 09:43 Liam Clarke-Hutchinson < liam.clarke@adscale.co.nz > wrote: > Hi Vishnu, > > I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't > auto-create topics on the target cluster. Does your target cluster on port > 9091 have the same topics created on it that exist on the source cluster on > port 9092? > > Also note that a Kafka consumer defaults to "latest" for auto offset reset > - so when starting MM 1 it won't automatically replicate all existing data, > only data created after it begins consuming. You can override this by > setting the consumer property "auto.offset.reset" to "earliest". > > Hope that helps, ...

Hi Vishnu, I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't auto-create topics on the target cluster. Does your target cluster on port 9091 have the same topics created on it that exist on the source cluster on port 9092? Also note that a Kafka consumer defaults to "latest" for auto offset reset - so when starting MM 1 it won't automatically replicate all existing data, only data created after it begins consuming. You can override this by setting the consumer property "auto.offset.reset" to "earliest". Hope that helps, Liam Clarke-Hutchinson On Thu, Apr 30, 2020 at 3:56 PM vishnu murali < vishnumurali9762@gmail.com > wrote: > i start the kafka mirror using the below configuration > > .\bin\windows\kafka-mirror-maker.bat --consumer.config > .\config\consumer.properties --producer.config .\config\producer.properties > --whitelist=".*" > > *Consumer.pro...

Thanks for the information. So, does it mean that the state store modified locally by each application is replicated to all other applications? If yes then does it happen in real time i.e. the state modified by an application is immediately reflected in the state store of the other application? On Thu, Apr 30, 2020 at 12:18 AM Matthias J. Sax < mjsax@apache.org > wrote: > This might help: > > https://stackoverflow.com/questions/40274884/is-kafka-stream-statestore-global-over-all-instances-or-just-local/40278168#40278168 > > Note, that "global stores" are also "local" (ie, on local disk). The > difference is really between being sharded or being replicated. > > > -Matthias > > On 4/29/20 11:28 AM, Bill Bejeck wrote: > > Hi Pushkar, > > > > There is a concept of a global store > > < > https://kafka.apache.org/25/javadoc/org/apache/kafka/streams/StreamsBuilder.html#addGlobal...

Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller < blak3mill3r@gmail.com > wrote: > Oh, and it looks like Confluent has released a newer replacement for > MirrorMaker called Replicator > > > https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html > > > > On Wed, Apr 29, 2020 at 6:49 PM Blake Miller < blak3mill3r@gmail.com > > wrote: > > > Hi Vishnu, > > > > Check out MirrorMaker > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > > > > This can do what you want. Note that the offsets are not copied, nor are > > the message timestamps. > > > > HTH > > > > > > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali ...