Hi Peter,
The biggest issue with Java serialization has always been compatibility
between versions of the classes between producer and consumer, it can be
avoided via very careful management, but it's a lot more painful (IMO) than
other serialisation formats.
If you're looking to avoid JSON, I recommend Avro and/or Protobuf, as the
Schema Registry developed by Confluent and associated Kafka producer
serializers/ consumer deserializers explicitly support JSON, Avro and
Protobuf, and (once again, in my opinion) using the schema registry really
takes the pain out of schema version compatibility management.
Kind regards,
Liam Clarke-Hutchinson
On Mon, 1 Feb. 2021, 3:49 am Peter Penzov, <peter.penzov@gmail.com> wrote:
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
>
> I managed to implement this working concept:
> https://github.com/rcbandit111/skyobject_engine
>
> In this code example I send several Java Objects under one Kafka Topic
> and response is sent back into another Kafka Topic.
>
> I would like to ask you what are the advantages and disadvantages in
> this design of sending data between Consumer and Producer? I want to
> use a serialized Java object, not JSON because I think it's more
> optimized for performance.
>
> Can you share your feedback, please?
>
> Also is there some better way to implement this? Like for example
> using Streaming API?
>
>
> BR,
> Peter
>
The biggest issue with Java serialization has always been compatibility
between versions of the classes between producer and consumer, it can be
avoided via very careful management, but it's a lot more painful (IMO) than
other serialisation formats.
If you're looking to avoid JSON, I recommend Avro and/or Protobuf, as the
Schema Registry developed by Confluent and associated Kafka producer
serializers/ consumer deserializers explicitly support JSON, Avro and
Protobuf, and (once again, in my opinion) using the schema registry really
takes the pain out of schema version compatibility management.
Kind regards,
Liam Clarke-Hutchinson
On Mon, 1 Feb. 2021, 3:49 am Peter Penzov, <peter.penzov@gmail.com> wrote:
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
>
> I managed to implement this working concept:
> https://github.com/rcbandit111/skyobject_engine
>
> In this code example I send several Java Objects under one Kafka Topic
> and response is sent back into another Kafka Topic.
>
> I would like to ask you what are the advantages and disadvantages in
> this design of sending data between Consumer and Producer? I want to
> use a serialized Java object, not JSON because I think it's more
> optimized for performance.
>
> Can you share your feedback, please?
>
> Also is there some better way to implement this? Like for example
> using Streaming API?
>
>
> BR,
> Peter
>
Comments
Post a Comment