Re: Kafka Streams upgrade.from config while upgrading to 2.1.0

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIzBAEBCgAdFiEESn/iOv2tmCkcP0KLVp2sL37kObwFAlyaTBgACgkQVp2sL37k
Obxn2g/+Pkqu2g6HGuWF6qam9/q8h+3pExYX62tER6WF7H+wJ+CfqVpc2fE+9Im+
RSNVgclNNaMpAsmQQX9trYI5NOEPhlFF01ZAuHrzZLe8OOivsPF/aHmmAUkbUuL+
pXqd81tfbJzG8J2w3YfIpHcj2HesDldFkbZCdsaylVBB5DPa/a2wWopJ0Rx8mUaC
T5vAijRPsV+e6kdLPTBn6oypvuyirKbeXNp7wjdHdwXIts3/YT9P9s8CjPV7cXkb
MUrplK78W/ctt8Jmdxvcj16L/gnMOBIFB+95HMjN2nw+READQ+uz/LdbVQP1T0Oe
1Vei68Ykt27T74eh6n+V/7qo911W6I+8jGGX7n4QuXBgCjoP1lG8ul+XRbg68L01
uq7Cv8A/wYGNXadOW428iUt8C7hioW6V++n85xPI91yslKl0ctARRzEjLHgzOAqy
xIkyj8Mkz2cuK9zSF3qtp6zL3vQl0z3DYpsGVbOonEpdJO5TSXahjUww//VrEZdG
u5ydNJd56Kl0F9C974cf3gkSCyNGr3lsQ6mMabDdwmANQaC0sMnQQiZi8ZHFV655
Y0Kpf1GbCFZW2RURM/RJbrMBFiqHEzPLTj5l+EX/Ydrd9MpKBFS9S+X96kt8DV6F
fO1BDXmQEBouaKU++B5rDMff1nhK0AI4k2pN6CsIUS+/xtHJLzg=
=oXDH
-----END PGP SIGNATURE-----
Not sure. How many instances to do you run? If it's only one, you don't
need the config.

-Matthias

On 3/26/19 5:17 AM, Anirudh Vyas wrote:
> Hi,
> I am in the process of upgrading my Kafka streams services from 1.1 to
> 2.1.0. I am following the upgrade guide:
> https://kafka.apache.org/20/documentation/streams/upgrade-guide .
>
> My service is running on kafka version 2.0 and using kafka streams 1.1.1. I
> updated my kafka-streams to 2.1.0 but DID NOT pass the config value
> `upgrade.from` (it is null), as can be verified from the logs
> ```
> [INFO ] 2019-03-26 18:05:49,550 [main]
> org.apache.kafka.streams.StreamsConfig:logAll: StreamsConfig values:
> application.id = application_id
> application.server =
> bootstrap.servers = [bootstrap-server-01:6667,
> bootstrap-server-02:6667]
> buffered.records.per.partition = 10000
> cache.max.bytes.buffering = 10485760
> client.id =
> commit.interval.ms = 15000
> connections.max.idle.ms = 540000
> default.deserialization.exception.handler = class
> org.apache.kafka.streams.errors.LogAndFailExceptionHandler
> default.key.serde = class
> org.apache.kafka.common.serialization.Serdes$ByteArraySerde
> default.production.exception.handler = class
> org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
> default.timestamp.extractor = class
> ziggurat.timestamp_transformer.IngestionTimeExtractor
> default.value.serde = class
> org.apache.kafka.common.serialization.Serdes$ByteArraySerde
> max.task.idle.ms = 0
> metadata.max.age.ms = 300000
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 30000
> num.standby.replicas = 0
> num.stream.threads = 3
> partition.grouper = class
> org.apache.kafka.streams.processor.DefaultPartitionGrouper
> poll.ms = 100
> processing.guarantee = at_least_once
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> replication.factor = 1
> request.timeout.ms = 40000
> retries = 0
> retry.backoff.ms = 100
> rocksdb.config.setter = null
> security.protocol = PLAINTEXT
> send.buffer.bytes = 131072
> state.cleanup.delay.ms = 600000
> state.dir = /tmp/kafka-streams
> topology.optimization = none
> upgrade.from = null
> windowstore.changelog.additional.retention.ms = 86400000
> ```
>
> The application is consuming messages as expected and is not failing.
>
> I even went through the steps mentioned in the upgrade guide and did 2
> rolling bounces with the correct config in place. Then I rolled back my
> application to Kafka streams 1.1.1 and it was running as expected.
>
> I went through KIP-268
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-268%3A+Simplify+Kafka+Streams+Rebalance+Metadata+Upgrade>
> and
> it mentions `Kafka Streams need to be configured with
> upgrade.from="<old.version>"for startup`, but the service runs fine even if
> I don't configure it.
> It also mentions `user prepares a second round of rebalance; this time, the
> configuration parameter upgrade.from must be removed for new startup` but
> the application runs as expected even if I don't remove the config
> parameter.
>
> So, my question is, what is the significance of the variable `upgrade.from`
> as it does not seem to have any effect on our kafka streams service.
>
> Regards,
> Anirudh
>