Hi, i have a question about kafka transaction hanging issue:
when i restart flink job, something wrong with kafka transaction. can't consumer in read_commit status.
1. find-hanging
./bin/kafka-transactions.sh --command-config ./config/clent_jaas.conf --bootstrap-server 172.18.8.18:9092,172.18.8.19:9092,172.18.8.20:9092 find-hanging --topic ods_csys_20220805_prd_01 --partition 0
Topic Partition ProducerId ProducerEpoch CoordinatorEpoch StartOffset LastTimestamp Duration(min)
ods_csys_20220805_prd_01 0 12241081 0 -1 13045569 1667275399011 21779
ods_csys_20220805_prd_01 0 12148922 0 -1 12863388 1667156500906 23761
ods_csys_20220805_prd_01 0 12224702 0 -1 13014865 1667252649837 22158
ods_csys_20220805_prd_01 0 12175551 0 -1 12908164 1667192889534 23154
ods_csys_20220805_prd_01 0 12210364 0 -1 13001172 1667237047285 22418
ods_csys_20220805_prd_01 0 12298428 0 -1 13156486 1667348702381 20557
2. abort use startoffset
3. kafka serser log
log:
logs/server.log.2022-08-12-11:[2022-08-12 11:06:16,519] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 6385510 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
logs/server.log.2022-11-03-08:[2022-11-03 08:24:02,923] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 12360851 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
logs/server.log.2022-11-10-13:[2022-11-10 13:10:32,774] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 12845103 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 6385510 is normal, 12360851 is hanging , 12845103 is normal
4. find more infomation
./bin/kafka-console-consumer.sh --bootstrap-server 172.18.8.18:9092 --consumer.config ./config/clent_jaas.conf --topic __transaction_state --formatter "kafka.coordinator.transaction.TransactionLog\$TransactionLogMessageFormatter" --from-beginning
i don't know how cause it ?
when i restart flink job, something wrong with kafka transaction. can't consumer in read_commit status.
1. find-hanging
./bin/kafka-transactions.sh --command-config ./config/clent_jaas.conf --bootstrap-server 172.18.8.18:9092,172.18.8.19:9092,172.18.8.20:9092 find-hanging --topic ods_csys_20220805_prd_01 --partition 0
Topic Partition ProducerId ProducerEpoch CoordinatorEpoch StartOffset LastTimestamp Duration(min)
ods_csys_20220805_prd_01 0 12241081 0 -1 13045569 1667275399011 21779
ods_csys_20220805_prd_01 0 12148922 0 -1 12863388 1667156500906 23761
ods_csys_20220805_prd_01 0 12224702 0 -1 13014865 1667252649837 22158
ods_csys_20220805_prd_01 0 12175551 0 -1 12908164 1667192889534 23154
ods_csys_20220805_prd_01 0 12210364 0 -1 13001172 1667237047285 22418
ods_csys_20220805_prd_01 0 12298428 0 -1 13156486 1667348702381 20557
2. abort use startoffset
3. kafka serser log
log:
logs/server.log.2022-08-12-11:[2022-08-12 11:06:16,519] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 6385510 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
logs/server.log.2022-11-03-08:[2022-11-03 08:24:02,923] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 12360851 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
logs/server.log.2022-11-10-13:[2022-11-10 13:10:32,774] INFO [TransactionCoordinator id=3] Initialized transactionalId ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 12845103 and producer epoch 0 on partition __transaction_state-49 (kafka.coordinator.transaction.TransactionCoordinator)
ods_csys_realtime_20220805_prd_trx_02-0-9860 with producerId 6385510 is normal, 12360851 is hanging , 12845103 is normal
4. find more infomation
./bin/kafka-console-consumer.sh --bootstrap-server 172.18.8.18:9092 --consumer.config ./config/clent_jaas.conf --topic __transaction_state --formatter "kafka.coordinator.transaction.TransactionLog\$TransactionLogMessageFormatter" --from-beginning
i don't know how cause it ?
Comments
Post a Comment