On Thu, Aug 26, 2021 at 4:44 PM Shantam Garg(Customer Service and Transact)
<shantam.garg@flipkart.com> wrote:
> Hello all,
>
> I have come across this behavior in our production cluster where the*
> Kafka consumers and heartbeat threads are getting stuck/hang-up* without
> any error.
>
> As the heartbeat threads as stuck - the session timeout is breached and
> the consumers are thrown on the live consumer group. I have tried debugging
> it by enabling debug logs for Coordinator and Consumer but couldn't find
> the reason for this, my hunch is that it's getting stuck in some deadlock
> but not sure what's causing it.
> Please let me know if anyone has any idea on how to debug this issue.
>
> Kafka version:* 2.7*
> Consumer client:* kafka-client:2.7*
>
> *Configs:*
> max.poll.interval.ms: 300000
> session.timeout.ms: 10000
> heartbeat.interval.ms: 3000
> fetch.min.bytes: 1
> fetch.max.bytes: 10485760
> (Note: I have tried increasing the session timeout to 60 seconds also but
> the same issue happens.)
>
>
> *Coordinator logs:*INFO [2021-08-26 13:38:10,136] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]:
> *Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> in group fkw_secor_prod_group_live has failed, removing it from the
> group INFO [2021-08-26 13:38:10,136] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Preparing
> to rebalance group fkw_secor_prod_group_live in state PreparingRebalance
> with old generation 1359 (__consumer_offsets-180) (reason: removing member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> on heartbeat expiration)*
> INFO [2021-08-26 13:38:10,140] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_31-c80c0c1d-4bd9-41a8-b6e3-6f7ecd7447b2
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,363] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_33-bed07f3c-3a10-4e51-bab6-d921de81083e
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,436] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_27-5f98ee05-633c-4cee-99b1-03e4f98ae52d
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,484] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_29-b5ddfc49-5ce1-41fa-b166-2222bcc1554d
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,545] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_34-88dd2f87-9b35-4966-9436-6ea4019e1c80
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,367] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_35-fa1db325-b70f-4bba-97c3-3170298cb194
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,421] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_30-68cdeafa-c6cc-4ac9-aa9c-6ed8b03e2b42
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,500] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_26-8bfb73de-7c1e-4bc2-a65d-4ce66dfb8cd8
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:13,055] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_28-fe52a1cf-3380-407c-91dd-9baa059e4bf8
> in group fkw_secor_prod_group_live has failed, removing it from the group
> * INFO [2021-08-26 13:38:13,056] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Group
> fkw_secor_prod_group_live with generation 1360 is now empty
> (__consumer_offsets-180)*
>
> *Last consumer Logs for member
> (fkw-secor-fkw-secor-prod-cluster-8344558_14763_32):*
> 2021-08-26 13:38:03,009 [Thread-10]
> (org.apache.kafka.clients.consumer.internals.Fetcher:266) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending READ_UNCOMMITTED
> IncrementalFetchRequest(toSend=(ing.fkw.demand.tracking.clickstream-7),
> toForget=(), implied=(ing.fkw.credit.tijori.LoanAccount-0,
> cdc.fkw.ful.ssms.shipment.shipment_wave-1,
> cdc.fkw.ful.inventory.fkw_wh_inventory.wid_wsn-0,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_inventory_issue-1,
> cdc.fkw.ful.picking.picking.product_short_qty_view-3,
> cdc.fkw.credit.tijori.ffb_tijori.CardPaymentInstrument-2,
> cdc.fkw.ful.ro.returns_b2c_production.qc_verifications-2,
> cdc.fkw.ful.staging.staging.consignment-4,
> cdc.fkw.ful.staging.staging.order_container-1,
> cdc.fkw.ful.qc.qc.qc_check_choices-0,
> cdc.fkw.ful.sourcemaster.source_master_v3.sources-3,
> cdc.fkw.shopping.pcm.ffb_pcm.media_type-0,
> cdc.fkw.ful.qc.qc.qc_answer_choices-0,
> cdc.fkw.ful.sourcemaster.source_master_v3.seller_ids-1,
> ing.fkw.shopping.catalog.taxonomy-1,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.entity_tags-4,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_failures-3,
> cdc.fkw.ful.receiving.flipkart_receiving.consignment-0,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_areas-3,
> cdc.fkw.ful.ro.returns_b2c_production.shipments-3,
> cdc.fkw.ful.qc.qc.qc_assets-1,
> cdc.fkw.ful.sourcemaster.source_master_v3.asset_metadata-3,
> cdc.fkw.ful.receiving.flipkart_receiving.wsn_receiving_details-4,
> cdc.fkw.ful.sortation.flipkart_sortation.task-3,
> ing.fkw.shopping.transact.orderAdjustments-4,
> cdc.fkw.credit.khaata.khaata.emi_installments-0,
> cdc.fkw.credit.khaata.khaata.transactions-1,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_asset_lock_details-3,
> cdc.fkw.ful.ssms.shipment.order_shipment-0,
> cdc.fkw.ful.ssms.shipment.order_shipment_v2-2,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_attributes-3,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_unit_attributes-6,
> cdc.fkw.ful.ssms.shipment.connection-1,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_unit_histories-6))
> to broker 10.48.130.211:9092 (id: 3 rack: null)
> 2021-08-26 13:38:03,009 [Thread-10]
> (org.apache.kafka.clients.NetworkClient:522) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending FETCH request with header
> RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710600) and timeout 30000 to node 3:
> {replica_id=-1,max_wait_ms=1000,min_bytes=1,max_bytes=10485760,isolation_level=0,session_id=1416968662,session_epoch=233604,topics=[{topic=ing.fkw.demand.tracking.clickstream,partitions=[{partition=7,current_leader_epoch=14,fetch_offset=4055570,last_fetched_epoch=-1,log_start_offset=-1,partition_max_bytes=1048576,_tagged_fields={}}],_tagged_fields={}}],forgotten_topics_data=[],rack_id=,_tagged_fields={}}
> 2021-08-26 13:38:03,010 [Thread-10]
> (com.pinterest.secor.reader.SecorKafkaMessageIterator:59) INFO Consumer
> poll called from memberId ::
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32
>
>
> *Last heartbeat logs:*2021-08-26 13:37:59,721
> [kafka-coordinator-heartbeat-thread | fkw_secor_prod_group_live]
> (org.apache.kafka.clients.NetworkClient:884) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710560):
> org.apache.kafka.common.requests.FetchResponse@5eaf4720
> 2021-08-26 13:37:59,721 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 2 response
> partition(s), 33 implied partition(s)
> 2021-08-26 13:37:59,722 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055557
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055558, lastStableOffset = 4055558,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=806)
> 2021-08-26 13:37:59,722 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 121914674 for partition cdc.fkw.ful.ssms.shipment.connection-1 returned
> fetch data (error=NONE, highWaterMark=121914737, lastStableOffset =
> 121914737, logStartOffset = 84087991, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=4320)
> *2021-08-26 13:38:00,134 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending Heartbeat request with
> generation 1359 and member id
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> to coordinator 10.49.115.249:9092 <http://10.49.115.249:9092> (id:
> 2147483646 rack: null)*
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:522)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending HEARTBEAT request with header
> RequestHeader(apiKey=HEARTBEAT, apiVersion=4,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710563) and timeout 30000 to node 2147483646:
> {group_id=fkw_secor_prod_group_live,generation_id=1359,member_id=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2,group_instance_id=null,_tagged_fields={}}
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 1 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710553):
> org.apache.kafka.common.requests.FetchResponse@547d0cb2
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 1 sent an incremental fetch
> response with throttleTimeMs = 0 for session 751564104 with 0 response
> partition(s), 56 implied partition(s)
> 2021-08-26 13:38:00,235 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710566):
> org.apache.kafka.common.requests.FetchResponse@4c8a341f
> 2021-08-26 13:38:00,236 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:00,237 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055560
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055561, lastStableOffset = 4055561,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=765)
> 2021-08-26 13:38:00,623 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 1 sent an incremental fetch
> response with throttleTimeMs = 0 for session 751564104 with 1 response
> partition(s), 55 implied partition(s)
> 2021-08-26 13:38:00,626 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 16596199 for partition
> cdc.fkw.ful.symphony.fulfillment_b2c_production.mappings-6 returned fetch
> data (error=NONE, highWaterMark=16596229, lastStableOffset = 16596229,
> logStartOffset = 14934897, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=1783)
> 2021-08-26 13:38:00,827 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710576):
> org.apache.kafka.common.requests.FetchResponse@4070cb5c
> 2021-08-26 13:38:00,829 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:00,829 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055563
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055564, lastStableOffset = 4055564,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=775)
> 2021-08-26 13:38:01,268 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710580):
> org.apache.kafka.common.requests.FetchResponse@64420235
> 2021-08-26 13:38:01,268 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:01,269 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055564
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055565, lastStableOffset = 4055565,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=782)
> 2021-08-26 13:38:02,774 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710597):
> org.apache.kafka.common.requests.FetchResponse@58668a93
> 2021-08-26 13:38:02,775 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:02,776 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 121915139 for partition cdc.fkw.ful.ssms.shipment.connection-1 returned
> fetch data (error=NONE, highWaterMark=121915161, lastStableOffset =
> 121915161, logStartOffset = 84087991, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=1510)
>
>
> Regards,
> Shantam Garg
>
>
>
--
*-----------------------------------------------------------------------------------------*
*This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee,
you should not disseminate, distribute or copy this email. Please notify
the sender immediately by email if you have received this email by mistake
and delete this email from your system. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this information is
strictly prohibited.*****
****
*Any views or opinions presented in this
email are solely those of the author and do not necessarily represent those
of the organization. Any information on shares, debentures or similar
instruments, recommended product pricing, valuations and the like are for
information purposes only. It is not meant to be an instruction or
recommendation, as the case may be, to buy or to sell securities, products,
services nor an offer to buy or sell securities, products or services
unless specifically stated to be so on behalf of the Flipkart group.
Employees of the Flipkart group of companies are expressly required not to
make defamatory statements and not to infringe or authorise any
infringement of copyright or any other legal right by email communications.
Any such communication is contrary to organizational policy and outside the
scope of the employment of the individual concerned. The organization will
not accept any liability in respect of such communication, and the employee
responsible will be personally liable for any damages or other liability
arising.*****
****
*Our organization accepts no liability for the
content of this email, or for the consequences of any actions taken on the
basis of the information *provided,* unless that information is
subsequently confirmed in writing. If you are not the intended recipient,
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited.*
_-----------------------------------------------------------------------------------------_
<shantam.garg@flipkart.com> wrote:
> Hello all,
>
> I have come across this behavior in our production cluster where the*
> Kafka consumers and heartbeat threads are getting stuck/hang-up* without
> any error.
>
> As the heartbeat threads as stuck - the session timeout is breached and
> the consumers are thrown on the live consumer group. I have tried debugging
> it by enabling debug logs for Coordinator and Consumer but couldn't find
> the reason for this, my hunch is that it's getting stuck in some deadlock
> but not sure what's causing it.
> Please let me know if anyone has any idea on how to debug this issue.
>
> Kafka version:* 2.7*
> Consumer client:* kafka-client:2.7*
>
> *Configs:*
> max.poll.interval.ms: 300000
> session.timeout.ms: 10000
> heartbeat.interval.ms: 3000
> fetch.min.bytes: 1
> fetch.max.bytes: 10485760
> (Note: I have tried increasing the session timeout to 60 seconds also but
> the same issue happens.)
>
>
> *Coordinator logs:*INFO [2021-08-26 13:38:10,136] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]:
> *Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> in group fkw_secor_prod_group_live has failed, removing it from the
> group INFO [2021-08-26 13:38:10,136] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Preparing
> to rebalance group fkw_secor_prod_group_live in state PreparingRebalance
> with old generation 1359 (__consumer_offsets-180) (reason: removing member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> on heartbeat expiration)*
> INFO [2021-08-26 13:38:10,140] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_31-c80c0c1d-4bd9-41a8-b6e3-6f7ecd7447b2
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,363] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_33-bed07f3c-3a10-4e51-bab6-d921de81083e
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,436] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_27-5f98ee05-633c-4cee-99b1-03e4f98ae52d
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,484] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_29-b5ddfc49-5ce1-41fa-b166-2222bcc1554d
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:10,545] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_34-88dd2f87-9b35-4966-9436-6ea4019e1c80
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,367] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_35-fa1db325-b70f-4bba-97c3-3170298cb194
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,421] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_30-68cdeafa-c6cc-4ac9-aa9c-6ed8b03e2b42
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:11,500] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_26-8bfb73de-7c1e-4bc2-a65d-4ce66dfb8cd8
> in group fkw_secor_prod_group_live has failed, removing it from the group
> INFO [2021-08-26 13:38:13,055] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Member
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_28-fe52a1cf-3380-407c-91dd-9baa059e4bf8
> in group fkw_secor_prod_group_live has failed, removing it from the group
> * INFO [2021-08-26 13:38:13,056] [executor-Heartbeat][]
> kafka.coordinator.group.GroupCoordinator - [GroupCoordinator 1]: Group
> fkw_secor_prod_group_live with generation 1360 is now empty
> (__consumer_offsets-180)*
>
> *Last consumer Logs for member
> (fkw-secor-fkw-secor-prod-cluster-8344558_14763_32):*
> 2021-08-26 13:38:03,009 [Thread-10]
> (org.apache.kafka.clients.consumer.internals.Fetcher:266) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending READ_UNCOMMITTED
> IncrementalFetchRequest(toSend=(ing.fkw.demand.tracking.clickstream-7),
> toForget=(), implied=(ing.fkw.credit.tijori.LoanAccount-0,
> cdc.fkw.ful.ssms.shipment.shipment_wave-1,
> cdc.fkw.ful.inventory.fkw_wh_inventory.wid_wsn-0,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_inventory_issue-1,
> cdc.fkw.ful.picking.picking.product_short_qty_view-3,
> cdc.fkw.credit.tijori.ffb_tijori.CardPaymentInstrument-2,
> cdc.fkw.ful.ro.returns_b2c_production.qc_verifications-2,
> cdc.fkw.ful.staging.staging.consignment-4,
> cdc.fkw.ful.staging.staging.order_container-1,
> cdc.fkw.ful.qc.qc.qc_check_choices-0,
> cdc.fkw.ful.sourcemaster.source_master_v3.sources-3,
> cdc.fkw.shopping.pcm.ffb_pcm.media_type-0,
> cdc.fkw.ful.qc.qc.qc_answer_choices-0,
> cdc.fkw.ful.sourcemaster.source_master_v3.seller_ids-1,
> ing.fkw.shopping.catalog.taxonomy-1,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.entity_tags-4,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_failures-3,
> cdc.fkw.ful.receiving.flipkart_receiving.consignment-0,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_areas-3,
> cdc.fkw.ful.ro.returns_b2c_production.shipments-3,
> cdc.fkw.ful.qc.qc.qc_assets-1,
> cdc.fkw.ful.sourcemaster.source_master_v3.asset_metadata-3,
> cdc.fkw.ful.receiving.flipkart_receiving.wsn_receiving_details-4,
> cdc.fkw.ful.sortation.flipkart_sortation.task-3,
> ing.fkw.shopping.transact.orderAdjustments-4,
> cdc.fkw.credit.khaata.khaata.emi_installments-0,
> cdc.fkw.credit.khaata.khaata.transactions-1,
> cdc.fkw.ful.inventory.fkw_wh_inventory.warehouse_asset_lock_details-3,
> cdc.fkw.ful.ssms.shipment.order_shipment-0,
> cdc.fkw.ful.ssms.shipment.order_shipment_v2-2,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_attributes-3,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_unit_attributes-6,
> cdc.fkw.ful.ssms.shipment.connection-1,
> cdc.fkw.ful.symphony.fulfillment_b2c_production.fulfill_item_unit_histories-6))
> to broker 10.48.130.211:9092 (id: 3 rack: null)
> 2021-08-26 13:38:03,009 [Thread-10]
> (org.apache.kafka.clients.NetworkClient:522) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending FETCH request with header
> RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710600) and timeout 30000 to node 3:
> {replica_id=-1,max_wait_ms=1000,min_bytes=1,max_bytes=10485760,isolation_level=0,session_id=1416968662,session_epoch=233604,topics=[{topic=ing.fkw.demand.tracking.clickstream,partitions=[{partition=7,current_leader_epoch=14,fetch_offset=4055570,last_fetched_epoch=-1,log_start_offset=-1,partition_max_bytes=1048576,_tagged_fields={}}],_tagged_fields={}}],forgotten_topics_data=[],rack_id=,_tagged_fields={}}
> 2021-08-26 13:38:03,010 [Thread-10]
> (com.pinterest.secor.reader.SecorKafkaMessageIterator:59) INFO Consumer
> poll called from memberId ::
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32
>
>
> *Last heartbeat logs:*2021-08-26 13:37:59,721
> [kafka-coordinator-heartbeat-thread | fkw_secor_prod_group_live]
> (org.apache.kafka.clients.NetworkClient:884) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710560):
> org.apache.kafka.common.requests.FetchResponse@5eaf4720
> 2021-08-26 13:37:59,721 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 2 response
> partition(s), 33 implied partition(s)
> 2021-08-26 13:37:59,722 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055557
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055558, lastStableOffset = 4055558,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=806)
> 2021-08-26 13:37:59,722 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 121914674 for partition cdc.fkw.ful.ssms.shipment.connection-1 returned
> fetch data (error=NONE, highWaterMark=121914737, lastStableOffset =
> 121914737, logStartOffset = 84087991, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=4320)
> *2021-08-26 13:38:00,134 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending Heartbeat request with
> generation 1359 and member id
> fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2
> to coordinator 10.49.115.249:9092 <http://10.49.115.249:9092> (id:
> 2147483646 rack: null)*
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:522)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Sending HEARTBEAT request with header
> RequestHeader(apiKey=HEARTBEAT, apiVersion=4,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710563) and timeout 30000 to node 2147483646:
> {group_id=fkw_secor_prod_group_live,generation_id=1359,member_id=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32-b153e17f-d864-4e0f-a362-996142211fb2,group_instance_id=null,_tagged_fields={}}
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 1 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710553):
> org.apache.kafka.common.requests.FetchResponse@547d0cb2
> 2021-08-26 13:38:00,135 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 1 sent an incremental fetch
> response with throttleTimeMs = 0 for session 751564104 with 0 response
> partition(s), 56 implied partition(s)
> 2021-08-26 13:38:00,235 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710566):
> org.apache.kafka.common.requests.FetchResponse@4c8a341f
> 2021-08-26 13:38:00,236 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:00,237 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055560
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055561, lastStableOffset = 4055561,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=765)
> 2021-08-26 13:38:00,623 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 1 sent an incremental fetch
> response with throttleTimeMs = 0 for session 751564104 with 1 response
> partition(s), 55 implied partition(s)
> 2021-08-26 13:38:00,626 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 16596199 for partition
> cdc.fkw.ful.symphony.fulfillment_b2c_production.mappings-6 returned fetch
> data (error=NONE, highWaterMark=16596229, lastStableOffset = 16596229,
> logStartOffset = 14934897, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=1783)
> 2021-08-26 13:38:00,827 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710576):
> org.apache.kafka.common.requests.FetchResponse@4070cb5c
> 2021-08-26 13:38:00,829 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:00,829 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055563
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055564, lastStableOffset = 4055564,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=775)
> 2021-08-26 13:38:01,268 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710580):
> org.apache.kafka.common.requests.FetchResponse@64420235
> 2021-08-26 13:38:01,268 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:01,269 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset 4055564
> for partition ing.fkw.demand.tracking.clickstream-7 returned fetch data
> (error=NONE, highWaterMark=4055565, lastStableOffset = 4055565,
> logStartOffset = 2977050, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=782)
> 2021-08-26 13:38:02,774 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live] (org.apache.kafka.clients.NetworkClient:884)
> DEBUG [Consumer clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Received FETCH response from node 3 for
> request with header RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> correlationId=710597):
> org.apache.kafka.common.requests.FetchResponse@58668a93
> 2021-08-26 13:38:02,775 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.FetchSessionHandler:463) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Node 3 sent an incremental fetch
> response with throttleTimeMs = 0 for session 1416968662 with 1 response
> partition(s), 34 implied partition(s)
> 2021-08-26 13:38:02,776 [kafka-coordinator-heartbeat-thread |
> fkw_secor_prod_group_live]
> (org.apache.kafka.clients.consumer.internals.Fetcher$1:315) DEBUG [Consumer
> clientId=fkw-secor-fkw-secor-prod-cluster-8344558_14763_32,
> groupId=fkw_secor_prod_group_live] Fetch READ_UNCOMMITTED at offset
> 121915139 for partition cdc.fkw.ful.ssms.shipment.connection-1 returned
> fetch data (error=NONE, highWaterMark=121915161, lastStableOffset =
> 121915161, logStartOffset = 84087991, preferredReadReplica = absent,
> abortedTransactions = null, divergingEpoch =Optional.empty,
> recordsSizeInBytes=1510)
>
>
> Regards,
> Shantam Garg
>
>
>
--
*-----------------------------------------------------------------------------------------*
*This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee,
you should not disseminate, distribute or copy this email. Please notify
the sender immediately by email if you have received this email by mistake
and delete this email from your system. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this information is
strictly prohibited.*****
****
*Any views or opinions presented in this
email are solely those of the author and do not necessarily represent those
of the organization. Any information on shares, debentures or similar
instruments, recommended product pricing, valuations and the like are for
information purposes only. It is not meant to be an instruction or
recommendation, as the case may be, to buy or to sell securities, products,
services nor an offer to buy or sell securities, products or services
unless specifically stated to be so on behalf of the Flipkart group.
Employees of the Flipkart group of companies are expressly required not to
make defamatory statements and not to infringe or authorise any
infringement of copyright or any other legal right by email communications.
Any such communication is contrary to organizational policy and outside the
scope of the employment of the individual concerned. The organization will
not accept any liability in respect of such communication, and the employee
responsible will be personally liable for any damages or other liability
arising.*****
****
*Our organization accepts no liability for the
content of this email, or for the consequences of any actions taken on the
basis of the information *provided,* unless that information is
subsequently confirmed in writing. If you are not the intended recipient,
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited.*
_-----------------------------------------------------------------------------------------_
Comments
Post a Comment