Re: Kubernetes Pod got restarted Terminating process due to signal SIGTERM (org.apache.kafka.common.utils.LoggingSignalHandler)
Hi Samir,
The SIGTERM is not originating from Kafka, it's being done to Kafka - this
is Kubernetes gracefully shutting down or restarting your pod. When a pod
is being shutdown, k8s sends a SIGTERM to all containers in that pod, which
will then forward it to the main process in each container. After a certain
period (default in Kubernetes is 30 seconds), if the pod hasn't shut down,
it will then send a SIGKILL. More info here:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
As to why the pod was restarted - that really depends on how your k8s
cluster is configured and administered, and really beyond the scope of a
Kafka mailing list. But I'll point you at this for some starting points for
investigation:
https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions
Lastly, I strongly recommend you configure "terminationGracePeriodSeconds"
for the pods running Kafka much higher, depending again, on how you're
using Kafka. One of our devops found out the hard way why you need to let
Kafka gracefully shut down - he was using systemctl to restart it, which
like k8s, first sends SIGTERM and then after a default of 90 seconds, sends
SIGKILL - our broker had reached a size where a graceful shutdown took
about 120 seconds, so systemctl was SIGKILLing it, which led to corruption
of log file indexes, which then had to be recovered on start-up, which
turned a normal 1 - 2 minute start-up period for the broker into a 2 hour
start-up period. So be very careful about how long a graceful shutdown of
your broker will actually take.
Kind regards,
Liam Clarke-Hutchinson
On Sat, Mar 28, 2020 at 3:49 AM Samir Tusharbhai Chauhan
<samir.tusharbhai.chauhan@prudential.com.sg.invalid> wrote:
> Hi,
>
> My pod got restarted with below message [2020-03-25 08:24:04,905] INFO
> Terminating process due to signal SIGTERM
> (org.apache.kafka.common.utils.LoggingSignalHandler).
>
>
>
> When do I get this situation? What should I look for to troubleshoot this?
>
> [2020-03-25 08:24:04,905] INFO Terminating process due to signal SIGTERM
> (org.apache.kafka.common.utils.LoggingSignalHandler)
>
> [2020-03-25 08:24:04,912] INFO [KafkaServer id=1001] shutting down
> (kafka.server.KafkaServer)
>
> [2020-03-25 08:24:04,913] INFO [KafkaServer id=1001] Starting controlled
> shutdown (kafka.server.KafkaServer)
>
> [2020-03-25 08:24:04,984] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions (kafka.server.ReplicaFetcherManager)
>
> [2020-03-25 08:24:04,984] INFO [ReplicaAlterLogDirsManager on broker 1001]
> Removed fetcher for partitions (kafka.server.ReplicaAlterLogDirsManager)
>
> [2020-03-25 08:24:04,987] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-1_reply-0
> (kafka.server.ReplicaFetcherManager)
>
> [2020-03-25 08:24:04,987] INFO [ReplicaAlterLogDirsManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-1_reply-0
> (kafka.server.ReplicaAlterLogDirsManager)
>
> [2020-03-25 08:24:04,989] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-0_reply-6
> (kafka.server.ReplicaFetcherManager)
>
>
>
> Samir Chauhan
>
> There's a reason we support Fair Dealing. YOU.
>
>
> This email and any files transmitted with it or attached to it (the
> [Email]) may contain confidential, proprietary or legally privileged
> information and is intended solely for the use of the individual or entity
> to whom it is addressed. If you are not the intended recipient of the
> Email, you must not, directly or indirectly, copy, use, print, distribute,
> disclose to any other party or take any action in reliance on any part of
> the Email. Please notify the system manager or sender of the error and
> delete all copies of the Email immediately.
>
> No statement in the Email should be construed as investment advice being
> given within or outside Singapore. Prudential Assurance Company Singapore
> (Pte) Limited (PACS) and each of its related entities shall not be
> responsible for any losses, claims, penalties, costs or damages arising
> from or in connection with the use of the Email or the information therein,
> in whole or in part. You are solely responsible for conducting any virus
> checks prior to opening, accessing or disseminating the Email.
>
> PACS (Company Registration No. 199002477Z) is a company incorporated under
> the laws of Singapore and has its registered office at 30 Cecil Street,
> #30-01, Prudential Tower, Singapore 049712.
>
> PACS is an indirect wholly owned subsidiary of Prudential plc of the
> United Kingdom. PACS and Prudential plc are not affiliated in any manner
> with Prudential Financial, Inc., a company whose principal place of
> business is in the United States of America.
>
The SIGTERM is not originating from Kafka, it's being done to Kafka - this
is Kubernetes gracefully shutting down or restarting your pod. When a pod
is being shutdown, k8s sends a SIGTERM to all containers in that pod, which
will then forward it to the main process in each container. After a certain
period (default in Kubernetes is 30 seconds), if the pod hasn't shut down,
it will then send a SIGKILL. More info here:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
As to why the pod was restarted - that really depends on how your k8s
cluster is configured and administered, and really beyond the scope of a
Kafka mailing list. But I'll point you at this for some starting points for
investigation:
https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions
Lastly, I strongly recommend you configure "terminationGracePeriodSeconds"
for the pods running Kafka much higher, depending again, on how you're
using Kafka. One of our devops found out the hard way why you need to let
Kafka gracefully shut down - he was using systemctl to restart it, which
like k8s, first sends SIGTERM and then after a default of 90 seconds, sends
SIGKILL - our broker had reached a size where a graceful shutdown took
about 120 seconds, so systemctl was SIGKILLing it, which led to corruption
of log file indexes, which then had to be recovered on start-up, which
turned a normal 1 - 2 minute start-up period for the broker into a 2 hour
start-up period. So be very careful about how long a graceful shutdown of
your broker will actually take.
Kind regards,
Liam Clarke-Hutchinson
On Sat, Mar 28, 2020 at 3:49 AM Samir Tusharbhai Chauhan
<samir.tusharbhai.chauhan@prudential.com.sg.invalid> wrote:
> Hi,
>
> My pod got restarted with below message [2020-03-25 08:24:04,905] INFO
> Terminating process due to signal SIGTERM
> (org.apache.kafka.common.utils.LoggingSignalHandler).
>
>
>
> When do I get this situation? What should I look for to troubleshoot this?
>
> [2020-03-25 08:24:04,905] INFO Terminating process due to signal SIGTERM
> (org.apache.kafka.common.utils.LoggingSignalHandler)
>
> [2020-03-25 08:24:04,912] INFO [KafkaServer id=1001] shutting down
> (kafka.server.KafkaServer)
>
> [2020-03-25 08:24:04,913] INFO [KafkaServer id=1001] Starting controlled
> shutdown (kafka.server.KafkaServer)
>
> [2020-03-25 08:24:04,984] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions (kafka.server.ReplicaFetcherManager)
>
> [2020-03-25 08:24:04,984] INFO [ReplicaAlterLogDirsManager on broker 1001]
> Removed fetcher for partitions (kafka.server.ReplicaAlterLogDirsManager)
>
> [2020-03-25 08:24:04,987] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-1_reply-0
> (kafka.server.ReplicaFetcherManager)
>
> [2020-03-25 08:24:04,987] INFO [ReplicaAlterLogDirsManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-1_reply-0
> (kafka.server.ReplicaAlterLogDirsManager)
>
> [2020-03-25 08:24:04,989] INFO [ReplicaFetcherManager on broker 1001]
> Removed fetcher for partitions pulse__flink-tm-0_reply-6
> (kafka.server.ReplicaFetcherManager)
>
>
>
> Samir Chauhan
>
> There's a reason we support Fair Dealing. YOU.
>
>
> This email and any files transmitted with it or attached to it (the
> [Email]) may contain confidential, proprietary or legally privileged
> information and is intended solely for the use of the individual or entity
> to whom it is addressed. If you are not the intended recipient of the
> Email, you must not, directly or indirectly, copy, use, print, distribute,
> disclose to any other party or take any action in reliance on any part of
> the Email. Please notify the system manager or sender of the error and
> delete all copies of the Email immediately.
>
> No statement in the Email should be construed as investment advice being
> given within or outside Singapore. Prudential Assurance Company Singapore
> (Pte) Limited (PACS) and each of its related entities shall not be
> responsible for any losses, claims, penalties, costs or damages arising
> from or in connection with the use of the Email or the information therein,
> in whole or in part. You are solely responsible for conducting any virus
> checks prior to opening, accessing or disseminating the Email.
>
> PACS (Company Registration No. 199002477Z) is a company incorporated under
> the laws of Singapore and has its registered office at 30 Cecil Street,
> #30-01, Prudential Tower, Singapore 049712.
>
> PACS is an indirect wholly owned subsidiary of Prudential plc of the
> United Kingdom. PACS and Prudential plc are not affiliated in any manner
> with Prudential Financial, Inc., a company whose principal place of
> business is in the United States of America.
>
Comments
Post a Comment