51CTO K8S常见故障速查指南常见的k8s pod crashloopbackoff排查

转载自博客：https://www.cnblogs.com/dalianpai/p/12258165.html

 方案1：

转载自博客：https://www.cnblogs.com/dalianpai/p/12258165.html

现象如下：

 
[root@k8s1 ~]# kubectl get pod
NAME                             READY   STATUS             RESTARTS   AGE
eureka-server-65695bbdc8-49b6v   0/1     CrashLoopBackOff   5          4m32s
[root@k8s1 ~]# kubectl get pod
NAME                             READY   STATUS             RESTARTS   AGE
eureka-server-65695bbdc8-49b6v   0/1     CrashLoopBackOff   5          5m16s
 

查找原因及解决如下：

 
[root@k8s1 ~]# kubectl describe pod eureka-server-65695bbdc8-49b6v
Name:           eureka-server-65695bbdc8-49b6v
Namespace:      default
Priority:       0
Node:           k8s3/192.168.180.144
Start Time:     Mon, 03 Feb 2020 21:03:24 +0800
Labels:         app=eureka-server
                pod-template-hash=65695bbdc8
Annotations:    <none>
Status:         Running
IP:             10.244.2.2
IPs:            <none>
Controlled By:  ReplicaSet/eureka-server-65695bbdc8
Containers:
  eureka-server:
    Container ID:   docker://0a8de1c10a8bc42aea722d4994f8b53c06820030de30a2c014f6600d7a98d09d
    Image:          112.124.104.171:1180/topcheer/eureka-server:1.0.1
    Image ID:       docker-pullable://112.124.104.171:1180/topcheer/eureka-server@sha256:b18dbacbc640c62b64d92f7240af0adcec2c4d9e870c60194d7ce9fa9d1e557c
    Port:           9010/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 03 Feb 2020 21:07:15 +0800
      Finished:     Mon, 03 Feb 2020 21:07:22 +0800
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tdbtd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-tdbtd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tdbtd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m30s                  default-scheduler  Successfully assigned default/eureka-server-65695bbdc8-49b6v to k8s3
  Normal   Pulled     2m21s (x5 over 4m29s)  kubelet, k8s3      Container image "112.124.104.171:1180/topcheer/eureka-server:1.0.1" already present on machine
  Normal   Created    2m21s (x5 over 4m28s)  kubelet, k8s3      Created container eureka-server
  Normal   Started    2m21s (x5 over 4m28s)  kubelet, k8s3      Started container eureka-server
  Warning  BackOff    105s (x10 over 4m6s)   kubelet, k8s3      Back-off restarting failed container
 
 
[root@k8s1 ~]# kubectl logs eureka-server-65695bbdc8-49b6v
2020-02-03 13:07:18.750  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2920e78d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.4.RELEASE)

2020-02-03 13:07:19.062  INFO 1 --- [           main] c.t.eureka.EurekaServerApplication       : No active profile set, falling back to default profiles: default
2020-02-03 13:07:20.402  WARN 1 --- [           main] o.s.boot.actuate.endpoint.EndpointId     : Endpoint ID 'service-registry' contains invalid characters, please migrate to a valid format.
2020-02-03 13:07:20.719  INFO 1 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=aab566da-2958-3e79-aa70-5d794b4e315b
2020-02-03 13:07:21.203  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2920e78d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-02-03 13:07:21.621  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 9010 (http)
2020-02-03 13:07:21.638  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-02-03 13:07:21.639  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.30]
2020-02-03 13:07:21.727  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-02-03 13:07:21.728  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2630 ms
2020-02-03 13:07:21.988  WARN 1 --- [           main] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
2020-02-03 13:07:21.989  INFO 1 --- [           main] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
2020-02-03 13:07:22.018  INFO 1 --- [           main] c.netflix.config.DynamicPropertyFactory  : DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@3c73951
2020-02-03 13:07:22.358 ERROR 1 --- [           main] o.s.b.web.embedded.tomcat.TomcatStarter  : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'traceFilterRegistration' defined in class path resource [org/springframework/cloud/netflix/eureka/server/EurekaServerAutoConfiguration.class]: Unsatisfied dependency expressed through method 'traceFilterRegistration' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'javax.servlet.Filter' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Qualifier(value=httpTraceFilter)}
2020-02-03 13:07:22.405  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2020-02-03 13:07:22.413  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
2020-02-03 13:07:22.442  INFO 1 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-02-03 13:07:22.778 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:156) ~[spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544) ~[spring-context-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
        at com.topcheer.eureka.EurekaServerApplication.main(EurekaServerApplication.java:13) [classes!/:0.0.1-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
 

才发现是boot的版本太高了，和boot有冲突。。。

 案例2：

K8S 启动POD遇到CRASHLOOPBACKOFF的解决方法与思路

转载自博客：https://www.freesion.com/article/83191218638/

一直正常运的k8s，集群节点没问题，但启动pod出现异常
等待中: CrashLoopBackOff

 

1.登陆此节点主机使用kubctl获取pod状态

kubectl get pod

 

 

kubectl logs elkhost-944bcbcd4-8n9nj

 

[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

错误原因在这里 经查elasticsearch运行要求:
vm.max_map_count内核参数必需大于262114，确认系统调过 $sysctl -w vm.max_map_count=262144
这里注意的问题是 上述属于临时性调整，主机重启后又恢复到默认状态，记久性修改 vi /etc/sysconfig 加入：

vm.max_map_count=262144
保存/etc/sysctl.conf,重新启动服务器以应用更改,或执行：sysctl -p以应用更改而不重新启动.他们将在重新启动时永久保持.
重启pod 进入控制台 查询状态 恢复

 4.总结 出现故障可能很多种 但要学查看日志 排查具体原因及应用所在

 

posted on 2023-09-04 21:10  luzhouxiaoshuai  阅读(270)  评论(0)    收藏  举报

导航