【解决】Error from server (ServiceUnavailable): the server is currently unable to handle the request

 【问题】部署kube-prometheus使用一段时间，apiserver调用特别卡（响应达到15秒），查看v1beta1.metrics.k8s.io发现无法请求apiserver

 

[root@k8s-master ~]# kubectl get --raw "/apis/metrics.k8s.io/v1beta1"
Error from server (ServiceUnavailable): the server is currently unable to handle the request
[root@k8s-master ~]# kubectl get apiservice | grep metrics.k8s.io
v1beta1.metrics.k8s.io monitoring/prometheus-adapter False (FailedDiscoveryCheck) 44s
[root@k8s-master ~]# kubectl describe apiservice v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{“apiVersion”:“apiregistration.k8s.io/v1”,“kind”:“APIService”,“metadata”:{“annotations”:{},“name”:“v1beta1.metrics.k8s.io”},“spec”:{"group…
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2020-04-21T09:30:16Z
Resource Version: 18821900
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
UID: d0ae5bbf-e63d-4a77-ad7d-7ab45266b1f9
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: prometheus-adapter
Namespace: monitoring
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2020-04-21T09:30:16Z
Message: failing or missing response from https://192.168.54.3:6443/apis/metrics.k8s.io/v1beta1: Get https://192.168.54.3:6443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available

 

【解决】为了方便连接apiserver，我们可以通过把prometheus-adapter部署到apiserver同一台服务器（即master节点，注意：高可用则需要部署到所有master节点）

 以下是我对 prometheus-adapter-deployment.yaml 文件的修改：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-adapter
  namespace: monitoring
spec:
  replicas: 3
  selector:
    matchLabels:
      name: prometheus-adapter
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        name: prometheus-adapter
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node-role.kubernetes.io/master
               operator: Exists
      containers:
      - args:
        - --cert-dir=/var/run/serving-cert
        - --config=/etc/adapter/config.yaml
        - --logtostderr=true
        - --metrics-relist-interval=1m
        - --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/
        - --secure-port=6443
        #image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
        image: registry.cn-shanghai.aliyuncs.com/leozhanggg/prometheus/k8s-prometheus-adapter-amd64:v0.5.0
        name: prometheus-adapter
        ports:
        - containerPort: 6443
        volumeMounts:
        - mountPath: /tmp
          name: tmpfs
          readOnly: false
        - mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: false
        - mountPath: /etc/adapter
          name: config
          readOnly: false
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: prometheus-adapter
      volumes:
      - emptyDir: {}
        name: tmpfs
      - emptyDir: {}
        name: volume-serving-cert
      - configMap:
          name: adapter-config
        name: config

 

【Github】Error from server (ServiceUnavailable): the server is currently unable to handle the request #505