删除命名空间失败
今天在作测试的时候,清理集群。就把没用的都清理掉包括命名空间。但是发现失败了,一直卡在终止状态。
导致删除失败的原因一般有两种:
1、命名空间下还有资源在用,如果有删除,命名空间自动消失(原因:API服务器强制要求,只有在名称空间为空的情况下,并且只有在名称空间的情况下才能从存储中删除名称空间。)
2、就是名称空间下没有资源
开始分析原因
正常可以有三种方式
1、查看控制器日志 kube-controller-manager
2、查看api-resources会有报错信息在下面
获取所有注册ns的资源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)接下来查看注册服务kubectl get apiservice
3、失败查看当前yaml文件,通过get获取。其中有status字段中的message会有相关信息。
kubectl get ns t1 -o yaml
删除了一些留了主要的部分
[root@ECS1 ~]# kubectl get ns t1 -o yaml
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2021-06-12T08:43:19Z"
deletionTimestamp: "2021-06-12T08:43:24Z"
name: t1
resourceVersion: "546044"
uid: 62f4616c-ea45-441f-8ccf-4079d35c2796
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: 'Discovery failed for some groups, 2 failing: unable to retrieve the
complete list of server APIs: discovery.k8s.io/v1: the server could not find
the requested resource, policy/v1: the server could not find the requested resource'
reason: DiscoveryFailed
status: "True"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content successfully removed
reason: ContentRemoved
status: "False"
type: NamespaceContentRemaining
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content-preserving finalizers finished
reason: ContentHasNoFinalizers
status: "False"
type: NamespaceFinalizersRemaining
phase: Terminating
问题直接定位到,这个资源找不到。并且在查看资源与查看注册资源时,输出最下面会有error。不处理上文故障的处理办法如下:
正文:
处理方法
[root@ECS1 ~]# kubectl get ns
NAME STATUS AGE
app-team1 Terminating 3d7h
default Active 3d19h
internal Active 2d23h
kube-node-lease Active 3d19h
kube-public Active 3d19h
kube-system Active 3d19h
[root@ECS1 ~]# kubectl delete ns/app-team1
namespace "app-team1" deleted
^C
[root@ECS1 ~]#
没办法只能手动停止,不然卡到你天荒地老。
找到一个神奇的地方找到这么一段话
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
大体意思就是如果删除了命名空间,在已经清除空间下所有对象后。空间还在,那么需要通过更新名称空间子资源来强制删除。这种方式通知名称空间控制器,我要从命名空间中删除终结器并且执行清理所有操作。
这东西用的是restful请求方式,但是我这不安全端口都封掉了,开个代理吧(可以选择用证书)
[root@ECS1 ~]# kubectl proxy --port=8081
Starting to serve on 127.0.0.1:8081
cat <<EOF | curl -X PUT \
localhost:8081/api/v1/namespaces/app-team1/finalize \
-H "Content-Type: application/json" \
--data-binary @-
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "app-team1"
},
"spec": {
"finalizers": null
}
}
EOF
查看结果
[root@ECS1 ~]# cat <<EOF | curl -X PUT \
> localhost:8081/api/v1/namespaces/app-team1/finalize \
> -H "Content-Type: application/json" \
> --data-binary @-
> {
> "kind": "Namespace",
> "apiVersion": "v1",
> "metadata": {
> "name": "app-team1"
> },
> "spec": {
> "finalizers": null
> }
> }
> EOF
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "app-team1",
"uid": "108e6665-9b70-422c-8f94-783347101836",
"resourceVersion": "533794",
"creationTimestamp": "2021-06-08T23:46:24Z",
"deletionTimestamp": "2021-06-12T06:27:33Z",
"managedFields": [
{
"manager": "curl",
"operation": "Update",
"apiVersion": "v1",
"time": "2021-06-12T06:58:32Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:status":{"f:phase":{}}}
}
]
},
"spec": {
},
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "True",
"lastTransitionTime": "2021-06-12T06:27:38Z",
"reason": "DiscoveryFailed",
"message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource"
},
{
"type": "NamespaceDeletionGroupVersionParsingFailure",
"status": "False",
"lastTransitionTime": "2021-06-12T06:27:38Z",
"reason": "ParsedGroupVersions",
"message": "All legacy kube types successfully parsed"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime":