kubectl apply源码分析

patch容易出现字段冲突

近期在使用client-go对某个k8s原生资源进行patch操作时,出现了字段冲突导致的patch失败问题,具体是patch尝试修改资源的某个字段的类型,比如将readiness probe的类型从tcp修改为httpGet,patch时希望修改probe类型但被认为是一种追加动作,导致apiserver端验证错误不允许为一种类型的probe指定多个handler:

当然,处理方式可以在patch数据中为要删除的readiness tcp probe加一个删除标记,这样patch请求到达apiserver的时候就可以被正确处理达到替换的目的:

"spec": {
   "containers":[
      {
         "name":"xxx",
         "readinessProbe":{
            "exec":nil, // delete
            "httpGet":{ // add
            }
         }
      }
   }]
}

给我带来的疑惑是使用kubectl apply时为什么就没这个问题呢?

kubectl apply使用3-way patch

kubectl apply命令会在要apply的资源对象上添加last-apply-configuration,表示最近一次通过kubectl apply更新的资源清单,如果某个资源一直都是通过apply来更新,那么ast-apply-configuration与对象一致

对于k8s原生的资源如deployment、pod等,kubectl apply时通过3-way patch生成strategicpatch类型的patch数据,其中:

注意如果是crd资源,用的应该是jsonmergepatch.CreateThreeWayJSONMergePatch

# staging/src/k8s.io/kubectl/pkg/cmd/apply/apply.go
// 根据original、modified、current三方数据生成最终patch请求的数据
if openapiPatch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, lookupPatchMeta, p.Overwrite); err != nil {
			fmt.Fprintf(errOut, "warning: error calculating patch from openapi spec: %v\n", err)
} else {
			patchType = types.StrategicMergePatchType
			patch = openapiPatch
}

current是集群中当前的资源数据:

    // info.Get通过RestClient请求api获取对象
		if err := info.Get(); err != nil {
      // err是not found error,说明是首次创建
			if !errors.IsNotFound(err) {
				return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:\n%s\nfrom server for:", info.String()), info.Source, err)
			}
			// Create the resource if it doesn't exist
			// First, update the annotation used by kubectl apply
      // 如果集群中当前的对象没有last-apply-configuration注解,那么先用这个对象本身生存anno并更新到集群
			if err := util.CreateApplyAnnotation(info.Object, unstructured.UnstructuredJSONScheme); err != nil {
				return cmdutil.AddSourceToErr("creating", info.Source, err)
			}
		}

modified是此次需要apply放入数据(比如-f指定的文件内容):

		// Get the modified configuration of the object. Embed the result
		// as an annotation in the modified configuration, so that it will appear
		// in the patch sent to the server.
   // 可以看看这个方法具体的实现,会把自身encode之后放到自己的last-apply-configuration之中(覆盖可能已经存在的这个anno)
		modified, err := util.GetModifiedConfiguration(info.Object, true, unstructured.UnstructuredJSONScheme)


original是集群中当前资源的LastAppliedConfigAnnotation数据:

// Retrieve the original configuration of the object from the annotation.
original, err := util.GetOriginalConfiguration(obj)

// GetOriginalConfiguration retrieves the original configuration of the object
// from the annotation, or nil if no annotation was found.
func GetOriginalConfiguration(obj runtime.Object) ([]byte, error) {
	annots, err := metadataAccessor.Annotations(obj)
	if err != nil {
		return nil, err
	}

	if annots == nil {
		return nil, nil
	}
 // 直接取的annotation
	original, ok := annots[v1.LastAppliedConfigAnnotation]
	if !ok {
		return nil, nil
	}

	return []byte(original), nil
}

有了这三方数据之后,strategicpatch.CreateThreeWayMergePatch方法就会产生最终要patch的数据

  • 根据集群中当前资源数据currentMap和此次要修改的数据Modified计算出那些字段是新增的,计算增量时忽略哪些要被删除的字段
    • 因为集群中的对象可能被修改过(人为或者某些组件)且这些修改不会更新last-apply-configuration anno,所以这里apply计算哪些字段是新增的时,就需要以集群当前状态和此次的apply数据modified来决定
  • 根据集群中当前资源的original(last-apply-configuration anno)数据和此次要修改的数据Modified计算出哪些字段是要删除的(设置为"-"),忽略增加的字段
    • kubectl apply认为冲突的字段应该通过相邻的两次apply操作来计算
// CreateThreeWayMergePatch reconciles a modified configuration with an original configuration,
// while preserving any changes or deletions made to the original configuration in the interim,
// and not overridden by the current configuration. All three documents must be passed to the
// method as json encoded content. It will return a strategic merge patch, or an error if any
// of the documents is invalid, or if there are any preconditions that fail against the modified
// configuration, or, if overwrite is false and there are conflicts between the modified and current
// configurations. Conflicts are defined as keys changed differently from original to modified
// than from original to current. In other words, a conflict occurs if modified changes any key
// in a way that is different from how it is changed in current (e.g., deleting it, changing its
// value). We also propagate values fields that do not exist in original but are explicitly
// defined in modified.
func CreateThreeWayMergePatch(original, modified, current []byte, schema LookupPatchMeta, overwrite bool, fns ...mergepatch.PreconditionFunc) ([]byte, error) {
  // 三方数据都反序列化为unstracture通用结构
	originalMap := map[string]interface{}{}
	if len(original) > 0 {
		if err := json.Unmarshal(original, &originalMap); err != nil {
			return nil, mergepatch.ErrBadJSONDoc
		}
	}

	modifiedMap := map[string]interface{}{}
	if len(modified) > 0 {
		if err := json.Unmarshal(modified, &modifiedMap); err != nil {
			return nil, mergepatch.ErrBadJSONDoc
		}
	}

	currentMap := map[string]interface{}{}
	if len(current) > 0 {
		if err := json.Unmarshal(current, &currentMap); err != nil {
			return nil, mergepatch.ErrBadJSONDoc
		}
	}

	// The patch is the difference from current to modified without deletions, plus deletions
	// from original to modified. To find it, we compute deletions, which are the deletions from
	// original to modified, and delta, which is the difference from current to modified without
	// deletions, and then apply delta to deletions as a patch, which should be strictly additive.
	deltaMapDiffOptions := DiffOptions{
		IgnoreDeletions: true,
		SetElementOrder: true,
	}
  // DiffOptions中IgnoreDeletions设置为true,根据集群中当前资源数据currentMap和此次要修改的数据计算出那些字段是新增的,
  // 计算增量时先忽略那些要被删除的
	deltaMap, err := diffMaps(currentMap, modifiedMap, schema, deltaMapDiffOptions)
	if err != nil {
		return nil, err
	}
	deletionsMapDiffOptions := DiffOptions{
		SetElementOrder:           true,
		IgnoreChangesAndAdditions: true,
	}
  // DiffOptions中IgnoreDeletions默认值为false,根据集群中当前资源的last-apply数据和此次要修改的数据计算出那些字段是要
  // 删除的,这里忽略增量的数据
  // 当有字段冲突时,这里会把original即上一次apply中的该字段标记为删除,deletionsMap中的值为nil
	deletionsMap, err := diffMaps(originalMap, modifiedMap, schema, deletionsMapDiffOptions)
	if err != nil {
		return nil, err
	}

	mergeOptions := MergeOptions{}
  // 将deletionsMap和deltaMap做一次合并,生成最终要patch的数据
	patchMap, err := mergeMap(deletionsMap, deltaMap, schema, mergeOptions)
	if err != nil {
		return nil, err
	}

	return json.Marshal(patchMap)
}
func diffMaps(original, modified map[string]interface{}, schema LookupPatchMeta, diffOptions DiffOptions) (map[string]interface{}, error) {
  // 记录结果
	patch := map[string]interface{}{}

	// Compare each value in the modified map against the value in the original map
  // 遍历originalMap这个unstrctureMap的每一个key
	for key, modifiedValue := range modified {
		originalValue, ok := original[key]
		if !ok {
			// Key was added, so add to patch
      // 如果value不存在于originalMap,但是存在于modifiedMap,并且IgnoreChangesAndAdditions为false
			if !diffOptions.IgnoreChangesAndAdditions {
        // 结果添加modifiedMap中的这个kv
				patch[key] = modifiedValue
			}
			continue
		}
		
    // original和modified中都有value,就看value是不是同一种类型
		if reflect.TypeOf(originalValue) != reflect.TypeOf(modifiedValue) {
			// Types have changed, so add to patch
      // 类型一样并且IgnoreChangesAndAdditions为false,那么结果添加modifiedMap中的这个kv
			if !diffOptions.IgnoreChangesAndAdditions {
				patch[key] = modifiedValue
			}
			continue
		}

		// Types are the same, so compare values
    // original和modified中都有value,就看value是同一种类型
    // 那么根据具体的类型,调用handleMapDiff或handleSliceDiff处理
		switch originalValueTyped := originalValue.(type) {
    // value的类型是一个复合结构
		case map[string]interface{}:
			modifiedValueTyped := modifiedValue.(map[string]interface{})
			err = handleMapDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
    // value的类型是一个slice切片结构
		case []interface{}:
			modifiedValueTyped := modifiedValue.([]interface{})
			err = handleSliceDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
		default:
    // 既不是map也不是slice,那么直接用modifiedValue替换originalValue
			replacePatchFieldIfNotEqual(key, originalValue, modifiedValue, patch, diffOptions)
		}
		if err != nil {
			return nil, err
		}
	}
	// 如果ignoreDeletions为false,那么遍历originalMap的每一个key,如果modefiedMap中不存在value,那么在最终的结果中
  // 标记该key为需要删除
	updatePatchIfMissing(original, modified, patch, diffOptions)
	return patch, nil
}

从上面的分析可以看出,kubect在apply时通过3-way patch的方式,可以计算出哪些字段是要新增的,哪些字段是要被删除的,以避免冲突的出现,如果original中的数据(last-apply)与modifed不能正确计算出要被删除的字段,也会出现apply失败的问题,比如资源通过kubectl create创建则没有last-apply-configuration注解,这个时候如果修改字段的值类型,即使通过kubectl apply也会失败。

posted @ 2021-08-01 13:58  JL_Zhou  阅读(987)  评论(0编辑  收藏  举报