Kubernetes源码阅读笔记——Controller Manager（之三）

前一篇文章中，我们探索了Informer工作的大致逻辑，提到了添加回调函数部分包含了三块，即：Informer的创建、函数调用的逻辑、以及回调函数本身。前两块已在前文谈到过，下面我们来看看第三块，即回调函数自身的处理逻辑：

一、回调函数

这里仍然以deployment为例。首先还是进入NewDeploymentController方法：

pkg/controller/deployment/deployment_controller.go

func NewDeploymentController(dInformer appsinformers.DeploymentInformer, rsInformer appsinformers.ReplicaSetInformer, podInformer coreinformers.PodInformer, client clientset.Interface) (*DeploymentController, error) {
	...
	dc := &DeploymentController{
		client:        client,
		eventRecorder: eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "deployment-controller"}),
		queue:         workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "deployment"),
	}
	dc.rsControl = controller.RealRSControl{
		KubeClient: client,
		Recorder:   dc.eventRecorder,
	}

	dInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    dc.addDeployment,
		UpdateFunc: dc.updateDeployment,
		
		DeleteFunc: dc.deleteDeployment,
	})
	rsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    dc.addReplicaSet,
		UpdateFunc: dc.updateReplicaSet,
		DeleteFunc: dc.deleteReplicaSet,
	})
	podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
		DeleteFunc: dc.deletePod,
	})

	dc.syncHandler = dc.syncDeployment
	dc.enqueueDeployment = dc.enqueue

	dc.dLister = dInformer.Lister()
	dc.rsLister = rsInformer.Lister()
	dc.podLister = podInformer.Lister()
	dc.dListerSynced = dInformer.Informer().HasSynced
	dc.rsListerSynced = rsInformer.Informer().HasSynced
	dc.podListerSynced = podInformer.Informer().HasSynced
	return dc, nil
}

我们看到，方法为Deployment和ReplicaSet的informer分别添加了增、改、删的回调函数，而为Pod的informer仅仅添加了删除的回调函数。这主要是针对策略为Recreate的Deployment（即所有旧Pod都删除后再创建新Pod），且仅当所有旧Pod都删除后才会进行下一步操作（马上提到）。对于RollingUpdate策略的Deployment，Pod数量的维持由dc的rsControl字段通过创建一个RSController来负责，不涉及进一步的操作。

这里说的进一步操作，指调用dc的enqueueDeployment方法，将Deployment加入队列中。比如，看addDeployment方法：

pkg/controller/deployment/deployment_controller.go

func (dc *DeploymentController) addDeployment(obj interface{}) {
	d := obj.(*apps.Deployment)
	klog.V(4).Infof("Adding deployment %s", d.Name)
	dc.enqueueDeployment(d)
}

再比如，看deleteDeployment方法：

pkg/controller/deployment/deployment_controller.go

func (dc *DeploymentController) deleteDeployment(obj interface{}) {
	d, ok := obj.(*apps.Deployment)
	if !ok {
		...
	}
	klog.V(4).Infof("Deleting deployment %s", d.Name)
	dc.enqueueDeployment(d)
}

7个回调函数都是这样，在最后调用了enqueueDeployment方法，将Deployment存入队列中。

enqueueDeployment已被设置为enqueue方法，所以看一下enqueue方法：

pkg/controller/deployment/deployment_controller.go

func (dc *DeploymentController) enqueue(deployment *apps.Deployment) {
	key, err := controller.KeyFunc(deployment)
	if err != nil {
		utilruntime.HandleError(fmt.Errorf("Couldn't get key for object %#v: %v", deployment, err))
		return
	}

	dc.queue.Add(key)
}

这里，KeyFunc的作用是将Deployment的namespace和name字段提取出来，并以namespace/name的格式返回字符串。而这个字符串则被存入队列中，以待下一步的处理。

所以，7个回调函数殊途同归，最终都是将相关的Deployment以namespace/name的格式存入了队列中。

二、Run

存入队列中的Deployment如何被处理呢？答案是在startDeploymentController方法中通过Go协程调用Run方法。现在我们来看一下Run方法：

pkg/controller/deployment/deployment_controller.go

func (dc *DeploymentController) Run(workers int, stopCh <-chan struct{}) {
    defer utilruntime.HandleCrash()
    defer dc.queue.ShutDown()

    klog.Infof("Starting deployment controller")
    defer klog.Infof("Shutting down deployment controller")

    if !controller.WaitForCacheSync("deployment", stopCh, dc.dListerSynced, dc.rsListerSynced, dc.podListerSynced) {
	return
    }

    for i := 0; i < workers; i++ {
	go wait.Until(dc.worker, time.Second, stopCh)
    }

    <-stopCh
}

其中，WaitForCacheSync方法是在对controller中Deployment、ReplicaSet和Pod的缓存进行同步，如果失败，则直接返回。

后面通过goroutine调用worker方法。worker方法则是调用了processNextWorkItem方法，在队列不为空的情况下持续取出队列中的下一个元素，并调用syncHandler方法执行操作。而前面的NewDeploymentController方法中，有一句dc.syncHandler = dc.syncDeployment，因此调用syncHandler方法即为调用syncDeployment方法。

三、syncDeployment

这个方法是对队列中Deployment元素的处理函数。方法中包含了对Deployment的一些操作，如获取ReplicaSet列表、进行更新和回滚等等。

pkg/controller/deployment/deployment_controller.go

func (dc *DeploymentController) syncDeployment(key string) error {
    startTime := time.Now()
    klog.V(4).Infof("Started syncing deployment %q (%v)", key, startTime)
    defer func() {
        klog.V(4).Infof("Finished syncing deployment %q (%v)", key, time.Since(startTime))
    }()

    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        return err
    }
    deployment, err := dc.dLister.Deployments(namespace).Get(name)
    if errors.IsNotFound(err) {
        klog.V(2).Infof("Deployment %v has been deleted", key)
        return nil
    }
    if err != nil {
        return err
    }

    // Deep-copy otherwise we are mutating our cache.
    // TODO: Deep-copy only when needed.
    d := deployment.DeepCopy()

    everything := metav1.LabelSelector{}
    if reflect.DeepEqual(d.Spec.Selector, &everything) {
        dc.eventRecorder.Eventf(d, v1.EventTypeWarning, "SelectingAll", "This deployment is selecting all pods. A non-empty selector is required.")
        if d.Status.ObservedGeneration < d.Generation {
            d.Status.ObservedGeneration = d.Generation
            dc.client.AppsV1().Deployments(d.Namespace).UpdateStatus(d)
        }
        return nil
    }

    // List ReplicaSets owned by this Deployment, while reconciling ControllerRef
    // through adoption/orphaning.
    rsList, err := dc.getReplicaSetsForDeployment(d)
    if err != nil {
        return err
    }
    // List all Pods owned by this Deployment, grouped by their ReplicaSet.
    // Current uses of the podMap are:
    //
    // * check if a Pod is labeled correctly with the pod-template-hash label.
    // * check that no old Pods are running in the middle of Recreate Deployments.
    podMap, err := dc.getPodMapForDeployment(d, rsList)
    if err != nil {
        return err
    }

    if d.DeletionTimestamp != nil {
        return dc.syncStatusOnly(d, rsList)
    }

    // Update deployment conditions with an Unknown condition when pausing/resuming
    // a deployment. In this way, we can be sure that we won't timeout when a user
    // resumes a Deployment with a set progressDeadlineSeconds.
    if err = dc.checkPausedConditions(d); err != nil {
        return err
    }

    if d.Spec.Paused {
        return dc.sync(d, rsList)
    }

    // rollback is not re-entrant in case the underlying replica sets are updated with a new
    // revision so we should ensure that we won't proceed to update replica sets until we
    // make sure that the deployment has cleaned up its rollback spec in subsequent enqueues.
    if getRollbackTo(d) != nil {
        return dc.rollback(d, rsList)
    }

    scalingEvent, err := dc.isScalingEvent(d, rsList)
    if err != nil {
        return err
    }
    if scalingEvent {
        return dc.sync(d, rsList)
    }

    switch d.Spec.Strategy.Type {
    case apps.RecreateDeploymentStrategyType:
        return dc.rolloutRecreate(d, rsList, podMap)
    case apps.RollingUpdateDeploymentStrategyType:
        return dc.rolloutRolling(d, rsList)
    }
    return fmt.Errorf("unexpected deployment strategy type: %s", d.Spec.Strategy.Type)
}

方法大约包含以下几部分：

（1）取出的元素为namespace/name格式，所以首先根据这两个字段获取deployment对象。

（2）检查Deployment的DeletionTimestamp、PauseCondition、ScalingEvent等字段的内容，并调用相应的处理函数，包括sync、syncStatusOnly等。

（3）判断Deployment的策略是RollingUpdate还是Recreate，并调用相应的处理函数，包括rolloutRolling等。

下面我们看几个处理函数。

四、处理函数

先来看一下syncStatusOnly方法：

pkg/controller/deployment/sync.go

// syncStatusOnly only updates Deployments Status and doesn't take any mutating actions.
func (dc *DeploymentController) syncStatusOnly(d *apps.Deployment, rsList []*apps.ReplicaSet) error {
    newRS, oldRSs, err := dc.getAllReplicaSetsAndSyncRevision(d, rsList, false)
    if err != nil {
        return err
    }

    allRSs := append(oldRSs, newRS)
    return dc.syncDeploymentStatus(allRSs, newRS, d)
}

正如注释所说，syncStatusOnly方法只做了一件事，就是调用syncDeploymentStatus方法，来同步Deployment的状态。此方法在后面分析。

再来看sync方法：

pkg/controller/deployment/sync.go

// sync is responsible for reconciling deployments on scaling events or when they
// are paused.
func (dc *DeploymentController) sync(d *apps.Deployment, rsList []*apps.ReplicaSet) error {
    newRS, oldRSs, err := dc.getAllReplicaSetsAndSyncRevision(d, rsList, false)
    if err != nil {
        return err
    }
    if err := dc.scale(d, newRS, oldRSs); err != nil {
        // If we get an error while trying to scale, the deployment will be requeued
        // so we can abort this resync
        return err
    }

    // Clean up the deployment when it's paused and no rollback is in flight.
    if d.Spec.Paused && getRollbackTo(d) == nil {
        if err := dc.cleanupDeployment(oldRSs, d); err != nil {
            return err
        }
    }

    allRSs := append(oldRSs, newRS)
    return dc.syncDeploymentStatus(allRSs, newRS, d)
}

如注释所说，此方法比syncStatusOnly方法多了对Deployment的scale和pause事件的处理，但最后仍然是调用syncDeploymentStatus方法，同步Deployment的状态。

再来看看syncDeploymentStatus方法：

pkg/controller/deployment/sync.go

func (dc *DeploymentController) syncDeploymentStatus(allRSs []*apps.ReplicaSet, newRS *apps.ReplicaSet, d *apps.Deployment) error {
	newStatus := calculateStatus(allRSs, newRS, d)

	if reflect.DeepEqual(d.Status, newStatus) {
		return nil
	}

	newDeployment := d
	newDeployment.Status = newStatus
	_, err := dc.client.AppsV1().Deployments(newDeployment.Namespace).UpdateStatus(newDeployment)
	return err
}

此方法首先检查新旧Deployment的状态是否一样，如果一样则直接返回。否则，就通过client连接API Server，修改Deployment的状态。这一步，就是controller处理deployment的最终目的。至于后面对Pod的具体操作，就是kubelet的职责了，而controller所负责的就是与API Server交互，并在API Server中更新deployment的状态。

rolloutRolling方法用于处理Deployment的rollingupdat事件，最后的本质仍是通过与API Server交互来更新Deployment的状态。

五、总结

至此，Controller Manager的大致运行流程就整理完了。

总结一下，Controller Manager本质上是kubernetes中众多controller的管理组件。每个controller在启动时都会运行自己的informer。这些informer通过list-watch机制，通过与API Server交互，获取监听资源的实时状态变化，并在FIFO队列中进行更新。

informer不断从FIFO队列中取出元素，一方面更新Thread Safe Store，另一方面调用具体controller的回调函数对元素进行处理。此外，informer会使用listener机制，从Thread Safe Store中取出内容，调用相应的回调函数进行处理。

以deployment controller为例，deployment的informer最终会将deployment存入队列中。队列中的元素取出后经过一系列deployment的scale、rollingupdate等处理，最终再次通过与API Server的交互，在API Server中更新处理后资源的状态。这就是一个controller的循环。其余的Controller大致也是依照这一流程在运行的。