kafka源码走读-coordinator

小白一个，今天第一次写博客，主要是开始接触kafka的工作，最近读了coordinator模块，阅读的是开源版本0.10.2.1，废话不多说，直接上菜吧。

本文主要介绍coordinator的基本作用以及相关rebalance的具体过程（由于是源码走读，篇幅会有些长，需要耐心，哈哈）。首先从概念上简单认识下相关的关键字：

1. consumer group：

• Consumer Group用group.id(String)作为全局唯一标识符
• 每个Group可以有零个、一个或多个Consumer Client
• 每个Group可以管理零个、一个或多个Topic
• Group下每个Consumer Client可同时订阅Topic的一个或多个Partition
• Group下同一个Partition只能被一个Client订阅，多Group下的Client订阅不受影响；因为如果一个Partition有多个Consumer，那么每个Consumer在该Partition上的Offset很可能会不一致，这样会导致在Rebalance后赋值处理的Client的消费起点发生混乱；与此同时，这种场景也不符合Kafka中Partition消息消费的一致性；因此在同一Group下一个Partition只能对应一个Consumer Client。

2. rebalance：

　　它本质上是一种协议，规定了一个 Consumer Group 下的所有 consumer 如何达成一致，来分配订阅 Topic 的每个分区。例如：某 Group 下有 20 个 consumer 实例，它订阅了一个具有 100 个 partition 的 Topic 。正常情况下，kafka 会为每个 Consumer 平均的分配 5 个分区。这个分配的过程就是 Rebalance。

　　需要注意的是，rebalance过程中group内的所有consumer会停止消费，非常影响kafka的消费性能，所以需要尽量避免rebalance的发生。

　　rebalance是本文的重点：先引用gaoyanliang的博客中的图，比较生动贴切，首先是join和sync过程：

　　1、Join 顾名思义就是加入组。这一步中，所有成员都向coordinator发送JoinGroup请求，请求加入消费组。一旦所有成员都发送了JoinGroup请求，coordinator会从中选择一个consumer担任leader的角色，并把组成员信息以及订阅信息发给leader

　　

 

　　2、Sync，这一步leader开始分配消费方案，即哪个consumer负责消费哪些topic的哪些partition。一旦完成分配，leader会将这个方案封装进SyncGroup请求中发给coordinator，非leader也会发SyncGroup请求，只是内容为空。coordinator接收到分配方案之后会把方案塞进SyncGroup的response中发给各个consumer。这样组内的所有成员就都知道自己应该消费哪些分区了。

 

　　然后是rebalance的场景分析：

　　1、新成员加入组

　　2、组成员崩溃

 

　　3、组成员主动离开组

 

 

 　　4、提交offset（这个不是引发rebalance的条件，可以看看这种场景下的流程分析）

 

 　　其中，还包括group订阅的topic发生变化；以及订阅的partition数发生变化（例如扩容），这些都会触发rebalance。以上就是发生rebalance的场景，图示非常生动形象，为了进一步细致的理解，就需要看代码的具体实现了。

3. 代码走读

coordinator服务主要分为client端和server端两部分，接下来我们直接上代码吧。

/**
     * Do one round of polling. In addition to checking for new data, this does any needed offset commits
     * (if auto-commit is enabled), and offset resets (if an offset reset policy is defined).
     * @param timeout The maximum time to block in the underlying call to {@link ConsumerNetworkClient#poll(long)}.
     * @return The fetched records (may be empty)
     */
    private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollOnce(long timeout) {
        coordinator.poll(time.milliseconds());

        // fetch positions if we have partitions we're subscribed to that we
        // don't know the offset for
        if (!subscriptions.hasAllFetchPositions())
            updateFetchPositions(this.subscriptions.missingFetchPositions());

        // if data is available already, return it immediately
        Map<TopicPartition, List<ConsumerRecord<K, V>>> records = fetcher.fetchedRecords();
        if (!records.isEmpty())
            return records;

        // send any new fetches (won't resend pending fetches)
        fetcher.sendFetches();

        long now = time.milliseconds();
        long pollTimeout = Math.min(coordinator.timeToNextPoll(now), timeout);

        client.poll(pollTimeout, now, new PollCondition() {
            @Override
            public boolean shouldBlock() {
                // since a fetch might be completed by the background thread, we need this poll condition
                // to ensure that we do not block unnecessarily in poll()
                return !fetcher.hasCompletedFetches();
            }
        });

        // after the long poll, we should check whether the group needs to rebalance
        // prior to returning data so that the group can stabilize faster
        if (coordinator.needRejoin())
            return Collections.emptyMap();

        return fetcher.fetchedRecords();
    }

 可以看到consumer在fetch数据之前首先调用了coordinator.poll(time.milliseconds())，这个就是coordinator的开始，进去看看做了什么吧。

/**
     * Poll for coordinator events. This ensures that the coordinator is known and that the consumer
     * has joined the group (if it is using group management). This also handles periodic（定期的） offset commits
     * if they are enabled.
     *
     * @param now current time in milliseconds
     */
    public void poll(long now) {
        // 先完成已提交offset的回调
        invokeCompletedOffsetCommitCallbacks();

        if (subscriptions.partitionsAutoAssigned() && coordinatorUnknown()) {
            // 自动分配partition模式要首先找到对应的coordinator，并建立连接
            ensureCoordinatorReady();
            now = time.milliseconds();
        }

        // 首先判断是否有需要rejoin group的情况发生，如果订阅的 partition 变化或则分配的 partition 变化时,需要 rejoin
        if (needRejoin()) {
            // due to a race condition between the initial metadata fetch and the initial rebalance,
            // we need to ensure that the metadata is fresh before joining initially. This ensures
            // that we have matched the pattern against the cluster's topics at least once before joining.
            // rejoin group 之前先刷新一下 metadata（对于 AUTO_PATTERN 而言）
            if (subscriptions.hasPatternSubscription())
                client.ensureFreshMetadata();

            // 确保 group 是 active; 加入 group; 分配订阅的 partition
            ensureActiveGroup();
            now = time.milliseconds();
        }

        // consumer poll心跳一次，用来记录两次poll之间的间隔
        pollHeartbeat(now);
        // ensureActiveGroup中成功结束时定了auto commit offset的时间，接下来完成commit offset
        maybeAutoCommitOffsetsAsync(now);
    }

从我的中文注释可以看到，首先完成已提交offset的回调，如果是自动分配partition的模式（注意，只有auto mode才需要接下来的coordinator服务以及needRejoin过程）要首先找到对应的coordinator，并建立连接。然后判断当前consumer是否有需要rejoin group，如果需要，确保 group 是 active，然后重新加入 group，分配得到订阅的partitions。然后consumer poll心跳一次，用来记录两次poll之间的间隔（为什么需要这个间隔，接下来再看，先记作mark1），最后由于在ensureActiveGroup中成功结束时定了auto commit offset的时间，到点完成commit offset。

首先是完成已提交offset的回调：

void invokeCompletedOffsetCommitCallbacks() {
        while (true) {
            OffsetCommitCompletion completion = completedOffsetCommits.poll();
            if (completion == null)
                break;
            completion.invoke();
        }
    }

进入invokeCompletedOffsetCommitCallbacks可以看到，是从一个线程安全的队列completedOffsetCommits中不断出队完成成功commit offset的回调，回调具体干了什么事，接下来看，记作mark2。

然后是确保coordinator服务处于ready状态：

/**
     * Block until the coordinator for this group is known and is ready to receive requests.
     */
    public synchronized void ensureCoordinatorReady() {
        // Using zero as current time since timeout is effectively infinite
        ensureCoordinatorReady(0, Long.MAX_VALUE);
    }

/**
     * Ensure that the coordinator is ready to receive requests.
     * @param startTimeMs Current time in milliseconds
     * @param timeoutMs Maximum time to wait to discover the coordinator
     * @return true If coordinator discovery and initial connection succeeded, false otherwise
     */
    protected synchronized boolean ensureCoordinatorReady(long startTimeMs, long timeoutMs) {
        long remainingMs = timeoutMs;

        while (coordinatorUnknown()) {
            // 找到coordinator
            RequestFuture<Void> future = lookupCoordinator();
            client.poll(future, remainingMs);

            if (future.failed()) {
                if (future.isRetriable()) {
                    remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
                    if (remainingMs <= 0)
                        break;

                    log.debug("Coordinator discovery failed for group {}, refreshing metadata", groupId);
                    client.awaitMetadataUpdate(remainingMs);
                } else
                    throw future.exception();
            } else if (coordinator != null && client.connectionFailed(coordinator)) {
                // we found the coordinator, but the connection has failed, so mark
                // it dead and backoff before retrying discovery
                coordinatorDead();
                time.sleep(retryBackoffMs);
            }

            remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
            if (remainingMs <= 0)
                break;
        }

        return !coordinatorUnknown();
    }

进去看到就是要确保找到该consumer group对应的coordinator服务，否则会一直block住，其中最主要的部分就是lookupCoordinator()，进去看看怎么找的。

protected synchronized RequestFuture<Void> lookupCoordinator() {
        if (findCoordinatorFuture == null) {
            // find a node to ask about the coordinator
            // 找负载最低的机器
            Node node = this.client.leastLoadedNode();
            if (node == null) {
                // TODO: If there are no brokers left, perhaps we should use the bootstrap set
                // from configuration?
                log.debug("No broker available to send GroupCoordinator request for group {}", groupId);
                return RequestFuture.noBrokersAvailable();
            } else
                // 给找到的机器发送获取该group的coordinator的request，回调成功赋给coordinator
                findCoordinatorFuture = sendGroupCoordinatorRequest(node);
        }
        return findCoordinatorFuture;
    }

/**
     * Discover the current coordinator for the group. Sends a GroupMetadata request to
     * one of the brokers. The returned future should be polled to get the result of the request.
     * @return A request future which indicates the completion of the metadata request
     */
    private RequestFuture<Void> sendGroupCoordinatorRequest(Node node) {
        // initiate the group metadata request
        log.debug("Sending GroupCoordinator request for group {} to broker {}", groupId, node);
        GroupCoordinatorRequest.Builder requestBuilder =
                new GroupCoordinatorRequest.Builder(this.groupId);
        return client.send(node, requestBuilder)
                     .compose(new GroupCoordinatorResponseHandler());
    }

可以看出，为了尽可能小的影响kafka的线上服务，它会先找一台负载最小的机器，然后在上面根据groupId给server端发送找该group coordinator的请求，我们去server端看看具体是怎么处理的。

def handleGroupCoordinatorRequest(request: RequestChannel.Request) {
    val groupCoordinatorRequest = request.body.asInstanceOf[GroupCoordinatorRequest]

    if (!authorize(request.session, Describe, new Resource(Group, groupCoordinatorRequest.groupId))) {
      val responseBody = new GroupCoordinatorResponse(Errors.GROUP_AUTHORIZATION_FAILED.code, Node.noNode)
      requestChannel.sendResponse(new RequestChannel.Response(request, responseBody))
    } else {
      // 根绝groupId找到对应的coordinator所在的partitionId
      val partition = coordinator.partitionFor(groupCoordinatorRequest.groupId)

      // get metadata (and create the topic if necessary)
      val offsetsTopicMetadata = getOrCreateGroupMetadataTopic(request.listenerName)

      val responseBody = if (offsetsTopicMetadata.error != Errors.NONE) {
        new GroupCoordinatorResponse(Errors.GROUP_COORDINATOR_NOT_AVAILABLE.code, Node.noNode)
      } else {
        
        // 根据找到的partitionId找到其leader所在的broker，该broker就是该group的coordinator所在的节点
        val coordinatorEndpoint = offsetsTopicMetadata.partitionMetadata().asScala
          .find(_.partition == partition)
          .map(_.leader())

        coordinatorEndpoint match {
          case Some(endpoint) if !endpoint.isEmpty =>
            new GroupCoordinatorResponse(Errors.NONE.code, endpoint)
          case _ =>
            new GroupCoordinatorResponse(Errors.GROUP_COORDINATOR_NOT_AVAILABLE.code, Node.noNode)
        }
      }

      debug("Sending consumer metadata %s for correlation id %d to client %s connectionId=%s ."
        .format(responseBody, request.header.correlationId, request.header.clientId, request.connectionId))
      requestChannel.sendResponse(new RequestChannel.Response(request, responseBody))
    }
  }

它首先根绝groupId找到对应的coordinator所在的partitionId，然后根据找到的partitionId找到其leader所在的broker，该broker就是该group的coordinator所在的节点，那么「groupId找到对应的coordinator所在的partitionId」这一步是怎么算出来的呢？看代码：

def partitionFor(groupId: String): Int = Utils.abs(groupId.hashCode) % groupMetadataTopicPartitionCount

原来是根据groupId的hashcode对「__consumer_offsets」这个topic的partition数求模，至于「__consumer_offsets」这个特殊的topic可以自行了解，不是本文的重点。然后server将GroupCoordinatorRequest的处理结果返回给client端，视线回到sendGroupCoordinatorRequest里，看看对返回结果怎么处理的：

private class GroupCoordinatorResponseHandler extends RequestFutureAdapter<ClientResponse, Void> {

        @Override
        public void onSuccess(ClientResponse resp, RequestFuture<Void> future) {
            log.debug("Received GroupCoordinator response {} for group {}", resp, groupId);

            GroupCoordinatorResponse groupCoordinatorResponse = (GroupCoordinatorResponse) resp.responseBody();
            // use MAX_VALUE - node.id as the coordinator id to mimic separate connections
            // for the coordinator in the underlying network client layer
            // TODO: this needs to be better handled in KAFKA-1935
            Errors error = Errors.forCode(groupCoordinatorResponse.errorCode());
            clearFindCoordinatorFuture();
            if (error == Errors.NONE) {
                synchronized (AbstractCoordinator.this) {
                    AbstractCoordinator.this.coordinator = new Node(
                            Integer.MAX_VALUE - groupCoordinatorResponse.node().id(),
                            groupCoordinatorResponse.node().host(),
                            groupCoordinatorResponse.node().port());
                    log.info("Discovered coordinator {} for group {}.", coordinator, groupId);
                    client.tryConnect(coordinator);
                    heartbeat.resetTimeouts(time.milliseconds());
                }
                future.complete(null);
            } else if (error == Errors.GROUP_AUTHORIZATION_FAILED) {
                future.raise(new GroupAuthorizationException(groupId));
            } else {
                log.debug("Group coordinator lookup for group {} failed: {}", groupId, error.message());
                future.raise(error);
            }
        }

        @Override
        public void onFailure(RuntimeException e, RequestFuture<Void> future) {
            clearFindCoordinatorFuture();
            super.onFailure(e, future);
        }
    }

可以看到client端拿到coordinator所在的机器信息，然后consumer network client去连接该机器，然后client端记录最近与server端连接session的时间。

rebalance过程

然后视线回到coordinator poll里，前两步走完了，该判断是否rejoin group了。

public boolean needRejoin() {
        if (!subscriptions.partitionsAutoAssigned())
            return false;

        // we need to rejoin if we performed the assignment and metadata has changed
        if (assignmentSnapshot != null && !assignmentSnapshot.equals(metadataSnapshot))
            return true;

        // we need to join if our subscription has changed since the last join
        if (joinedSubscription != null && !joinedSubscription.equals(subscriptions.subscription()))
            return true;

        return super.needRejoin();
    }

可以看出，只有auto assign模式的才可能需要，如果订阅的topic list发生变化、订阅的topic的partition发生变化，就需要rejoin，也就是要进行rebalance过程，其实还有个条件会触发，就是当group里有新的consumer加入或者有consumer主动退出或挂掉，这种情况接下来会遇到，暂定mark3吧。

如果需要rejoin，对于 AUTO_PATTERN 而言，需要先刷新一下metadata，然后就进入最重要的ensureActiveGroup()，它会确保group是active，然后完成rebalance过程中的两个重点：rejoin和sync，接下来重点介绍。

/**
     * Ensure that the group is active (i.e. joined and synced)
     */
    public void ensureActiveGroup() {
        // always ensure that the coordinator is ready because we may have been disconnected
        // when sending heartbeats and does not necessarily require us to rejoin the group.
        log.warn("starting ensureActiveGroup with coordinator {} for group {}", coordinator, groupId);
        long tsEnsureCoordinatorReady = System.currentTimeMillis();
        ensureCoordinatorReady();
        long tsStartHeartbeatThreadIfNeeded = System.currentTimeMillis();
        log.warn("ensureActiveGroup#ensureCoordinatorReady cost: {} with coordinator {} for group {}",
                tsStartHeartbeatThreadIfNeeded - tsEnsureCoordinatorReady, coordinator, groupId);
        startHeartbeatThreadIfNeeded();
        long tsJoinGroupIfNeeded = System.currentTimeMillis();
        log.warn("ensureActiveGroup#startHeartbeatThreadIfNeeded cost: {} with coordinator {} for group {}",
                tsJoinGroupIfNeeded - tsStartHeartbeatThreadIfNeeded, coordinator, groupId);
        joinGroupIfNeeded();
        log.warn("ensureActiveGroup#joinGroupIfNeeded cost: {} with coordinator {} for group {}",
                System.currentTimeMillis() - tsJoinGroupIfNeeded, coordinator, groupId);
        log.warn("finish ensureActiveGroup with coordinator {} for group {}", coordinator, groupId);
    }

可以看到之前还是需要确认coordinator已经处于ready状态，如果心跳线程没有起来则先起一个心跳线程，然后就进行join group的过程了。

void joinGroupIfNeeded() {
        long ts = System.currentTimeMillis();
        int loops = 0;
        while (needRejoin() || rejoinIncomplete()) {
            loops++;
            ensureCoordinatorReady();

            // call onJoinPrepare if needed. We set a flag to make sure that we do not call it a second
            // time if the client is woken up before a pending rebalance completes. This must be called
            // on each iteration of the loop because an event requiring a rebalance (such as a metadata
            // refresh which changes the matched subscription set) can occur while another rebalance is
            // still in progress.
            if (needsJoinPrepare) {
                long tsOnJoinPrepare = System.currentTimeMillis();
                onJoinPrepare(generation.generationId, generation.memberId);
                log.warn("onJoinPrepare cost: {} with coordinator {} for group {}",
                        System.currentTimeMillis() - tsOnJoinPrepare, coordinator, groupId);
                needsJoinPrepare = false;
            }


            // join group，future.value是分配结果，完成从client端发送join group请求到收到server端发来的assignment
            RequestFuture<ByteBuffer> future = initiateJoinGroup();
            long tsPoll = System.currentTimeMillis();
            client.poll(future);
            log.warn("client.poll cost: {} with coordinator {} for group {}",
                    System.currentTimeMillis() - tsPoll, coordinator, groupId);
            resetJoinGroupFuture();

            if (future.succeeded()) {
                needsJoinPrepare = true;
                long tsOnJoinComplete = System.currentTimeMillis();
                // 更新metadata，并设定下次auto commit offset的时间
                onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
                log.warn("onJoinComplete cost: {} with coordinator {} for group {}",
                        System.currentTimeMillis() - tsOnJoinComplete, coordinator, groupId);
            } else {
                RuntimeException exception = future.exception();
                if (exception instanceof UnknownMemberIdException ||
                        exception instanceof RebalanceInProgressException ||
                        exception instanceof IllegalGenerationException) {
                    log.warn("generation.memberId={} UnknownMemberIdException={} "
                                    + "RebalanceInProgressException={} "
                                    + "IllegalGenerationException={} with coordinator {} for group {}",
                            generation.memberId, exception instanceof UnknownMemberIdException,
                            exception instanceof RebalanceInProgressException,
                            exception instanceof IllegalGenerationException, coordinator, groupId);
                    continue;
                } else if (!future.isRetriable())
                    throw exception;
                time.sleep(retryBackoffMs);
            }
        }
        log.warn("joinGroupIfNeeded cost: {}, loops: {}, with coordinator {} for group {}"
                , System.currentTimeMillis() - ts, loops, coordinator, groupId);
    }

看了下，还是需要确认coordinator已经处于ready状态，然后要经过一次join之前的准备过程，进去JoinPrepare(generation.generationId, generation.memberId)里，看看在join之前做什么准备工作（擦屁股）？

protected void onJoinPrepare(int generation, String memberId) { // join之前先同步commit offset
        // commit offsets prior to rebalance if auto-commit enabled
        // 发送给server端commit offset
        maybeAutoCommitOffsetsSync(rebalanceTimeoutMs);

        // execute the user's callback before rebalance
        ConsumerRebalanceListener listener = subscriptions.listener();
        log.info("Revoking(吊销) previously assigned partitions {} for group {}", subscriptions.assignedPartitions(), groupId);
        try {
            Set<TopicPartition> revoked = new HashSet<>(subscriptions.assignedPartitions());
            listener.onPartitionsRevoked(revoked);
        } catch (WakeupException | InterruptException e) {
            throw e;
        } catch (Exception e) {
            log.error("User provided listener {} for group {} failed on partition revocation",
                    listener.getClass().getName(), groupId, e);
        }

        // join之前先全部归零，都不是leader，等待server端回复join group中包含谁是leader的信息
        isLeader = false;
        subscriptions.resetGroupSubscription();
    }

整体来看就是在join之前，要把该consumer之前的offset同步提交了，然后添加listener，用来在别处监听保存join之前的consumer的订阅状态。然后置为非leader（最终是不是group内里的consumer leader，需要稍后分析，记为mark4），最后将group的订阅topic list置为当前consumer订阅的topic list。可以先看下是怎么提交offset的。

private void maybeAutoCommitOffsetsSync(long timeoutMs) {
        if (autoCommitEnabled) {
            Map<TopicPartition, OffsetAndMetadata> allConsumedOffsets = subscriptions.allConsumed();
            try {
                log.debug("Sending synchronous auto-commit of offsets {} for group {}", allConsumedOffsets, groupId);
                if (!commitOffsetsSync(allConsumedOffsets, timeoutMs)) // 发送给server端commit offset
                    log.debug("Auto-commit of offsets {} for group {} timed out before completion",
                            allConsumedOffsets, groupId);
            } catch (WakeupException | InterruptException e) {
                log.debug("Auto-commit of offsets {} for group {} was interrupted before completion",
                        allConsumedOffsets, groupId);
                // rethrow wakeups since they are triggered by the user
                throw e;
            } catch (Exception e) {
                // consistent with async auto-commit failures, we do not propagate the exception
                log.warn("Auto-commit of offsets {} failed for group {}: {}", allConsumedOffsets, groupId,
                        e.getMessage());
            }
        }
    }

可以看到先取得该consumer消费所有partition的offset，然后发给server端commitOffsetsSync提交，去server端看看是怎么实现commit offset的。

public boolean commitOffsetsSync(Map<TopicPartition, OffsetAndMetadata> offsets, long timeoutMs) {
        invokeCompletedOffsetCommitCallbacks();

        if (offsets.isEmpty())
            return true;

        long now = time.milliseconds();
        long startMs = now;
        long remainingMs = timeoutMs;
        do {
            if (coordinatorUnknown()) {
                if (!ensureCoordinatorReady(now, remainingMs))
                    return false;

                remainingMs = timeoutMs - (time.milliseconds() - startMs);
            }

            // 构造OffsetCommitRequest请求，发送给服务端GroupCoordinator
            RequestFuture<Void> future = sendOffsetCommitRequest(offsets);
            // 等待请求完成
            client.poll(future, remainingMs);

            if (future.succeeded()) {
                if (interceptors != null)
                    interceptors.onCommit(offsets);
                return true;
            }

            if (!future.isRetriable())
                throw future.exception();

            time.sleep(retryBackoffMs);

            now = time.milliseconds();
            remainingMs = timeoutMs - (now - startMs);
        } while (remainingMs > 0);

        return false;
    }

/**
     * Commit offsets for the specified list of topics and partitions. This is a non-blocking call
     * which returns a request future that can be polled in the case of a synchronous commit or ignored in the
     * asynchronous case.
     *
     * @param offsets The list of offsets per partition that should be committed.
     * @return A request future whose value indicates whether the commit was successful or not
     */
    private RequestFuture<Void> sendOffsetCommitRequest(final Map<TopicPartition, OffsetAndMetadata> offsets) {
        if (offsets.isEmpty())
            return RequestFuture.voidSuccess();

        Node coordinator = coordinator();
        if (coordinator == null)
            return RequestFuture.coordinatorNotAvailable();

        // create the offset commit request
        Map<TopicPartition, OffsetCommitRequest.PartitionData> offsetData = new HashMap<>(offsets.size());
        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
            OffsetAndMetadata offsetAndMetadata = entry.getValue();
            if (offsetAndMetadata.offset() < 0) {
                return RequestFuture.failure(new IllegalArgumentException("Invalid offset: " + offsetAndMetadata.offset()));
            }
            offsetData.put(entry.getKey(), new OffsetCommitRequest.PartitionData(
                    offsetAndMetadata.offset(), offsetAndMetadata.metadata()));
        }

        final Generation generation;
        if (subscriptions.partitionsAutoAssigned())
            generation = generation();
        else
            generation = Generation.NO_GENERATION;

        // if the generation is null, we are not part of an active group (and we expect to be).
        // the only thing we can do is fail the commit and let the user rejoin the group in poll()
        // 不是stable状态，已经脱离group，需重置generation
        if (generation == null) {
            log.warn("sendOffsetCommitRequest generation is null. with {} to coordinator {} for group {}", offsets, coordinator, groupId);
            resetGeneration();
            return RequestFuture.failure(new CommitFailedException());
        }

        OffsetCommitRequest.Builder builder =
                new OffsetCommitRequest.Builder(this.groupId, offsetData).
                        setGenerationId(generation.generationId).
                        setMemberId(generation.memberId).
                        setRetentionTime(OffsetCommitRequest.DEFAULT_RETENTION_TIME);

        log.trace("Sending OffsetCommit request with {} to coordinator {} for group {}", offsets, coordinator, groupId);

        return client.send(coordinator, builder)
                .compose(new OffsetCommitResponseHandler(offsets));
    }

/**
    * Handle an offset commit request
    */
  def handleOffsetCommitRequest(request: RequestChannel.Request) {
    val header = request.header
    val offsetCommitRequest = request.body.asInstanceOf[OffsetCommitRequest]

    // reject the request if not authorized to the group
    if (!authorize(request.session, Read, new Resource(Group, offsetCommitRequest.groupId))) {
      val errorCode = new JShort(Errors.GROUP_AUTHORIZATION_FAILED.code)
      val results = offsetCommitRequest.offsetData.keySet.asScala.map { topicPartition =>
        (topicPartition, errorCode)
      }.toMap
      val response = new OffsetCommitResponse(results.asJava)
      requestChannel.sendResponse(new RequestChannel.Response(request, response))
    } else {
      val (existingAndAuthorizedForDescribeTopics, nonExistingOrUnauthorizedForDescribeTopics) = offsetCommitRequest.offsetData.asScala.toMap.partition {
        case (topicPartition, _) => {
          val authorizedForDescribe = authorize(request.session, Describe, new Resource(auth.Topic, topicPartition.topic))
          val exists = metadataCache.contains(topicPartition.topic)
          if (!authorizedForDescribe && exists)
            debug(s"Offset commit request with correlation id ${header.correlationId} from client ${header.clientId} " +
              s"on partition $topicPartition failing due to user not having DESCRIBE authorization, but returning UNKNOWN_TOPIC_OR_PARTITION")
          authorizedForDescribe && exists
        }
      }

      val (authorizedTopics, unauthorizedForReadTopics) = existingAndAuthorizedForDescribeTopics.partition {
        case (topicPartition, _) => authorize(request.session, Read, new Resource(auth.Topic, topicPartition.topic))
      }

      // the callback for sending an offset commit response
      def sendResponseCallback(commitStatus: immutable.Map[TopicPartition, Short]) {
        val combinedCommitStatus = commitStatus.mapValues(new JShort(_)) ++
          unauthorizedForReadTopics.mapValues(_ => new JShort(Errors.TOPIC_AUTHORIZATION_FAILED.code)) ++
          nonExistingOrUnauthorizedForDescribeTopics.mapValues(_ => new JShort(Errors.UNKNOWN_TOPIC_OR_PARTITION.code))

        if (isDebugEnabled)
          combinedCommitStatus.foreach { case (topicPartition, errorCode) =>
            if (errorCode != Errors.NONE.code) {
              debug(s"Offset commit request with correlation id ${header.correlationId} from client ${header.clientId} " +
                s"on partition $topicPartition failed due to ${Errors.forCode(errorCode).exceptionName}")
            }
          }
        val response = new OffsetCommitResponse(combinedCommitStatus.asJava)
        requestChannel.sendResponse(new RequestChannel.Response(request, response))
      }

      if (authorizedTopics.isEmpty)
        sendResponseCallback(Map.empty)
      else if (header.apiVersion == 0) {
        // for version 0 always store offsets to ZK
        val responseInfo = authorizedTopics.map {
          case (topicPartition, partitionData) =>
            val topicDirs = new ZKGroupTopicDirs(offsetCommitRequest.groupId, topicPartition.topic)
            try {
              if (partitionData.metadata != null && partitionData.metadata.length > config.offsetMetadataMaxSize)
                (topicPartition, Errors.OFFSET_METADATA_TOO_LARGE.code)
              else {
                zkUtils.updatePersistentPath(s"${topicDirs.consumerOffsetDir}/${topicPartition.partition}", partitionData.offset.toString)
                (topicPartition, Errors.NONE.code)
              }
            } catch {
              case e: Throwable => (topicPartition, Errors.forException(e).code)
            }
        }
        sendResponseCallback(responseInfo)
      } else {
        // for version 1 and beyond store offsets in offset manager

        // compute the retention time based on the request version:
        // if it is v1 or not specified by user, we can use the default retention
        val offsetRetention =
        if (header.apiVersion <= 1 ||
          offsetCommitRequest.retentionTime == OffsetCommitRequest.DEFAULT_RETENTION_TIME)
          coordinator.offsetConfig.offsetsRetentionMs
        else
          offsetCommitRequest.retentionTime

        // commit timestamp is always set to now.
        // "default" expiration timestamp is now + retention (and retention may be overridden if v2)
        // expire timestamp is computed differently for v1 and v2.
        //   - If v1 and no explicit commit timestamp is provided we use default expiration timestamp.
        //   - If v1 and explicit commit timestamp is provided we calculate retention from that explicit commit timestamp
        //   - If v2 we use the default expiration timestamp
        val currentTimestamp = time.milliseconds
        val defaultExpireTimestamp = offsetRetention + currentTimestamp
        val partitionData = authorizedTopics.mapValues { partitionData =>
          val metadata = if (partitionData.metadata == null) OffsetMetadata.NoMetadata else partitionData.metadata
          new OffsetAndMetadata(
            offsetMetadata = OffsetMetadata(partitionData.offset, metadata),
            commitTimestamp = currentTimestamp,
            expireTimestamp = {
              if (partitionData.timestamp == OffsetCommitRequest.DEFAULT_TIMESTAMP)
                defaultExpireTimestamp
              else
                offsetRetention + partitionData.timestamp
            }
          )
        }

        // call coordinator to handle commit offset
        coordinator.handleCommitOffsets(
          offsetCommitRequest.groupId,
          offsetCommitRequest.memberId,
          offsetCommitRequest.generationId,
          partitionData,
          sendResponseCallback)
      }
    }
  }

代码很长，在server端的前面部分可以简单的理解为request数据的处理，关键看后面的coordinator.handleCommitOffsets的实现。

def handleCommitOffsets(groupId: String,
                          memberId: String,
                          generationId: Int,
                          offsetMetadata: immutable.Map[TopicPartition, OffsetAndMetadata],
                          responseCallback: immutable.Map[TopicPartition, Short] => Unit) {
    if (!isActive.get) {
      responseCallback(offsetMetadata.mapValues(_ => Errors.GROUP_COORDINATOR_NOT_AVAILABLE.code))
    } else if (!isCoordinatorForGroup(groupId)) {
      responseCallback(offsetMetadata.mapValues(_ => Errors.NOT_COORDINATOR_FOR_GROUP.code))
    } else if (isCoordinatorLoadingInProgress(groupId)) {
      responseCallback(offsetMetadata.mapValues(_ => Errors.GROUP_LOAD_IN_PROGRESS.code))
    } else {
      groupManager.getGroup(groupId) match {
        case None =>
          if (generationId < 0) {
            // the group is not relying on Kafka for group management, so allow the commit
            val group = groupManager.addGroup(new GroupMetadata(groupId))
            doCommitOffsets(group, memberId, generationId, offsetMetadata, responseCallback)
          } else {
            // or this is a request coming from an older generation. either way, reject the commit
            responseCallback(offsetMetadata.mapValues(_ => Errors.ILLEGAL_GENERATION.code))
          }

        case Some(group) =>
          doCommitOffsets(group, memberId, generationId, offsetMetadata, responseCallback)
      }
    }
  }

简而言之，就是如果group在server端不存在，groupmanager会创建一个，然后doCommitOffsets真正实现提交。

private def doCommitOffsets(group: GroupMetadata,
                      memberId: String,
                      generationId: Int,
                      offsetMetadata: immutable.Map[TopicPartition, OffsetAndMetadata],
                      responseCallback: immutable.Map[TopicPartition, Short] => Unit) {
    var delayedOffsetStore: Option[DelayedStore] = None

    group synchronized {
      if (group.is(Dead)) {
        responseCallback(offsetMetadata.mapValues(_ => Errors.UNKNOWN_MEMBER_ID.code))
      } else if (generationId < 0 && group.is(Empty)) {
        // the group is only using Kafka to store offsets
        delayedOffsetStore = groupManager.prepareStoreOffsets(group, memberId, generationId,
          offsetMetadata, responseCallback)
      } else if (group.is(AwaitingSync)) {
        responseCallback(offsetMetadata.mapValues(_ => Errors.REBALANCE_IN_PROGRESS.code))
      } else if (!group.has(memberId)) {
        responseCallback(offsetMetadata.mapValues(_ => Errors.UNKNOWN_MEMBER_ID.code))
      } else if (generationId != group.generationId) {
        responseCallback(offsetMetadata.mapValues(_ => Errors.ILLEGAL_GENERATION.code))
      } else {
        val member = group.get(memberId)
        completeAndScheduleNextHeartbeatExpiration(group, member)
        delayedOffsetStore = groupManager.prepareStoreOffsets(group, memberId, generationId,
          offsetMetadata, responseCallback)
      }
    }

    // store the offsets without holding the group lock
    delayedOffsetStore.foreach(groupManager.store)
  }

主要干了两件事，completeAndScheduleNextHeartbeatExpiration(group, member)和groupManager.prepareStoreOffsets，分别看看具体实现：

/**
   * Complete existing DelayedHeartbeats for the given member and schedule the next one
   */
  private def completeAndScheduleNextHeartbeatExpiration(group: GroupMetadata, member: MemberMetadata) {
    // complete current heartbeat expectation
    // 先完成这次heartbeat
    member.latestHeartbeat = time.milliseconds()
    val memberKey = MemberKey(member.groupId, member.memberId)
    heartbeatPurgatory.checkAndComplete(memberKey)

    // reschedule the next heartbeat expiration deadline
    val newHeartbeatDeadline = member.latestHeartbeat + member.sessionTimeoutMs
    val delayedHeartbeat = new DelayedHeartbeat(this, group, member, newHeartbeatDeadline, member.sessionTimeoutMs)
    // 尝试完成heartbeat，否则就添加到watcher
    heartbeatPurgatory.tryCompleteElseWatch(delayedHeartbeat, Seq(memberKey))
  }

可以看到server端先完成了本次心跳，可以理解为这是client端consumer发来commit offset请求，server感知了这件事，收到新的一次心跳，认为发来request的这个consumer还活着。完成本地心跳后，会设置下次consumer发来心跳的deadline时间，并watch这个consumer member。此时，completeAndScheduleNextHeartbeatExpiration这件事干完了，接下来prepareStoreOffsets以及之后的delayedOffsetStore.foreach(groupManager.store)可以理解为将offset值存到metadata topic【__consumer_offsets】中。

然后视线回到join prepare，准备工作完成了，就该进入最最核心的一行代码：RequestFuture<ByteBuffer> future = initiateJoinGroup();在里面，完成了client端请求join group、coordinator server端选leader，leader计算tp分配方案assignment发给coordinator server，server再将assignment发送给group内的所有consumer。来看看具体实现吧：

private synchronized RequestFuture<ByteBuffer> initiateJoinGroup() {
        long ts = System.currentTimeMillis();
        // we store the join future in case we are woken up by the user after beginning the
        // rebalance in the call to poll below. This ensures that we do not mistakenly attempt
        // to rejoin before the pending rebalance has completed.
        // 不会在pending的rebalance完成之前错误的再次rejoin
        if (joinFuture == null) {
            // fence off the heartbeat thread explicitly so that it cannot interfere with the join group.
            // Note that this must come after the call to onJoinPrepare since we must be able to continue
            // sending heartbeats if that callback takes some time.
            long tsDisableHeartbeatThread = System.currentTimeMillis();
            // 加入组的过程停掉心跳
            disableHeartbeatThread();
            log.warn("disableHeartbeatThread cost: {} with coordinator {} for group {}",
                    System.currentTimeMillis() - tsDisableHeartbeatThread, coordinator, groupId);

            state = MemberState.REBALANCING;
            // 发送join group请求给服务端，中间包括server端回复join，然后leader consumer分配tp，和其它
            // consumer再往server端发送sync请求，并收到assignment回复
            joinFuture = sendJoinGroupRequest();
            joinFuture.addListener(new RequestFutureListener<ByteBuffer>() {
                @Override
                public void onSuccess(ByteBuffer value) {
                    // handle join completion in the callback so that the callback will be invoked
                    // even if the consumer is woken up before finishing the rebalance
                    synchronized (AbstractCoordinator.this) {
                        log.warn("Successfully joined group {} with generation {}", groupId,
                                generation.generationId);
                        state = MemberState.STABLE;
                        AbstractCoordinator.this.rejoinNeeded = false;

                        if (heartbeatThread != null)
                            // 加入组成功后再次启动心跳
                            heartbeatThread.enable();
                    }
                }

                @Override
                public void onFailure(RuntimeException e) {
                    // we handle failures below after the request finishes. if the join completes
                    // after having been woken up, the exception is ignored and we will rejoin
                    synchronized (AbstractCoordinator.this) {
                        state = MemberState.UNJOINED;
                    }
                }
            });
        }
        log.warn("initiateJoinGroup cost: {} with coordinator {} for group {}",
                System.currentTimeMillis() - ts, coordinator, groupId);
        return joinFuture;
    }

具体做了这几件事：先停掉consumer端的心跳，将client置为rebalancing的状态，然后发送join group请求给服务端，中间包括server端回复join，然后leader consumer分配tp，和其它consumer再往server端发送sync请求，并收到assignment回复，并给这个过程加上监听，成功就将client置为stable状态，启动心跳线程。

好吧，核心还在下一层代码，😓

/**
     * Join the group and return the assignment for the next generation. This function handles both
     * JoinGroup and SyncGroup, delegating to {@link #performAssignment(String, String, Map)} if
     * elected leader by the coordinator.
     * @return A request future which wraps the assignment returned from the group leader
     */
    private RequestFuture<ByteBuffer> sendJoinGroupRequest() {
        if (coordinatorUnknown())
            return RequestFuture.coordinatorNotAvailable();

        // send a join group request to the coordinator
        log.info("(Re-)joining group {}", groupId);
        JoinGroupRequest.Builder requestBuilder = new JoinGroupRequest.Builder(
                groupId,
                this.sessionTimeoutMs,
                this.generation.memberId,
                protocolType(),
                metadata()).setRebalanceTimeout(this.rebalanceTimeoutMs);

        log.debug("Sending JoinGroup ({}) to coordinator {}", requestBuilder, this.coordinator);
        long ts = System.currentTimeMillis();
        return client.send(coordinator, requestBuilder)
                .compose(new JoinGroupResponseHandler(ts));
    }

def handleJoinGroupRequest(request: RequestChannel.Request) {
    val joinGroupRequest = request.body.asInstanceOf[JoinGroupRequest]

    // the callback for sending a join-group response
    def sendResponseCallback(joinResult: JoinGroupResult) {
      val members = joinResult.members map { case (memberId, metadataArray) => (memberId, ByteBuffer.wrap(metadataArray)) }
      val responseBody = new JoinGroupResponse(request.header.apiVersion, joinResult.errorCode, joinResult.generationId,
        joinResult.subProtocol, joinResult.memberId, joinResult.leaderId, members.asJava)

      debug("Sending join group response %s for correlation id %d to client %s connectionId=%s ."
        .format(responseBody, request.header.correlationId, request.header.clientId, request.connectionId))
      requestChannel.sendResponse(new RequestChannel.Response(request, responseBody))
    }

    // the callback for limit group no response
    def noOperationCallback(): Unit = {
      requestChannel.noOperation(request.processor, request)
    }

    if (!authorize(request.session, Read, new Resource(Group, joinGroupRequest.groupId()))) {
      val responseBody = new JoinGroupResponse(
        request.header.apiVersion,
        Errors.GROUP_AUTHORIZATION_FAILED.code,
        JoinGroupResponse.UNKNOWN_GENERATION_ID,
        JoinGroupResponse.UNKNOWN_PROTOCOL,
        JoinGroupResponse.UNKNOWN_MEMBER_ID, // memberId
        JoinGroupResponse.UNKNOWN_MEMBER_ID, // leaderId
        Collections.emptyMap())
      requestChannel.sendResponse(new RequestChannel.Response(request, responseBody))
    } else {
      // let the coordinator to handle join-group
      val protocols = joinGroupRequest.groupProtocols().asScala.map(protocol =>
        (protocol.name, Utils.toArray(protocol.metadata))).toList
      coordinator.handleJoinGroup(
        joinGroupRequest.groupId,
        joinGroupRequest.memberId,
        request.header.clientId,
        request.session.clientAddress.toString,
        joinGroupRequest.rebalanceTimeout,
        joinGroupRequest.sessionTimeout,
        joinGroupRequest.protocolType,
        protocols,
        sendResponseCallback,
        noOperationCallback)
    }
  }

  def handleJoinGroup(groupId: String,
                      memberId: String,
                      clientId: String,
                      clientHost: String,
                      rebalanceTimeoutMs: Int,
                      sessionTimeoutMs: Int,
                      protocolType: String,
                      protocols: List[(String, Array[Byte])],
                      responseCallback: JoinCallback,
                      noOperationCallback: NoOpCallback) {
    if (!groupManager.isGroupBan(groupId)) {

      if (!isActive.get) {
        responseCallback(joinError(memberId, Errors.GROUP_COORDINATOR_NOT_AVAILABLE.code))
      } else if (!validGroupId(groupId)) {
        responseCallback(joinError(memberId, Errors.INVALID_GROUP_ID.code))
      } else if (!isCoordinatorForGroup(groupId)) {
        responseCallback(joinError(memberId, Errors.NOT_COORDINATOR_FOR_GROUP.code))
      } else if (isCoordinatorLoadingInProgress(groupId)) {
        responseCallback(joinError(memberId, Errors.GROUP_LOAD_IN_PROGRESS.code))
      } else if (sessionTimeoutMs < groupConfig.groupMinSessionTimeoutMs ||
        sessionTimeoutMs > groupConfig.groupMaxSessionTimeoutMs) {
        responseCallback(joinError(memberId, Errors.INVALID_SESSION_TIMEOUT.code))
      } else {
        // only try to create the group if the group is not unknown AND
        // the member id is UNKNOWN, if member is specified but group does not
        // exist we should reject the request
        groupManager.getGroup(groupId) match {
          case None =>
            if (memberId != JoinGroupRequest.UNKNOWN_MEMBER_ID) {
              responseCallback(joinError(memberId, Errors.UNKNOWN_MEMBER_ID.code))
            } else {
              val group = groupManager.addGroup(new GroupMetadata(groupId))
              doJoinGroup(group, memberId, clientId, clientHost, rebalanceTimeoutMs, sessionTimeoutMs, protocolType, protocols, responseCallback)
            }

          case Some(group) =>
            doJoinGroup(group, memberId, clientId, clientHost, rebalanceTimeoutMs, sessionTimeoutMs, protocolType, protocols, responseCallback)
        }
      }
    } else {
      noOperationCallback()
    }
  }

先重点看最后server端怎么处理join group request的吧。同样还是没有group先创建group的元信息，然后do join group：

private def doJoinGroup(group: GroupMetadata,
                          memberId: String,
                          clientId: String,
                          clientHost: String,
                          rebalanceTimeoutMs: Int,
                          sessionTimeoutMs: Int,
                          protocolType: String,
                          protocols: List[(String, Array[Byte])],
                          responseCallback: JoinCallback) {
    group synchronized {
      if (!group.is(Empty) && (group.protocolType != Some(protocolType) || !group.supportsProtocols(protocols.map(_._1).toSet))) {
        // if the new member does not support the group protocol, reject it
        responseCallback(joinError(memberId, Errors.INCONSISTENT_GROUP_PROTOCOL.code))
      } else if (memberId != JoinGroupRequest.UNKNOWN_MEMBER_ID && !group.has(memberId)) {
        // if the member trying to register with a un-recognized id, send the response to let
        // it reset its member id and retry
        responseCallback(joinError(memberId, Errors.UNKNOWN_MEMBER_ID.code))
      } else {
        group.currentState match {
          case Dead =>
            // if the group is marked as dead, it means some other thread has just removed the group
            // from the coordinator metadata; this is likely that the group has migrated to some other
            // coordinator OR the group is in a transient unstable phase. Let the member retry
            // joining without the specified member id,
            responseCallback(joinError(memberId, Errors.UNKNOWN_MEMBER_ID.code))

          case PreparingRebalance =>
            if (memberId == JoinGroupRequest.UNKNOWN_MEMBER_ID) {
              addMemberAndRebalance(rebalanceTimeoutMs, sessionTimeoutMs, clientId, clientHost, protocolType, protocols, group, responseCallback)
            } else {
              val member = group.get(memberId)
              updateMemberAndRebalance(group, member, protocols, responseCallback)
            }

          case AwaitingSync =>
            if (memberId == JoinGroupRequest.UNKNOWN_MEMBER_ID) {
              addMemberAndRebalance(rebalanceTimeoutMs, sessionTimeoutMs, clientId, clientHost, protocolType, protocols, group, responseCallback)
            } else {
              val member = group.get(memberId)
              if (member.matches(protocols)) {
                // member is joining with the same metadata (which could be because it failed to
                // receive the initial JoinGroup response), so just return current group information
                // for the current generation.
                responseCallback(JoinGroupResult(
                  members = if (memberId == group.leaderId) {
                    group.currentMemberMetadata
                  } else {
                    Map.empty
                  },
                  memberId = memberId,
                  generationId = group.generationId,
                  subProtocol = group.protocol,
                  leaderId = group.leaderId,
                  errorCode = Errors.NONE.code))
              } else {
                // member has changed metadata, so force a rebalance
                group.metrics.memberMetadataChange.mark()
                updateMemberAndRebalance(group, member, protocols, responseCallback)
              }
            }

          case Empty | Stable => {
            if(group.currentState == Empty){
              group.continueMetric()
            }
            if (memberId == JoinGroupRequest.UNKNOWN_MEMBER_ID) {
              // if the member id is unknown, register the member to the group
              addMemberAndRebalance(rebalanceTimeoutMs, sessionTimeoutMs, clientId, clientHost, protocolType, protocols, group, responseCallback)
            } else {
              val member = group.get(memberId)
              if (memberId == group.leaderId || !member.matches(protocols)) {
                // force a rebalance if a member has changed metadata or if the leader sends JoinGroup.
                // The latter allows the leader to trigger rebalances for changes affecting assignment
                // which do not affect the member metadata (such as topic metadata changes for the consumer)
                group.metrics.memberMetadataChange.mark()
                updateMemberAndRebalance(group, member, protocols, responseCallback)
              } else {
                // for followers with no actual change to their metadata, just return group information
                // for the current generation which will allow them to issue SyncGroup
                responseCallback(JoinGroupResult(
                  members = Map.empty,
                  memberId = memberId,
                  generationId = group.generationId,
                  subProtocol = group.protocol,
                  leaderId = group.leaderId,
                  errorCode = Errors.NONE.code))
              }
            }
          }
        }

        if (group.is(PreparingRebalance))
          joinPurgatory.checkAndComplete(GroupKey(group.groupId))
      }
    }
  }

可以看到正如文章开始的server端group的状态机一样，do join group只能在group为empty、preparing rebalancing、awaiting sync状态时做。addMemberAndRebalance和updateMemberAndRebalance几乎一样，以addMemberAndRebalance为例分析一下具体过程：

private def addMemberAndRebalance(rebalanceTimeoutMs: Int,
                                    sessionTimeoutMs: Int,
                                    clientId: String,
                                    clientHost: String,
                                    protocolType: String,
                                    protocols: List[(String, Array[Byte])],
                                    group: GroupMetadata,
                                    callback: JoinCallback) = {
    // use the client-id with a random id suffix as the member-id
    val memberId = clientId + "-" + group.generateMemberIdSuffix
    val member = new MemberMetadata(memberId, group.groupId, clientId, clientHost, rebalanceTimeoutMs,
      sessionTimeoutMs, protocolType, protocols)
    member.awaitingJoinCallback = callback
    if (groupConfig.configWhiteList.containsKey(group.groupId) ||
      group.allMembers.size <= groupConfig.groupMaxMemberNum) {
      // 已经选好leader，如果开始group的member是空，第一个member就是leader
      group.add(member)
      group.metrics.memberAdd.mark()
      // 构建好memberid并判断group的member数是否符合要求后，就可以进行rebalance了
      maybePrepareRebalance(group)
    } else {
      // 加入黑名单
      groupManager.markGroupMemberTooMuch(group.groupId,joinPurgatory)
    }
    //这里如果使用了这个返回值则会出现问题。
    member
  }

注意看group.add(member)这一行，其实leader consumer的选择策略就在这一行（此处回应mark4），如果group为空，新加入的第一个就是leader，否则，server端的coordinator记忆的还是之前的那个leader，或者如果之前的leader离开group了，coordinator也会很简单的选择剩下consumer中memberId最开头的。这在group.add和group.remove里有实现。然后就是maybePrepareRebalance了，可以进去看看：

private def maybePrepareRebalance(group: GroupMetadata) {
    group synchronized {
      if (group.canRebalance)
        prepareRebalance(group)
    }
  }

  private def prepareRebalance(group: GroupMetadata) {
    // if any members are awaiting sync, cancel their request and have them rejoin
    // preparerebalance之前可能是AwaitingSync、stable、empty，这是AwaitingSync的情况
    if (group.is(AwaitingSync))
      //完成assinment的重置和心跳完成并调度下次
      resetAndPropagateAssignmentError(group, Errors.REBALANCE_IN_PROGRESS)

    group.transitionTo(PreparingRebalance)
    group.metrics.groupRebalanced.mark()
    info("Preparing to restabilize group %s with old generation %s".format(group.groupId, group.generationId))

    val rebalanceTimeout = group.rebalanceTimeoutMs
    val delayedRebalance = new DelayedJoin(this, group, rebalanceTimeout)
    val groupKey = GroupKey(group.groupId)
    // 在rebalanceTimeout之前完成rebalance，并向members发送谁是leader的response
    joinPurgatory.tryCompleteElseWatch(delayedRebalance, Seq(groupKey))
  }

可以看到，如果当前group的state是awaiting syn的状态，现在又有join group，那么consumer们等待同步之前assignment的过程肯定要被中断，所以需要重置下上轮leader算的assignment，并且group内的所有members完成心跳，设定下次心跳时间。然后正式将group的state转为preparing rebalance的状态，最后设定一个delayjoin，它的具体含义就是：如果group内的members都报道了就开始执行join group操作，如果在deadline内还有未报道的member，coordinator会认为这个member死掉了，踢出group，将前来报道的members进行join group操作。具体join group的结果就是谁是leader，给leader发送的就是该group的metadata，非leader发送的就是空包！

很啰嗦？上代码吧！首先看resetAndPropagateAssignmentError的具体实现：

private def resetAndPropagateAssignmentError(group: GroupMetadata, error: Errors) {
    assert(group.is(AwaitingSync))
    // 需要rejoin，之前要清空各个member的assignment
    group.allMemberMetadata.foreach(_.assignment = Array.empty[Byte])
    // 发送给client，说明正在处于rebalance中，并完成心跳调度下一次心跳
    propagateAssignment(group, error)
  }

  private def propagateAssignment(group: GroupMetadata, error: Errors) {
    for (member <- group.allMemberMetadata) {
      if (member.awaitingSyncCallback != null) {
        // error是none时，发送给client的是assignment
        member.awaitingSyncCallback(member.assignment, error.code)
        member.awaitingSyncCallback = null

        // reset the session timeout for members after propagating the member's assignment.
        // This is because if any member's session expired while we were still awaiting either
        // the leader sync group or the storage callback, its expiration will be ignored and no
        // future heartbeat expectations will not be scheduled.
        completeAndScheduleNextHeartbeatExpiration(group, member)
      }
    }
  }

/**
   * Complete existing DelayedHeartbeats for the given member and schedule the next one
   */
  private def completeAndScheduleNextHeartbeatExpiration(group: GroupMetadata, member: MemberMetadata) {
    // complete current heartbeat expectation
    // 先完成这次heartbeat
    member.latestHeartbeat = time.milliseconds()
    val memberKey = MemberKey(member.groupId, member.memberId)
    heartbeatPurgatory.checkAndComplete(memberKey)

    // reschedule the next heartbeat expiration deadline
    val newHeartbeatDeadline = member.latestHeartbeat + member.sessionTimeoutMs
    val delayedHeartbeat = new DelayedHeartbeat(this, group, member, newHeartbeatDeadline, member.sessionTimeoutMs)
    // 尝试完成heartbeat，否则就添加到watcher
    heartbeatPurgatory.tryCompleteElseWatch(delayedHeartbeat, Seq(memberKey))
  }

然后是delay operation的实现，

  def tryCompleteJoin(group: GroupMetadata, forceComplete: () => Boolean) = {
    group synchronized {
      if (group.notYetRejoinedMembers.isEmpty)
        forceComplete()
      else false
    }
  }

在deadline时间内，会有两次tryCompleteJoin，如果members都已经rejoin了或超时，就会执行forceComplete，里面包含了join的处理细节：

def forceComplete(): Boolean = {
    if (completed.compareAndSet(false, true)) {
      // cancel the timeout timer
      cancel()
      onComplete()
      true
    } else {
      false
    }
  }

def onCompleteJoin(group: GroupMetadata) {
    var delayedStore: Option[DelayedStore] = None
    group synchronized {
      // remove any members who haven't joined the group yet
      val notYetRejoinedMembers = group.notYetRejoinedMembers
      group.metrics.memberNotSendJoinBeforeJoinComplete.mark(notYetRejoinedMembers.size)

      // 对于那些还没rejoin的member，从group中移出
      notYetRejoinedMembers.foreach { failedMember =>
        group.remove(failedMember.memberId)
        // TODO: cut the socket connection to the client
      }

      if (group.notYetRejoinedMembers.nonEmpty) {
        warn("onCompleteJoin group=" + group.groupId + " notYetRejoinedMembers=" + group.notYetRejoinedMembers.mkString(","))
      }

      if (!group.is(Dead)) {
        // generation加1，根据members有无转换group state到awatingsync或empty
        group.initNextGeneration()
        if (group.is(Empty)) {
          info(s"Group ${group.groupId} with generation ${group.generationId} is now empty")

          delayedStore = groupManager.prepareStoreGroup(group, Map.empty, error => {
            if (error != Errors.NONE) {
              // we failed to write the empty group metadata. If the broker fails before another rebalance,
              // the previous generation written to the log will become active again (and most likely timeout).
              // This should be safe since there are no active members in an empty generation, so we just warn.
              warn(s"Failed to write empty metadata for group ${group.groupId}: ${error.message}")
            }
          })
        } else {
          info(s"Stabilized group ${group.groupId}. generation ${group.generationId}. member size ${group.allMembers.size}. group leader info ${group.get(group.leaderId).toString}")

          // trigger the awaiting join group response callback for all the members after rebalancing
          for (member <- group.allMemberMetadata) {
            assert(member.awaitingJoinCallback != null)
            // 给group中的members发送join-group的response
            val joinResult = JoinGroupResult(
              members=if (member.memberId == group.leaderId) { group.currentMemberMetadata } else { Map.empty },
              memberId=member.memberId,
              generationId=group.generationId,
              subProtocol=group.protocol,
              leaderId=group.leaderId,
              errorCode=Errors.NONE.code)

            member.awaitingJoinCallback(joinResult)
            member.awaitingJoinCallback = null
            // 完成这次心跳并调度下次心跳
            completeAndScheduleNextHeartbeatExpiration(group, member)
          }
        }
      }
    }

    // call without holding the group lock
    delayedStore.foreach(groupManager.store)
  }

可以看到，会先算出group内有哪些还没join的member，把他们从group内踢出去，更新generation，将leaderId发送给所有members，将group的metadata发送给leader，每个member都会心跳一次，并更新下次心跳时间。

---------------这是join和sync的分界线----------------

至此，server端对于join group的请求处理完毕，client端会有join group response的处理，再回到client端看收到response是怎么处理的？

private class JoinGroupResponseHandler extends CoordinatorResponseHandler<JoinGroupResponse, ByteBuffer> {

        private final long tsBegin;

        JoinGroupResponseHandler(long tsBegin) {
            this.tsBegin = tsBegin;
        }

        @Override
        public void handle(JoinGroupResponse joinResponse, RequestFuture<ByteBuffer> future) {
            Errors error = Errors.forCode(joinResponse.errorCode());
            if (error == Errors.NONE) {
                log.debug("Received successful JoinGroup response for group {}: {}", groupId, joinResponse);
                sensors.joinLatency.record(response.requestLatencyMs());

                synchronized (AbstractCoordinator.this) {
                    if (state != MemberState.REBALANCING) {
                        // if the consumer was woken up before a rebalance completes, we may have already left
                        // the group. In this case, we do not want to continue with the sync group.
                        future.raise(new UnjoinedGroupException());
                    } else {
                        AbstractCoordinator.this.generation = new Generation(joinResponse.generationId(),
                                joinResponse.memberId(), joinResponse.groupProtocol());
                        if (joinResponse.isLeader()) {
                            // leader consumer具体的分配过程，并将assignment发送sync请求给server，server将
                            // assignment再回复回来
                            onJoinLeader(joinResponse).chain(future);
                        } else {
                            // 非leader的consumer包装个空集合发送sync请求给server，server将assignment再回复回来
                            onJoinFollower().chain(future);
                        }
                        log.warn("JoinGroup cost: {} with coordinator {} for group {}",
                                System.currentTimeMillis() - tsBegin, coordinator, groupId);
                    }
                }
            } else if (error == Errors.GROUP_LOAD_IN_PROGRESS) {
                log.warn("Attempt to join group {} rejected since coordinator {} is loading the group.", groupId,
                        coordinator());
                // backoff and retry
                future.raise(error);
            } else if (error == Errors.UNKNOWN_MEMBER_ID) {
                // reset the member id and retry immediately
                resetGeneration();
                log.warn("Attempt to join group {} failed due to unknown member id.", groupId);
                future.raise(Errors.UNKNOWN_MEMBER_ID);
            } else if (error == Errors.GROUP_COORDINATOR_NOT_AVAILABLE
                    || error == Errors.NOT_COORDINATOR_FOR_GROUP) {
                // re-discover the coordinator and retry with backoff
                coordinatorDead();
                log.warn("Attempt to join group {} failed due to obsolete coordinator information: {}", groupId, error.message());
                future.raise(error);
            } else if (error == Errors.INCONSISTENT_GROUP_PROTOCOL
                    || error == Errors.INVALID_SESSION_TIMEOUT
                    || error == Errors.INVALID_GROUP_ID) {
                // log the error and re-throw the exception
                log.error("Attempt to join group {} failed due to fatal error: {}", groupId, error.message());
                future.raise(error);
            } else if (error == Errors.GROUP_AUTHORIZATION_FAILED) {
                future.raise(new GroupAuthorizationException(groupId));
            } else {
                // unexpected error, throw the exception
                future.raise(new KafkaException("Unexpected error in join group response: " + error.message()));
            }
        }
    }

可以看到consumer收到response后会判断自己是不是leader，如果是，它会根据收到的group metadata计算出partition具体怎么分配的（assignment），然后将assignment发送给server端，server端再将这个assignment同步给所有consumer，这样所有consumer就知道自己负责消费哪些partition，如果自己不是leader的话，也同样会给server端发sync请求，只不过是个空请求。

先看下非leader干的什么事：

private RequestFuture<ByteBuffer> onJoinFollower() {
        // send follower's sync group with an empty assignment
        SyncGroupRequest.Builder requestBuilder =
                new SyncGroupRequest.Builder(groupId, generation.generationId, generation.memberId,
                        Collections.<String, ByteBuffer>emptyMap());
        log.debug("Sending follower SyncGroup for group {} to coordinator {}: {}", groupId, this.coordinator,
                requestBuilder);
        return sendSyncGroupRequest(requestBuilder);
    }

像上面说的一样，发送个空map给server端，由于leader计算完assignment也会往server端发sync请求，所以这里先不展示sync的处理过程，马上就会解释。

然后就是leader consumer的处理过程：

private RequestFuture<ByteBuffer> onJoinLeader(JoinGroupResponse joinResponse) {
        try {
            // perform the leader synchronization and send back the assignment for the group
            // consumer leader执行tp的assignment策略
            Map<String, ByteBuffer> groupAssignment = performAssignment(joinResponse.leaderId(), joinResponse.groupProtocol(),
                    joinResponse.members());

            // 带着assignment发给server端sync请求
            SyncGroupRequest.Builder requestBuilder =
                    new SyncGroupRequest.Builder(groupId, generation.generationId, generation.memberId, groupAssignment);
            log.debug("Sending leader SyncGroup for group {} to coordinator {}: {}",
                    groupId, this.coordinator, requestBuilder);
            // sync请求的回复，包含着assignment
            return sendSyncGroupRequest(requestBuilder);
        } catch (RuntimeException e) {
            return RequestFuture.failure(e);
        }
    }

他首先会计算assignment，具体的分配策略有三种：MockPartitionAssignor、RangePartitionAssignor、RoundRobinAssignor，具体可以自己搜索学习下，不是本文的重点。然后就是给server端发sync请求。

private RequestFuture<ByteBuffer> sendSyncGroupRequest(SyncGroupRequest.Builder requestBuilder) {
        if (coordinatorUnknown())
            return RequestFuture.coordinatorNotAvailable();
        long ts = System.currentTimeMillis();
        return client.send(coordinator, requestBuilder)
                .compose(new SyncGroupResponseHandler(ts));
    }

再跳到server端看怎么处理sync请求的吧：

def handleSyncGroupRequest(request: RequestChannel.Request) {
    val syncGroupRequest = request.body.asInstanceOf[SyncGroupRequest]

    def sendResponseCallback(memberState: Array[Byte], errorCode: Short) {
      val responseBody = new SyncGroupResponse(errorCode, ByteBuffer.wrap(memberState))
      debug("Sending sync group response %s for correlation id %d to client %s connectionId=%s ."
        .format(responseBody, request.header.correlationId, request.header.clientId, request.connectionId))

      requestChannel.sendResponse(new Response(request, responseBody))
    }

    // the callback for limit group no response
    def noOperationCallback(): Unit = {
      requestChannel.noOperation(request.processor, request)
    }

    if (!authorize(request.session, Read, new Resource(Group, syncGroupRequest.groupId()))) {
      sendResponseCallback(Array[Byte](), Errors.GROUP_AUTHORIZATION_FAILED.code)
    } else {
      coordinator.handleSyncGroup(
        syncGroupRequest.groupId(),
        syncGroupRequest.generationId(),
        syncGroupRequest.memberId(),
        syncGroupRequest.groupAssignment().asScala.mapValues(Utils.toArray),
        sendResponseCallback,
        noOperationCallback
      )
    }
  }

def handleSyncGroup(groupId: String,
                      generation: Int,
                      memberId: String,
                      groupAssignment: Map[String, Array[Byte]],
                      responseCallback: SyncCallback,
                      noOperationCallback: NoOpCallback) {
    if (!groupManager.isGroupBan(groupId)) {

      if (!isActive.get) {
        responseCallback(Array.empty, Errors.GROUP_COORDINATOR_NOT_AVAILABLE.code)
      } else if (!isCoordinatorForGroup(groupId)) {
        responseCallback(Array.empty, Errors.NOT_COORDINATOR_FOR_GROUP.code)
      } else {
        groupManager.getGroup(groupId) match {
          case None => responseCallback(Array.empty, Errors.UNKNOWN_MEMBER_ID.code)
          case Some(group) => doSyncGroup(group, generation, memberId, groupAssignment, responseCallback)
        }
      }
    } else {
      noOperationCallback()
    }
  }

private def doSyncGroup(group: GroupMetadata,
                          generationId: Int,
                          memberId: String,
                          groupAssignment: Map[String, Array[Byte]],
                          responseCallback: SyncCallback) {
    var delayedGroupStore: Option[DelayedStore] = None

    group synchronized {
      if (!group.has(memberId)) {
        responseCallback(Array.empty, Errors.UNKNOWN_MEMBER_ID.code)
      } else if (generationId != group.generationId) {
        responseCallback(Array.empty, Errors.ILLEGAL_GENERATION.code)
      } else {
        group.currentState match {
          case Empty | Dead =>
            responseCallback(Array.empty, Errors.UNKNOWN_MEMBER_ID.code)

          case PreparingRebalance =>
            responseCallback(Array.empty, Errors.REBALANCE_IN_PROGRESS.code)

          case AwaitingSync =>
            group.get(memberId).awaitingSyncCallback = responseCallback

            // if this is the leader, then we can attempt to persist state and transition to stable
            if (memberId == group.leaderId) {
              info(s"Assignment received from leader for group ${group.groupId} for generation ${group.generationId}")

              // fill any missing members with an empty assignment
              val missing = group.allMembers -- groupAssignment.keySet
              val assignment = groupAssignment ++ missing.map(_ -> Array.empty[Byte]).toMap

              // 将相关group metadata的信息做成消息形式，在后面会存到__consumer_offset中
              delayedGroupStore = groupManager.prepareStoreGroup(group, assignment, (error: Errors) => {
                group synchronized {
                  // another member may have joined the group while we were awaiting this callback,
                  // so we must ensure we are still in the AwaitingSync state and the same generation
                  // when it gets invoked. if we have transitioned to another state, then do nothing
                  // 不是AwaitingSync状态什么也不干
                  if (group.is(AwaitingSync) && generationId == group.generationId) {
                    if (error != Errors.NONE) {
                      // AwaitingSync时如果有其它member进来，要转为preparerebalance
                      resetAndPropagateAssignmentError(group, error)
                      maybePrepareRebalance(group)
                    } else {
                      // 给每个member分配leader计算的assignment，回复给client，并完成心跳，同时调度下次
                      setAndPropagateAssignment(group, assignment)
                      group.transitionTo(Stable)
                      group.latestStableTime.set(System.currentTimeMillis())
                    }
                  }
                }
              })
            }

          case Stable =>
            // if the group is stable, we just return the current assignment
            val memberMetadata = group.get(memberId)
            // 回复给client，并完成心跳，同时调度下次
            responseCallback(memberMetadata.assignment, Errors.NONE.code)
            completeAndScheduleNextHeartbeatExpiration(group, group.get(memberId))
        }
      }
    }

    // store the group metadata without holding the group lock to avoid the potential
    // for deadlock if the callback is invoked holding other locks (e.g. the replica
    // state change lock)
    delayedGroupStore.foreach(groupManager.store)
  }

可以直接看server端的处理逻辑：

 如果不是leader consumer发来的什么也不做，是leader的话，则将assignment做成消息的形式，在后面会存到__consumer_offset中，在这个过程中，如果有其他member中途进来，就需要再次清空assignment，重新调度心跳线程并prepare rebalance，如果过程顺利执行，就给每个member分配leader计算的assignment，回复给client，将group state转为stable，并心跳一次，同时调度下次心跳时间，这个将assignment回复给client的过程需要具体看下：

// 给每个member分配leader计算的assignment，并完成心跳，同时调度下次
  private def setAndPropagateAssignment(group: GroupMetadata, assignment: Map[String, Array[Byte]]) {
    assert(group.is(AwaitingSync))
    group.allMemberMetadata.foreach(member => member.assignment = assignment(member.memberId))
    // 发送给client的是assignment,并完成心跳调度下一次心跳
    propagateAssignment(group, Errors.NONE)
  }

private def propagateAssignment(group: GroupMetadata, error: Errors) {
    for (member <- group.allMemberMetadata) {
      if (member.awaitingSyncCallback != null) {
        member.awaitingSyncCallback(member.assignment, error.code)
        member.awaitingSyncCallback = null

        // reset the session timeout for members after propagating the member's assignment.
        // This is because if any member's session expired while we were still awaiting either
        // the leader sync group or the storage callback, its expiration will be ignored and no
        // future heartbeat expectations will not be scheduled.
        completeAndScheduleNextHeartbeatExpiration(group, member)
      }
    }
  }

其中，member.awaitingSyncCallback(member.assignment, error.code)就将每个member对应的assignment回调到client端了，为什么这么说呢？可以在doSyncGroup中找到group.get(memberId).awaitingSyncCallback = responseCallback这行代码，回调方法就是responseCallback，它从何而来呢？向前追溯到server端最开始的地方：

def handleSyncGroupRequest(request: RequestChannel.Request) {
    val syncGroupRequest = request.body.asInstanceOf[SyncGroupRequest]

    def sendResponseCallback(memberState: Array[Byte], errorCode: Short) {
      val responseBody = new SyncGroupResponse(errorCode, ByteBuffer.wrap(memberState))
      debug("Sending sync group response %s for correlation id %d to client %s connectionId=%s ."
        .format(responseBody, request.header.correlationId, request.header.clientId, request.connectionId))

      requestChannel.sendResponse(new Response(request, responseBody))
    }

    // the callback for limit group no response
    def noOperationCallback(): Unit = {
      requestChannel.noOperation(request.processor, request)
    }

    if (!authorize(request.session, Read, new Resource(Group, syncGroupRequest.groupId()))) {
      sendResponseCallback(Array[Byte](), Errors.GROUP_AUTHORIZATION_FAILED.code)
    } else {
      coordinator.handleSyncGroup(
        syncGroupRequest.groupId(),
        syncGroupRequest.generationId(),
        syncGroupRequest.memberId(),
        syncGroupRequest.groupAssignment().asScala.mapValues(Utils.toArray),
        sendResponseCallback,
        noOperationCallback
      )
    }
  }

就是这个sendResponseCallback，它会将member state，也就是该member的assignment装进去，作为response返回给client。那么client收到该response怎么做呢？视线再拉回到client端初始发送sync request的地方吧：

private RequestFuture<ByteBuffer> sendSyncGroupRequest(SyncGroupRequest.Builder requestBuilder) {
        if (coordinatorUnknown())
            return RequestFuture.coordinatorNotAvailable();
        long ts = System.currentTimeMillis();
        return client.send(coordinator, requestBuilder)
                .compose(new SyncGroupResponseHandler(ts));
    }

    private class SyncGroupResponseHandler extends CoordinatorResponseHandler<SyncGroupResponse, ByteBuffer> {

        private final long tsBegin;

        SyncGroupResponseHandler(long tsBegin) {
            this.tsBegin = tsBegin;
        }

        @Override
        public void handle(SyncGroupResponse syncResponse,
                           RequestFuture<ByteBuffer> future) {
            Errors error = Errors.forCode(syncResponse.errorCode());
            if (error == Errors.NONE) {
                sensors.syncLatency.record(response.requestLatencyMs());
                future.complete(syncResponse.memberAssignment());
                log.warn("SyncGroup cost: {} with coordinator {} for group {}",
                        System.currentTimeMillis() - tsBegin, coordinator, groupId);
            } else {
                requestRejoin();

                if (error == Errors.GROUP_AUTHORIZATION_FAILED) {
                    future.raise(new GroupAuthorizationException(groupId));
                } else if (error == Errors.REBALANCE_IN_PROGRESS) {
                    log.warn("SyncGroup for group {} failed due to coordinator rebalance", groupId);
                    future.raise(error);
                } else if (error == Errors.UNKNOWN_MEMBER_ID
                        || error == Errors.ILLEGAL_GENERATION) {
                    log.warn("SyncGroup for group {} failed due to {}", groupId, error);
                    resetGeneration();
                    future.raise(error);
                } else if (error == Errors.GROUP_COORDINATOR_NOT_AVAILABLE
                        || error == Errors.NOT_COORDINATOR_FOR_GROUP) {
                    log.warn("SyncGroup for group {} failed due to {}", groupId, error);
                    coordinatorDead();
                    future.raise(error);
                } else {
                    future.raise(new KafkaException("Unexpected error from SyncGroup: " + error.message()));
                }
            }
        }
    }

一行代码：future.complete(syncResponse.memberAssignment())，表明这个future以该consumer的assignment的值返回的，再向上回溯到joinGroupIfNeeded：

void joinGroupIfNeeded() {
        long ts = System.currentTimeMillis();
        int loops = 0;
        while (needRejoin() || rejoinIncomplete()) {
            loops++;
            ensureCoordinatorReady();

            // call onJoinPrepare if needed. We set a flag to make sure that we do not call it a second
            // time if the client is woken up before a pending rebalance completes. This must be called
            // on each iteration of the loop because an event requiring a rebalance (such as a metadata
            // refresh which changes the matched subscription set) can occur while another rebalance is
            // still in progress.
            if (needsJoinPrepare) {
                long tsOnJoinPrepare = System.currentTimeMillis();
                onJoinPrepare(generation.generationId, generation.memberId);
                log.warn("onJoinPrepare cost: {} with coordinator {} for group {}",
                        System.currentTimeMillis() - tsOnJoinPrepare, coordinator, groupId);
                needsJoinPrepare = false;
            }


            // join group，future.value是分配结果，完成从client端发送join group请求到收到server端发来的assignment
            RequestFuture<ByteBuffer> future = initiateJoinGroup();
            long tsPoll = System.currentTimeMillis();
            client.poll(future);
            log.warn("client.poll cost: {} with coordinator {} for group {}",
                    System.currentTimeMillis() - tsPoll, coordinator, groupId);
            resetJoinGroupFuture();

            if (future.succeeded()) {
                needsJoinPrepare = true;
                long tsOnJoinComplete = System.currentTimeMillis();
                // 更新metadata，并设定下次auto commit offset的时间
                onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
                log.warn("onJoinComplete cost: {} with coordinator {} for group {}",
                        System.currentTimeMillis() - tsOnJoinComplete, coordinator, groupId);
            } else {
                RuntimeException exception = future.exception();
                if (exception instanceof UnknownMemberIdException ||
                        exception instanceof RebalanceInProgressException ||
                        exception instanceof IllegalGenerationException) {
                    log.warn("generation.memberId={} UnknownMemberIdException={} "
                                    + "RebalanceInProgressException={} "
                                    + "IllegalGenerationException={} with coordinator {} for group {}",
                            generation.memberId, exception instanceof UnknownMemberIdException,
                            exception instanceof RebalanceInProgressException,
                            exception instanceof IllegalGenerationException, coordinator, groupId);
                    continue;
                } else if (!future.isRetriable())
                    throw exception;
                time.sleep(retryBackoffMs);
            }
        }
        log.warn("joinGroupIfNeeded cost: {}, loops: {}, with coordinator {} for group {}"
                , System.currentTimeMillis() - ts, loops, coordinator, groupId);
    }

onJoinComplete这个方法用到了future.value的值，也就是consumer的assignment值，进去看下干什么了：

protected void onJoinComplete(int generation,
                                  String memberId,
                                  String assignmentStrategy,
                                  ByteBuffer assignmentBuffer) {
        // only the leader is responsible for monitoring for metadata changes (i.e. partition changes)
        if (!isLeader) {
            log.warn("onJoinComplete is follower, generation=" + generation + " memberId=" + memberId + " group=" + groupId);
            assignmentSnapshot = null;
        } else {
            log.warn("onJoinComplete is leader, generation=" + generation + " memberId=" + memberId + " group=" + groupId);
        }

        PartitionAssignor assignor = lookupAssignor(assignmentStrategy);
        if (assignor == null)
            throw new IllegalStateException("Coordinator selected invalid assignment protocol: " + assignmentStrategy);

        Assignment assignment = ConsumerProtocol.deserializeAssignment(assignmentBuffer);

        // set the flag to refresh last committed offsets
        // 允许从服务端获取最近一次提交的offset
        subscriptions.needRefreshCommits();

        // update partition assignment
        subscriptions.assignFromSubscribed(assignment.partitions());

        // check if the assignment contains some topics that were not in the original
        // subscription, if yes we will obey what leader has decided and add these topics
        // into the subscriptions as long as they still match the subscribed pattern
        //
        // TODO this part of the logic should be removed once we allow regex on leader assign
        // 分配到tp中有新的topic，需要加到subscriptions里
        Set<String> addedTopics = new HashSet<>();
        for (TopicPartition tp : subscriptions.assignedPartitions()) {
            if (!joinedSubscription.contains(tp.topic()))
                addedTopics.add(tp.topic());
        }

        if (!addedTopics.isEmpty()) {
            Set<String> newSubscription = new HashSet<>(subscriptions.subscription());
            Set<String> newJoinedSubscription = new HashSet<>(joinedSubscription);
            newSubscription.addAll(addedTopics);
            newJoinedSubscription.addAll(addedTopics);

            this.subscriptions.subscribeFromPattern(newSubscription);
            this.joinedSubscription = newJoinedSubscription;
        }

        // update the metadata and enforce a refresh to make sure the fetcher can start
        // fetching data in the next iteration
        // 更新metadata
        this.metadata.setTopics(subscriptions.groupSubscription());
        client.ensureFreshMetadata();

        // give the assignor a chance to update internal state based on the received assignment
        assignor.onAssignment(assignment);

        // reschedule the auto commit starting from now
        this.nextAutoCommitDeadline = time.milliseconds() + autoCommitIntervalMs;

        // execute the user's callback after rebalance
        ConsumerRebalanceListener listener = subscriptions.listener();
        log.info("Setting newly assigned partitions {} for group {}", subscriptions.assignedPartitions(), groupId);
        try {
            Set<TopicPartition> assigned = new HashSet<>(subscriptions.assignedPartitions());
            listener.onPartitionsAssigned(assigned);
        } catch (WakeupException | InterruptException e) {
            throw e;
        } catch (Exception e) {
            log.error("User provided listener {} for group {} failed on partition assignment",
                    listener.getClass().getName(), groupId, e);
        }
    }

可以看出，用assignment更新了consumer的subscriptions，同时可能订阅了新的topic，还更新了该consumer订阅的topic list，client端更新metadata值，重新计算下次auto commit offset。至此，needJoin的逻辑全部执行完毕，回到poll中，还剩下两步：

public void poll(long now) {
        // 先完成已提交offset的回调
        invokeCompletedOffsetCommitCallbacks();

        if (subscriptions.partitionsAutoAssigned() && coordinatorUnknown()) {
            // 自动分配partition模式要首先找到对应的coordinator，并建立连接
            ensureCoordinatorReady();
            now = time.milliseconds();
        }

        // 首先判断是否有需要rejoin group的情况发生，如果订阅的 partition 变化或则分配的 partition 变化时,需要 rejoin
        if (needRejoin()) {
            // due to a race condition between the initial metadata fetch and the initial rebalance,
            // we need to ensure that the metadata is fresh before joining initially. This ensures
            // that we have matched the pattern against the cluster's topics at least once before joining.
            // rejoin group 之前先刷新一下 metadata（对于 AUTO_PATTERN 而言）
            if (subscriptions.hasPatternSubscription())
                client.ensureFreshMetadata();

            // 确保 group 是 active; 加入 group; 分配订阅的 partition
            ensureActiveGroup();
            now = time.milliseconds();
        }

        // consumer poll心跳一次，用来记录两次poll之间的间隔
        pollHeartbeat(now);
        // ensureActiveGroup中成功结束时定了auto commit offset的时间，接下来完成commit offset
        maybeAutoCommitOffsetsAsync(now);
    }

consumer 需要poll心跳一次，用来记录两次poll之间的间隔，这个还是mark1的问题，马上回应。然后就是由于在ensureActiveGroup中成功结束时定了auto commit offset的时间，接下来完成commit offset，具体看下怎么commit的：

private void maybeAutoCommitOffsetsAsync(long now) {
        if (autoCommitEnabled) {
            if (!subscriptions.partitionsAutoAssigned()) {
                if (now >= nextAutoCommitDeadline) {
                    this.nextAutoCommitDeadline = now + autoCommitIntervalMs;
                    doAutoCommitOffsetsAsync();
                }
            } else {
                if (coordinatorUnknown()) {
                    this.nextAutoCommitDeadline = now + retryBackoffMs;
                } else if (now >= nextAutoCommitDeadline) {
                    this.nextAutoCommitDeadline = now + autoCommitIntervalMs;
                    doAutoCommitOffsetsAsync();
                }
            }
        }
    }

private void doAutoCommitOffsetsAsync() {
        Map<TopicPartition, OffsetAndMetadata> allConsumedOffsets = subscriptions.allConsumed();
        log.debug("Sending asynchronous auto-commit of offsets {} for group {}", allConsumedOffsets, groupId);

        commitOffsetsAsync(allConsumedOffsets, new OffsetCommitCallback() {
            @Override
            public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
                if (exception != null) {
                    log.warn("Auto-commit of offsets {} failed for group {}: {}, isLeader:{}",
                            offsets, groupId, isLeader, exception.getMessage());
                    if (exception instanceof RetriableException)
                        nextAutoCommitDeadline = Math.min(time.milliseconds() + retryBackoffMs, nextAutoCommitDeadline);
                } else {
                    log.debug("Completed auto-commit of offsets {} for group {}", offsets, groupId);
                }
            }
        });
    }

public void commitOffsetsAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, final OffsetCommitCallback callback) {
        // 所有tp commit offset完成回调
        invokeCompletedOffsetCommitCallbacks();

        if (!coordinatorUnknown()) {
            doCommitOffsetsAsync(offsets, callback);
        } else {
            // we don't know the current coordinator, so try to find it and then send the commit
            // or fail (we don't want recursive retries which can cause offset commits to arrive
            // out of order). Note that there may be multiple offset commits chained to the same
            // coordinator lookup request. This is fine because the listeners will be invoked in
            // the same order that they were added. Note also that AbstractCoordinator prevents
            // multiple concurrent coordinator lookup requests.
            // 一个coordinator可能会处理多个commit offset，按顺序处理
            pendingAsyncCommits.incrementAndGet();
            lookupCoordinator().addListener(new RequestFutureListener<Void>() {
                @Override
                public void onSuccess(Void value) {
                    pendingAsyncCommits.decrementAndGet();
                    doCommitOffsetsAsync(offsets, callback);
                }

                @Override
                public void onFailure(RuntimeException e) {
                    pendingAsyncCommits.decrementAndGet();
                    completedOffsetCommits.add(new OffsetCommitCompletion(callback, offsets, new RetriableCommitFailedException(e)));
                }
            });
        }

        // ensure the commit has a chance to be transmitted (without blocking on its completion).
        // Note that commits are treated as heartbeats by the coordinator, so there is no need to
        // explicitly allow heartbeats through delayed task execution.
        client.pollNoWakeup();
    }

可以在最后看到，这是个异步提交的过程，一个coordinator可能会处理多个commit offset，按顺序处理，具体的提交细节还是回到了doCommitOffsetsAsync，这个在上面提到过，在看一下：

private void doCommitOffsetsAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, final OffsetCommitCallback callback) {
        this.subscriptions.needRefreshCommits();
        // 发送给server请求commit offset
        RequestFuture<Void> future = sendOffsetCommitRequest(offsets);
        final OffsetCommitCallback cb = callback == null ? defaultOffsetCommitCallback : callback;
        // 成功/失败的回调
        future.addListener(new RequestFutureListener<Void>() {
            @Override
            public void onSuccess(Void value) {
                if (interceptors != null)
                    interceptors.onCommit(offsets);

                completedOffsetCommits.add(new OffsetCommitCompletion(cb, offsets, null));
            }

            @Override
            public void onFailure(RuntimeException e) {
                Exception commitException = e;

                if (e instanceof RetriableException)
                    commitException = new RetriableCommitFailedException(e);

                completedOffsetCommits.add(new OffsetCommitCompletion(cb, offsets, commitException));
            }
        });
    }

还是会向Server端发提交offset的请求，最后将回调方法包装为一个OffsetCommitCompletion放到completedOffsetCommits里。回调方法的实现具体就是doAutoCommitOffsetsAsync里的new OffsetCommitCallback()的实现，看到了这里，completedOffsetCommits是不是很熟悉，它呼应了mark2。

最后再关注一下client端的heartbeat的run方法：

public void run() {
            try {
                log.debug("Heartbeat thread for group {} started", groupId);
                while (true) {
                    synchronized (AbstractCoordinator.this) {
                        if (closed)
                            return;

                        if (!enabled) {
                            AbstractCoordinator.this.wait();
                            continue;
                        }

                        if (state != MemberState.STABLE) {
                            // the group is not stable (perhaps because we left the group or because the coordinator
                            // kicked us out), so disable heartbeats and wait for the main thread to rejoin.
                            disable();
                            continue;
                        }

                        client.pollNoWakeup();
                        long now = time.milliseconds();

                        if (coordinatorUnknown()) {
                            if (findCoordinatorFuture == null)
                                lookupCoordinator();
                            else
                                AbstractCoordinator.this.wait(retryBackoffMs);
                        } else if (heartbeat.sessionTimeoutExpired(now)) {
                            // the session timeout has expired without seeing a successful heartbeat, so we should
                            // probably make sure the coordinator is still healthy.
                            // server端coordinator超时没有返回心跳
                            log.warn("(heartbeat.sessionTimeout,generation={}, groupId={}",generation,groupId);
                            coordinatorDead();
                        } else if (heartbeat.pollTimeoutExpired(now)) { // consumer在poll间隔内超时没有poll了，估计leave group了
                            // the poll timeout has expired, which means that the foreground thread has stalled
                            // in between calls to poll(), so we explicitly leave the group.
                            log.warn("(heartbeat.pollTimeout,generation={}, groupId={}",generation,groupId);
                            maybeLeaveGroup();
                        } else if (!heartbeat.shouldHeartbeat(now)) {
                            // poll again after waiting for the retry backoff in case the heartbeat failed or the
                            // coordinator disconnected
                            AbstractCoordinator.this.wait(retryBackoffMs);
                        } else {
                            heartbeat.sentHeartbeat(now);

                            // 向server端发心跳请求
                            sendHeartbeatRequest().addListener(new RequestFutureListener<Void>() {
                                @Override
                                public void onSuccess(Void value) {
                                    synchronized (AbstractCoordinator.this) {
                                        // server端收到心跳并回复
                                        heartbeat.receiveHeartbeat(time.milliseconds());
                                    }
                                }

                                @Override
                                public void onFailure(RuntimeException e) {
                                    synchronized (AbstractCoordinator.this) {
                                        if (e instanceof RebalanceInProgressException) {
                                            // it is valid to continue heartbeating while the group is rebalancing. This
                                            // ensures that the coordinator keeps the member in the group for as long
                                            // as the duration of the rebalance timeout. If we stop sending heartbeats,
                                            // however, then the session timeout may expire before we can rejoin.
                                            heartbeat.receiveHeartbeat(time.milliseconds());
                                        } else {
                                            heartbeat.failHeartbeat();

                                            // wake up the thread if it's sleeping to reschedule the heartbeat
                                            AbstractCoordinator.this.notify();
                                        }
                                    }
                                }
                            });
                        }
                    }
                }
            } catch (InterruptedException | InterruptException e) {
                Thread.interrupted();
                log.error("Unexpected interrupt received in heartbeat thread for group {}", groupId, e);
                this.failed.set(new RuntimeException(e));
            } catch (RuntimeException e) {
                log.error("Heartbeat thread for group {} failed due to unexpected error" , groupId, e);
                this.failed.set(e);
            } finally {
                log.debug("Heartbeat thread for group {} has closed", groupId);
            }
        }

    }

可以看到heartbeat.sessionTimeoutExpired(now)，这个表示的是现在距离上次与server端的coordinator的session连接时间，也就是coordinator返回心跳超时了，会认为coordinator挂掉，导致之后又需要lookup coordinator。而heartbeat.pollTimeoutExpired(now)呼应了mark1，如果client端两次poll心跳超时，coordinator会认为该consumer离开group了，做leave group的处理，同理由client端向server端发送leave group request，server端仍然做一个rebalance处理，这就呼应了mark3，正常情况下，client端心跳一次，并向server端发送心跳请求，server端做心跳处理，如果在心跳超时范围内，则成功返回给client端一个心跳。

 

作者写篇博客不容易，觉得有用的话赏点小钱请作者喝杯可乐吧：