Hbase flusher源码解析(flush全代码流程解析)

版权声明:本文为博主原创文章,遵循版权协议,转载请附上原文出处链接和本声明。

在介绍HBASE flush源码之前,我们先在逻辑上大体梳理一下,便于后续看代码。flush的整体流程分三个阶段

  1.第一阶段:prepare阶段,这个阶段主要是将当前memstore的内存结构做snapshot。HBASE写入内存的数据结构(memstore以及snapshot)是跳跃表,用的是jdk自带的ConcurrentSkipListMap结构。这个过程其实就是将memstore赋值给snapshot,并构造一个新的memstore。

  2.第二阶段:flushcache阶段,这个阶段主要是将第一阶段生成的snapshot flush到disk,但是注意这里是将其flush到temp文件,此时并没有将生成的hfile move到store实际对应的cf路径下,move是发生在第三阶段。

  3.第三阶段:commit阶段。这个阶段主要是将第二阶段生成的hfile move最终正确的位置。

上面是HBASE flush的逻辑流程,flush是region级别,涉及到的类很多,下面我们开始介绍一下Flush相关的操作

flush线程启动

  • 在regionserver启动时,会调用startServiceThread方法启动一些服务线程,其中
// Cache flushing
protected MemStoreFlusher cacheFlusher;
。。。。。省略。。。。。。
private void startServiceThreads() throws IOException { 。。。。其他代码省略。。。 this.cacheFlusher.start(uncaughtExceptionHandler); }
  •  而cacheFlusher是MemStoreFlusher类的实例,在梳理上述逻辑之前首先介绍两个MemStoreFlusher的变量
  •  //该变量是一个BlockingQueue<FlushQueueEntry>类型的变量。
      // 主要存储了FlushRegionEntry类型刷新请求实例,以及一个唤醒队列WakeupFlushThread实例对象。
      private final BlockingQueue<FlushQueueEntry> flushQueue =
        new DelayQueue<FlushQueueEntry>();
      //同时也会把加入到flushqueue中的requst加入到regionsInQueue中。
      private final Map<HRegion, FlushRegionEntry> regionsInQueue =
        new HashMap<HRegion, FlushRegionEntry>();
  • MemStoreFlusher的start方法如下:
 synchronized void start(UncaughtExceptionHandler eh) {
    ThreadFactory flusherThreadFactory = Threads.newDaemonThreadFactory(
        server.getServerName().toShortString() + "-MemStoreFlusher", eh);
    for (int i = 0; i < flushHandlers.length; i++) {
      flushHandlers[i] = new FlushHandler("MemStoreFlusher." + i);
      flusherThreadFactory.newThread(flushHandlers[i]);
      flushHandlers[i].start();
    }
  }

  会根据配置flusher.handler.count生成相应个数的flushHandler线程。然后对每一个flushHandler线程调用start方法。我们继续看一下flushHandler。

private class FlushHandler extends HasThread {
private FlushHandler(String name) {
super(name);
}
@Override
public void run() {
//如果server正常没有stop
while (!server.isStopped()) {
FlushQueueEntry fqe = null;
try {
wakeupPending.set(false); // allow someone to wake us up again
//阻塞队列的poll方法,如果没有会阻塞在这
fqe = flushQueue.poll(threadWakeFrequency, TimeUnit.MILLISECONDS);
if (fqe == null || fqe instanceof WakeupFlushThread) {
// 如果没有flush request或者flush request是一个全局flush的request。
if (isAboveLowWaterMark()) {
// 检查所有的memstore是否超过max_heap * hbase.regionserver.global.memstore.lowerLimit配置的值,默认0.35
// 超过配置的最小memstore的值,flush最大的一个memstore的region
LOG.debug("Flush thread woke up because memory above low water="
+ TraditionalBinaryPrefix.long2String(globalMemStoreLimitLowMark, "", 1));

if (!flushOneForGlobalPressure()) {
// 如果没有任何Region需要flush,但已经超过了lowerLimit。
// 这种情况不太可能发生,除非可能会在关闭整个服务器时发生,即有另一个线程正在执行flush regions。
// 只里只需要sleep一下,然后唤醒任何被阻塞的线程再次检查。
// Wasn't able to flush any region, but we're above low water mark
// This is unlikely to happen, but might happen when closing the
// entire server - another thread is flushing regions. We'll just
// sleep a little bit to avoid spinning, and then pretend that
// we flushed one, so anyone blocked will check again
Thread.sleep(1000);
wakeUpIfBlocking();
}
// Enqueue another one of these tokens so we'll wake up again
wakeupFlushThread();
}
//阻塞超时后也会继续continue
continue;
}
// 如果是正常的flush request
// 单个region memstore大小超过hbase.hregion.memstore.flush.size配置的值,默认128M,执行flush操作
FlushRegionEntry fre = (FlushRegionEntry) fqe;
if (!flushRegion(fre)) {
break;
}
} catch (InterruptedException ex) {
continue;
} catch (ConcurrentModificationException ex) {
continue;
} catch (Exception ex) {
LOG.error("Cache flusher failed for entry " + fqe, ex);
if (!server.checkFileSystem()) {
break;
}
}
}
//结束MemStoreFlusher的线程调用,通常是regionserver stop,这个是在while循环之外的
synchronized (regionsInQueue) {
regionsInQueue.clear();
flushQueue.clear();
}

// Signal anyone waiting, so they see the close flag
wakeUpIfBlocking();
LOG.info(getName() + " exiting");
}

  现在我们看是看梳理一下FlusherHandler的run方法的逻辑

  1. 只要rs不挂,就一直循环判断有没有flushrequest
  2. 通过flushqueue.poll来阻塞,应该flushqueue是阻塞队列,当队列为空时会阻塞,直到超时。
  3. 如果不为空,取出一个request,调用MemStoreFlusher.flushRegion(fre)

Flush流程

 

  可见是调用的MemStoreFlusher.flushRegion方法进行flush的,我们继续跟进flushRegion一探究竟。
private boolean flushRegion(final FlushRegionEntry fqe) {
    //在FlushQueueEntry中取出region信息
    HRegion region = fqe.region;
    //如果region不是metaregion并且含有太多的storefile,则随机blcoking.
    //tooManyStoreFiles默认的阈值时7,同时也要看hbase.hstore.blockingStoreFiles配置的值,没有配置取默认值7
    if (!region.getRegionInfo().isMetaRegion() &&
        isTooManyStoreFiles(region)) {

      //判断是否已经wait了设置的时间
      if (fqe.isMaximumWait(this.blockingWaitTime)) {
        LOG.info("Waited " + (EnvironmentEdgeManager.currentTime() - fqe.createTime) +
          "ms on a compaction to clean up 'too many store files'; waited " +
          "long enough... proceeding with flush of " +
          region.getRegionNameAsString());
      } else {
        // If this is first time we've been put off, then emit a log message.
        //如果当前flush是第一次加入到flush queue
        if (fqe.getRequeueCount() <= 0) {
          // Note: We don't impose blockingStoreFiles constraint on meta regions
          LOG.warn("Region " + region.getRegionNameAsString() + " has too many " +
            "store files; delaying flush up to " + this.blockingWaitTime + "ms");
          //flush前判断该region是否需要split,如果不需要split,同时因为又太多的storefiles,因此调用过一次compact
          if (!this.server.compactSplitThread.requestSplit(region)) {
            try {
              this.server.compactSplitThread.requestSystemCompaction(
                  region, Thread.currentThread().getName());
            } catch (IOException e) {
              LOG.error(
                "Cache flush failed for region " + Bytes.toStringBinary(region.getRegionName()),
                RemoteExceptionHandler.checkIOException(e));
            }
          }
        }

        // Put back on the queue.  Have it come back out of the queue
        // after a delay of this.blockingWaitTime / 100 ms.
        //如果有too manyfile的region已经超过了随机延迟的时间,加入flushqueue队列,唤醒handler开始flush
        this.flushQueue.add(fqe.requeue(this.blockingWaitTime / 100));
        // Tell a lie, it's not flushed but it's ok
        return true;
      }
    }
    //正常情况下的flush
    return flushRegion(region, false, fqe.isForceFlushAllStores());
  }

  该方法中会判断要flush的region是否有过多的hfile,如果是则随机wait一定的时间。wait完成后加入flushqueue唤醒handler开始flush。在正常的情况下最终是调用MemStoreFlusher的重载函数flushRgion(region,flase, isForceFlushAllStores),那我们继续跟进该重载函数。

private boolean flushRegion(final HRegion region, final boolean emergencyFlush,
      boolean forceFlushAllStores) {
    long startTime = 0;
    //枷锁
    synchronized (this.regionsInQueue) {
      //在regioninQueue中移除该region
      FlushRegionEntry fqe = this.regionsInQueue.remove(region);
      // Use the start time of the FlushRegionEntry if available
      if (fqe != null) {
        startTime = fqe.createTime;
      }
      if (fqe != null && emergencyFlush) {
        // Need to remove from region from delay queue.  When NOT an
        // emergencyFlush, then item was removed via a flushQueue.poll.
        flushQueue.remove(fqe);
     }
    }
    if (startTime == 0) {
      // Avoid getting the system time unless we don't have a FlushRegionEntry;
      // shame we can't capture the time also spent in the above synchronized
      // block
      startTime = EnvironmentEdgeManager.currentTime();
    }
    lock.readLock().lock();
    try {
      notifyFlushRequest(region, emergencyFlush);
      //最终是调用region的flushcache
      HRegion.FlushResult flushResult = region.flushcache(forceFlushAllStores);
      boolean shouldCompact = flushResult.isCompactionNeeded();
      // We just want to check the size
      boolean shouldSplit = region.checkSplit() != null;
      if (shouldSplit) {
        this.server.compactSplitThread.requestSplit(region);
      } else if (shouldCompact) {
        server.compactSplitThread.requestSystemCompaction(
            region, Thread.currentThread().getName());
      }
      if (flushResult.isFlushSucceeded()) {
        long endTime = EnvironmentEdgeManager.currentTime();
        server.metricsRegionServer.updateFlushTime(endTime - startTime);
      }
    } catch (DroppedSnapshotException ex) {
      // Cache flush can fail in a few places. If it fails in a critical
      // section, we get a DroppedSnapshotException and a replay of wal
      // is required. Currently the only way to do this is a restart of
      // the server. Abort because hdfs is probably bad (HBASE-644 is a case
      // where hdfs was bad but passed the hdfs check).
      server.abort("Replay of WAL required. Forcing server shutdown", ex);
      return false;
    } catch (IOException ex) {
      LOG.error("Cache flush failed" +
        (region != null ? (" for region " + Bytes.toStringBinary(region.getRegionName())) : ""),
        RemoteExceptionHandler.checkIOException(ex));
      if (!server.checkFileSystem()) {
        return false;
      }
    } finally {
      lock.readLock().unlock();
      wakeUpIfBlocking();
    }
    return true;
  }

  其他无关的代码这里不再细说,之间看标红的位置,核心逻辑在这里,可以看到是调用的region.flushcache(isForceFlushAllStores),因此flush是region级别。同时在flush完成后会判断是否需要进行split,如果不需要split会将判断是否需要compact。继续跟进看下里面做了啥。

//flush cache,参数意义为是否需要flush所有的store
    public FlushResult flushcache(boolean forceFlushAllStores) throws IOException {
        // fail-fast instead of waiting on the lock
        //判断当前region是否处于closing状态,
        if (this.closing.get()) {
            String msg = "Skipping flush on " + this + " because closing";
            LOG.debug(msg);
            return new FlushResult(FlushResult.Result.CANNOT_FLUSH, msg);
        }
        MonitoredTask status = TaskMonitor.get().createStatus("Flushing " + this);
        status.setStatus("Acquiring readlock on region");
        // block waiting for the lock for flushing cache
        //此处加了锁
        lock.readLock().lock();
        try {
            if (this.closed.get()) {
                String msg = "Skipping flush on " + this + " because closed";
                LOG.debug(msg);
                status.abort(msg);
                return new FlushResult(FlushResult.Result.CANNOT_FLUSH, msg);
            }
            if (coprocessorHost != null) {
                status.setStatus("Running coprocessor pre-flush hooks");
                coprocessorHost.preFlush();
            }
            // TODO: this should be managed within memstore with the snapshot, updated only after flush
            // successful
            if (numMutationsWithoutWAL.get() > 0) {
                numMutationsWithoutWAL.set(0);
                dataInMemoryWithoutWAL.set(0);
            }
            synchronized (writestate) {
                //此次flush之前 该region并没有在flush,是否还处于write状态
                if (!writestate.flushing && writestate.writesEnabled) {
                    this.writestate.flushing = true;
                } else {//否则表示该region正处于flushing状态或者不可写,abort flush
                    if (LOG.isDebugEnabled()) {
                        LOG.debug("NOT flushing memstore for region " + this
                                + ", flushing=" + writestate.flushing + ", writesEnabled="
                                + writestate.writesEnabled);
                    }
                    String msg = "Not flushing since "
                            + (writestate.flushing ? "already flushing"
                            : "writes not enabled");
                    status.abort(msg);
                    return new FlushResult(FlushResult.Result.CANNOT_FLUSH, msg);
                }
            }

            try {
                //根据参数forceFlushAllStores判断是否需要所有的store都进行flush,否侧按照flush策略进行选择
                //非全局flush的选择策略:flushSizeLowerBound是参数hbase.hregion.percolumnfamilyflush.size.lower.bound,默认16M或者不满足大小,
//但是该memstore足够老 Collection<Store> specificStoresToFlush = forceFlushAllStores ? stores.values() : flushPolicy.selectStoresToFlush(); //调用internalFlushcache进行flush FlushResult fs = internalFlushcache(specificStoresToFlush, status); if (coprocessorHost != null) { status.setStatus("Running post-flush coprocessor hooks"); coprocessorHost.postFlush(); } status.markComplete("Flush successful"); return fs; } finally { synchronized (writestate) { writestate.flushing = false; this.writestate.flushRequested = false; writestate.notifyAll(); } } } finally { lock.readLock().unlock(); status.cleanup(); } }

  核心逻辑在FlushResult fs = internalFlushcache(specificStoresToFlush, status);里面涉及到了具体的三个阶段,其中prepare的第一阶段是调用了region.internalPrepareFlushCache()实现的,第二阶段flush以及第三阶段commit阶段,是通过internalFlushAndCommit()进行的。我们现在看下具体的internalFlushCache方法的逻辑:

protected FlushResult internalFlushcache(final WAL wal, final long myseqid,
                                             final Collection<Store> storesToFlush, MonitoredTask status) throws IOException {
        //internalPrepareFlushCache执行snapshot,打快照
        PrepareFlushResult result
                = internalPrepareFlushCache(wal, myseqid, storesToFlush, status, false);
        //返回的result中的result是null.因此会执行internalFlushchacheAndCommit方法执行第二和第三阶段。
        if (result.result == null) {
            return internalFlushCacheAndCommit(wal, status, result, storesToFlush);
        } else {
            return result.result; // early exit due to failure from prepare stage
        }
    }

  现在我们看一下第一阶段: internalPrepareFlushCache。里面有一把region级别的updatelock。,这个里面代码比较多,可以先忽略不重要的部分

 //该方法用来执行flush的prepare阶段
    protected PrepareFlushResult internalPrepareFlushCache(
            final WAL wal, final long myseqid, final Collection<Store> storesToFlush,
            MonitoredTask status, boolean isReplay)
            throws IOException {

        if (this.rsServices != null && this.rsServices.isAborted()) {
            // Don't flush when server aborting, it's unsafe
            throw new IOException("Aborting flush because server is aborted...");
        }
        //便于计算flush耗时,记录开始时间
        final long startTime = EnvironmentEdgeManager.currentTime();
        // If nothing to flush, return, but we need to safely update the region sequence id
        //如果当前memstroe为空,不执行flush,但是要更新squenid
        if (this.memstoreSize.get() <= 0) {
            // Take an update lock because am about to change the sequence id and we want the sequence id
            // to be at the border of the empty memstore.
            MultiVersionConsistencyControl.WriteEntry w = null;
            this.updatesLock.writeLock().lock();
            try {
                if (this.memstoreSize.get() <= 0) {
                    // Presume that if there are still no edits in the memstore, then there are no edits for
                    // this region out in the WAL subsystem so no need to do any trickery clearing out
                    // edits in the WAL system. Up the sequence number so the resulting flush id is for
                    // sure just beyond the last appended region edit (useful as a marker when bulk loading,
                    // etc.)
                    // wal can be null replaying edits.
                    if (wal != null) {
                        w = mvcc.beginMemstoreInsert();
                        long flushSeqId = getNextSequenceId(wal);
                        FlushResult flushResult = new FlushResult(
                                FlushResult.Result.CANNOT_FLUSH_MEMSTORE_EMPTY, flushSeqId, "Nothing to flush");
                        w.setWriteNumber(flushSeqId);
                        mvcc.waitForPreviousTransactionsComplete(w);
                        w = null;
                        return new PrepareFlushResult(flushResult, myseqid);
                    } else {
                        return new PrepareFlushResult(
                                new FlushResult(FlushResult.Result.CANNOT_FLUSH_MEMSTORE_EMPTY, "Nothing to flush"),
                                myseqid);
                    }
                }
            } finally {
                this.updatesLock.writeLock().unlock();
                if (w != null) {
                    mvcc.advanceMemstore(w);
                }
            }
        }

        if (LOG.isInfoEnabled()) {
            LOG.info("Started memstore flush for " + this + ", current region memstore size "
                    + StringUtils.byteDesc(this.memstoreSize.get()) + ", and " + storesToFlush.size() + "/"
                    + stores.size() + " column families' memstores are being flushed."
                    + ((wal != null) ? "" : "; wal is null, using passed sequenceid=" + myseqid));
            // only log when we are not flushing all stores.
            //当不是flush所有的store时,打印log
            if (this.stores.size() > storesToFlush.size()) {
                for (Store store : storesToFlush) {
                    LOG.info("Flushing Column Family: " + store.getColumnFamilyName()
                            + " which was occupying "
                            + StringUtils.byteDesc(store.getMemStoreSize()) + " of memstore.");
                }
            }
        }
        // Stop updates while we snapshot the memstore of all of these regions' stores. We only have
        // to do this for a moment.  It is quick. We also set the memstore size to zero here before we
        // allow updates again so its value will represent the size of the updates received
        // during flush
        //停止写入,直到memstore的snapshot完成。
        MultiVersionConsistencyControl.WriteEntry w = null;
        // We have to take an update lock during snapshot, or else a write could end up in both snapshot
        // and memstore (makes it difficult to do atomic rows then)
        status.setStatus("Obtaining lock to block concurrent updates");
        // block waiting for the lock for internal flush
        //获取update的写锁
        this.updatesLock.writeLock().lock();
        status.setStatus("Preparing to flush by snapshotting stores in " +
                getRegionInfo().getEncodedName());
        //用于统计flush的所有的store的memtore内存大小之和
        long totalFlushableSizeOfFlushableStores = 0;
        //记录所有flush的store的cfname
        Set<byte[]> flushedFamilyNames = new HashSet<byte[]>();
        for (Store store : storesToFlush) {
            flushedFamilyNames.add(store.getFamily().getName());
        }
        //storeFlushCtxs,committedFiles,storeFlushableSize,比较重要的是storeFlushCtxs和committedFiles。他们都被定义为以CF做key的TreeMap,
        // 分别代表了store的CF实际执行(StoreFlusherImpl)和最终刷写的HFlile文件。
        //其中storeFlushContext的实现类StoreFlusherImpl里包含了flush相关的核心操作:prepare,flushcache,commit,abort等。
//所以这里保存的是每一个store的flush实例,后面就是通过这里的StoreFlushContext进行flush的 TreeMap<byte[], StoreFlushContext> storeFlushCtxs = new TreeMap<byte[], StoreFlushContext>(Bytes.BYTES_COMPARATOR);
//用来存储每个store和它对应的hdfs commit路径的映射 TreeMap<byte[], List<Path>> committedFiles = new TreeMap<byte[], List<Path>>( Bytes.BYTES_COMPARATOR); // The sequence id of this flush operation which is used to log FlushMarker and pass to // createFlushContext to use as the store file's sequence id. long flushOpSeqId = HConstants.NO_SEQNUM; long flushedSeqId = HConstants.NO_SEQNUM; // The max flushed sequence id after this flush operation. Used as completeSequenceId which is // passed to HMaster. byte[] encodedRegionName = getRegionInfo().getEncodedNameAsBytes(); long trxId = 0; try { try { w = mvcc.beginMemstoreInsert(); if (wal != null) { if (!wal.startCacheFlush(encodedRegionName, flushedFamilyNames)) { // This should never happen. String msg = "Flush will not be started for [" + this.getRegionInfo().getEncodedName() + "] - because the WAL is closing."; status.setStatus(msg); return new PrepareFlushResult(new FlushResult(FlushResult.Result.CANNOT_FLUSH, msg), myseqid); } flushOpSeqId = getNextSequenceId(wal); long oldestUnflushedSeqId = wal.getEarliestMemstoreSeqNum(encodedRegionName); // no oldestUnflushedSeqId means we flushed all stores. // or the unflushed stores are all empty. flushedSeqId = (oldestUnflushedSeqId == HConstants.NO_SEQNUM) ? flushOpSeqId : oldestUnflushedSeqId - 1; } else { // use the provided sequence Id as WAL is not being used for this flush. flushedSeqId = flushOpSeqId = myseqid; } //循环遍历region下面的storeFile,为每个storeFile生成了一个StoreFlusherImpl类, // 生成MemStore的快照就是调用每个StoreFlusherImpl的prepare方法生成每个storeFile的快照, // 至于internalFlushCacheAndCommit中的flush和commti行为也是调用了region中每个storeFile的flushCache和commit接口。 for (Store s : storesToFlush) { //用于统计flush的所有的store的memtore内存大小之和,而不是snapshot的getCellsCount() totalFlushableSizeOfFlushableStores += s.getFlushableSize(); //为每一个store生成自己的storeFlushImpl storeFlushCtxs.put(s.getFamily().getName(), s.createFlushContext(flushOpSeqId)); //此时还没有生成flush的hfile路径 committedFiles.put(s.getFamily().getName(), null); // for writing stores to WAL } // write the snapshot start to WAL if (wal != null && !writestate.readOnly) { FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.START_FLUSH, getRegionInfo(), flushOpSeqId, committedFiles); // no sync. Sync is below where we do not hold the updates lock //这里只是向wal中写入begin flush的marker,真正的sync在后面做,因为这里加了update的写锁,所有耗时操作都不在这里进行 trxId = WALUtil.writeFlushMarker(wal, this.htableDescriptor, getRegionInfo(), desc, sequenceId, false); } // Prepare flush (take a snapshot)这里的StoreFlushContext就是StoreFlusherImpl for (StoreFlushContext flush : storeFlushCtxs.values()) { //迭代region下的每一个store,把memstore下的kvset复制到memstore的snapshot中并清空kvset的值 //把memstore的snapshot复制到HStore的snapshot中 flush.prepare();//其prepare方法就是调用store的storeFlushImpl的snapshot方法生成快照 } } catch (IOException ex) { if (wal != null) { if (trxId > 0) { // check whether we have already written START_FLUSH to WAL try { FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.ABORT_FLUSH, getRegionInfo(), flushOpSeqId, committedFiles); WALUtil.writeFlushMarker(wal, this.htableDescriptor, getRegionInfo(), desc, sequenceId, false); } catch (Throwable t) { LOG.warn("Received unexpected exception trying to write ABORT_FLUSH marker to WAL:" + StringUtils.stringifyException(t)); // ignore this since we will be aborting the RS with DSE. } } // we have called wal.startCacheFlush(), now we have to abort it wal.abortCacheFlush(this.getRegionInfo().getEncodedNameAsBytes()); throw ex; // let upper layers deal with it. } } finally { //做完snapshot释放锁,此时不会阻塞业务的读写操作了 this.updatesLock.writeLock().unlock(); } String s = "Finished memstore snapshotting " + this + ", syncing WAL and waiting on mvcc, flushsize=" + totalFlushableSizeOfFlushableStores; status.setStatus(s); if (LOG.isTraceEnabled()) LOG.trace(s); // sync unflushed WAL changes // see HBASE-8208 for details if (wal != null) { try { wal.sync(); // ensure that flush marker is sync'ed } catch (IOException ioe) { LOG.warn("Unexpected exception while wal.sync(), ignoring. Exception: " + StringUtils.stringifyException(ioe)); } } // wait for all in-progress transactions to commit to WAL before // we can start the flush. This prevents // uncommitted transactions from being written into HFiles. // We have to block before we start the flush, otherwise keys that // were removed via a rollbackMemstore could be written to Hfiles. w.setWriteNumber(flushOpSeqId); mvcc.waitForPreviousTransactionsComplete(w); // set w to null to prevent mvcc.advanceMemstore from being called again inside finally block w = null; } finally { if (w != null) { // in case of failure just mark current w as complete mvcc.advanceMemstore(w); } } return new PrepareFlushResult(storeFlushCtxs, committedFiles, startTime, flushOpSeqId, flushedSeqId, totalFlushableSizeOfFlushableStores);

  在具体看StoreFlushContext.prepare()之前,我们先看一下StoreFlushContext接口的说明,如上所述,StoreFlushImpl是Store的内部类,继承自StoreFlushContext。

interface StoreFlushContext {


  void prepare();


  void flushCache(MonitoredTask status) throws IOException;


  boolean commit(MonitoredTask status) throws IOException;

  
  void replayFlush(List<String> fileNames, boolean dropMemstoreSnapshot) throws IOException;


  void abort() throws IOException;


  List<Path> getCommittedFiles();
}

  现在我们回过头来继续看internalPrepareFlushcache中标红的flush.prepare();

 

public void prepare() {
            //在region调用storeFlusherImpl的prepare的时候,前面提到是在region的update.write.lock中的,因此这里面所有的耗时操作都会影响业务正在进行的读写操作.
            //在snapshot中的逻辑中只是将memstore的跳跃表赋值给snapshot的跳跃表,在返回memstoresnapshot的时候,调用的snapshot的size()方法
            this.snapshot = memstore.snapshot();
            //MemstoreSnapshot的getCellsCount方法即在memstore的shapshot中返回的MemStoresnapshot中传入的snapshot.size()值,时间复杂度是o(n)
            this.cacheFlushCount = snapshot.getCellsCount();
            this.cacheFlushSize = snapshot.getSize();
            committedFiles = new ArrayList<Path>(1);
        }

  我们看下memstore的snapshot方法

public MemStoreSnapshot snapshot() {
    // If snapshot currently has entries, then flusher failed or didn't call
    // cleanup.  Log a warning.
    if (!this.snapshot.isEmpty()) {
      LOG.warn("Snapshot called again without clearing previous. " +
          "Doing nothing. Another ongoing flush or did we fail last attempt?");
    } else {
      this.snapshotId = EnvironmentEdgeManager.currentTime();
      //memstore使用的mem大小
      this.snapshotSize = keySize();
      if (!this.cellSet.isEmpty()) {
        //这里的cellset就是memstore内存中的数据
        this.snapshot = this.cellSet;
//构造一个新的cellset存储数据 this.cellSet = new CellSkipListSet(this.comparator); this.snapshotTimeRangeTracker = this.timeRangeTracker; this.timeRangeTracker = new TimeRangeTracker(); // Reset heap to not include any keys this.size.set(DEEP_OVERHEAD); this.snapshotAllocator = this.allocator; // Reset allocator so we get a fresh buffer for the new memstore if (allocator != null) { String className = conf.get(MSLAB_CLASS_NAME, HeapMemStoreLAB.class.getName()); this.allocator = ReflectionUtils.instantiateWithCustomCtor(className, new Class[] { Configuration.class }, new Object[] { conf }); } timeOfOldestEdit = Long.MAX_VALUE; } }

      prepare中的snapshot.getCellsCount();我们重点说一下,hbase的内存存储写入的数据使用的是跳跃表的数据结构,实现是使用jdk自带的ConcurrentSkipListMap。在hbase的MemStore(默认是DefaultMemStore)实现中有两个环境变量,分别是ConcurrentSkipListMap类型的cellset和snapshot。cellset用来存储写入到memstore的数据,snapshot是在flush的第一阶段是将cellset赋值用的。因此这个的getCellsCount()方法最终调用的是concurrentSkipListMap.size(),concurrentSkipListMap并没有一个原子变量来报错map的大小,因为这里为了并发,同时该操作也不常用。因此concurrentSkipListMap.size()是遍历整个跳跃表获取size大小。

  继续回到internalPrepareFlushCache中,对每一个store调用完prepare后,就将updatelock进行unlock。并返回一个PrepareFlushResult。继续往上走,
回到internalFlushCache方法。执行完internalPrepareFlushcache后走的是internalFlushAndCommit方法。继续跟进:
protected FlushResult internalFlushCacheAndCommit(
            final WAL wal, MonitoredTask status, final PrepareFlushResult prepareResult,
            final Collection<Store> storesToFlush)
            throws IOException {

        // prepare flush context is carried via PrepareFlushResult
        //进行flush的store的cf:storeFlushImpl映射
        TreeMap<byte[], StoreFlushContext> storeFlushCtxs = prepareResult.storeFlushCtxs;
        //flush生成的hfile的路径,当前key是有的,为cf,但是List<Path>为null,是在internalPrepareFlushCache中初始化的
        TreeMap<byte[], List<Path>> committedFiles = prepareResult.committedFiles;
        long startTime = prepareResult.startTime;
        long flushOpSeqId = prepareResult.flushOpSeqId;
        long flushedSeqId = prepareResult.flushedSeqId;
        long totalFlushableSizeOfFlushableStores = prepareResult.totalFlushableSize;

        String s = "Flushing stores of " + this;
        status.setStatus(s);
        if (LOG.isTraceEnabled()) LOG.trace(s);

        // Any failure from here on out will be catastrophic requiring server
        // restart so wal content can be replayed and put back into the memstore.
        // Otherwise, the snapshot content while backed up in the wal, it will not
        // be part of the current running servers state.
        boolean compactionRequested = false;
        try {
            // A.  Flush memstore to all the HStores.
            // Keep running vector of all store files that includes both old and the
            // just-made new flush store file. The new flushed file is still in the
            // tmp directory.
            //迭代region下的每一个store,调用HStore.storeFlushImpl.flushCache方法,把store中snapshot的数据flush到hfile中,当然这里是flush到temp文件中,最终是通过commit将其移到正确的路径下
            //
            //
            for (StoreFlushContext flush : storeFlushCtxs.values()) {
                flush.flushCache(status);
            }

            // Switch snapshot (in memstore) -> new hfile (thus causing
            // all the store scanners to reset/reseek).
            Iterator<Store> it = storesToFlush.iterator();
            // stores.values() and storeFlushCtxs have same order
            for (StoreFlushContext flush : storeFlushCtxs.values()) {
                boolean needsCompaction = flush.commit(status);
                if (needsCompaction) {
                    compactionRequested = true;
                }
                committedFiles.put(it.next().getFamily().getName(), flush.getCommittedFiles());
            }
            storeFlushCtxs.clear();

            // Set down the memstore size by amount of flush.
            this.addAndGetGlobalMemstoreSize(-totalFlushableSizeOfFlushableStores);

            if (wal != null) {
                // write flush marker to WAL. If fail, we should throw DroppedSnapshotException
                FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.COMMIT_FLUSH,
                        getRegionInfo(), flushOpSeqId, committedFiles);
                WALUtil.writeFlushMarker(wal, this.htableDescriptor, getRegionInfo(),
                        desc, sequenceId, true);
            }
        } catch (Throwable t) {
            // An exception here means that the snapshot was not persisted.
            // The wal needs to be replayed so its content is restored to memstore.
            // Currently, only a server restart will do this.
            // We used to only catch IOEs but its possible that we'd get other
            // exceptions -- e.g. HBASE-659 was about an NPE -- so now we catch
            // all and sundry.
            if (wal != null) {
                try {
                    FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.ABORT_FLUSH,
                            getRegionInfo(), flushOpSeqId, committedFiles);
                    WALUtil.writeFlushMarker(wal, this.htableDescriptor, getRegionInfo(),
                            desc, sequenceId, false);
                } catch (Throwable ex) {
                    LOG.warn("Received unexpected exception trying to write ABORT_FLUSH marker to WAL:" +
                            StringUtils.stringifyException(ex));
                    // ignore this since we will be aborting the RS with DSE.
                }
                wal.abortCacheFlush(this.getRegionInfo().getEncodedNameAsBytes());
            }
            DroppedSnapshotException dse = new DroppedSnapshotException("region: " +
                    Bytes.toStringBinary(getRegionName()));
            dse.initCause(t);
            status.abort("Flush failed: " + StringUtils.stringifyException(t));
            throw dse;
        }

        // If we get to here, the HStores have been written.
        if (wal != null) {
            wal.completeCacheFlush(this.getRegionInfo().getEncodedNameAsBytes());
        }

        // Record latest flush time
        for (Store store : storesToFlush) {
            this.lastStoreFlushTimeMap.put(store, startTime);
        }

        // Update the oldest unflushed sequence id for region.
        this.maxFlushedSeqId = flushedSeqId;

        // C. Finally notify anyone waiting on memstore to clear:
        // e.g. checkResources().
        synchronized (this) {
            notifyAll(); // FindBugs NN_NAKED_NOTIFY
        }

        long time = EnvironmentEdgeManager.currentTime() - startTime;
        long memstoresize = this.memstoreSize.get();
        String msg = "Finished memstore flush of ~"
                + StringUtils.byteDesc(totalFlushableSizeOfFlushableStores) + "/"
                + totalFlushableSizeOfFlushableStores + ", currentsize="
                + StringUtils.byteDesc(memstoresize) + "/" + memstoresize
                + " for region " + this + " in " + time + "ms, sequenceid="
                + flushOpSeqId + ", compaction requested=" + compactionRequested
                + ((wal == null) ? "; wal=null" : "");
        LOG.info(msg);
        status.setStatus(msg);

        return new FlushResult(compactionRequested ? FlushResult.Result.FLUSHED_COMPACTION_NEEDED :
                FlushResult.Result.FLUSHED_NO_COMPACTION_NEEDED, flushOpSeqId);
    }

  我们就只看其中两个方法:flush.flushcache和flush.commit。这里的flush即StoreFlushImpl。flushcache方法是用来执行第二阶段,commit用来执行第三阶段。

 public void flushCache(MonitoredTask status) throws IOException {
            //返回的是snapshotflush到临时文件后,最终需要移到的正确路径
            tempFiles = HStore.this.flushCache(cacheFlushSeqNum, snapshot, status);
        }

  转到store的flushcache方法

protected List<Path> flushCache(final long logCacheFlushId, MemStoreSnapshot snapshot,
                                    MonitoredTask status) throws IOException {
        // If an exception happens flushing, we let it out without clearing
        // the memstore snapshot.  The old snapshot will be returned when we say
        // 'snapshot', the next time flush comes around.
        // Retry after catching exception when flushing, otherwise server will abort
        // itself
        StoreFlusher flusher = storeEngine.getStoreFlusher();
        IOException lastException = null;
        for (int i = 0; i < flushRetriesNumber; i++) {
            try {
                //调用StoreFlusher.flushsnapshot方法将snapshotflush到temp文件
                List<Path> pathNames = flusher.flushSnapshot(snapshot, logCacheFlushId, status);
                Path lastPathName = null;
                try {
                    for (Path pathName : pathNames) {
                        lastPathName = pathName;
                        validateStoreFile(pathName);
                    }
                    return pathNames;
                } catch (Exception e) {
                    LOG.warn("Failed validating store file " + lastPathName + ", retrying num=" + i, e);
                    if (e instanceof IOException) {
                        lastException = (IOException) e;
                    } else {
                        lastException = new IOException(e);
                    }
                }
            } catch (IOException e) {
                LOG.warn("Failed flushing store file, retrying num=" + i, e);
                lastException = e;
            }
            if (lastException != null && i < (flushRetriesNumber - 1)) {
                try {
                    Thread.sleep(pauseTime);
                } catch (InterruptedException e) {
                    IOException iie = new InterruptedIOException();
                    iie.initCause(e);
                    throw iie;
                }
            }
        }
        throw lastException;
    }

 其中标红的部分是主要的逻辑。首先通过storeEngine.getStoreFlusher获取flush的实例,实际包括了sync到disk的writer以及append等操作。这里不再展开说明。我们重点看一下for循环中的flusher.flushSnapshot方法,涉及到一个重要的环境变量cellsCount

public List<Path> flushSnapshot(MemStoreSnapshot snapshot, long cacheFlushId,
      MonitoredTask status) throws IOException {
    ArrayList<Path> result = new ArrayList<Path>();
    //这里会调用snapshot的getCellsCount方法,之所以这里提了这个方法,是因为其实一个prepare阶段耗时较大的过程。
    int cellsCount = snapshot.getCellsCount();
    if (cellsCount == 0) return result; // don't flush if there are no entries

    // Use a store scanner to find which rows to flush.
    long smallestReadPoint = store.getSmallestReadPoint();
    InternalScanner scanner = createScanner(snapshot.getScanner(), smallestReadPoint);
    if (scanner == null) {
      return result; // NULL scanner returned from coprocessor hooks means skip normal processing
    }

    StoreFile.Writer writer;
    try {
      // TODO:  We can fail in the below block before we complete adding this flush to
      //        list of store files.  Add cleanup of anything put on filesystem if we fail.
      synchronized (flushLock) {
        status.setStatus("Flushing " + store + ": creating writer");
        // Write the map out to the disk
        //这里传入的cellsCount实际并没有用,可能是预置的变量?
        writer = store.createWriterInTmp(
            cellsCount, store.getFamily().getCompression(), false, true, true);
        writer.setTimeRangeTracker(snapshot.getTimeRangeTracker());
        IOException e = null;
        try {
          //真正的将snapshot写入临时文件
          performFlush(scanner, writer, smallestReadPoint);
        } catch (IOException ioe) {
          e = ioe;
          // throw the exception out
          throw ioe;
        } finally {
          if (e != null) {
            writer.close();
          } else {
            finalizeWriter(writer, cacheFlushId, status);
          }
        }
      }
    } finally {
      scanner.close();
    }
    LOG.info("Flushed, sequenceid=" + cacheFlushId +", memsize="
        + StringUtils.humanReadableInt(snapshot.getSize()) +
        ", hasBloomFilter=" + writer.hasGeneralBloom() +
        ", into tmp file " + writer.getPath());
    result.add(writer.getPath());
    return result;
  }

  可以看到store.createWriterInTmp中使用了该变量,继续跟进

public StoreFile.Writer createWriterInTmp(long maxKeyCount, Compression.Algorithm compression,
                                              boolean isCompaction, boolean includeMVCCReadpoint, boolean includesTag)
            throws IOException {
。。。。。忽略不重要逻辑。。。。。 //这里传入的maxkeyCount没有用 StoreFile.Writer w = new StoreFile.WriterBuilder(conf, writerCacheConf, this.getFileSystem()) .withFilePath(fs.createTempName()) .withComparator(comparator) .withBloomType(family.getBloomFilterType()) .withMaxKeyCount(maxKeyCount) .withFavoredNodes(favoredNodes) .withFileContext(hFileContext) .build(); return w; }

  可见将cellscount以参数的形式传给了writer。然后执行performFlush方法,该方法通过scanner遍历,然后使用hfile.writer将数据罗盘。我们看一下Writer中将cellscount用来干啥了。在整个writer中只有这两个地方用到了

generalBloomFilterWriter = BloomFilterFactory.createGeneralBloomAtWrite(
          conf, cacheConf, bloomType,
          (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
this.deleteFamilyBloomFilterWriter = BloomFilterFactory
            .createDeleteBloomAtWrite(conf, cacheConf,
                (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);  

继续跟进这两个

 public static BloomFilterWriter createDeleteBloomAtWrite(Configuration conf,
      CacheConfig cacheConf, int maxKeys, HFile.Writer writer) {
    if (!isDeleteFamilyBloomEnabled(conf)) {
      LOG.info("Delete Bloom filters are disabled by configuration for "
          + writer.getPath()
          + (conf == null ? " (configuration is null)" : ""));
      return null;
    }

    float err = getErrorRate(conf);

    int maxFold = getMaxFold(conf);
    // In case of compound Bloom filters we ignore the maxKeys hint.
    CompoundBloomFilterWriter bloomWriter = new CompoundBloomFilterWriter(getBloomBlockSize(conf),
        err, Hash.getHashType(conf), maxFold, cacheConf.shouldCacheBloomsOnWrite(),
        KeyValue.RAW_COMPARATOR);
    writer.addInlineBlockWriter(bloomWriter);
    return bloomWriter;
  }

  可见maxKeys没有使用,另一个方法同理,所以这里的cellscount变量在flush的第二阶段没有使用。

  到现在为止我们判断出在第二阶段cellcount没有使用,我们继续跟进第三阶段:回到internalFlushAndCOmmit中的flush.commit(status)
public boolean commit(MonitoredTask status) throws IOException {
            if (this.tempFiles == null || this.tempFiles.isEmpty()) {
                return false;
            }
            List<StoreFile> storeFiles = new ArrayList<StoreFile>(this.tempFiles.size());
            for (Path storeFilePath : tempFiles) {
                try {
                    storeFiles.add(HStore.this.commitFile(storeFilePath, cacheFlushSeqNum, status));
                } catch (IOException ex) {
                    LOG.error("Failed to commit store file " + storeFilePath, ex);
                    // Try to delete the files we have committed before.
                    for (StoreFile sf : storeFiles) {
                        Path pathToDelete = sf.getPath();
                        try {
                            sf.deleteReader();
                        } catch (IOException deleteEx) {
                            LOG.fatal("Failed to delete store file we committed, halting " + pathToDelete, ex);
                            Runtime.getRuntime().halt(1);
                        }
                    }
                    throw new IOException("Failed to commit the flush", ex);
                }
            }

            for (StoreFile sf : storeFiles) {
                if (HStore.this.getCoprocessorHost() != null) {
                    HStore.this.getCoprocessorHost().postFlush(HStore.this, sf);
                }
                committedFiles.add(sf.getPath());
            }

            HStore.this.flushedCellsCount += cacheFlushCount;
            HStore.this.flushedCellsSize += cacheFlushSize;

            // Add new file to store files.  Clear snapshot too while we have the Store write lock.
            return HStore.this.updateStorefiles(storeFiles, snapshot.getId());
        }

  第三阶段比较简单,将flush的文件移动到hdfs正确的路径下。同时可见在这里用到了cellscount。这里是赋值给store的flushedCellsCount,这里主要是用来进行metric收集flushedCellsSize的。根据经验这个metric可忽略,未使用过。

总结

这里之所以总是提到cellscount变量,是因为给其赋值调用ConcurrentSkipListMap.size()方法在flush的第一阶段中最耗时的,同时持有hbase region 级别的updatelock,但是通过梳理并没有太大的用处,可以干掉。否则会因此一些毛刺,pct99比较高。已有patch,但是是应用在2.+的版本的、

整个flush的流程就结束了,如有不对的地方,欢迎指正。欢迎加微信相互交流:940184856

 

posted @ 2019-10-14 11:28  Evil_XJZ  阅读(...)  评论(... 编辑 收藏