HBase compaction有什么用
这篇文章主要为大家展示了"HBase compaction有什么用"，内容简而易懂，条理清晰，希望能够帮助大家解决疑惑，下面让小编带领大家一起研究并学习一下"HBase compaction有什么用"这篇文章吧。
compaction是将多个HFile合并为一个HFile操作，
进行compaction有如下几个作用：
减少HFile文件的个数，
HFile减少可能提高读取性能，
清除过期和删除数据。
compaction有二种类型：minor 和 major
1）Minor操作只用来做部分文件的合并操作以及包括minVersion=0并且设置ttl的过期版本清理，不做任何删除数据、多版本数据的清理工作。
2）Major操作是对Region下的HStore下的所有StoreFile执行合并操作，会做删除数据，多版本数据清理工作，最终的结果是整理合并出一个文件。
HRegionServer在启动的时候，会启动compactionChecker线程，compactionChecker会检测region是否需要compaction。
主要执行的逻辑如下：
protected void chore() { for (HRegion r : this.instance.onlineRegions.values()) { if (r == null) continue; for (Store s : r.getStores().values()) { try { long multiplier = s.getCompactionCheckMultiplier(); assert multiplier > 0; if (iteration % multiplier != 0) continue; if (s.needsCompaction()) { // Queue a compaction. Will recognize if major is needed. this.instance.compactSplitThread.requestSystemCompaction(r, s, getName() + " requests compaction"); } else if (s.isMajorCompaction()) { if (majorCompactPriority == DEFAULT_PRIORITY || majorCompactPriority > r.getCompactPriority()) { this.instance.compactSplitThread.requestCompaction(r, s, getName() + " requests major compaction; use default priority", null); } else { this.instance.compactSplitThread.requestCompaction(r, s, getName() + " requests major compaction; use configured priority", this.majorCompactPriority, null); } } } catch (IOException e) { LOG.warn("Failed major compaction check on " + r, e); } } } iteration = (iteration == Long.MAX_VALUE) ? 0 : (iteration + 1); } }如代码，遍历onlineRegions，获取每个region的Store进行判断，其中needsCompaction逻辑如下：
public boolean needsCompaction(final CollectionstoreFiles, final List filesCompacting) { int numCandidates = storeFiles.size() - filesCompacting.size(); return numCandidates >= comConf.getMinFilesToCompact(); }
minFilesToCompact由hbase.hstore.compaction.min(老版本是：hbase.hstore.compactionThreshold)控制，默认值为3，即store下面的storeFiles数量 减去 正在compaction的数量 >=3是，需要做compaction。
needsCompaction为true后，会调用compactSplitThread.requestSystemCompaction方法，发送compaction请求，具体内容会在compactSplitThread线程里分析。
needsCompaction为false后，会判断是否为isMajorCompaction，具体逻辑如下：
/* * @param filesToCompact Files to compact. Can be null. * @return True if we should run a major compaction. */ public boolean isMajorCompaction(final CollectionfilesToCompact) throws IOException { boolean result = false; long mcTime = getNextMajorCompactTime(filesToCompact); if (filesToCompact == null || filesToCompact.isEmpty() || mcTime == 0) { return result; } // TODO: Use better method for determining stamp of last major (HBASE-2990) long lowTimestamp = StoreUtils.getLowestTimestamp(filesToCompact); long now = System.currentTimeMillis(); if (lowTimestamp > 0l && lowTimestamp < (now - mcTime)) { // Major compaction time has elapsed. long cfTtl = this.storeConfigInfo.getStoreFileTtl(); if (filesToCompact.size() == 1) { // Single file StoreFile sf = filesToCompact.iterator().next(); Long minTimestamp = sf.getMinimumTimestamp(); long oldest = (minTimestamp == null) ? Long.MIN_VALUE : now - minTimestamp.longValue(); if (sf.isMajorCompaction() && (cfTtl == HConstants.FOREVER || oldest < cfTtl)) { if (LOG.isDebugEnabled()) { LOG.debug("Skipping major compaction of " + this + " because one (major) compacted file only and oldestTime " + oldest + "ms is < ttl=" + cfTtl); } } else if (cfTtl != HConstants.FOREVER && oldest > cfTtl) { LOG.debug("Major compaction triggered on store " + this + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; } } else { if (LOG.isDebugEnabled()) { LOG.debug("Major compaction triggered on store " + this + "; time since last major compaction " + (now - lowTimestamp) + "ms"); } result = true; } } return result; }
以上是"HBase compaction有什么用"这篇文章的所有内容，感谢各位的阅读！相信大家都有了一定的了解，希望分享的内容对大家有所帮助，如果还想学习更多知识，欢迎关注行业资讯频道！
