MySQL 5.5: InnoDB Change Buffering

To speed up bulk loading of data, InnoDB implements an insert buffer, a special index in the InnoDB system tablespace that buffers modifications to secondary indexes when the leaf pages are not in the buffer pool. Batched merges from the insert buffer to the index pages result in less random access patterns than when updating the pages directly. This speeds up the operation on hard disks.

In MySQL 5.5, the insert buffer has been extended to a change buffer, which covers all modifications of secondary index leaf pages. This will improve the performance of bulk deletes and updates, transaction rollback and the purging of deleted records (reducing the “purge lag”).

To assess the benefits of the extended buffering, you may want to run benchmarks with the settingsinnodb_change_buffering=all, innodb_change_buffering=inserts, and innodb_change_buffering=none. Users of solid-state storage, where random reads are about as fast as sequential reads, might benefit from disabling the buffering altogether.

Read on to learn how the change buffering works.

Operations on Secondary Indexes

 

InnoDB can perform three kinds of modifications on secondary index records. If the affected index page is not in the buffer pool, the modifications can be buffered in the change buffer. When an index lookup or scan needs a page that is not in the buffer pool, the page will be read from the tablespace and any buffered changes will be merged to it.

The following operations can modify secondary index pages:

Insert

Inserting a record; supported in all versions of InnoDB

Delete-mark

Marking a record for deletion

Purge

Removing a deleted record that is no longer accessible by active transactions

Before MySQL 5.5, UPDATE, DELETE and purge operations were performed directly on the index pages, resulting in random-access I/O. In MySQL 5.5, all of these operations can be buffered.

Implications of InnoDB Multiversioning

In InnoDB, there are two types of indexes: the clustered index B-tree, where the records are stored in the PRIMARY KEYorder, and secondary index B-trees, which identify rows by primary key. InnoDB multiversion concurrency control (MVCC) treats these indexes differently.

Records in the clustered index can be updated in place, and their hidden system columns DB_TRX_ID, DB_ROLL_PTRpoint to undo log entries from which earlier versions can be reconstructed. InnoDB secondary index records do not contain any system columns, and their data is never updated in place. An UPDATE of an indexed column requires the operations Delete-mark(old), Insert(new) and eventually Purge(old) in the secondary index. An UPDATE of a PRIMARY KEY results in Delete-mark, Insert and eventually Purge in all indexes.

When a secondary index record has been marked for deletion or when the page has been updated by a newer transaction, InnoDB will look up the clustered index record. In the clustered index, it suffices to check the DB_TRX_IDand only retrieve the correct version from the undo log when the record was modified after the reading transaction started.

To Buffer or not to Buffer

When a page is in the buffer pool, it will always be updated directly. When a page is loaded to the buffer pool, any buffered changes will be merged to it, so that users never see unmerged changes.

Because change buffering works on individual leaf pages, we cannot buffer changes that would result into page splits or merges, but must perform such changes on the B-tree pages directly.

The insert buffer bitmap keeps track on the available space on pages and prevents overflows when buffering inserts. Delete-marking records can always be buffered, because the flag will be updated in place. Purging a delete-marked record could result in an empty page, something that we do not allow. We determine the non-emptiness of a page from previously buffered operations on the same page. If there are no previously buffered operations, the purge will have to load the index page to the buffer pool.

InnoDB refuses to buffer an operation when the on-disk change buffer tree would grow bigger than ⅓ of the in-memory buffer pool (innodb_buffer_pool_size). This might be a good rule-of-thumb, but some setups could benefit from the ability of setting the change buffer size independently of the buffer pool size.

Conclusion

The InnoDB change buffer is a persistent data structure and a complex mechanism that comes into play when the workload does not fit in the buffer pool. Because it trades random I/O with a larger amount of sequential I/O, it speeds up operation on hard disks, where random access is much slower than sequential access.

On solid-state storage, there is not much difference between sequential and random access times. Change buffering may still be useful if writes to solid-state storage are expensive, either in terms of speed or the consumption of limited program/erase cycles. Change buffering could reduce the write load on user tablespaces and cause more writes to the system tablespace (which contains the insert buffer) and the redo log. These should be placed on a hard disk.

 

change buffering

The general term for the features involving the change buffer, consisting of insert buffering, delete buffering, and purge buffering. Index changes resulting from SQL statements, which could normally involve random I/O operations, are held back and performed periodically by a background thread. This sequence of operations can write the disk blocks for a series of index values more efficiently than if each value were written to disk immediately. Controlled by the innodb_change_bufferingand innodb_change_buffer_max_size configuration options.

 

insert buffering

The technique of storing secondary index changes due to INSERT operations in the insert buffer rather than writing them immediately, so that the physical writes can be performed to minimize random I/O. It is one of the types of change buffering; the others are delete buffering and purge buffering.

Insert buffering is not used if the secondary index is unique, because the uniqueness of new values cannot be verified before the new entries are written out. Other kinds of change buffering do work for unique indexes.

 

unique index

An index on a column or set of columns that have a unique constraint. Because the index is known not to contain any duplicate values, certain kinds of lookups and count operations are more efficient than in the normal kind of index. Most of the lookups against this type of index are simply to determine if a certain value exists or not. The number of values in the index is the same as the number of rows in the table, or at least the number of rows with non-null values for the associated columns.

The insert buffering optimization does not apply to unique indexes. As a workaround, you can temporarily set unique_checks=0 while doing a bulk data load into an InnoDB table.

 

delete buffering

The technique of storing index changes due to DELETE operations in the insert buffer rather than writing them immediately, so that the physical writes can be performed to minimize random I/O. (Because delete operations are a two-step process, this operation buffers the write that normally marks an index record for deletion.) It is one of the types of change buffering; the others are insert buffering and purge buffering.

 

purge buffering

The technique of storing index changes due to DELETE operations in the insert buffer rather than writing them immediately, so that the physical writes can be performed to minimize random I/O. (Because delete operations are a two-step process, this operation buffers the write that normally purges an index record that was previously marked for deletion.) It is one of the types of change buffering; the others are insert buffering. and delete buffering

 

When INSERT, UPDATE, and DELETE operations are done to a table, often the values of indexed columns (particularly the values of secondary keys) are not in sorted order, requiring substantial I/O to bring secondary indexes up to date. InnoDB has an insert buffer that caches changes to secondary index entries when the relevantpage is not in the buffer pool, thus avoiding I/O operations by not reading in the page from the disk. The buffered changes are merged when the page is loaded to the buffer pool, and the updated page is later flushed to disk using the normal mechanism. The InnoDB main thread merges buffered changes when the server is nearly idle, and during a slow shutdown.

Because it can result in fewer disk reads and writes, this feature is most valuable for workloads that are I/O-bound, for example applications with a high volume of DML operations such as bulk inserts.

However, the insert buffer occupies a part of the buffer pool, reducing the memory available to cache data pages. If the working set almost fits in the buffer pool, or if your tables have relatively few secondary indexes, it may be useful to disable insert buffering. If the working set entirely fits in the buffer pool, insert buffering does not impose any extra overhead, because it only applies to pages that are not in the buffer pool.

 

 innodb_change_buffering

Command-Line Format	--innodb_change_buffering=#
Option-File Format	innodb_change_buffering
System Variable Name	innodb_change_buffering
Variable Scope	Global
Dynamic Variable	Yes
	Permitted Values (<= 5.5.3)
	Type	enumeration
	Default	inserts
	Valid Values	inserts
		none
	Permitted Values (>= 5.5.4)
	Type	enumeration
	Default	all
	Valid Values	inserts
		deletes
		purges
		changes
		all
		none

Whether InnoDB performs change buffering, an optimization that delays write operations to secondary indexes so that the I/O operations can be performed sequentially. The permitted values are inserts (buffer insert operations), deletes (buffer delete operations; strictly speaking, the writes that mark index records for later deletion during a purge operation), changes (buffer insert and delete-marking operations), purges (buffer purge operations, the writes when deleted index entries are finally garbage-collected), all (buffer insert, delete-marking, and purge operations) and none (do not buffer any operations). The default is all. 

innodb_change_buffer_max_size

Introduced	5.6.2
Command-Line Format	--innodb_change_buffer_max_size=#
Option-File Format	innodb_change_buffer_max_size
System Variable Name	innodb_change_buffer_max_size
Variable Scope	Global
Dynamic Variable	Yes
	Permitted Values
	Type	numeric
	Default	25
	Range	0 .. 50

Maximum size for the InnoDB change buffer, as a percentage of the total size of the buffer pool. You might increase this value for a MySQL server with heavy insert, update, and delete activity, or decrease it for a MySQL server with unchanging data used for reporting. 

 

 

参考：

https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_5_innodb_change

http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_change_buffering

http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_insert_buffer

http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_unique_index

http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_delete_buffering

http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_purge_buffering

http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_change_buffer_max_size

http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_change_buffering

http://dev.mysql.com/doc/innodb/1.1/en/innodb-performance-change_buffering.html#