MDEV-37070 per-index adaptive hash index parameters#4507
Conversation
|
|
|
|
|
Yes, the combinations I'm tuning the test. diff --git a/mysql-test/suite/innodb/t/index_ahi_option.test b/mysql-test/suite/innodb/t/index_ahi_option.test
index 5b86c2a250d..7c92eab581e 100644
--- a/mysql-test/suite/innodb/t/index_ahi_option.test
+++ b/mysql-test/suite/innodb/t/index_ahi_option.test
@@ -46,8 +46,8 @@ INSERT INTO t1 VALUES (50001, 50, 1, 1), (50002, 50, 2, 2),
--echo #
SET GLOBAL innodb_monitor_enable = module_adaptive_hash;
-let $warm_up_rounds= 200;
-let $query_rounds= 10;
+let $warm_up_rounds= 0;
+let $query_rounds= 140;
--echo # Warming up AHI
let $i = $warm_up_rounds;
@@ -194,7 +194,7 @@ let $val0 = $val1;
ALTER TABLE t1 adaptive_hash_index='ON';
--echo # Warming up AHI
-let $i = 200;
+let $i = $warm_up_rounds;
while ($i)
{
--disable_query_logI'll run again the tests with |
dr-m
left a comment
dr-m
left a comment
There was a problem hiding this comment.
It’s a great idea to combine several fields into a single 32-bit atomic field. But, some "pretty" code can actually result in some quite ugly machine code (with conditional branches).
storage/innobase/include/dict0mem.h
Outdated
| - bit 23: fields valid | ||
| - bit 24: bytes valid | ||
| - bit 25: left valid | ||
| - bits [26, 31]: spare (6 bits) |
There was a problem hiding this comment.
It could be better to allocate something useful (such as part of the enabled/disabled preference) at the most significant bit. For example, on x86, the test instruction would allow the most significant bit to be copied to a flag.
There was a problem hiding this comment.
I kept this as is in the last version, since I did not want to change too much the new inline bool is_enabled(const dict_index_t *index) const noexcept logic. Is this acceptable?
iMineLink
force-pushed
the
MDEV-37070
branch
3 times, most recently
from
e445a83 to
769793f
Compare
Thirunarayanan
force-pushed
the
MDEV-37070
branch
from
769793f to
fbc8869
Compare
iMineLink
force-pushed
the
MDEV-37070
branch
from
fbc8869 to
972c1fd
Compare
|
All the commits have been squashed into one with description updated accordingly. @Thirunarayanan I believe review can start anyway since there's a lot to cover, I'll address the TODO and ping you when it's done if you agree. Thanks. |
iMineLink
force-pushed
the
MDEV-37070
branch
2 times, most recently
from
fbdc7b9 to
67a57fc
Compare
|
@Thirunarayanan the TODO has been addressed by adding a comment in |
Thirunarayanan
left a comment
Thirunarayanan
left a comment
There was a problem hiding this comment.
Few test case scenarios are missing
- ALTER TABLE AHI=0 with concurrent AHI Access
- Buffer pool resize with ahi setting change
- Crash recovery with ahi just to see how it retains for the table metadata
- AHI in partition table
- Verify altering ahi is instant DDL
- Boundary check for complete_fields, bytes_from_incomplete_fields
|
|
||
| innobase_copy_frm_flags_from_table_share( | ||
| ctx->new_table, altered_table->s); | ||
| innobase_copy_frm_flags_from_table( |
There was a problem hiding this comment.
IIUC, alter table ahi=0 is no rebuild operation. When do we drop ahi entries of the indexes if we disable ahi for the table? Do we do it lazily somewhere? Since it is commit phase, there will be no other thread will read from ahi at this point
There was a problem hiding this comment.
Current behavior will keep the entries in the buffer pool until the pages are evicted.
In btr_search_update_hash_on_insert(), the stale entries will still be updated, unless we add check that btr_search.is_enabled(index) in the early return path:
if (!index)
return;
Currently, the tests run ALTER TABLE with ALGORITHM=INSTANT, LOCK=NONE, for example:
ALTER TABLE t1 adaptive_hash_index=OFF, ALGORITHM=INSTANT, LOCK=NONE;
If we drop the AHI entries on ALTER TABLE, it may delay the DDL execution, but will clean earlier the buffer pool and AHI partition.
I suppose the change in btr_search_update_hash_on_insert() may be implemented anyway.
What is the preferred way to proceed here?
There was a problem hiding this comment.
4f12d3c added early return in btr_search_update_hash_on_insert() if the AHI index is disabled for the index, preventing maintenance for indices that won't use the AHI.
| @param innodb_table InnoDB table definition | ||
| @param option_struct MariaDB table options | ||
| @param table MariaDB table handle */ | ||
| static void innodb_ahi_enable(dict_table_t *innodb_table, |
There was a problem hiding this comment.
Are we missing the boundary check like
CREATE TABLE t1 (
id INT PRIMARY KEY,
col1 VARCHAR(100),
INDEX idx3 (col1)
complete_fields=0
bytes_from_incomplete_field=0
) ENGINE=InnoDB;
Why complete_fields and bytes_from_incomplete_field is 0?
CREATE TABLE t2 (
id INT PRIMARY KEY,
col1 INT,
INDEX idx2 (col1)
complete_fields=5
) ENGINE=InnoDB;
Even though index has 1 field. We should also check whether bytes_from_incomplete_field is used when complete_field + 1 exist. Does it make sense to use bytes_from_incomplete_field for integer? (should we restrict bytes_from_incomplete_field to text, varchar, blob, varbinary?)
There was a problem hiding this comment.
There is a test in index_ahi_option_debug.test that the complete_fields=0 bytes_from_incomplete_field=0 results in no AHI usage:
ALTER TABLE t1_ahi_yes ADD INDEX idx_6 (col2, col1) complete_fields=0 bytes_from_incomplete_field=0 for_equal_hash_point_to_last_record=0;
ALTER TABLE t1_ahi_yes ADD INDEX idx_7 (col3, col2) complete_fields=0 bytes_from_incomplete_field=0 for_equal_hash_point_to_last_record=1;
...
CALL run_and_check_idx(240, 6, 't1_ahi_yes');
result_msg
# No AHI used in SELECT (idx_6)
CALL run_and_check_idx(240, 7, 't1_ahi_yes');
result_msg
# No AHI used in SELECT (idx_7)
However, you are right that we can disable the AHI earlier to avoid useless runtime overhead. It's a useless combination.
In the same test, combinations the exceed the valid boundaries are tested as well:
ALTER TABLE t1_ahi_yes ADD INDEX idx_4 (col2) complete_fields=2;
ALTER TABLE t1_ahi_yes ADD INDEX idx_5 (col3) bytes_from_incomplete_field=5;
...
CALL run_and_check_idx(240, 4, 't1_ahi_yes');
result_msg
# No AHI used in SELECT (idx_4)
CALL run_and_check_idx(240, 5, 't1_ahi_yes');
result_msg
# No AHI used in SELECT (idx_5)
...
It seems those result in no AHI usage as well.
innodb_ahi_enable() is void, it may be complex to propagate a failure to the caller.
What could be done, is adjusting the values internally in innodb_ahi_enable(), to not provide "extra" fields/bytes nor useless combination complete_fields=0 bytes_from_incomplete_field=0 to btr_search_info_update_hash(), I'll check it.
I think that using bytes_from_incomplete_field for integer may be interesting if one only wants to hash the high bytes (big endian storage), which is a strange use case, but legit.
There was a problem hiding this comment.
4f12d3c adds self-correction of the options in innodb_ahi_enable(). The only option which is not corrected is bytes_from_incomplete_field when referring to a valid field exceeding its width, since rec_fold() is already clamping the n_bytes value to the maximum allowed.
|
Please note that there are some mroonga related failures introduced by this PR that need to be addressed before merging. |
Added ADAPTIVE_HASH_INDEX=DEFAULT|YES|NO table and index option to InnoDB.
The table and index options only have an effect if InnoDB adaptive hash
index feature is enabled.
- Having the ADAPTIVE_HASH_INDEX TABLE option set to NO will disable
adaptive hash index for all indexes in the table that does not have
the index option adaptive_hash_index=yes.
- Having the ADAPTIVE_HASH_INDEX TABLE option set to YES will enable the
adaptive hash index for all indexes in the table that does not have
the index option adaptive_hash_index=no.
- Using adaptive_hash_index=default deletes the old setting.
- One can also use OFF/ON as the options. This is to make it work similar
as other existing options.
- innodb.adaptive_hash_index has been changed from a bool to an enum with
values OFF, ON and IF_SPECIFIED. If IF_SPECIFIED is used, adaptive
hash index are only used for tables and indexes that specifies
adaptive_hash_index=on.
- The following new options can be used to further optimize adaptive hash
index for an index (default is unset/auto for all of them):
- complete_fields:
- 0 to the number of columns the key is defined on (max 64)
- bytes_from_incomplete_field:
- This is only usable for memcmp() comparable index fields, such as
VARBINARY or INT. For example, a 3-byte prefix on an INT will
return an identical hash value for 0‥255, another one for 256‥511,
and so on.
- Range is min 0 max 16383.
- for_equal_hash_point_to_last_record
- Default is unset/auto, NO points to the first record, known as
left_side in the code; YES points to the last record.
Example: we have an INT column with the values 1,4,10 and bytes=3,
will that hash value point to the record 1 or the record 10?
Note: all values will necessarily have the same hash value
computed on the big endian byte prefix 0x800000, for all of the
values 0x80000001, 0x80000004, 0x8000000a. InnoDB inverts the
sign bit in order to have memcmp() compatible comparison
Example:
CREATE TABLE t1 (a int primary key, b varchar(100), c int,
index (b) adaptive_hash_index=no, index (c))
engine=innodb, adaptive_hash_index=yes;
Notable changes in InnoDB
- btr_search.enabled was changed from a bool to a ulong to be
able to handle options OFF, ON as IF_ENABLED. ulong is needed
to compile with MariaDB enum variables.
- To be able to find all instances where btr_search.enabled was used
I changed all code to use btr_search.get_enabled() when accessing
the value and used btr_search.is_enabled(index) to test if AHI is
enabled for the index.
- btr_search.enable() was changed to always take two parameters,
resize and value of enabled. This was needed as enabled can now
have values 0, 1, and 2.
- store all AHI related options in per-index `dict_index_t::ahi`
bit-packed 32-bit atomic field `ahi_enabled_fixed_mask`
- static assertions and debug assertions ensure that all options fit
into the 32-bit field
- packing details:
- `enabled`, `adaptive_hash_index` (first 2 bits)
- `fields`, `complete_fields` (7 bit)
- `bytes`, `bytes_from_incomplete_field` (14 bits)
- `left`, `~for_equal_hash_point_to_last_record` (1 bit)
- `is_fields_set`, `fields` set flag (1 bit)
- `is_bytes_set`, `bytes` set flag (1 bit)
- `is_left_set`, `left` set flag (last 1 bit)
- 5 bits spare after `is_left_set`
- manipulation of the bit-packed field avoids usage of branches or
conditional instructions to minimize the performance impact of
the new options
- in `btr_search_update_hash_ref` apply the per-index AHI options
using bit-masking to override internal heuristic values with user
preferences
- add `innodb.index_ahi_option` test:
- test a combination of per-table and per-index AHI options
- use a stored procedure which checks if AHI is used during a burst of
index lookups checking delta in `adaptive_hash_searches` InnoDB
monitor variable
- test that the maximum number of fields per (secondary) index is 64
(32+32)
- add `innodb.index_ahi_option_debug` test:
- test debug builds with `index_ahi_option_debug_check` debug variable
enabled to verify that the proper per-index AHI options are applied
during index lookups
- test that illegal per-index AHI are non-destructive and just lead to
no AHI usage
Visible user changes:
- select @@global.adaptive_hash_index will now return a string instead
of 0 or 1.
Other notable changes:
- In `sql/create_options.cc`:
- In function `parse_engine_part_options` do allocate table options
in share root to avoid MSAN/ASAN errors due to use after free of
`option_struct` in test `parts.partition_special_innodb`.
- In function `set_one_value` avoid reading after the end of the
current string.
Co-authored-by: Monty <monty@mariadb.org>
Co-authored-by: Marko Mäkelä <marko.makela@mariadb.com>
Co-authored-by: Alessandro Vetere <iminelink@gmail.com>
Co-authored-by: Thirunarayanan Balathandayuthapani <thiru@mariadb.com>
5 participants
Description
This is based on work by @montywi. Some InnoDB changes will be needed, as noted in MDEV-37070.
Release Notes
TBD
How can this PR be tested?
Basing the PR against the correct MariaDB version
mainbranch.PR quality check