Functional index iterator is not stable

[Totktonada]

So we can crash tarantool and observe other unwanted effects. Consider tarantool/tarantool#6786 for details.

See tarantool/doc#2102 and further links for some kind of definition of the 'stable iterator' term. Say, you open an iterator by a space index, moving on it and changing something in the space. The iterator is 'stable' when it is not invalidated by the changes in the space. It always has a predictable behavior and don't lead to crash, data corruption or other undesirable effects.

The most common use case for a stable iterator is to delete tuples while iterating over a space:

for _, tuple in box.space.s.index.foo:pairs() do
    box.space.s:delete({tuple[1]})
end

However it is not the only use case.

The problem can be reproduced on a debug tarantool build. It reproduces for me on expirationd 1.1.1-44-g838c2d1 on the test_memtx_tree_functional_index test case.

$ cd /home/alex/projects/tarantool-meta/r/t-6
$ cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_BACKTRACE=ON -DENABLE_DIST=ON -DENABLE_FEEDBACK_DAEMON=OFF -DENABLE_BUNDLED_LIBCURL=OFF && make -j
$ cd /home/alex/projects/tarantool-meta/expirationd
$ export PATH="/home/alex/projects/tarantool-meta/r/t-6/src:${PATH}"
$ luatest -c -v -p test_memtx_tree_functional_index
<...>
    custom_index.engine:"memtx".index_type:"TREE".test_memtx_tree_functional_index ... 
2022-01-15 01:59:22.159 [7804] main/110/guardian of "clean_all" I> expiration: task "clean_all" restarted
tarantool: /home/alex/p/tarantool-meta/r/t-6/src/box/tuple_compare.cc:1394: int func_index_compare(tuple*, hint_t, tuple*, hint_t, key_def*) [with bool is_nullable = false; hint_t = long unsigned int]: Assertion `mp_typeof(*key_b) == MP_ARRAY' failed.
Aborted (core dumped)

According to what I said here, a year ago we didn't lean on the tree iterator stability, so the test would likely pass. So I guess we can consider this bug as a regression. I didn't check, though.

It seems, we should switch to the old 'get batch, process, get next batch using GT iterator' iteration way for functional indexes.

It was not catched by tests before, because we have no test runs on a debug tarantool build in CI. It seems, it is must have for expirationd, where we lean on non-trivial index guarantees. I filed #102 to add testing on the debug build into CI.

[Totktonada]

Functional indexes are supported since:

$ (needle='box: introduce functional indexes in memxtx'; for branch in 1.7 1.9 1.10 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 master; do printf "${branch}\t"; commit_description=$(git log --oneline "origin/${branch}" | grep -F "${needle}"); [ -n "${commit_description}" ] && echo "$(git describe --long --always $(echo "${commit_description}" | cut -d" " -f1)) $(echo "${commit_description}" | cut -d" " -f2-)" || echo "(not found)"; done)

1.7	(not found)
1.9	(not found)
1.10	(not found)
2.1	(not found)
2.2	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.3	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.4	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.5	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.6	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.7	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
2.8	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx
master	2.2.0-618-g4177fe175 box: introduce functional indexes in memxtx

The crash was fixed in:

$ (needle='memtx: fix use after free in functional index iterator'; for branch in 1.7 1.9 1.10 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 master; do printf "${branch}\t"; commit_description=$(git log --oneline "origin/${branch}" | grep -F "${needle}"); [ -n "${commit_description}" ] && echo "$(git describe --long --always $(echo "${commit_description}" | cut -d" " -f1)) $(echo "${commit_description}" | cut -d" " -f2-)" || echo "(not found)"; done)

1.7	(not found)
1.9	(not found)
1.10	(not found)
2.1	(not found)
2.2	(not found)
2.3	(not found)
2.4	(not found)
2.5	(not found)
2.6	(not found)
2.7	(not found)
2.8	2.8.3-30-g31096a1b7 memtx: fix use after free in functional index iterator
master	2.10.0-beta2-82-g51d0a9b22 memtx: fix use after free in functional index iterator

How we can handle it in expirationd:

1. Switch to the 'get batch, process, get next batch using GT iterator' iteration implementation for functional indexes.
2. The same, but only for old tarantool versions.
3. Disable support of functional indexes for old tarantool versions (but allow to forcefully enable it using an option).

[oleg-jukovec]

The problem was from July 2019 to February 2022. That's a lot of time. The fix is fresh, there are many versions without it.

So, at the moment, as a client package, we need to support all versions with functional indexes.
Also, I think we need a simple canonical implementation and a deprecated workaround for old Tarantools. We will remove the workaround in the future.

So I prefer:
2. The same, but only for old tarantool versions.

[Totktonada]

@oleg-jukovec I'm ok with this approach.

[oleg-jukovec]

@Totktonada

Unfortunately, this approach probably won't work. The reason in this pull request: #50

The problem is that at the moment it is impossible to iterate through a non-unique index using index:select without hacks:
tarantool/tarantool#5282

So I suggest to move to the next approach:

Disable support of functional indexes for old tarantool versions (but allow to forcefully enable it using an option).

Please, see my draft pull request: #108