[Upgrade] ER_ACCESS_DENIED: Session access to universe '' is denied for user 'replicator'

[ligurio]

Tarantool version:
1.6.9-94-ga5da60b (installed using official RPM packages)

OS version:
CentOS Linux release 7.7.1908 (Core)

Bug description:
auth failed after upgrade tarantool instance

Steps to reproduce:

1. install tarantool 1.6:

$ cat /etc/yum.repos.d/tarantool_1.6.repo 
[tarantool1.6]
name=Tarantool 1.6
baseurl=https://download.tarantool.org/tarantool/1.6/el/7/x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
$ yum install -y tarantool-queue.noarch tarantool.x86_64

2. setup a cluster with at least one master and some replicas (see initialization Lua scripts below)
3. add yum repo file and upgrade tarantool RPM package using yum update -y tarantool
4. try to start new tarantool instance
5. new instance failed to join to the cluster:

2020-04-28 12:01:43.493 [10527] main/105/applier/[email protected]:33 I> subscribed
2020-04-28 12:01:43.494 [10527] main/105/applier/[email protected]:33 I> remote vclock {1: 376999} local vclock {1: 375406}
2020-04-28 12:01:43.494 [10527] main/108/applier/[email protected]:33 I> authenticated
2020-04-28 12:01:43.494 [10527] main/108/applier/[email protected]:33 I> subscribed
2020-04-28 12:01:43.494 [10527] main/108/applier/[email protected]:33 I> remote vclock {1: 376999} local vclock {1: 375406}
2020-04-28 12:01:43.495 [10527] main/106/applier/[email protected]:33 I> failed to authenticate
2020-04-28 12:01:43.495 [10527] main/106/applier/[email protected]:33 xrow.c:962 E> ER_ACCESS_DENIED: Session access to universe '' is denied for user 'replicator'
2020-04-28 12:01:43.495 [10527] main/106/applier/[email protected]:33 I> will retry every 1.00 second
2020-04-28 12:01:43.500 [10527] main/107/applier/[email protected]:33 I> failed to authenticate
2020-04-28 12:01:43.500 [10527] main/107/applier/[email protected]:33 xrow.c:962 E> ER_ACCESS_DENIED: Session access to universe '' is denied for user 'replicator'
2020-04-28 12:01:43.500 [10527] main/107/applier/[email protected]:33 I> will retry every 1.00 second

Optional (but very desirable):

master1.6-init.lua:

queue = require('queue')
local fiber = require('fiber')

-- файл экземпляра для мастера
box.cfg{
  listen = 3301,
  replication_source = {'replicator:[email protected]:3301',
                        'replicator:[email protected]:3302',
                        'replicator:[email protected]:3303',
                        'replicator:[email protected]:3304',
                        'replicator:[email protected]:3305'}
  -- slab_alloc_arena = 1,
  read_only = false
}

box.once("schema", function()
  box.schema.user.create('replicator', {password = 'password'})
  box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
  box.schema.space.create("test")
  box.space.test:create_index("primary")
  print('box.once executed on master')
  queue.create_tube('mail_msg', 'fifottl')
end)

local producer = fiber.create(function()
    while true do
        queue.tube.mail_msg:put('1')
        fiber.sleep(0.2)
    end
end)

console = require('console')
console.start()

replica1.6-init.lua:

box.cfg{
  listen = 3302,
  replication_source = {'replicator:[email protected]:3301',
                        'replicator:[email protected]:3303',
                        'replicator:[email protected]:3304',
                        'replicator:[email protected]:3305'},
  slab_alloc_arena = 1,
  read_only = true
}
box.once("schema", function()
  box.schema.user.create('replicator', {password = 'password'})
  box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
  box.schema.space.create("test")
  box.space.test:create_index("primary")
  print('box.once executed on replica 2')
end)

console = require('console')
console.start()

replica1.10-init.lua

box.cfg{
  listen = 3307,
  replication = {
                        'replicator:[email protected]:3301',  -- мастер 1.6
                        'replicator:[email protected]:3302',  -- реплика 1.6
                        'replicator:[email protected]:3303',  -- реплика 1.6
                        'replicator:[email protected]:3304',  -- реплика 1.6
                        'replicator:[email protected]:3305',  -- реплика 1.6
                        'replicator:[email protected]:3304',  -- реплика 1.6
                        'replicator:[email protected]:3305',  -- реплика 1.6
                        --'replicator:[email protected]:3306',  -- мастер 1.10
                        --'replicator:[email protected]:3307',  -- реплика 1.10
                        --'replicator:[email protected]:3308',  -- реплика 1.10
                        --'replicator:[email protected]:3309',  -- реплика 1.10
                        --'replicator:[email protected]:3310'}, -- реплика 1.10
                        },
  slab_alloc_arena = 1,
  read_only = true
}
box.once("schema", function()
  box.schema.user.create('replicator', {password = 'password'})
  box.schema.user.grant('replicator', 'replication') -- настроить роль для репликации
  box.schema.space.create("test")
  box.space.test:create_index("primary")
  print('box.once executed on replica 7')
end)

console = require('console')
console.start()

tarantool 1.10 version:

$ tarantool -v
Tarantool 1.10.6-3-gaacc444
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON
Compiler: /opt/rh/devtoolset-8/root/usr/bin/cc /opt/rh/devtoolset-8/root/usr/bin/c++
C_FLAGS:-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic  -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -fno-gnu89-inline -Wno-cast-function-type
CXX_FLAGS:-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic  -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-cast-function-type

Optional (but very desirable):

[Totktonada]

NB: 1.8.3-223-g02adb2a8e commit merges 'session' privilege support (which was introduced in 1.7.7-128-g74ab44ae7) into 1.8 w/o autogranting 'session' privilege for compatibility with 1.7.6 and below.

diff --cc src/box/alter.cc
index 5b2807e1d,74043b814..afd4c2108
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@@ -2351,7 -2554,18 +2453,7 @@@ on_replace_dd_priv(struct trigger * /* 
  
        if (new_tuple != NULL && old_tuple == NULL) {   /* grant */
                priv_def_create_from_tuple(&priv, new_tuple);
-               priv_def_check(&priv);
 -              /*
 -               * Add system privileges explicitly to the
 -               * universe grant issued prior to 1.7.7 in
 -               * case upgrade script has not been invoked.
 -               */
 -              if (priv.object_type == SC_UNIVERSE &&
 -                  dd_version_id < version_id(1, 7, 7)) {
 -
 -                      priv.access |= PRIV_S;
 -                      priv.access |= PRIV_U;
 -              }
+               priv_def_check(&priv, PRIV_GRANT);
                grant_or_revoke(&priv);
                struct trigger *on_rollback =
                        txn_alter_trigger_new(revoke_priv, NULL);

Note: 1.8 becomes 2.0/2.1/2.*; 1.7 evolves into 1.9/1.10.

However, yep, if you're test 1.6 vs 1.10 it should not affect the result. And it seems that this code only works on changes in privileges.