test-run worker hangs, when an instance does not enter to the event loop

[ligurio]

How to reproduce

test reproducer (test path xlog/hang.test.lua):

test_run = require('test_run').new()
test_run:cmd('create server replica with rpl_master=default, script="xlog/replica.lua"')
test_run:cmd('start server replica')
test_run:cmd('cleanup server replica')
test_run:cmd('delete server replica')

test hang due to absence read access (see log var/001_xlog/replica.log):

...
2021-03-19 18:11:44.865 [1987357] main/103/replica I> connected to 1 replicas
2021-03-19 18:11:44.865 [1987357] main/103/replica I> bootstrapping replica from 8a89e9f8-7364-4d6a-96cd-d2c8fe5b93bb at unix/:/home/sergeyb/sources/MRG/tarantool/build/test/var/001_xlog/xlog.socket-iproto
2021-03-19 18:11:44.865 [1987357] main/112/applier/unix/:/home/sergeyb/sources/MRG/tarantool/build/test/var/001_xlog/xlog.socket-iproto I> can't read row
2021-03-19 18:11:44.865 [1987357] main/112/applier/unix/:/home/sergeyb/sources/MRG/tarantool/build/test/var/001_xlog/xlog.socket-iproto session.cc:332 E> ER_ACCESS_DENIED: Read access to universe '' is denied for user 'guest'
2021-03-19 18:11:44.865 [1987357] main/112/applier/unix/:/home/sergeyb/sources/MRG/tarantool/build/test/var/001_xlog/xlog.socket-iproto I> will retry every 1.00 second

test is passed if add line box.schema.user.grant('guest', 'replication') on top of test

Versions

test-run 5941741

tarantool --version:

Tarantool 2.8.0-134-g81c663335
Target: Linux-x86_64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=ON
Compiler: /usr/bin/cc /usr/bin/c++
C_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror
CXX_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror

[Totktonada]

This test should hang: we unable to bootstrap the replica, because it is unable to join to the master (because of lack of the grant).

However the problem is that a test timeout does not work and the test-run worker 'hangs'. It seems, it is because of time.sleep(<...>) usage, which does not work well with gevent cooperative greenlets. At least, the following patch seems to fix the problem:

diff --git a/lib/tarantool_server.py b/lib/tarantool_server.py
index f611dba..ee83489 100644
--- a/lib/tarantool_server.py
+++ b/lib/tarantool_server.py
@@ -449,7 +449,7 @@ class TarantoolLog(object):
         while True:
             if os.path.exists(self.path):
                 break
-            time.sleep(0.001)
+            gevent.sleep(0.001)
 
         with open(self.path, 'r') as f:
             f.seek(self.log_begin, os.SEEK_SET)
@@ -460,7 +460,7 @@ class TarantoolLog(object):
                         raise TarantoolStartError(name)
                 log_str = f.readline()
                 if not log_str:
-                    time.sleep(0.001)
+                    gevent.sleep(0.001)
                     f.seek(cur_pos, os.SEEK_SET)
                     continue
                 if re.findall(msg, log_str):

I guess we can reproduce the similar behaviour around other time.sleep(<...>) usages.

[Totktonada]

However it would be good to have some timeout for tarantool instance startup and report a meaningful error in the case.