Fixed memory corruption bug with yielding from Lua code and using the LuaJIT2 VM.

[Neopallium]

If you turn on LUA_USE_APICHECK in LuaJIT2, then an exception will be raised in lua_settop(cc, 0) when called after the Lua code yields. I do not see any reason why lua_settop() should be call when Lua yields.

[chaoslawful]

The problem you described can't be reproduced in our environment. I checked LuaJIT2's apicheck mechanism, and found that using lua_settop(L, 0) should not cause any exceptions. Could you provide more information about your compiling environment (especially LuaJIT2.0 source revision you used) and a minimum test to reproduce it?

[Neopallium]

I am using super-nginx with the latest HEAD revision of LuaJIT2. The crash/api exception happens when running the /lua/ or /recur examples. Also the memory corruption doesn't seem to happen when nginx is compiled with -O1 or lower, but only with -O2 or higher. I am using gcc 4.4.5 glibc 2.12 on Linux 2.6.36.

The crash happens at: ngx_http_handler() in nginx/src/http/ngx_http_core_module.c:824. The value at r->main_conf[0] is over written in side a memalign() call from nginx's pool allocator (I have used the no-pool patch to remove the pool allocator to confirm the problem wasn't with the pool allocator) called at line nginx/src/http/ngx_http_request.c:436. I wasn't sure how memalign()'s internal state was being corrupted, but when I removed the lua_setup(cc, 0) call the problem disapeared (which I removed because LuaJIT2 with api check turned on was raising an exception).

Maybe the root cause of the memory corruption is from somewhere else and causing some corruption in the LuaJIT2 state.

[Neopallium]

I did some more testing with nginx compiled at -O1 instead of -O2 or higher the api check in LuaJIT2 isn't triggered when calling lua_settop(). The crash might be cause by some optimization bug in gcc. I will try compiling with LLVM's clang compiler.

[Neopallium]

I have now isolated the bug down to gcc's -fcaller-saves optimization. I think this optimization is incompatible/conflicting with LuaJIT2. The crash can be re-created with "-O1 -fcaller-saves". It is possible to use higher optimization levels (-O2 or even -O3) by disabling that optimization with "-fno-caller-saves". I will report this to Mike Pall.

I am closing this pull request, since the problem isn't with lua_settop(). Sorry for the noise.