x64: Fix __call metamethod return dispatch.

After linking new cframe to the chain KBASEa still stores the address of
the previous one. If the execution proceeds to <lj_vmeta_call> KBASE
value (i.e. low 32 bits of the stored address) might be equal to the
current BASE address value so the execution takes the invalid path. Such
address parts clashing occurs only on x86_64 platform with disabled
LJ_GC64, so 64-bit registers have to be compared in x64 build.

NB: Though there is only 32-bit load to restore BASE value prior to the
comparison, the high 32 bits of RDX are reset to zeros, according to x86
long mode semantics.

Igor Munkin:
* backported the original patch to tarantool/luajit repo
* extended the original commit message with the rationale

For more info and explanation see LuaJIT#636.

Relates to tarantool/tarantool#4518
Relates to tarantool/tarantool#4649

Reviewed-by: Sergey Kaplun <[email protected]>
Reviewed-by: Sergey Ostanevich <[email protected]>
Signed-off-by: Igor Munkin <[email protected]>

Author: Mike Pall

src/vm_x86.dasc

@@ -1374,7 +1374,11 @@ static void build_subroutines(BuildCtx *ctx)
   |  mov LFUNC:RB, [RA-8]
   |  add NARGS:RD, 1
   |  // This is fragile. L->base must not move, KBASE must always be defined.
+  |.if x64
+  |  cmp KBASEa, rdx			// Continue with CALLT if flag set.
+  |.else
   |  cmp KBASE, BASE			// Continue with CALLT if flag set.
+  |.endif
   |  je ->BC_CALLT_Z
   |  mov BASE, RA
   |  ins_call				// Otherwise call resolved metamethod.