asm_conv: Assertion `((IRType)((ir->t).irt & IRT_TYPE)) != st' failed.

[spacewander]

I discovered a (possible) JIT bug when I wrote a LuaJIT implementation of Rabin-Karp algorithm.

#!/usr/bin/env luajit
require "table.new"
local str_byte = string.byte
local bit = require "bit"
local ffi = require "ffi"

ffi.cdef[[
    typedef unsigned int u32;
    int memcmp(const void *s1, const void *s2, size_t n);
]]
local prime = 16777619

-- returns the hash and the appropriate multiplicative factor for use in Rabin-Karp algorithm.
local function prepare_hash(s)
    local hash = ffi.cast("u32", 0)
    local pow = ffi.cast("u32", 1)
    local sq = ffi.cast("u32", prime)

    ----[[ comment out this block makes the bug disappear
    for i = #s, 1, -1 do
        hash = ffi.cast("u32", hash * prime) + str_byte(s, i, i)
    end

    local i = #s
    while i > 0 do
        if bit.band(i, 1) ~= 0 then
            pow = ffi.cast("u32", pow * sq)
        end
        sq = ffi.cast("u32", sq * sq)
        i = bit.rshift(i, 1)
    end
    --]]

    return hash, pow
end

local function rfind(haystack, needle)
    local nlen = #needle
    local hlen = #haystack
    if nlen == 0 then
        return hlen
    end

    prepare_hash(needle)
    -- ... other stuff
end

local max = 1000
for case = 1, max do
    local size = math.random(1, 20)
    local sub_end = math.random(1, size)
    local sub_begin = math.random(1, sub_end)
    local buf = table.new(size, 0)
    for i = 1, size do
        buf[i] = math.random(33, 126)
    end
    local raw = string.char(unpack(buf))
    local sub = string.sub(raw, sub_begin, sub_end)
    rfind(raw, sub)
end

Run the script above: luajit x.lua
Got:

luajit: lj_asm_x86.h:813: asm_conv: Assertion `((IRType)((ir->t).irt & IRT_TYPE)) != st' failed.

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f444e2a1801 in __GI_abort () at abort.c:79
#2  0x00007f444e29139a in __assert_fail_base (fmt=0x7f444e4187d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x55af7f4f0698 "((IRType)((ir->t).irt & IRT_TYPE)) != st", file=file@entry=0x55af7f4ef615 "lj_asm_x86.h",
    line=line@entry=813, function=function@entry=0x55af7f4ef090 <__PRETTY_FUNCTION__.6710> "asm_conv") at assert.c:92
#3  0x00007f444e291412 in __GI___assert_fail (assertion=assertion@entry=0x55af7f4f0698 "((IRType)((ir->t).irt & IRT_TYPE)) != st",
    file=file@entry=0x55af7f4ef615 "lj_asm_x86.h", line=line@entry=813,
    function=function@entry=0x55af7f4ef090 <__PRETTY_FUNCTION__.6710> "asm_conv") at assert.c:101
#4  0x000055af7f4b482b in asm_conv (as=<optimized out>, ir=0x7f444ef82828) at lj_asm_x86.h:813
#5  0x000055af7f4b8b24 in asm_ir (ir=0x7f444ef82828, as=0x7ffc953930f0) at lj_asm.c:1719
#6  lj_asm_trace (J=J@entry=0x7f444efff688, T=T@entry=0x7f444efff688) at lj_asm.c:2358
#7  0x000055af7f4846d8 in trace_state (L=0x7f444efff378, dummy=<optimized out>, ud=0x7f444efff688) at lj_trace.c:698
#8  0x000055af7f490969 in lj_vm_cpcall () at buildvm_x86.dasc:1321
#9  0x000055af7f485516 in lj_trace_ins (J=J@entry=0x7f444efff688, pc=pc@entry=0x7f444f00bb28) at lj_trace.c:730
#10 0x000055af7f472258 in lj_dispatch_ins (L=0x7f444efff378, pc=0x7f444f00bb2c) at lj_dispatch.c:424
#11 0x000055af7f4924ab in lj_vm_inshook () at buildvm_x86.dasc:2588
#12 0x000055af7f47a720 in lua_pcall (L=0x7f444efff378, nargs=0, nresults=-1, errfunc=<optimized out>) at lj_api.c:1129
#13 0x000055af7f46c102 in docall (L=0x7f444efff378, narg=0, clear=0) at luajit.c:121
#14 0x000055af7f46cf71 in handle_script (argx=<optimized out>, L=0x7f444efff378) at luajit.c:292
#15 pmain (L=0x7f444efff378) at luajit.c:553
#16 0x000055af7f490573 in lj_BC_FUNCC () at buildvm_x86.dasc:898
#17 0x000055af7f47a7fb in lua_cpcall (L=<optimized out>, func=<optimized out>, ud=<optimized out>) at lj_api.c:1153
#18 0x000055af7f46baef in main (argc=2, argv=0x7ffc953938b8) at luajit.c:582

My LuaJIT is built by:

make -j8 TARGET_STRIP=@: CCDEBUG=-g Q= XCFLAGS='-DLUA_USE_APICHECK -DLUA_USE_ASSERT -g -fno-omit-frame-pointer'

The same bug can be reproduced in moonjit, so I think it is a new one.

[spacewander]

Fixed in moonjit@51ef005

[MikePall]

Fixed. Thanks!