parse_llvm_nm_symbols: when symbol name contains ":" character error occurs

[SageWu]

emsdk version: 2.0.31
in /emsdk/upstream/emscripten/tools/building.py
parse_llvm_nm_symbols function
When the symbol contains ":" character the entry_pos variable will be wrong value. And that result in line[entry_pos + 13:] access wrong index.

error stack:

Traceback (most recent call last):
  File "*/emsdk/upstream/emscripten/em++.py", line 14, in <module>
    sys.exit(emcc.run(sys.argv))
  File "*/emsdk/upstream/emscripten/emcc.py", line 1091, in run
    phase_calculate_system_libraries(state, linker_arguments, linker_inputs, newargs)
  File "*/emsdk/python/3.9.2-1_64bit/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "*/emsdk/upstream/emscripten/emcc.py", line 2487, in phase_calculate_system_libraries
    extra_files_to_link += system_libs.calculate([f for _, f in sorted(linker_inputs)] + extra_files_to_link, forced=state.forced_stdlibs)
  File "*/emsdk/upstream/emscripten/tools/system_libs.py", line 1535, in calculate
    handle_reverse_deps(input_files)
  File "*/emsdk/upstream/emscripten/tools/system_libs.py", line 1506, in handle_reverse_deps
    symbolses = building.llvm_nm_multiple([os.path.abspath(t) for t in input_files])
  File "*/emsdk/python/3.9.2-1_64bit/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "*/emsdk/upstream/emscripten/tools/building.py", line 195, in llvm_nm_multiple
    for key, value in parse_llvm_nm_symbols(results.stdout).items():
  File "*/emsdk/upstream/emscripten/tools/building.py", line 566, in parse_llvm_nm_symbols
    status = line[entry_pos + 11] # Skip address, which is always fixed-length 8 chars.
IndexError: string index out of range

[sbc100]

Do you have symbols that contain :? is that possible? It might be useful if you could attach the output of llvm-nm. Sadly to do that you would need to hack building.py and do something like print(output) at the top of parse_llvm_nm_symbols. If you could attach that output to this bug it would make it much easier to debug.

@juj, any ideas?

[SageWu]

Do you have symbols that contain :? is that possible? It might be useful if you could attach the output of llvm-nm. Sadly to do that you would need to hack building.py and do something like print(output) at the top of parse_llvm_nm_symbols. If you could attach that output to this bug it would make it much easier to debug.

@juj, any ideas?

.a:LibTess2.cpp.o: U _invoke_i32%"struct.tess2::TESSmesh"

[juj]

yikes... :/

Parsing these lines is tricky with the colons, because of Windows "C:" path names (though now I also notice that it is possible to have a colon in path name on macOS and Linux, e.g. a:b is a valid directory name on these OSes.. that will not work in current parsing either)

Another issue here is that this parsing loop is quite hot, it can be executed hundreds of thousands of times in a link of a large project. Hence the parsing logic needs to be mindful about performance.

Maybe the parsing scheme would be best to look for the index of the first "" sequence in the llvm-nm output of each line, and split the line into filename and symbol parts based on that.

It would be great to have a minimal test case that produces this failure. Would you be able to reduce your code down to one?

I tried with

namespace a
{
	struct b
	{
		struct c
		{
			void bar() {}
		};
	};
}

a::b::c foo;
int main()
{
	foo.bar();
}

but that does not seem to reproduce it at least. I wonder what code pattern is present there that generates such a symbol name in LLVM.

[SageWu]

@juj That symbol should came from library developers in other company department write. Now I can't get the actual code produces that symbol. Sorry.

[sbc100]

Where did LibTess2.cpp.o come from? I'm pretty sure symbols like that should never exist with recent versions of llvm. Those _invoke_ symbols should get converted to valid symbol names.

See the llvm change and corresponding binaryen change:
https://reviews.llvm.org/D88697
WebAssembly/binaryen#3191

This makes me thing that that object file must have been built before https://reviews.llvm.org/D88697, with an older version of llvm. Even if we fix the nm issue it will not be usable linkable because the name of the _invoke_ function doesn't match what emscripten expects.

In short, I think you need to rebuild that library with an newer version of llvm/emsdk.

[SageWu]

@sbc100 Thanks, I will have a look at that. Because LibTess2.cpp comes from sdk developed by other department colleagues, I will try to talk with them for solving the problem.

[PyryM]

I've hit this issue as well when exporting .bc files from Terra, which produces names like $anon (junk/wasm_helloworld.t:7) for anonymous functions. Since both filenames and symbol names can potentially have colons (and spaces, and most other characters), I don't think there's a 100% reliable way to parse the output of llvm-nm as is, and the utility doesn't seem to have a more structured output format like JSON.

However, at the cost of running llvm-nm twice I think it would work to:

1. First run llvm-nm -j ... to get a list of just the symbol names
2. Then run the full llvm-nm command as before
3. When parsing each line first strip away the symbol name from (1) from the right side of (2)

[gsopu8065]

I get the same error with the latest emscripten version.

Traceback (most recent call last):
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/em++.py", line 14, in <module>
    sys.exit(emcc.run(sys.argv))
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/emcc.py", line 1168, in run
    phase_calculate_system_libraries(state, linker_arguments, linker_inputs, newargs)
  File "/Users/jack/Amazon-Demo/emsdk/python/3.9.2_64bit/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/emcc.py", line 2726, in phase_calculate_system_libraries
    extra_files_to_link += system_libs.calculate(all_linker_inputs, newargs, forced=state.forced_stdlibs)
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/tools/system_libs.py", line 1885, in calculate
    handle_reverse_deps(input_files)
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/tools/system_libs.py", line 1729, in handle_reverse_deps
    symbolses = building.llvm_nm_multiple([os.path.abspath(t) for t in input_files])
  File "/Users/jack/Amazon-Demo/emsdk/python/3.9.2_64bit/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/tools/building.py", line 184, in llvm_nm_multiple
    for key, value in parse_llvm_nm_symbols(results.stdout).items():
  File "/Users/jack/Amazon-Demo/emsdk/upstream/emscripten/tools/building.py", line 574, in parse_llvm_nm_symbols
    status = line[entry_pos + 11] # Skip address, which is always fixed-length 8 chars.
IndexError: string index out of range

[juj]

While the root cause might be what @sbc100 commented above,

In short, I think you need to rebuild that library with an newer version of llvm/emsdk.

, if you have a repro zip of artifacts and command line to reproduce that you can attach, I can look into if I can find a way to unambiguously parse the multiple : symbols.

[sbc100]

We don't need to support symbols with ":" in them.. but it would be good to be able to detect and reject them with a better error message perhaps.

[kleisauke]

Was this fixed with commit cba63c7?

[sbc100]

Yes. Thanks for pointing that out.