Crash in vc4_dri.so

[Hixie]

Just filing this here FYI, I'll investigate more thoroughly myself when I have time. I made a basic animated Flutter app and left it running overnight, when I came back it had crashed. I ran it under gdb and this is what I found. If there's any commands you'd like me to run before I get around to debugging it myself, let me know and I can run them.

Thread 3 "flutter-pi" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x6faff330 (LWP 10083)]
0x75d86360 in ?? () from /usr/lib/arm-linux-gnueabihf/dri/vc4_dri.so
(gdb) bt
#0  0x75d86360 in  () at /usr/lib/arm-linux-gnueabihf/dri/vc4_dri.so
#1  0x00000000 in  ()
(gdb) 

[ardera]

vc4_dri.so is the mesa VideoCore 4 graphics driver. I just compiled mesa & flutter engine with debug symbols yesterday. Here's a link so you don't need to compile them yourself. After extraction, don't forget to rename to libflutter_engine.so to libflutter_engine.so.debug, and then just point LD_LIBRARY_PATH to the directory of libflutter_engine.so.debug and the mesa3d-install/lib/arm-linux-gnueabihf. You also need to point LIBGL_DRIVERS_PATH to the dri directory inside the mesa binaries.

Btw, this is latest master mesa (it's recommended to test against latest master before submitting bug reports to mesa). I also have mesa 20.2.X available if you want. Though it's also pretty easy to compile yourself. The only real gotcha is that you must include the kmsro driver when configuring the build, even though you only want to build for VC4 / V3D, otherwise it'll try to use software rendering.

Otherwise, you can also send me the asset bundle / app source so I can reproduce it. I have run flutter apps using flutter-pi overnight in the past, and I didn't get the error, so it's likely not reproducible by any app just running a long time

[Hixie]

Ok I unzipped that link in ~/temp, renamed the .so to .so.debug, and ran the following command to launch the app under gdb. I'll report back when/if it crashes! Let me know if I did anything wrong (I'm not 100% sure how to check that I got the env variables right).

ianh@burmilla:~/dev/laundry_room_console $ LD_LIBRARY_PATH=~/temp:~/temp/mesa3d-install/lib/arm-linux-gnueabihf LIBGL_DRIVERS_PATH=~/temp/mesa3d-install/lib/arm-linux-gnueabihf/dri gdb --args ~/dev/flutter-pi/out/flutter-pi --release .
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/ianh/dev/flutter-pi/out/flutter-pi...done.
(gdb) r
Starting program: /home/ianh/dev/flutter-pi/out/flutter-pi --release .
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[flutter-pi] WARNING: display has non-square pixels. Non-square-pixels are not supported by flutter.
===================================
display mode:
  resolution: 800 x 480
  refresh rate: 60Hz
  physical size: 155mm x 86mm
  flutter device pixel ratio: 1.358234
===================================
EGL information:
  version: 1.4
  vendor: "Mesa Project"
  client extensions: "EGL_EXT_client_extensions EGL_EXT_device_base EGL_EXT_device_enumeration EGL_EXT_device_query EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses EGL_KHR_debug EGL_EXT_platform_device EGL_MESA_platform_gbm EGL_KHR_platform_gbm EGL_MESA_platform_surfaceless"
  display extensions: "EGL_ANDROID_blob_cache EGL_ANDROID_native_fence_sync EGL_EXT_buffer_age EGL_EXT_image_dma_buf_import EGL_EXT_image_dma_buf_import_modifiers EGL_KHR_cl_event2 EGL_KHR_config_attribs EGL_KHR_create_context EGL_KHR_create_context_no_error EGL_KHR_fence_sync EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_image EGL_KHR_image_base EGL_KHR_image_pixmap EGL_KHR_no_config_context EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_EXT_pixel_format_float EGL_KHR_wait_sync EGL_MESA_configless_context EGL_MESA_drm_image EGL_MESA_image_dma_buf_export EGL_MESA_query_driver "
===================================
OpenGL ES information:
  version: "OpenGL ES 2.0 Mesa 21.0.0-devel (git-34a35d8f2e)"
  shading language version: "OpenGL ES GLSL ES 1.0.16"
  vendor: "Broadcom"
  renderer: "VC4 V3D 2.1"
  extensions: "GL_EXT_blend_minmax GL_EXT_multi_draw_arrays GL_EXT_texture_format_BGRA8888 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth24 GL_OES_element_index_uint GL_OES_fbo_render_mipmap GL_OES_mapbuffer GL_OES_rgb8_rgba8 GL_OES_stencil8 GL_OES_texture_3D GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_EGL_image GL_OES_depth_texture GL_AMD_performance_monitor GL_OES_packed_depth_stencil GL_OES_get_program_binary GL_APPLE_texture_max_level GL_EXT_discard_framebuffer GL_EXT_read_format_bgra GL_EXT_frag_depth GL_NV_fbo_color_attachments GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_vertex_array_object GL_ANGLE_pack_reverse_row_order GL_EXT_occlusion_query_boolean GL_EXT_unpack_subimage GL_NV_draw_buffers GL_NV_read_buffer GL_NV_read_depth GL_NV_read_depth_stencil GL_NV_read_stencil GL_EXT_draw_buffers GL_EXT_map_buffer_range GL_KHR_debug GL_KHR_texture_compression_astc_ldr GL_NV_pixel_buffer_object GL_OES_required_internalformat GL_OES_surfaceless_context GL_EXT_separate_shader_objects GL_EXT_compressed_ETC1_RGB8_sub_texture GL_EXT_draw_elements_base_vertex GL_EXT_texture_border_clamp GL_KHR_context_flush_control GL_OES_draw_elements_base_vertex GL_OES_texture_border_clamp GL_KHR_no_error GL_KHR_texture_compression_astc_sliced_3d GL_KHR_parallel_shader_compile GL_MESA_tile_raster_order "
===================================
[New Thread 0x703b7320 (LWP 16582)]
[New Thread 0x6f9ff320 (LWP 16583)]
[New Thread 0x6efff320 (LWP 16584)]
[New Thread 0x6dbfb320 (LWP 16585)]
[New Thread 0x6d3fa320 (LWP 16586)]
[New Thread 0x6c9ff320 (LWP 16587)]
[New Thread 0x6bfff320 (LWP 16588)]
[New Thread 0x6fbb6320 (LWP 16589)]
detected VideoCore IV as underlying graphics chip, and VC4 as the driver.
Reporting modified GL_EXTENSIONS string that doesn't contain non-working extensions.
[New Thread 0x6b1ff320 (LWP 16590)]
[New Thread 0x6b17e320 (LWP 16591)]
flutter: remy: connected
[Thread 0x6b1ff320 (LWP 16590) exited]
[Thread 0x6b17e320 (LWP 16591) exited]

[Hixie]

Interesting, I'm getting a bunch of these followed by it not drawing everything on the screen:

Mesa: User error: GL_OUT_OF_MEMORY in glTexImage
Mesa: User error: GL_OUT_OF_MEMORY in glTexImage2D

...hasn't crashed yet, but could be related; maybe in the non-debug build we don't check if we're running out of memory and just crash? (We're working on making debugging memory issues easier over in Flutter land, so that should help debug why I'm running out of memory here.)

[ardera]

I actually just fixed a bug 10 hours ago that could cause an OpenGL OOM (see issue #133). Can you check you're on latest master? Maybe it's fixed already. Though the issue that was fixed manifested itself a bit differently. There was no segmentation fault inside the graphics driver, the program was just killed by the linux OOM killer.

Ok I unzipped that link in ~/temp, renamed the .so to .so.debug, and ran the following command to launch the app under gdb. I'll report back when/if it crashes! Let me know if I did anything wrong (I'm not 100% sure how to check that I got the env variables right).

Looks good to me. If you want to be absolutely certain, you can try to list some source code inside gdb, or see if the function declarations look complete.

maybe in the non-debug build we don't check if we're running out of memory and just crash? (We're working on making debugging memory issues easier over in Flutter land, so that should help debug why I'm running out of memory here.)

That's possible. If you want to debug it, your best bet is asan. Sadly valgrind doesn't work that well on raspberry pi or ARM in general. Many uncommon ARM instructions are unsupported and valgrind will inevitably terminate on the first unknown instruction.

[Hixie]

That does seem to have fixed it! I'll reopen if I see it break again!