Some issues related to older i915 rendering on a Celeron G470

[Abnormalpingu]

Hi,

We have some x86 Shuttle pc's and a variety of older intel based hardware out in the wild, our plan is to network boot all our clients from a central local pxe server in read only, as all we need is all the clients starting up, logging in and starting our app (based on flutter as you might've already guessed).

The rpi4's will be booted with raspian (latest stable) 64-bit, with no desktop environment, it works perfectly.
The shuttle's and the older "custom" machines will be running debian 11 64-bit (latest stable) - the debian network boot version is built using debootstrap just to minimize it as much as possible.
Initial testing on the shuttle machines works perferctly after we installed the libdrm-intel1 package.

Our older intel hardware did not go as we hoped, and there are some stuff we'd really appreciate some pointers and ultimately some help getting to work.

We first tried networkbooting the old client like the already working Shuttle, OS wise it works fine, but when we get to flutter-pi and running our app we ran into issues.
Just to test stuff (and without the network boot complexities) I've installed Debian 11 from scratch on the older machine.

When starting the app we see the following; link

If we try removing the 'libdrm-intel1' package we see the following; link

(just to illustrate the intel package actually does something)

I did some digging and came over the 'kmscube' package, just to see if it renders anything at all, which also works when we do 'kmscube --atomic'

Here is the output from kmscube with the libdrm-intel1 package installed; link
Here is the output without the libdrm-intel1 package installed; link

Our "older intel"-stuff runs an ASUS H61M-F motherboard with a Celeron G470 CPU
After som digging the CPU has a i915 (driver mentions i965 but not sure if this is relevant, see the log) hd gpu (apu*), which should be sufficient for our requirements (the app doesn't have any overly complicated graphics).

The app is built on a windows 10 box using (powershell; Flutter build foldername)
Flutter 3.7.3
Dart 2.19.2
DevTools 2.20.1

Debian 11 client is running 5.10.0-20-amd64
Flutter-pi is built from https://github.com/ardera/flutter-pi
https://github.com/sony/flutter-embedded-linux/releases/download/b24591ed32/elinux-x64-debug.zip
https://raw.githubusercontent.com/ardera/flutter-engine-binaries-for-arm/main/flutter_embedder.h
https://github.com/ardera/flutter-engine-binaries-for-arm/raw/main/arm64/icudtl.dat

Installation is done according to the documentation provided on this github.
(sorry for the lengthy post)

[ardera]

Hey,

if kmscube can output things, flutter-pi probably just does something wrong when speaking to the modesetting device. (Sometimes video drivers have very specific constraints that userspace applications like flutter-pi need to fulfill)

Can you do the following:

• Upload some details about your video driver using the instructions here: https://drmdb.emersion.fr/
  • (not technically required, just to keep the database up to date :) )
• Install libdrm-tests and run modetest, and send me the full output of that
• run:

  echo 0x1FF | sudo tee /sys/module/drm/parameters/debug # Enable verbose DRM logging
  sudo dmesg -C # Clear kernel logs
  dmesg -w >dmesg.log & # Continuously write DRM logs to a file, in the background
  flutter-pi ... # Run flutter-pi again, manually terminate it after a few errors have happened
  fg # Kill dmesg with Ctrl+C
  echo 0 | sudo tee /sys/module/drm/parameters/debug # Disable DRM logging

• and send me the dmesg.log file

[Abnormalpingu]

Hi,

yeah that's what i thought too, my skillset is somewhat limited with Linux so thank you so much for the quick and detailed reply !

I was able to dump the driver details which can be reached here
The modtest output looks like this
I was able to create the dmesg.log which is attached to this post.

-Dan
dmesg.log

[ardera]

Okay, seems like your display controller doesn't support an ARGB as the pixel format for the output planes.

Should be easy to fix

[Abnormalpingu]

Fantastic news, I was worried we had hit a hardware limitation, which wouldn't have suprised me due to the age of the hardware.
Is there something specific we can do in our app to remedy this or is this a flutter-pi specific issue ?

[ardera]

This is a flutter-pi specific issue, but if you want to try running without a real fix in place, you can try running with --pixelformat XRGB8888 :)

[Abnormalpingu]

Allright, Thank you! :)
I will try this alternative method until a real fix is in place (take your time!)

Not sure if you're aware but there seems to be a version issue between Dart and flutter-pi - just rebuilt our app and i get this;
[ERROR:flutter/shell/common/shell.cc(89)] Dart Error: Can't load Kernel binary: Invalid SDK Hash
I was stupid and updated stuff on my build machine, but i reckon this will be an issue for any new users of flutter-pi. I might be wrong though.

[ardera]

np, I already have it almost done

That issue is actually mentioned in the readme: https://github.com/ardera/flutter-pi#building-the-asset-bundle
TL;DR if you use the 3.7.3 engine binaries on the target device, you also need to use the 3.7.3 SDK on your development machine. If there's a slight mismatch it won't work. If you upgrade your SDK without also upgrading the engine binaries, that won't work.

Maybe I should clarify that in the readme, because a lot of people seem to run into that problem

[Abnormalpingu]

Great, I've solved the issue, I just didn't read closely enough, thanks for the tip!

I've rebuild the app and tested the --pixelformat XRGB8888 option and I still get the same message on the older hardware, so guess I'll just wait until you're done.

[ardera]

I've rebuild the app and tested the --pixelformat XRGB8888 option and I still get the same message on the older hardware

Interesting, can you try with the feature/gstreamer-video-player branch? (it should work without the --pixelformat option there)

[Abnormalpingu]

Hello again,

I've tried doing this without much success, seems I'm unable to build the gstreamer-video-player branch correctly. I'm pretty sure it's something I'm doing that's wrong though.

(First I made sure the dependencies for the gstreamer video stuff was installed)
I deleted the old flutter-pi git cloned folder, then did;

git clone -b feature/gstreamer-video-player https://github.com/ardera/flutter-pi.git
cd flutter-pi/
mkdir build
cd build
cmake ..

and I get this error

CMakeError.log
CMakeOutput.log

Cmake is installed;
apt install cmake:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
cmake is already the newest version (3.18.4-2+deb11u1).
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

[Abnormalpingu]

Hi, sorry for the spam :(

As suspected it was something wrong with cmake in my end, I was able to fix it by installing the 'build-essential' package.

However, we seem to have a different issue when building it - see here

[ardera]

Are you sure you're building the latest sources? That error was fixed by a commit some time ago

The lines in the error message look different now:

flutter-pi/include/pixel_format.h

Lines 36 to 50 in 8eb4ccd

	enum pixfmt {
	kRGB565_FpiPixelFormat,
	kARGB4444_FpiPixelFormat,
	kXRGB4444_FpiPixelFormat,
	kARGB1555_FpiPixelFormat,
	kXRGB1555_FpiPixelFormat,
	kARGB8888_FpiPixelFormat,
	kXRGB8888_FpiPixelFormat,
	kBGRA8888_FpiPixelFormat,
	kBGRX8888_FpiPixelFormat,
	kRGBA8888_FpiPixelFormat,
	kRGBX8888_FpiPixelFormat,
	kMax_PixFmt = kRGBX8888_FpiPixelFormat,
	kCount_PixFmt = kMax_PixFmt + 1
	};

Maybe try deleting & cloning again

[Abnormalpingu]

Well, that explains it, I did in fact remove the old sources, then I cloned;
git clone -b feature/gstreamer-video-player https://github.com/ardera/flutter-pi.git
Took a look at the source files and they are indeed not updated.

I looked around at the other branches and found a recently updated one;
git clone -b feature/gstreamer-player-from-pipeline https://github.com/ardera/flutter-pi.git

It builds allthough I'm getting a lot of warnings, but it completed, was able to 'make install', and what do you know, tried starting our flutter app and it works!

Amazing work, thank you so much for the help and patience!

[ardera]

Nice! Glad it works now!

Can you send me the warnings you're getting?

[Abnormalpingu]

Absolutely,
here are the warnings :)

[ardera]

Thanks, I'll look into it

[Abnormalpingu]

Hello again,

so it seems there are more hardware variations than we first expected out in the wild from our client, so we seem to have some more issues with rendering, I'm hoping you're able to help us out again.. :) Let me know if you want me to create a seperate post for this and I'll delete this post.

I am able to get kmscube running as before, I assume this is a good sign, that the issue is related to flutter-pi and not the hardware itself. I'll post the information you requested last time here;

Driver details on drmdb - here
dmesg.log - here
modetest output - here

The main differences now is that this is an AMD Sempron 145 cpu w/Radeon R600 onboard gpu. We're still running debian 11 etc. The celeron hw is still working perfectly after your earlier fix.

I've installed the libdrm-amdgpu1 instead of libdrm-intel1;

'apt install libgl1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdrm-amdgpu1 libdrm-dev libgbm-dev fontconfig libsystemd-dev libinput-dev libudev-dev libxkbcommon-dev' (and of course all the gstreamer stuff).
I am building the flutter-pi client from the gstereamer-player-from-pipeline branch

Radeon requires firmware to work in linux, so I've added the contrib and non-free repositories and installed;
'apt install firmware-amd-graphics'

The output I'm getting from flutter-pi is;
[flutter-pi] WARNING: display didn't provide valid physical dimensions. The device-pixel ratio will default to 1.0, which may not be the fitting device-pixel ratio for your display.
[flutter-pi] Could not create GBM Surface. gbm_surface_create_with_modifiers. Will attempt with gbm_surface_create: Function not implemented
[modesetting] Could not find a suitable unused DRM plane for pushing the framebuffer.
[compositor] Couldn't add fb layer.
[compositor] Could not present backing store. rendertarget->present: Input/output error
[modesetting] Could not commit display update. drmModePageFlip: No such file or directory
[compositor] Could not present frame. kms_req_commit_blocking: No such file or directory
[modesetting] Could not find a suitable unused DRM plane for pushing the framebuffer.
[compositor] Couldn't add fb layer.
[compositor] Could not present backing store. rendertarget->present: Input/output error
[modesetting] Could not commit display update. drmModePageFlip: No such file or directory
[compositor] Could not present frame. kms_req_commit_blocking: No such file or directory

Should you need anything else please let me know !

-Dan

[ardera]

Oof, legacy KMS. That's pretty old 😄

Can you try the feature/compositor-ng branch, configure with cmake ... -DCMAKE_BUILD_TYPE=Debug -DDEBUG_DRM_PLANE_ALLOCATIONS=On ... and send me the output of that?

That'd help me see what's going on. Thanks!

[Abnormalpingu]

Yeah, this stuff is ancient, it will be replaced with RPI's as soon as they are back in stock, seems pretty much impossible to be able to get any these days, especially when we need about a thousand of them..

Anyway, sorry for the delay we've had some days off due to our consitution day :)

I did the following;

git clone --single-branch --branch feature/compositor-ng https://github.com/ardera/flutter-pi
made a build folder, cd'ed into that and did;
cmake .. -DCMAKE_BUILD_TYPE=Debug -DDEBUG_DRM_PLANE_ALLOCATIONS=On
Build is completed OK

output from running our app, also this is being ran as root, just to avoid any permission issues temporarily.

flutter-pi client/ > tmp.txt

[locales] Warning: The system has no configured locale. The default "C" locale may or may not be supported by the app.
[keyboard] Could not load keyboard configuration from "/etc/default/keyboard". Default keyboard config will be used. load_file: No such file or directory
[flutter-pi] WARNING: display didn't provide valid physical dimensions. The device-pixel ratio will default to 1.0, which may not be the fitting device-pixel ratio for your display.
[flutter-pi] Could not create GBM Surface. gbm_surface_create_with_modifiers. Will attempt with gbm_surface_create: Function not implemented
[modesetting] Could not find a suitable unused DRM plane for pushing the framebuffer.
[compositor] Couldn't add fb layer.
[compositor] Could not present backing store. rendertarget->present: Input/output error
[modesetting] Could not commit display update. drmModePageFlip: No such file or directory
[compositor] Could not present frame. kms_req_commit_blocking: No such file or directory
[modesetting] Could not find a suitable unused DRM plane for pushing the framebuffer.
[compositor] Couldn't add fb layer.
[compositor] Could not present backing store. rendertarget->present: Input/output error
[modesetting] Could not commit display update. drmModePageFlip: No such file or directory
[compositor] Could not present frame. kms_req_commit_blocking: No such file or directory
^C

Thanks again for your help and patience !

[ardera]

Can you try cloning and reconfiguring again? The output looks a bit different than what expected. I removed the [modesetting], [compositor] style logging in fd8ee91 :)

So it should print stuff like compositor.c: ..., modesetting.c: ... now. Also not sure debug mode is enabled correctly, it should output some debug information for locale, egl, opengl and display mode at the start in debug mode.

I'm also not sure if > will capture stderr too. Can you try &> to capture both stdout and stderr? Thanks! (Or did you redirect to tmp.txt but send me the console output (not the contents of tmp.txt) instead?)

[Abnormalpingu]

Hello again !

Allright, I've recloned and rebuild flutter-pi as before, also changed to &> file.txt
I also double-checked if it output all the console text, which it does :)
I'm not really sure if debug is enabled correctly, as before I do the following as before in flutter-pi\build\ -
cmake .. -DCMAKE_BUILD_TYPE=Debug -DDEBUG_DRM_PLANE_ALLOCATIONS=On

Here's a paste of the output, hopefully this helps !

edit: uh, the formatting somehow doesn't want to cooperate when pasting (even as code), attached tmp2.txt to the post instead ;)

tmp2.txt

[ardera]

Is it possible you accidentally used the master branch this time? 😜 The output should definitely look different on the feature/compositor-ng branch. Try cloning with git clone -b feature/compositor-ng https://github.com/ardera/flutter-pi

What does git log in the cloned directory say for you?

[Abnormalpingu]

Doesn't look like it, but I've been known to do.. things :D

git log from what i tried yesterday - gitlog1.txt
just tried using git with '-b' and not '--single-branch --branch' options - gitlog2.txt

gitlog1.txt
gitlog2.txt

[ardera]

maybe you're somehow not executing the flutter-pi you've built? 😅 because that's definitely the flutter-pi master branch output

[Abnormalpingu]

You're absolutely correct, I'll stop git cloning other branches directly from the client, it obviously doesn't work for some reason.
I'll just download a zip and transfer it manually to the client to make sure in the future, sorry for the confusion :)

Got the expected output now;
log.txt

[ardera]

Okay, apparently the feature/compositor-ng branch had a bug and it didn't even reach to point where to problem ocurred previously. I fixed it now, can you try again with the latest commit?

And also record the dmesg log with:

echo 0x1FF | sudo tee /sys/module/drm/parameters/debug # Enable verbose DRM logging
sudo dmesg -C # Clear kernel logs
dmesg -w >dmesg.log & # Continuously write DRM logs to a file, in the background
flutter-pi ... # Run flutter-pi again, manually terminate it after a few errors have happened
fg # Kill dmesg with Ctrl+C
echo 0 | sudo tee /sys/module/drm/parameters/debug # Disable DRM logging

(and still configure with -DCMAKE_BUILD_TYPE=Debug -DDEBUG_DRM_PLANE_ALLOCATIONS=On too)

Thanks!

[Abnormalpingu]

Hi!
I see, well something interesting happened it seems, because it works now! :)
As you requested I updated to the new commit in compositor-ng branch, compiled with the debug flags and so on

I might also be having issues with an old nvidia client, I'll do some testing and report back, I'll be sure to use the correct branch and log as we've done here this time :P

Attaching the dmesg.log
dmesg.log

Thanks again!

[Abnormalpingu]

Hello again,

so I've been testing out the nvidia based client, that seems to have some issues as well (shocker :D)
One odd thing is that when running kmscube, it works fine, but when ctrl-c'ing the screen remains black, this might of course be unrelated to what we're trying to do. Just a strange occurence.
When starting the flutter-pi client the screen goes black, but it runs, so not sure what's going on here. I though it might be the resolution but the test monitor is the same and the resolution is still 1280x1024 when flutter-pi starts up. I tried starting it in 1024x768 and the same thing happened.

Main hw differences;
AMD Athlon(tm) II X2 250 CPU and GeForce 7025 / nForce 630a (rev a2)

Installed libdrm-nouveau2 which is for nvidia gpus (instead of libdrm-intel1 and libdrm-radeon1)
Using the compositor-ng branch (actually I've copied the actual build files from the amd client to make sure it's the same, then rebuilt on the nvidia client with the debug flags and so on)

I'm unable to post the drm-info to the db-site as the clients lives in a airgapped network, but i just piped the output to a file instead, hopefully this is ok.
drm_info.txt
modetest.txt
dmesg.log

This should be the last client we need, as these are the most numerous in the existing deployment, thanks again for your patience and help!

[ardera]

Ah, I know what causes this. Flutter-pi tries to use ARGB but your GPU only supports XRGB for scanout.
Can you try if it works with latest feature/compositor-ng?

[Abnormalpingu]

Hi,

Interesting, I downloaded the latest compositor-ng (manually) and built it on the client (with the debug flags) and I could for a split second see graphics, then it goes back to black, system is not crashing or hanging, but needs to be rebooted to regain a video output. So it's absolutely a step in the right direction :)
I also tried starting flutter-pi with the different XRGB-pixelformats manually without any difference (worth a shot)

I've attached the dmesg logfile in case you'd need it

dmesg.log

From the console I see some relevant text too;
Use the -d commandline parameter to specify the physical dimensions of your display.
display mode:
resolution: 1280 x 1024
refresh rate: 60.019740Hz
physical size: 0mm x 0mm
flutter device pixel ratio: 1.000000
pixel format: (any)
/home/test/flutter-pi/src/keyboard.c: Could not load keyboard configuration from "/etc/default/keyboard". Default keyboard config will be used. load_file: No such file or directory
/home/test/flutter-pi/src/flutter-pi.c: Info: Could not load flutter engine. dlopen("libflutter_engine.so.debug"): libflutter_engine.so.debug: cannot open shared object file: No such file or directory.
/home/test/flutter-pi/src/pluginregistry.c: Initialized plugins: services, text input, raw keyboard plugin, gstreamer video_player, audioplayers,
/home/test/flutter-pi/src/window.c: Flutter requested render surface before supplying surface dimensions.
/home/test/flutter-pi/src/gl_renderer.c: Choosing EGL config with pixel format ARGB 8:8:8:8...
flutter: The Dart VM service is listening on http://127.0.0.1:38917/toBTcVMAOwE=/
flutter: Main.dart file:///home/test/
flutter: GAMEMANAGER.INITIATE
flutter: ApplicationMenuViewModel._initiate
flutter: Trying to get audio implementation platform linux
flutter: AudioImplementationLinuxLibao
ALSA lib pcm_hw.c:1829:(_snd_pcm_hw_open) Invalid value for card
couldn't open play stream: No such file or directory
flutter: Automuting sound on startup to avoid mental issues
/home/test/flutter-pi/src/modesetting.c: checking if plane with id 37 qualifies...
/home/test/flutter-pi/src/modesetting.c: does qualify.
flutter: ConnectionState HubConnectionState.Connecting
../../third_party/skia/src/gpu/ganesh/SurfaceDrawContext.cpp:2110: fatal error: "assert(copy)"
Illegal instruction

[ardera]

../../third_party/skia/src/gpu/ganesh/SurfaceDrawContext.cpp:2110: fatal error: "assert(copy)"
Illegal instruction

That seems like a GL driver problem :/
Only way to debug this would be to build the flutter engine with debug symbols and step through the relevant skia code. I'm currently building the 3.10.4 engine with debug symbols here: https://github.com/ardera/flutter-ci/actions/runs/5242452070?pr=5

Ah wait, you're on x86, in that case my engine binaries probably won't help. But if you're building the engine yourself you can build it with debug symbols by adding --no-stripped --unoptimized --runtime-mode debug to the gn invocation

[Abnormalpingu]

Gotcha, well, we've tried building the engine-binaries for x64 and we seem to have issues getting this to build correctly..
Would it be possible for you to build these binaries for x64 and provide them to us maybe? If you should need somewhere to store them we can provide somewhere to do that (or whatever works for you).
Just for our temporary testing of course, we'd look further into building our own at a later point, just to save some time in our troubleshooting efforts.

We also upgraded to flutter 3.10.4 and rebuilt our app.
When we build the engine we only find one libflutter_engine.so file, I see in your builds you get them with .debug, .release and so forth. We can also find the flutter_embedder.h which we tried installing, as well as renaming the so-files. Not sure if when you build the engine the output is the same as the folder/file-structure of the git or you pick out the files manually and push them.

[ardera]

Sure, here: https://1drv.ms/u/s!AsNHmDqMr-aVg65exuQp7lRxiSmsuA?e=Lsh22W

It's built with:

./src/flutter/tools/gn \
  --runtime-mode debug \
  --embedder-for-target \
  --disable-desktop-embeddings \
  --no-build-glfw-shell \
  --no-build-embedder-examples \
  --no-goma \
  --unoptimized --no-stripped

and some custom modifications so it's built with -ggdb and -fdebug-default-version=4 (some gdbs have issues with DWARF v5) instead of -g2:

diff --git a/build/config/compiler/BUILD.gn b/build/config/compiler/BUILD.gn
index 4c9a1ee..b74ab87 100644
--- a/build/config/compiler/BUILD.gn
+++ b/build/config/compiler/BUILD.gn
@@ -951,7 +951,7 @@ config("symbols") {
     }
     ldflags = [ "/DEBUG" ]
   } else {
-    cflags = [ "-g2" ]
+    cflags = [ "-ggdb", "-fdebug-default-version=4" ]
   }
 }