[webgl] enabling WEBGL_USE_SHAPES_UNIFORMS results in model execution returning null values

[vladmandic]

user reported issue with my library and after a while it was traced down to WEBGL_USE_SHAPES_UNIFORMS
which i have enabled by default for over half a year now due to massive performance advantages

issue itself is that model execution shows no issues and it returns tensors in expected shape,
but tensor data is an array with all null values instead of expected float32 values

what is strange that on majority of systems there are no issues (plus i cannot reproduce locally)
and there is only single (quite small) affected model out of 10+ used

affected model

• https://vladmandic.github.io/human/models/faceres.json

very simple code reproduction (again, i cannot reproduce myself)
(it runs a model using predefined image input and checks output validity)

• https://vladmandic.github.io/human/test/issue-faceres.html?default: returns arrays with correct float values
• https://vladmandic.github.io/human/test/issue-faceres.html?uniforms: returns array with all null values

test has tfjs debug enabled and you can see all ops in the browser console

environment:

• tfjs: 3.20.0 (older versions are also affected)
• browser: chrome 104 and 105
• os: windows 10 pro

it's reported on two different systems using different graphics adapters (AMD Radeon and nVidia RTX)
both affected systems report no issues looking at chrome://gpu and webgl v2 is working fine

only other thing worth noting is that user is using non-latin os locale

link to original issue: vladmandic/human#291

[qjia7]

@vladmandic I also can't reproduce it on my windows machine.
@shurshilov I need your help to narrow down this issue to see which op's uniforms have the problem.
Can you paste the console info in https://vladmandic.github.io/human/test/issue-faceres.html?default and https://vladmandic.github.io/human/test/issue-faceres.html?uniforms separately so that we may know from which op it becomes abnormal?
And I am also curious if all models will be impacted by WEBGL_USE_SHAPES_UNIFORMS in your environment or only faceres?

And please also follow below steps to verify whether the uniform APIs for webgl1 works well in your system:

1. Open https://registry.khronos.org/webgl/sdk/tests/webgl-conformance-tests.html?version=1.0.4 for webgl1 and https://registry.khronos.org/webgl/sdk/tests/webgl-conformance-tests.html?version=2.0.1 for webgl2
2. Search all/conformance/uniforms and click the run button before it. Then see if all tests are pass.

[vladmandic]

@qjia7 i can confirm that library runs multiple models before this one and all return expected results - this is the only affected model out of several.

[shurshilov]

https://vladmandic.github.io/human/test/issue-faceres.html?uniforms
Hi, sorry for delay
vladmandic.github.io-1662639126523.log
vladmandic.github.io-1662639083084.log

[vladmandic]

@qjia7 just a quick look - while there are no errors in default run (pass), when uniforms are enabled (fail) there are 160 instances of Found NaN in the result of '<op>'

[qjia7]

@vladmandic I see the first NaN op is Minimum in the log. I assume it's the first error happened. The second parameter of Minimum is a scalar. So I made some changes to process the scalar in #6828. And also added some env flags to specifically enable/disable some ops (unary ops, binary ops, encoded matrix)' uniforms. I need your help to apply PR #6828 and build it. Then use this webgl in your test and let @shurshilov have another try.

Here I may need you do several things (All below tests are still based on WEBGL_USE_SHAPES_UNIFORMS enabled).

1. Directly apply PR webgl: Fix NaN issue #6828 and try it. In this version, it only do the scalar changes. All the new added flags keep unchanged. If it works, it means the error is caused by scalar.
2. Based on 1), disable WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS. And test again. If it works, it means the error is caused by EncodeMatrixPackedProgram.
3. Based on 2), disable WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS. And test again. If it works, it means the error is caused by binary ops.
4. Based on 3), disable WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS. And test again. If it works, it means the error is caused by unary ops.

And please help dump the test log info in each step.

PS. I will be OOO until Sep, 13.

[vladmandic]

ive built the custom tfjs from @qjia7 branch and created tests at https://github.com/vladmandic/tfjs-fix-uniforms

note that tests use url parser to set tfjs flags so you can try any combo of flags
but what was asked here is listed in the readme

@shurshilov please run tests in live tests section and upload the each log from browser console here

[shurshilov]

https://vladmandic.github.io/tfjs-fix-uniforms/index.html
[vl
vladmandic.github.io-1662733013714.log
https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true
vladmandic.github.io-1662733064991.log
https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=false
vladmandic.github.io-1662733166647.log
https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=false&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=false&WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=false
vladmandic.github.io-1662733190344.log

[shurshilov]

https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=false&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=false
vladmandic.github.io-1662733322570.log

[vladmandic]

@qjia7 based on cursory look, issue seems to be in in
BinaryOpPackedProgram which relies on pre-set outShape for checks and then calls standard BinaryOpProgram

looking at logs:

• pass: default
• fail: uniforms=true
• fail: uniforms=true, matrix=false
• pass: uniforms=true, matrix=false, binary=false
• pass: uniforms=true, matrix=false, binary=false, unary=false

@shurshilov to have a confirmation, can you try with just that single variation disabled?

https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=false&WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true

[qjia7]

@vladmandic @shurshilov Thanks for helping digging this. Now I have strong feeling that the NaN data is caused by Minimum/Maximum's isnan check. In this CHECK_NAN_SNIPPET_PACKED, you can see when isNaN.xxx is true, it will directly wirte NAN result.

To confirm this, @vladmandic can you also try to disable WEBGL_PACK? Once WEBGL_PACK is disabled, it will call unpacked program. For Minimum, the CHECK_NAN_SNIPPE will keep A or B's value instead of write NaN. In this case, I suppose the result will not be all NaN but some incorrect values.

If above assumption is correct, maybe just one small test Minimum with the same shape a: 4D 1,112,112,64 b: 0D in the model can reproduce this issue.

[vladmandic]

@qjia7 sure, any flag can just be added to url, so those tests are easy.
@shurshilov tests that should run (ignore previous one, its included here):

1. all uniforms enabled except binary shapes
   https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=false&WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true
2. all uniforms enabled with webgl pack disabled
   https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=true&WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true&WEBGL_PACK=false
3. all uniforms enabled with webgl pack for binary ops disabled
   https://vladmandic.github.io/tfjs-fix-uniforms/index.html?WEBGL_USE_SHAPES_UNIFORMS=true&WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true&WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=true&WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true&WEBGL_PACK=true&WEBGL_PACK_BINARY_OPERATIONS=false

[shurshilov]

2.

3.

[vladmandic]

@qjia7 seems your hunch is on track, closest isolation is WEBGL_PACK_BINARY_OPERATIONS=false - this results in no errors

1. all uniforms enabled except binary shapes

test completes with NaN values, but the values are incorrect!
this is a strange one as model completes without errors, but values returned are waaay off.

WEBGL_USE_SHAPES_UNIFORMS=true
WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true
WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=false
WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true

2. all uniforms enabled with webgl pack disabled

test completes and results are correct

WEBGL_USE_SHAPES_UNIFORMS=true
WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true
WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=true
WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true
WEBGL_PACK=false

3. all uniforms enabled with webgl pack for binary ops disabled

test completes and results are correct

WEBGL_USE_SHAPES_UNIFORMS=true
WEBGL_ENABLE_ENCODE_MATRIX_SHAPES_UNIFORMS=true
WEBGL_ENABLE_BINARY_SHAPES_UNIFORMS=true
WEBGL_ENABLE_UNARY_SHAPES_UNIFORMS=true
WEBGL_PACK=true
WEBGL_PACK_BINARY_OPERATIONS=false

btw, although we're getting closer to understanding the issue, i still have no idea why its manifesting only on some systems?