WASM build #262

agriyakhetarpal · 2025-02-27T22:29:30Z

Closes #234

agriyakhetarpal · 2025-02-27T22:33:08Z

I've been testing it a bit through a PR in my fork, and it all builds well, up to one point in the Cython sources where it fails in step 107/114: https://github.com/agriyakhetarpal/python-flint/actions/runs/13576857716/job/37954995494?pr=1

with the following error trace:

FAILED: src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o 
/tmp/tmp8t79rr74/cc -Isrc/flint/types/_gr.cpython-312-wasm32-emscripten.so.p -Isrc/flint/types -I../src/flint/types -I/opt/hostedtoolcache/Python/3.12.9/x64/include/python3.12 -I/home/runner/work/python-flint/python-flint/wasm-library-dir/include -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o -MF src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o.d -o src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o -c src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c
src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c:19051:65: error: incompatible integer to pointer conversion passing 'mp_limb_t' (aka 'unsigned long') to parameter of type 'const fmpz *' (aka 'const long *') [-Wint-conversion]
 19051 |   gr_ctx_init_fq_nmod(__pyx_v_ctx->__pyx_base.__pyx_base.ctx_t, __pyx_v_p, __pyx_v_d, __pyx_v_name);
       |                                                                 ^~~~~~~~~
/home/runner/work/python-flint/python-flint/wasm-library-dir/include/flint/gr.h:1366:53: note: passing argument to parameter 'p' here
 1366 | void gr_ctx_init_fq_nmod(gr_ctx_t ctx, const fmpz_t p, slong d, const char * var);
      |                                                     ^
src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c:19150:65: error: incompatible integer to pointer conversion passing 'mp_limb_t' (aka 'unsigned long') to parameter of type 'const fmpz *' (aka 'const long *') [-Wint-conversion]
 19150 |   gr_ctx_init_fq_zech(__pyx_v_ctx->__pyx_base.__pyx_base.ctx_t, __pyx_v_p, __pyx_v_d, __pyx_v_name);
       |                                                                 ^~~~~~~~~
/home/runner/work/python-flint/python-flint/wasm-library-dir/include/flint/gr.h:1367:53: note: passing argument to parameter 'p' here
 1367 | void gr_ctx_init_fq_zech(gr_ctx_t ctx, const fmpz_t p, slong d, const char * var);
      |                                                     ^
2 errors generated.

agriyakhetarpal · 2025-02-27T23:05:08Z

FWIW; I hardly have any Cython development experience, so I will have to wrap my head around this a bit, unless someone has an idea for a fix. The basis for the failure here is that WASM is stricter about type conversions than compilation for other platforms, which is why this mismatch is causing errors.

agriyakhetarpal · 2025-02-27T23:07:22Z

Okay – the cause of the error, more specifically, is that gr_fq_nmod_ctx and gr_fq_zech_ctx pass a ulong parameter p

python-flint/src/flint/types/_gr.pxd

Lines 1189 to 1202 in 34dbb5d

    
           @cython.no_gc 
        
           cdef class gr_fq_zech_ctx(gr_scalar_ctx): 
        
               cdef ulong p 
        
               cdef slong d 
        
               @staticmethod 
        
               cdef inline gr_fq_zech_ctx _new(ulong p, slong d, char* name): 
        
                   cdef gr_fq_zech_ctx ctx 
        
                   ctx = gr_fq_zech_ctx.__new__(gr_fq_zech_ctx) 
        
                   ctx.p = p 
        
                   ctx.d = d 
        
                   gr_ctx_init_fq_zech(ctx.ctx_t, p, d, name) 
        
                   ctx._init = True 
        
                   return ctx

but flint/gr.h has fmpz_t in version 3.0.1: https://github.com/flintlib/flint/blob/c05007be863b1aae7221f11a9c135d673d968638/src/gr.h#L1366-L1367

and this has apparently been fixed in version v3.1.0. I'll try updating flint to v3.1.0 (before proceeding to try out v3.1.2, if I do)

agriyakhetarpal · 2025-02-27T23:32:30Z

Successful build: https://github.com/agriyakhetarpal/python-flint/actions/runs/13577635441?pr=1 ✨

The next task is to run tests, which I can take a look at later in the day.

oscarbenjamin · 2025-02-28T00:02:20Z

I haven't looked through this yet but thanks!

oscarbenjamin · 2025-02-28T00:04:53Z

but flint/gr.h has fmpz_t in version 3.0.1:

Best to keep with latest FLINT version for the pyodide build. There is no need to support a range of versions in pyodide. Generally python-flint tracks the latest version but has some #ifdef type stuff for older versions so that someone could build against them. No one is going to build against old versions in WASM though.

agriyakhetarpal · 2025-02-28T00:11:15Z

Sounds good to me! I think the tests will have a bunch of function signature mismatches, though 😬 based on what I noticed through my fork's run. Also, building itself takes ~20 minutes in total – we could get to half of that if we have prebuilt WASM binaries. I guess they could be hosted somewhere in a GitHub release when things are ready.

agriyakhetarpal · 2025-03-03T11:34:51Z

This can take a bit of time. I'm disabling tests and running it in on my fork to see how widespread the signature mismatches are, and I managed to get test_all.py to run till the end – now we have some more in test_docstrings.py. My next tasks will be to debug these mismatches, which I haven't had fun doing, as, i. compilation of these libraries to WASM is messy on a macOS and ii. a test here contains several assert statements, instead of several tests containing a few of them. Please bear with me as I proceed with this slowly and steadily. We could also wholly ignore test_docstrings.py if you're not too keen on that being tested. (I think it's good to test, though).

oscarbenjamin · 2025-03-03T11:46:48Z

I don't quite understand what the problem with the tests is. Is it just caused by the Cython/C signatures not matching?

There is now FLINT 3.2.0-rc1. We could perhaps update all the Cython declarations to that version and use that for the WASM build. I'm not sure when 3.2.0 final will be released but we would want to update to that straight away when it is.

agriyakhetarpal · 2025-03-03T12:30:18Z

I don't quite understand what the problem with the tests is. Is it just caused by the Cython/C signatures not matching?

Yes, majorly. If there's even a slight mismatch between the upstream signatures for Flint and the ones we have here, WASM's type safety guarantees that it will terminate the program, which makes Pyodide raise a fatal error. These can happen at the time of linking (like when it happened with version 3.0.1 above), or at runtime, like we are facing now. I have to drop these completely instead of xfailing them as the running Pyodide interpreter is no longer possible to be used after it has encountered fatal errors.

There are so many of these with SciPy, especially from its wrapping of Fortran libraries that we can't compile to WASM directly without having to f2c them over. I hope the new Flang from LLVM-19 can do this better: pyodide/pyodide#5268

There is now FLINT 3.2.0-rc1. We could perhaps update all the Cython declarations to that version and use that for the WASM build. I'm not sure when 3.2.0 final will be released but we would want to update to that straight away when it is.

Yes, this sounds good to me, and I hope they've fixed a few of these upstream. I could start another PR for this when I have enough time to do so; I assume it will help me learn Cython a bit. :)

oscarbenjamin · 2025-03-03T12:36:43Z

I've opened gh-264 to bump the FLINT version to 3.2.0-rc1 and update all of the Cython declarations.

The Cython declarations are set from the FLINT docs by running the bin/all_rst_to_pxd.py script. There can often be mismatches resulting from parsing the signatures from the docs rather than the actual C code e.g. the docs may be inaccurate but usually in the Cython -> C -> compiled extension modules the differences even out largely because a lot of FLINT's functions are not actually called directly by python-flint.

oscarbenjamin · 2025-03-05T19:07:22Z

I've opened gh-264 to bump the FLINT version to 3.2.0-rc1 and update all of the Cython declarations.

I've just merged this to main. If you rebase/merge with main then this PR will get the updated declarations.

There might still be some wrong declarations that we can fix manually for now but then upstream to FLINT afterwards.

oscarbenjamin · 2025-05-26T18:18:01Z

This is what I got from emmake make tests, it didn't succeed:

Maybe this can fix the immediate blocker:

diff --git a/src/mag/test/t-d_log_lower_bound.c b/src/mag/test/t-d_log_lower_bound.c
index 27c8bd4d3..94a8f2175 100644
--- a/src/mag/test/t-d_log_lower_bound.c
+++ b/src/mag/test/t-d_log_lower_bound.c
@@ -16,8 +16,8 @@
 
 /* Defined in t-d_log_lower_bound.c, t-d_log_upper_bound.c, t-set_d_2exp_fmpz.c
  * and t-set_d.c */
-#ifndef d_randtest2
-#define d_randtest2 d_randtest2
+#ifndef _d_randtest2
+#define _d_randtest2
 /* XXX: d_randtest is not good enough */
 
 #define EXP_MINUS_32 2.3283064365386962891e-10
diff --git a/src/mag/test/t-d_log_upper_bound.c b/src/mag/test/t-d_log_upper_bound.c
index bf01748f5..4f5126742 100644
--- a/src/mag/test/t-d_log_upper_bound.c
+++ b/src/mag/test/t-d_log_upper_bound.c
@@ -16,8 +16,8 @@
 
 /* Defined in t-d_log_lower_bound.c, t-d_log_upper_bound.c, t-set_d_2exp_fmpz.c
  * and t-set_d.c */
-#ifndef d_randtest2
-#define d_randtest2 d_randtest2
+#ifndef _d_randtest2
+#define _d_randtest2
 /* XXX: d_randtest is not good enough */
 
 #define EXP_MINUS_32 2.3283064365386962891e-10
diff --git a/src/mag/test/t-set_d.c b/src/mag/test/t-set_d.c
index f112ccc27..0a560a0af 100644
--- a/src/mag/test/t-set_d.c
+++ b/src/mag/test/t-set_d.c
@@ -16,8 +16,8 @@
 
 /* Defined in t-d_log_lower_bound.c, t-d_log_upper_bound.c, t-set_d_2exp_fmpz.c
  * and t-set_d.c */
-#ifndef d_randtest2
-#define d_randtest2 d_randtest2
+#ifndef _d_randtest2
+#define _d_randtest2
 /* XXX: d_randtest is not good enough */
 
 #define EXP_MINUS_32 2.3283064365386962891e-10
diff --git a/src/mag/test/t-set_d_2exp_fmpz.c b/src/mag/test/t-set_d_2exp_fmpz.c
index ff402e8e1..14622f477 100644
--- a/src/mag/test/t-set_d_2exp_fmpz.c
+++ b/src/mag/test/t-set_d_2exp_fmpz.c
@@ -16,8 +16,8 @@
 
 /* Defined in t-d_log_lower_bound.c, t-d_log_upper_bound.c, t-set_d_2exp_fmpz.c
  * and t-set_d.c */
-#ifndef d_randtest2
-#define d_randtest2 d_randtest2
+#ifndef _d_randtest2
+#define _d_randtest2
 /* XXX: d_randtest is not good enough */
 
 #define EXP_MINUS_32 2.3283064365386962891e-10

oscarbenjamin · 2025-05-26T19:10:20Z

For the remaining test failures then we have this one in test_fq_default_poly:

python-flint/src/flint/test/test_all.py

Line 4731 in 590c7d6

assert raises(lambda: f.inverse_mod(2*f), ValueError)

That implies that this function is not raising ValueError which is worrying:

After 60c291e I can reproduce the test_fq_default failure locally (in a non-WASM build) as now shown in CI but only with when running python -m flint.test and not via pytest or spin test.

I think that this is something else, not directly related to emscripten/WASM and it is probably to do with caching and comparison of context objects...

agriyakhetarpal · 2025-05-26T20:16:13Z

This is what I got from emmake make tests, it didn't succeed:

Maybe this can fix the immediate blocker:

🤔 this had no effect, unfortunately. It's also possible that I may have omitted a --disable-shared somewhere, but I can try again tomorrow…

oscarbenjamin · 2025-05-26T20:22:00Z

This is what I got from emmake make tests, it didn't succeed:

Maybe this can fix the immediate blocker:

🤔 this had no effect, unfortunately.

When you get a chance this is my simpler idea:

diff --git a/src/mag/test/main.c b/src/mag/test/main.c
index 4549b0155..56274b90b 100644
--- a/src/mag/test/main.c
+++ b/src/mag/test/main.c
@@ -24,8 +24,6 @@
 #include "t-cosh.c"
 #include "t-div.c"
 #include "t-div_lower.c"
-#include "t-d_log_lower_bound.c"
-#include "t-d_log_upper_bound.c"
 #include "t-dump_file.c"
 #include "t-dump_str.c"
 #include "t-exp.c"
@@ -54,8 +52,6 @@
 #include "t-root.c"
 #include "t-rsqrt.c"
 #include "t-rsqrt_lower.c"
-#include "t-set_d_2exp_fmpz.c"
-#include "t-set_d.c"
 #include "t-set_ui.c"
 #include "t-set_ui_lower.c"
 #include "t-sinh.c"

It looks like your pretty close with make tests. It builds about 100 test programs and you have about 80.

agriyakhetarpal · 2025-05-26T20:29:14Z

Yes, I think not compiling the files is fine for now ;)

Some more mismatches popped up, if this helps:

make: make tests
  CC  mag/test/main.c
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:78:5: error: use of undeclared identifier 'test_mag_d_log_lower_bound'; did you mean 'mag_d_log_lower_bound'?
   78 |     TEST_FUNCTION(mag_d_log_lower_bound),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:168:1: note: expanded from here
  168 | test_mag_d_log_lower_bound
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/mag.h:538:8: note: 'mag_d_log_lower_bound' declared here
  538 | double mag_d_log_lower_bound(double x);
      |        ^
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:79:5: error: use of undeclared identifier 'test_mag_d_log_upper_bound'; did you mean 'mag_d_log_upper_bound'?
   79 |     TEST_FUNCTION(mag_d_log_upper_bound),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:171:1: note: expanded from here
  171 | test_mag_d_log_upper_bound
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/mag.h:537:8: note: 'mag_d_log_upper_bound' declared here
  537 | double mag_d_log_upper_bound(double x);
      |        ^
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:108:5: error: use of undeclared identifier 'test_mag_set_d_2exp_fmpz'; did you mean 'test_mag_add_2exp_fmpz'?
  108 |     TEST_FUNCTION(mag_set_d_2exp_fmpz),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:258:1: note: expanded from here
  258 | test_mag_set_d_2exp_fmpz
      | ^~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/t-add_2exp_fmpz.c:16:1: note: 'test_mag_add_2exp_fmpz' declared here
   16 | TEST_FUNCTION_START(mag_add_2exp_fmpz, state)
      | ^
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:62:5: note: expanded from macro 'TEST_FUNCTION_START'
   62 | int CAT(test, label)(void)                              \
      |     ^
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^
<scratch space>:272:1: note: expanded from here
  272 | test_mag_add_2exp_fmpz
      | ^
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:109:5: error: use of undeclared identifier 'test_mag_set_d'; did you mean 'test_mag_get_d'?
  109 |     TEST_FUNCTION(mag_set_d),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:261:1: note: expanded from here
  261 | test_mag_set_d
      | ^~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/t-get_d.c:17:1: note: 'test_mag_get_d' declared here
   17 | TEST_FUNCTION_START(mag_get_d, state)
      | ^
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:62:5: note: expanded from macro 'TEST_FUNCTION_START'
   62 | int CAT(test, label)(void)                              \
      |     ^
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^
<scratch space>:17:1: note: expanded from here
   17 | test_mag_get_d
      | ^
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:78:5: error: incompatible function pointer types initializing 'int (*)(void)' with an expression of type 'double (double)' [-Wincompatible-function-pointer-types]
   78 |     TEST_FUNCTION(mag_d_log_lower_bound),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:168:1: note: expanded from here
  168 | test_mag_d_log_lower_bound
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c:79:5: error: incompatible function pointer types initializing 'int (*)(void)' with an expression of type 'double (double)' [-Wincompatible-function-pointer-types]
   79 |     TEST_FUNCTION(mag_d_log_upper_bound),
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/test_helpers.h:59:32: note: expanded from macro 'TEST_FUNCTION'
   59 | #define TEST_FUNCTION(label) { CAT(test, label), #label }
      |                                ^~~~~~~~~~~~~~~~
/Users/agriyakhetarpal/Desktop/flint/src/templates.h:15:18: note: expanded from macro 'CAT'
   15 | #define CAT(X,Y) X##_##Y
      |                  ^~~~~~~
<scratch space>:171:1: note: expanded from here
  171 | test_mag_d_log_upper_bound
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~
6 errors generated.
emcc: error: '/Users/agriyakhetarpal/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/Users/agriyakhetarpal/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -Wall -Werror=newline-eof -Werror=implicit-function-declaration -std=c11 -pedantic -O3 -g3 -I/Users/agriyakhetarpal/Desktop/flint/src -Isrc -I/Users/agriyakhetarpal/Desktop/flint/wasm_libs/include -I/Users/agriyakhetarpal/Desktop/flint/wasm_libs/include -DBUILDING_FLINT -MMD -MP -MF build/mag/test/main.d -c /Users/agriyakhetarpal/Desktop/flint/src/mag/test/main.c -o /var/folders/b3/2bq1m1_50bs4c7305j8vxcqr0000gn/T/emscripten_temp_2bwp0ahc/main_0.o' failed (returned 1)
make: *** [build/mag/test/main] Error 1
emmake: error: 'make tests' failed (returned 2)

Otherwise, I shall build/test from a clean state again.

oscarbenjamin · 2025-05-26T20:35:15Z

More deletions are needed. This time I checked it compiles and runs:

diff --git a/src/mag/test/main.c b/src/mag/test/main.c
index 4549b0155..0789414fc 100644
--- a/src/mag/test/main.c
+++ b/src/mag/test/main.c
@@ -24,8 +24,6 @@
 #include "t-cosh.c"
 #include "t-div.c"
 #include "t-div_lower.c"
-#include "t-d_log_lower_bound.c"
-#include "t-d_log_upper_bound.c"
 #include "t-dump_file.c"
 #include "t-dump_str.c"
 #include "t-exp.c"
@@ -54,8 +52,6 @@
 #include "t-root.c"
 #include "t-rsqrt.c"
 #include "t-rsqrt_lower.c"
-#include "t-set_d_2exp_fmpz.c"
-#include "t-set_d.c"
 #include "t-set_ui.c"
 #include "t-set_ui_lower.c"
 #include "t-sinh.c"
@@ -79,8 +75,6 @@ test_struct tests[] =
     TEST_FUNCTION(mag_cosh),
     TEST_FUNCTION(mag_div),
     TEST_FUNCTION(mag_div_lower),
-    TEST_FUNCTION(mag_d_log_lower_bound),
-    TEST_FUNCTION(mag_d_log_upper_bound),
     TEST_FUNCTION(mag_dump_file),
     TEST_FUNCTION(mag_dump_str),
     TEST_FUNCTION(mag_exp),
@@ -109,8 +103,6 @@ test_struct tests[] =
     TEST_FUNCTION(mag_root),
     TEST_FUNCTION(mag_rsqrt),
     TEST_FUNCTION(mag_rsqrt_lower),
-    TEST_FUNCTION(mag_set_d_2exp_fmpz),
-    TEST_FUNCTION(mag_set_d),
     TEST_FUNCTION(mag_set_ui),
     TEST_FUNCTION(mag_set_ui_lower),
     TEST_FUNCTION(mag_sinh),

agriyakhetarpal · 2025-05-26T21:36:11Z

Ah, yes, this time it compiles. Here is the output from emmake make check:

make: make check
add_ssaaaa...
add_ssaaaa                                        0.00   (PASS)
add_sssaaaaaa...
add_sssaaaaaa                                     0.09   (PASS)
flint_clz...
flint_clz                                         0.00   (PASS)
flint_ctz...
flint_ctz                                         0.00   (PASS)
flint_fprintf...
flint_fprintf                                     0.00   (FAIL)
Could not open temporary file "tmp"
Aborted()
/Users/agriyakhetarpal/Desktop/flint/build/test/main:380
  var e = new WebAssembly.RuntimeError(what);
          ^

RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
    at abort (/Users/agriyakhetarpal/Desktop/flint/build/test/main:380:11)
    at __abort_js (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3388:7)
    at main.wasm.abort (wasm://wasm/main.wasm-083fa0d6:wasm-function[8641]:0x4647b4)
    at main.wasm.flint_abort (wasm://wasm/main.wasm-083fa0d6:wasm-function[40]:0x90a0)
    at main.wasm.main (wasm://wasm/main.wasm-083fa0d6:wasm-function[38]:0x9010)
    at Module._main (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3728:102)
    at callMain (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3754:15)
    at doRun (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3793:24)
    at run (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3806:5)
    at removeRunDependency (/Users/agriyakhetarpal/Desktop/flint/build/test/main:348:7)

Node.js v20.18.0
make: *** [build/test/main_TEST_RUN] Error 1
emmake: error: 'make check' failed (returned 2)

The failing test is fine; we can't access the file system without mounting a NodeFS or a similar one. I don't think we should dwell on it.

Note that I had to manually modify the generated Makefile as it did not interpret that build/test/main was a JavaScript file, so I had to prepend node before the %_TEST_RUN: % rules. It should build main.js if the HOST_OS matches emscripten and use Node.js in that case by default – but that was just some misconfiguration that we can deal with. At least from this, we can see that most tests pass and WASM compilation works.

~~I hope we are not missing any tests, though.~~ Looks like we are :/

oscarbenjamin · 2025-05-26T21:44:51Z

~~I hope we are not missing any tests, though.~~ Looks like we are :/

Yes, usually there are more than 5 tests!

agriyakhetarpal · 2025-05-27T08:29:53Z

I added NODERAWFS=1 to my TESTCFLAGS for allowing access to my file system, and this has let the tests go forward much more:

make: make check
add_ssaaaa...
add_ssaaaa                                        0.00   (PASS)
add_sssaaaaaa...
add_sssaaaaaa                                     0.04   (PASS)
flint_clz...
flint_clz                                         0.00   (PASS)
flint_ctz...
flint_ctz                                         0.00   (PASS)
flint_fprintf...
flint_fprintf                                     0.01   (PASS)
flint_printf...
flint_printf                                          (SKIPPED)
memory_manager...
memory_manager                                    0.00   (PASS)
sdiv_qrnnd...
sdiv_qrnnd                                        0.01   (PASS)
smul_ppmm...
smul_ppmm                                         0.01   (PASS)
flint_sort...
flint_sort                                        0.00   (PASS)
sub_dddmmmsss...
sub_dddmmmsss                                     0.05   (PASS)
sub_ddmmss...
sub_ddmmss                                        0.00   (PASS)
udiv_qrnnd...
udiv_qrnnd                                        0.01   (PASS)
udiv_qrnnd_preinv...
udiv_qrnnd_preinv                                 0.00   (PASS)
umul_ppmm...
umul_ppmm                                         0.01   (PASS)
thread_pool...
thread_pool                                       0.10   (PASS)
thread_support_parallel_binary_splitting...
thread_support_parallel_binary_splitting          0.00   (PASS)
thread_support_parallel_do...
thread_support_parallel_do                        0.00   (PASS)
n_addmod...
n_addmod                                          0.00   (PASS)
n_cbrt_binary_search...
n_cbrt_binary_search                              0.00   (PASS)
n_cbrt...
n_cbrt                                            0.01   (PASS)
n_cbrt_chebyshev_approx...
n_cbrt_chebyshev_approx                           0.00   (PASS)
n_cbrtrem...
n_cbrtrem                                         0.01   (PASS)
n_clog_2exp...
n_clog_2exp                                       0.00   (PASS)
n_clog...
n_clog                                            0.00   (PASS)
compute_primes...
Aborted(OOM)
/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:380
  var e = new WebAssembly.RuntimeError(what);
          ^

RuntimeError: Aborted(OOM). Build with -sASSERTIONS for more info.
    at abort (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:380:11)
    at abortOnCannotGrowMemory (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:3949:7)
    at _emscripten_resize_heap (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:3955:7)
    at main.wasm.sbrk (wasm://wasm/main.wasm-084fa6f2:wasm-function[8876]:0x4812bd)
    at main.wasm.emscripten_builtin_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[8868]:0x47f4c6)
    at main.wasm._flint_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[155]:0x186f6)
    at main.wasm.flint_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[147]:0x1850f)
    at main.wasm.n_compute_primes (wasm://wasm/main.wasm-084fa6f2:wasm-function[189]:0x1a0b1)
    at main.wasm.n_primes_arr_readonly (wasm://wasm/main.wasm-084fa6f2:wasm-function[271]:0x1fe54)
    at main.wasm.test_compute_primes (wasm://wasm/main.wasm-084fa6f2:wasm-function[27]:0x50f5)

Node.js v20.18.0
make: *** [build/ulong_extras/test/main_TEST_RUN] Error 1
emmake: error: 'make check' failed (returned 2)

The relevant section of the main file that Emscripten has set up looks like this (lines 3944–3956), but I don't think this helps much:

  var getHeapMax = () =>
      HEAPU8.length;
  var _emscripten_get_heap_max = () => getHeapMax();

  var abortOnCannotGrowMemory = (requestedSize) => {
      abort('OOM');
    };
  var _emscripten_resize_heap = (requestedSize) => {
      var oldSize = HEAPU8.length;
      // With CAN_ADDRESS_2GB or MEMORY64, pointers are already unsigned.
      requestedSize >>>= 0;
      abortOnCannotGrowMemory(requestedSize);
    };

And from the CI jobs for FLINT, I see that none of the nmod_vec_, flint_mpn, fmpz, fmpq, arb, gr_ tests ran; this isn't the full test suite either, so it would be nice if there were a pytest-like way to skip tests that we know are going to abort the program without having to recompile everything, or to allow a selective portion of the tests of our choice to run, similar to the -k option.

oscarbenjamin · 2025-05-27T10:09:55Z

it would be nice if there were a pytest-like way to skip tests that we know are going to abort the program without having to recompile everything, or to allow a selective portion of the tests of our choice to run, similar to the -k option.

I'm not sure what options there are but you can just run the individual test files like:

$ build/acb/test/main
acb_acos...
acb_acos                                          0.00   (PASS)
acb_acosh...
acb_acosh                                         0.00   (PASS)
acb_agm1...
acb_agm1                                          0.15   (PASS)
acb_agm...
acb_agm                                           0.09   (PASS)
...

You can list the hundred or so test executables and maybe put them in a script and comment out the ones you don't want:

$ ls build/*/test/main
build/acb_calc/test/main               build/fq_default_mat/test/main
build/acb_dft/test/main                build/fq_default_poly_factor/test/main
build/acb_dirichlet/test/main          build/fq_default_poly/test/main
build/acb_elliptic/test/main           build/fq_default/test/main
...

Given the things that currently don't work in this PR I would be most interested in acb_modular, nf, nf_elem, and the gr_* tests.

oscarbenjamin · 2025-05-27T11:35:55Z

Okay, so the CI job passes now. I think that a good milestone here would be to get something merged so that we have emscripten running in CI and nightly wheels being uploaded even if there are some outstanding known issues.

I suggest we:

Preserve this PR with its many commits and long discussion intact for posterity.
Open a new PR that squashes the commits and does any remaining clean ups.
Then I will follow up after with something to clean up the test changes and come up with a better solution for the doctests that are skipped or sorting of polynomial factors.

Lastly what remains is to see if we can identify and fix the problems with acb_modular_hilbert_class_poly and gr_nf but unless we can quickly identify the issue there I think it is better to leave that for later.

agriyakhetarpal · 2025-05-27T13:12:36Z

Sounds good to me! I will create another PR from here and also obtain some additional WASM test results from FLINT-main in a new issue – hopefully by tonight or early tomorrow. As most of the tests pass, I think the next steps, after yours, are to:

get patches for updated versions of FLINT and python-flint accordingly to https://github.com/pyodide/pyodide-recipes/ so that we can ensure that they are included in Pyodide 0.28 (which will come in a few weeks from now).
use Pyodide 0.28 for testing SymPy in its Pyodide CI, and include the python-flint tests that were being skipped because it's not installed, and address any problems there (perhaps it makes sense to address problems here first, I can't say)
Install python-flint alongside SymPy for SymPy Live, and try to resolve Display the SymPy version in the shell prompt sympy/live#23

fredrik-johansson · 2025-05-28T10:22:36Z

Not sure how urgent this is: should we hold up the 3.3 release of FLINT to try to get all tests to pass on WASM first, or is it OK if we release 3.3 now and push any further fixes for WASM compatibility to a future release?

oscarbenjamin · 2025-05-28T11:15:35Z

I think for now we can track FLINT's main branch for the WASM build so I think the FLINT release doesn't matter. At least the current state in this PR worked good enough with FLINT main and so if FLINT 3.3 is released now then it should be possible to get at least as good as we have here by pinning to FLINT 3.3.

How long it would take to iron these things out is unknown so it is probably best not to wait on it.

agriyakhetarpal added 6 commits February 28, 2025 02:15

Add a CI job to build libgmp, libmpfr, flint, and python-flint

7072379

Find MPFR

a0c5c63

Don't override Python's PKG_CONFIG_PATH

ffb88c4

Debug

d19caaf

Try to export WASM_LIBRARY_DIR properly

443d008

Fix missing WASM lib dir

44ef505

Bump flint to 3.1.0

42f4069

Run the test suite

304a55c

agriyakhetarpal added 7 commits February 28, 2025 05:42

Bump flint to version 3.1.2

52a5674

is is/is not for comparison with NoneType

5583688

Temporarily disable test_polys (FSM)

cf5f246

Temporarily disable test_factor_poly_mpoly

a3af36a

Cache WASM libs for CI speedups

35f811e

Temporarily disable test_fq_default

bcfeb29

Temporarily disable test_fq_default_poly

5dc6770

agriyakhetarpal force-pushed the bld/wasm branch from d4d9652 to 5dc6770 Compare March 3, 2025 10:59

agriyakhetarpal added 2 commits March 6, 2025 01:02

Merge branch 'main' into bld/wasm

920e9fa

Bump to flint version 3.2.0-rc1 for WASM build

db1ef76

Sort factors from nmod_poly.factor

232f4e1

oscarbenjamin added 2 commits May 26, 2025 19:38

Split out failing test

60c291e

Include all tests

5245adb

Fix randomised tests

aaf3ab6

Sort factors in fq_default_poly.factor

b93a5c5

oscarbenjamin added 2 commits May 26, 2025 21:40

skip flakey doctests

b178ec9

skip flakey tests

83dcae6

oscarbenjamin closed this May 26, 2025

oscarbenjamin reopened this May 26, 2025

oscarbenjamin added 2 commits May 26, 2025 22:15

Reenable fmpz_poly.hilbert_class doctests

220ff1a

skip another flakey doctest

1eee325

oscarbenjamin added 2 commits May 27, 2025 11:24

Disable hilbert class poly doctests again

02d33a2

Skip flakey doctest

cf39a48

agriyakhetarpal closed this May 27, 2025

agriyakhetarpal reopened this May 27, 2025

agriyakhetarpal closed this May 27, 2025

WASM build #262

WASM build #262

Uh oh!

Conversation

agriyakhetarpal commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agriyakhetarpal commented Feb 27, 2025

Uh oh!

agriyakhetarpal commented Feb 27, 2025

Uh oh!

agriyakhetarpal commented Feb 27, 2025

Uh oh!

agriyakhetarpal commented Feb 27, 2025

Uh oh!

oscarbenjamin commented Feb 28, 2025

Uh oh!

oscarbenjamin commented Feb 28, 2025

Uh oh!

agriyakhetarpal commented Feb 28, 2025

Uh oh!

agriyakhetarpal commented Mar 3, 2025

Uh oh!

oscarbenjamin commented Mar 3, 2025

Uh oh!

agriyakhetarpal commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oscarbenjamin commented Mar 3, 2025

Uh oh!

oscarbenjamin commented Mar 5, 2025

Uh oh!

oscarbenjamin commented May 26, 2025

Uh oh!

oscarbenjamin commented May 26, 2025

Uh oh!

agriyakhetarpal commented May 26, 2025

Uh oh!

oscarbenjamin commented May 26, 2025

Uh oh!

agriyakhetarpal commented May 26, 2025

Uh oh!

oscarbenjamin commented May 26, 2025

Uh oh!

agriyakhetarpal commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oscarbenjamin commented May 26, 2025

Uh oh!

agriyakhetarpal commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oscarbenjamin commented May 27, 2025

Uh oh!

oscarbenjamin commented May 27, 2025

Uh oh!

agriyakhetarpal commented May 27, 2025

Uh oh!

fredrik-johansson commented May 28, 2025

Uh oh!

oscarbenjamin commented May 28, 2025

Uh oh!

Uh oh!

agriyakhetarpal commented Feb 27, 2025 •

edited

Loading

agriyakhetarpal commented Mar 3, 2025 •

edited

Loading

agriyakhetarpal commented May 26, 2025 •

edited

Loading

agriyakhetarpal commented May 27, 2025 •

edited

Loading