ECM segfaults in Cygwin64

[embray]

Running ECM with almost any inputs results in a segfault on Cygwin64.

This is because any time it enters one of its hand-written assembly routines it needs to use the appropriate calling convention (Microsoft x64 convention on Windows), but it is only picking the correct Windows calling convention when compiling with mingw. It should do the same with Cygwin as well.

One possible workaround without patching would be to configure ECM with --disable-asm-redc to disable the custom assembly code entirely. But this is also overkill--the code works just fine as long as it's configured correctly for the platform.

Submitted upstream at https://gforge.inria.fr/tracker/index.php?func=detail&aid=20697&group_id=135&atid=623

Upstream: Fixed upstream, but not in a stable release.

CC: @jpflori @tscrim

Component: porting: Cygwin

Author: Erik Bray

Branch: a4adca5

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/21223

[embray]

Author: Erik Bray

[embray]

Branch: u/embray/ticket-21223

[embray]

New commits:

c80277a

Make sure the WINDOWS64_ABI flag is set on Cygwin as well, otherwise the assembly routines use the wrong calling convention.

[embray]

Commit: c80277a

[embray]

Description changed:

--- 
+++ 
@@ -1,3 +1,5 @@
 Running ECM with almost any inputs results in a segfault on Cygwin64.
 
-More details soon.
+This is because any time it enters one of its hand-written assembly routines it needs to use the appropriate calling convention (Microsoft x64 convention on Windows), but it is only picking the correct Windows calling convention when compiling with mingw.  It should do the same with Cygwin as well.
+
+One possible workaround without patching would be to configure ECM with `--disable-asm-redc` to disable the custom assembly code entirely.  But this is also overkill--the code works just fine as long as it's configured correctly for the platform.

[embray]

Upstream: Not yet reported upstream; Will do shortly.

[nexttime]

comment:3

ECM 7.x got "revived", see #20385. 7.0.3 is still kind of beta/rc though I'd say.

[sagetrac-git]

Changed commit from c80277a to 6361e6c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6361e6c

Bump patch level

[nexttime]

comment:5

... as is the branch on #20385; i.e., not really "needs review" yet, but you could try whether that has been fixed upstream already, probably with vanilla 7.0.3.

[nexttime]

comment:6

Replying to @nexttime:

you could try whether that has been fixed upstream already, probably with vanilla 7.0.3.

Nope, in 7.0.3 (now configure.ac) there's still:

if test "x$enable_asm_redc" = xyes; then
# do the necessary definitions and includes
  AC_DEFINE([USE_ASM_REDC],1,[Define to 1 to use asm redc])
  test "x$CCAS" != x || CCAS="$CC -c"
  AC_SUBST([CCAS])
  GMP_PROG_M4
  GMP_ASM_UNDERSCORE
  GMP_ASM_TEXT
  GMP_ASM_GLOBL
  GMP_ASM_TYPE

  case $host in
    *-*-mingw32) GMP_DEFINE([WINDOWS64_ABI], 1)
                 AC_DEFINE([WINDOWS64_ABI], 1,[Define to 1 if x86_64 mulredc*() functions should be called with Windows ABI]);;
    *) ;;
  esac

  case $host in
    pentium3-*-*)
      echo "WARNING: Your processor is recognized as Pentium3."
      echo "         The asm code uses SSE2, and therefore it might"
      echo "         fail if your proc is indeed a P3, and not a"
      echo "         Pentium M. If you have compilation problems,"
      echo "         consider using --disable-asm-redc." ;;
    *)
  esac
fi
AM_CONDITIONAL([ENABLE_ASM_REDC], [test "x$enable_asm_redc" = xyes])