Still can't compile 0.3.7 with PGI (19.10)

[christopherwharrop-noaa]

As reported in issues #2236 and #2237 version 0.3.7 has problems with PGI.

Now that I have access to PGI 19.10, which fixes the MMX instructions problems, I tried again to build 0.3.7 after applying changes in #2223 (#2240 is no longer needed due to fixes in 19.10). I have tried both the llvm and nollvm compilers. They both produce the same result.

Now, the compile fails with (for example):

pgcc -c -O2 -DMAX_STACK_ALLOC=2048 -mp -tp p7-64 -DF_INTERFACE_PGI -fPIC -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\" -DASMNAME=dtrsm_iun\
ucopy -DASMFNAME=dtrsm_iunucopy_ -DNAME=dtrsm_iunucopy_ -DCNAME=dtrsm_iunucopy -DCHAR_NAME=\"dtrsm_iunucopy_\" -DCHAR_CNAME=\"dtrsm_iunucopy\" -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX  -DDOUBLE -UCOMPLEX -UOU\
TER -ULOWER -DUNIT generic/trsm_uncopy_4.c -o dtrsm_iunucopy.o
<inline asm>:2:13: error: invalid operand for instruction
        addq    $128, %dl
                      ^~~
<inline asm>:5:15: error: invalid base+index expression
        vmulpd  -128(%dl), %xmm0, %xmm4

The invalid base+index expression error was mentioned in the discussion in #2237 here.

Does anyone know if a solution to the invalid base+index expression error was ever found? Or if there were other modifications found that were needed to get a clean build with PGI 19.10?

[martin-frbg]

Weird I'm fairly certain I test-build on SkylakeX when 19.10 became available. Can you try with a snapshot of the current develop branch ?

[christopherwharrop-noaa]

I just tried the current develop (8d2a796) with several compilers and the only one that succeeded was gcc-4.8.5. All the rest died with some version of undefined reference error, I think during builds for tests. It does appear, though, that the invalid base+index expression is gone.

PGI 19.10 failed with:

make -j 80 -C test all
make[1]: Entering directory `/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/test'
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat1.f  -o sblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat1.f  -o dblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat1.f  -o cblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat1.f  -o zblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat2.f  -o sblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat2.f  -o dblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat2.f  -o cblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat2.f  -o zblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat3.f  -o sblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat3.f  -o dblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat3.f  -o cblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat3.f  -o zblat3.o
pgfortran -O2 -tp p7-64 -mp   -o zblat1 zblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o sblat1 sblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o cblat1 cblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o dblat1 dblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
../libopenblas_haswellp-r0.3.8.dev.a(blas_server.o): In function `exec_blas':
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_penter'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_lcpu'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_ncpus'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:351: undefined reference to `_mp_pexit'
make[1]: *** [zblat1] Error 2
make[1]: *** Waiting for unfinished jobs....

gcc 8.3.0 and 9.2.0 failed with:

gcc -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -march=skylake-avx512 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_skylakexp-r0.3.8.dev.so -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib64/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../..  -lgfortran -lm -lgomp -lquadmath -lm -lpthread -lc   && echo OK.
/tmp/ccFAaQY2.o: In function `main':
linktest.c:(.text.startup+0x2f4e): undefined reference to `clagge_'
linktest.c:(.text.startup+0x2f55): undefined reference to `claghe_'
linktest.c:(.text.startup+0x2f5c): undefined reference to `clagsy_'
linktest.c:(.text.startup+0x2f63): undefined reference to `clahilb_'
linktest.c:(.text.startup+0x2f6a): undefined reference to `clakf2_'
linktest.c:(.text.startup+0x2f71): undefined reference to `clarge_'
linktest.c:(.text.startup+0x2f78): undefined reference to `clarnd_'
linktest.c:(.text.startup+0x2f7f): undefined reference to `claror_'
linktest.c:(.text.startup+0x2f86): undefined reference to `clarot_'
linktest.c:(.text.startup+0x2f8d): undefined reference to `clatm1_'
linktest.c:(.text.startup+0x2f94): undefined reference to `clatm2_'
linktest.c:(.text.startup+0x2f9b): undefined reference to `clatm3_'
linktest.c:(.text.startup+0x2fa2): undefined reference to `clatm5_'
linktest.c:(.text.startup+0x2fa9): undefined reference to `clatm6_'
linktest.c:(.text.startup+0x2fb0): undefined reference to `clatme_'
linktest.c:(.text.startup+0x2fb7): undefined reference to `clatmr_'
linktest.c:(.text.startup+0x2fbe): undefined reference to `clatms_'
linktest.c:(.text.startup+0x2fc5): undefined reference to `clatmt_'
linktest.c:(.text.startup+0x302e): undefined reference to `dlagge_'
linktest.c:(.text.startup+0x3035): undefined reference to `dlagsy_'
linktest.c:(.text.startup+0x303c): undefined reference to `dlahilb_'
linktest.c:(.text.startup+0x3043): undefined reference to `dlakf2_'
linktest.c:(.text.startup+0x304a): undefined reference to `dlaran_'
linktest.c:(.text.startup+0x3051): undefined reference to `dlarge_'
linktest.c:(.text.startup+0x3058): undefined reference to `dlarnd_'
linktest.c:(.text.startup+0x305f): undefined reference to `dlaror_'
linktest.c:(.text.startup+0x3066): undefined reference to `dlarot_'
linktest.c:(.text.startup+0x306d): undefined reference to `dlatm1_'
linktest.c:(.text.startup+0x3074): undefined reference to `dlatm2_'
linktest.c:(.text.startup+0x307b): undefined reference to `dlatm3_'
linktest.c:(.text.startup+0x3082): undefined reference to `dlatm5_'
linktest.c:(.text.startup+0x3089): undefined reference to `dlatm6_'
linktest.c:(.text.startup+0x3090): undefined reference to `dlatm7_'
linktest.c:(.text.startup+0x3097): undefined reference to `dlatme_'
linktest.c:(.text.startup+0x309e): undefined reference to `dlatmr_'
linktest.c:(.text.startup+0x30a5): undefined reference to `dlatms_'
linktest.c:(.text.startup+0x30ac): undefined reference to `dlatmt_'
linktest.c:(.text.startup+0x310e): undefined reference to `slagge_'
linktest.c:(.text.startup+0x3115): undefined reference to `slagsy_'
linktest.c:(.text.startup+0x311c): undefined reference to `slahilb_'
linktest.c:(.text.startup+0x3123): undefined reference to `slakf2_'
linktest.c:(.text.startup+0x312a): undefined reference to `slaran_'
linktest.c:(.text.startup+0x3131): undefined reference to `slarge_'
linktest.c:(.text.startup+0x3138): undefined reference to `slarnd_'
linktest.c:(.text.startup+0x313f): undefined reference to `slaror_'
linktest.c:(.text.startup+0x3146): undefined reference to `slarot_'
linktest.c:(.text.startup+0x314d): undefined reference to `slatm1_'
linktest.c:(.text.startup+0x3154): undefined reference to `slatm2_'
linktest.c:(.text.startup+0x315b): undefined reference to `slatm3_'
linktest.c:(.text.startup+0x3162): undefined reference to `slatm5_'
linktest.c:(.text.startup+0x3169): undefined reference to `slatm6_'
linktest.c:(.text.startup+0x3170): undefined reference to `slatm7_'
linktest.c:(.text.startup+0x3177): undefined reference to `slatme_'
linktest.c:(.text.startup+0x317e): undefined reference to `slatmr_'
linktest.c:(.text.startup+0x3185): undefined reference to `slatms_'
linktest.c:(.text.startup+0x318c): undefined reference to `slatmt_'
linktest.c:(.text.startup+0x31ee): undefined reference to `zlagge_'
linktest.c:(.text.startup+0x31f5): undefined reference to `zlaghe_'
linktest.c:(.text.startup+0x31fc): undefined reference to `zlagsy_'
linktest.c:(.text.startup+0x3203): undefined reference to `zlahilb_'
linktest.c:(.text.startup+0x320a): undefined reference to `zlakf2_'
linktest.c:(.text.startup+0x3211): undefined reference to `zlarge_'
linktest.c:(.text.startup+0x3218): undefined reference to `zlarnd_'
linktest.c:(.text.startup+0x321f): undefined reference to `zlaror_'
linktest.c:(.text.startup+0x3226): undefined reference to `zlarot_'
linktest.c:(.text.startup+0x322d): undefined reference to `zlatm1_'
linktest.c:(.text.startup+0x3234): undefined reference to `zlatm2_'
linktest.c:(.text.startup+0x323b): undefined reference to `zlatm3_'
linktest.c:(.text.startup+0x3242): undefined reference to `zlatm5_'
linktest.c:(.text.startup+0x3249): undefined reference to `zlatm6_'
linktest.c:(.text.startup+0x3250): undefined reference to `zlatme_'
linktest.c:(.text.startup+0x3257): undefined reference to `zlatmr_'
linktest.c:(.text.startup+0x325e): undefined reference to `zlatms_'
linktest.c:(.text.startup+0x3265): undefined reference to `zlatmt_'

Intel 17.0.5.239, 18.0.5.274, and 19.0.5.281 all failed during testing with

icc -O2 -DMAX_STACK_ALLOC=2048 -wd981 -fopenmp -DF_INTERFACE_INTEL -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_haswellp-r0.3.8.dev.so -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/../tbb/lib/intel64_lin/gcc4.4 -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/lib/intel64_lin -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/tbb/lib/intel64/gcc4.7 -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/mkl/lib/intel64_lin -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/compiler/lib/intel64_lin -L/apps/intel/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/../tbb/lib/intel64_lin/gcc4.4/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/lib/intel64_lin/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/tbb/lib/intel64/gcc4.7/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/mkl/lib/intel64_lin/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/compiler/lib/intel64_lin/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib  -lifport -lifcoremt -limf -lsvml -lm -lipgo -liomp5 -lirc -lpthread -lsvml -lc -lirc_s -ldl -lc  -lgfortran && echo OK.
../libopenblas_haswellp-r0.3.8.dev.so: undefined reference to `xerbl_'
make[1]: *** [../libopenblas_haswellp-r0.3.8.dev.so] Error 1
make[1]: Leaving directory `/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/intel/18.0.5.274/exports'
make: *** [shared] Error 2

[martin-frbg]

Your PGI seems to be looking for things from libpgmp.so but I wonder why and how it manages to
do this selectively for only some of the tests. Even less sure what to make of the other two - the gcc8/9 linktest seems to be missing all functions from the (netlib) LAPACK "matgen" helpers for the included regression tests that should be exactly the same as seen by gcc-4.8.5, Intel seems to have hit a mutilated reference to xerbla somewhere that nobody else noticed)

[christopherwharrop-noaa]

I had set USE_OPENMP=1 in my make command. I am going to remove that and try them all again to see if that makes any difference.

[martin-frbg]

BTW the trsm_uncopy_4.c that seemed to cause the assembler error should be unchanged from 0.3.7 (note the "invalid operand for instruction" would have been generated by pgcc itself from the C source !) so either the compiler options are different now (in a way we neglected to document in the earlier tickets) or your build is (sometimes?) using an older assembler (or even an older pgcc).
Anything special about that system, like network file systems mounted on demand or from a distributed storage that could be out of sync ?

[martin-frbg]

Hmm. I can force the "invalid base+index" on Haswell by removing the -D__MMX__ -Mnollvm from the CFLAGS in Makefile.system (the bits from #2240, so apparently it is not fixed completely despite #2237 (comment) @cparrott73 )

[cparrott73]

I don't believe the fix was propagated to the NoLLVM compilers, but I would have to double check with our developer here. In general, the NoLLVM compilers are deprecated and will not see any new releases after 20.4 or so.

[martin-frbg]

@cparrott73 it works with both options set but fails without, so it seems to be the llvm variant is the one that is (still?) having problems here.

[cparrott73]

Okay, sorry about that - I guess I misread the above. I assume you are referring to the inline assembly issue above, then? I'll probably need to file a new bug report on that one. I'll work to reproduce it here first.

[martin-frbg]

Build of current (unmodified) "develop" with the NoLLVM compiler works, but the openblas_utest binary gets built as a dummy binary with none of the tests. (I think this is a regression, but cannot be sure and would need to reinstall PGI 19.4 first to confirm). Building with the LLVM version appeared to work for the most part but here the openblas_utest binary segfaults on startup in _mp_preinit2 (in what was not supposed to be an OpenMP build)

And I just confirmed that a gcc 9.2.0 build is not missing any bits from MATGEN/TESTING in linktest here. No icc on that system at the moment to reproduce any "xerbl_"

[cparrott73]

I have opened TRS#28084 on this issue against PGI 19.10 here. I will let you know when I have more information.

[martin-frbg]

Thank you very much. BTW I can also reproduce the link error (undefined references to _mp_penter etc as noted above) when trying to compile with USE_OPENMP=1. Symbols mp_penter, mp_lcpu, mp_ncpus and mp_pexit do not appear to be defined in any of the libraries installed with 19.10 ?

[christopherwharrop-noaa]

I am getting some strange inconsistent behavior. The gcc failures puzzled me so I reran them and am now seeing them succeed/fail randomly. I suspect it might be an issue with parallel make, but I'm not 100% sure. I'm running my make command with -j1 to try to eliminate build issues, but it appears that the tests use -j 80 despite my efforts:

[Christopher.W.Harrop@Hera:hfe04 xml]$ grep "make " ../log/openblas/openblas_8d2a796_gcc*.log
../log/openblas/openblas_8d2a796_gcc_4.8.5.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../slaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../slaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../dlaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../dlaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../claswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../claswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../zlaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../zlaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C SRC
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C TESTING/MATGEN
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C LAPACKE
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C src
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C utils
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C test all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C utest all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C ctest all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:To install the library, you can run "make PREFIX=/path/to/your/installation install".
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran PREFIX=/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0 install
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -f Makefile.install install
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:Generating OpenBLASConfig.cmake in /scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0/lib/cmake/openblas
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:Generating OpenBLASConfigVersion.cmake in /scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0/lib/cmake/openblas

This might be affecting my other builds too. I'm going to go digging to see if I can turn off the parallel make.

[martin-frbg]

You should be able to override this in Makefile.rule if commandline or CFLAGS has no effect.

[cparrott73]

I'll take a look at the OpenMP issue here, but just FYI:

pgfortran -O2 -tp p7-64 -mp   -o sblat1 sblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o cblat1 cblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o dblat1 dblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc 

Explicitly appending the OpenMP runtime libraries on the compile line here is redundant - they are implied by -mp being present when linking. In fact, there is actually an alternate OpenMP implentation, which is available in 19.10 as libnvomp. libnvomp will be the default in the 20.x releases, so explicitly appending these libraries will break then.

If you want to test it out with 19.10, you can specify -mp=nvomp instead of just -mp on the command line.