SIGSEGV on Fedora 30 / PHP7.3.6 with Phalcon and Opcache enabled

[virgofx]

Hopefully this is the right spot for bug report ---

cc: @remicollet

Running a Fedora:30 container. (Should also be reproducible anywhere ... just change the PID to the master PHP-FPM process)

1. Fedora:30 container
2. Install Remi repo + PHP7.3 + opcache + enable opcache
3. Graceful restarts of the worker processes (not master) occur successfully when executing a SIGUSR2 to the master process. (e.g. kill -USR2 1 from within container). This works successfully when opcache is not enabled. (all the child processes restart, master stays alive, same PID 1). However, with opcache enabled. It kills the master process:

---

Side Note: Appears the debug info installed all the 7.3.6-1 locations (Not 7.3.6-3) hence the file not found reference. Shouldn't make a different regardless as I believe that's just trying to help the output. The backtrace should be the same.

[root@2fb3b586750c debug]# ls -alh | grep php-7.3
drwxr-xr-x 13 root root 4.0K Jun 26 21:29 php-7.3.6-1.fc30.x86_64
drwxr-xr-x 11 root root 4.0K Jun 26 21:46 php73-php-7.3.6-1.fc30.remi.x86_64

[root@2fb3b586750c debug]# rpm -qa | grep php-debuginfo
php73-php-debuginfo-7.3.6-1.fc30.remi.x86_64
php-debuginfo-7.3.6-3.fc30.remi.x86_64

Maybe a separate issue when generating updated/repeat builds the debug info has the updated name but puts files in the .1?

---

kill -USR2 1 (Opcache enabled)

Program received signal SIGUSR2, User defined signal 2.
0x00007fe18f4c35fb in epoll_wait (epfd=9, events=0x55d1cd458ae0, maxevents=maxevents@entry=101, timeout=timeout@entry=1000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30        return SYSCALL_CANCEL (epoll_wait, epfd, events, maxevents, timeout);
(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
zend_hash_destroy (ht=ht@entry=0x55d1cd266e40) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend_hash.c:1451
1451    /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend_hash.c: No such file or directory.
(gdb) bt
#0  zend_hash_destroy (ht=ht@entry=0x55d1cd266e40) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend_hash.c:1451
#1  0x000055d1cca18204 in destroy_zend_class (zv=<optimized out>) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend_opcode.c:317
#2  0x000055d1cca32d0d in zend_hash_destroy (ht=0x55d1cd023730) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend_hash.c:1418
#3  0x000055d1cca232ca in zend_shutdown () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/Zend/zend.c:982
#4  0x000055d1cc9c50f2 in php_module_shutdown () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/main/main.c:2489
#5  php_module_shutdown () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/main/main.c:2463
#6  0x000055d1ccab99cd in fpm_php_cleanup (which=<optimized out>, arg=<optimized out>) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_php.c:198
#7  0x000055d1ccab20bd in fpm_cleanups_run (type=type@entry=8) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_cleanup.c:43
#8  0x000055d1ccaba654 in fpm_pctl_exec () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_process_ctl.c:98
#9  fpm_pctl_action_last () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_process_ctl.c:109
#10 0x000055d1ccabb2db in fpm_pctl (action=2, new_state=0) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_process_ctl.c:256
#11 fpm_pctl_child_exited () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_process_ctl.c:256
#12 0x000055d1ccab1bcf in fpm_children_bury () at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_children.c:266
#13 0x000055d1ccab6736 in fpm_got_signal (ev=<optimized out>, which=<optimized out>, arg=<optimized out>)
    at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_events.c:75
#14 0x000055d1ccac48be in fpm_event_epoll_wait (queue=<optimized out>, timeout=<optimized out>) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/events/epoll.c:145
#15 0x000055d1ccab71b1 in fpm_event_loop (err=err@entry=0) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_events.c:409
#16 0x000055d1ccab1487 in fpm_run (max_requests=0x7ffd1f591ddc) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm.c:113
#17 0x000055d1cc8932ce in main (argc=2, argv=0x7ffd1f592328) at /usr/src/debug/php-7.3.6-3.fc30.remi.x86_64/sapi/fpm/fpm/fpm_main.c:1864

php -m

[PHP Modules]
apcu
bz2
Core
curl
date
dom
exif
fileinfo
filter
gettext
gmp
hash
iconv
igbinary
intl
json
libxml
mailparse
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
phalcon
Phar
readline
redis
Reflection
session
SimpleXML
sockets
SPL
sqlite3
standard
tokenizer
v8js
xdebug
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zlib

[Zend Modules]
Xdebug
Zend OPcache

php --version

PHP 7.3.6 (cli) (built: May 28 2019 09:32:59) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.6, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.7.2, Copyright (c) 2002-2019, by Derick Rethans

php --ri "Zend OPcache"

Zend OPcache

Opcode Caching => Disabled
Optimization => Disabled
SHM Cache => Enabled
File Cache => Disabled
Startup Failed => Opcode Caching is disabled for CLI

Directive => Local Value => Master Value
opcache.enable => On => On
opcache.use_cwd => On => On
opcache.validate_timestamps => On => On
opcache.validate_permission => Off => Off
opcache.validate_root => Off => Off
opcache.dups_fix => Off => Off
opcache.revalidate_path => Off => Off
opcache.log_verbosity_level => 1 => 1
opcache.memory_consumption => 128 => 128
opcache.interned_strings_buffer => 32 => 32
opcache.max_accelerated_files => 5000 => 5000
opcache.max_wasted_percentage => 5 => 5
opcache.consistency_checks => 0 => 0
opcache.force_restart_timeout => 180 => 180
opcache.revalidate_freq => 2 => 2
opcache.file_update_protection => 2 => 2
opcache.preferred_memory_model => no value => no value
opcache.blacklist_filename => /etc/php.d/opcache*.blacklist => /etc/php.d/opcache*.blacklist
opcache.max_file_size => 0 => 0
opcache.protect_memory => 0 => 0
opcache.save_comments => 1 => 1
opcache.optimization_level => 0 => 0xFFFFFBFF
opcache.opt_debug_level => 0 => 0
opcache.enable_file_override => Off => Off
opcache.enable_cli => Off => Off
opcache.error_log => no value => no value
opcache.restrict_api => no value => no value
opcache.lockfile_path => /tmp => /tmp
opcache.file_cache => no value => no value
opcache.file_cache_only => 0 => 0
opcache.file_cache_consistency_checks => 1 => 1
opcache.huge_code_pages => On => On

[remicollet]

Can you please try 7.3.7RC3 in "remi-php73-test" repository, which includes some opcache fix.

Notice: the segfault occurs in shutdown, so requests have been served (of course this is not normal)

As the segfault occurs in the shutdown process, the memory corruption can be raise by any installed modules... and some are not part of php, will be nice to check if same happen without them (phalcon, v8js, igbinary...)

A better place for this is https://bugs.php.net/ (as this doesn't seems related to the packaging)

[virgofx]

Will try the 73-test with 7.3.7RC3 and update tomorrow. Thanks for pointers.

[remicollet]

Also rpm -qa php\*

-1 release is from official Fedora repository, -3 from my repo.

[remicollet]

Please also notes that there is no "graceful" restart in FPM for now only some WIP upstream

[virgofx]

Segfaulting on 7.3.7RC3 also :( Removed v8js as well. Left Phalcon in. Will retry again without Phalcon... but second I disable opcache, it works

[root@390d6dc8e1cb app]# php --version
PHP 7.3.7RC3 (cli) (built: Jun 20 2019 14:11:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.7RC3, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.7RC3, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.7.2, Copyright (c) 2002-2019, by Derick Rethans
[root@390d6dc8e1cb app]# kill -USR2 1

php-fpm exited with code 139

I realize the USR2 is not a well-defined method for restarting. Just trying to fix some things in development inside of a container without stopping the container.

Specifically, using gettext and I have a synced folder in docker if changes are made to the .po files ... it automatically recompiles the .mo files. However, the .mo binary files are somehow cached in PHP-FPM worker-land (part of the way gettext optimizes) -- which is great.. Need away to restart/clear that in development mode in a container to get updated values :/

[remicollet]

I understand... gettext...

IIRC you also have some issue when the system locale is not installed (which happens since glibc-langpack was splitted)

This is probably the main reason why lot of projects stay away of gettext (see mysqladmin/motranslator or zendframework/zend-i18n), as restart is not possible in shared hosting

[virgofx]

Interesting ... it works with Phalcon disabled and Opcache enabled. (Currently testing 7.3.7RC3) maybe I go back to stable land 7.3.6 and retry as well to confirm it's an issue with [Phalcon] <--> [Opcache]

I know @sergeyklay was going to backport a SEGFAULT issue with Zephir and tag Phalcon 3.4.4 -- I wonder if it's related. I'll PM him tomorrow to see if he can get that tagged, then hopefully you can rebuild it in your repo too.

[remicollet]

Looks like a know issue phalcon/cphalcon#14160

[remicollet]

I will try to run a test build with upstream patch

[remicollet]

Can you please test with php-phalcon3-3.4.3-3.fc30.remi.7.3.x86_64 (from remi-test)

P.S. only base package for F30, tell me if another build needed

[remicollet]

Fixed by https://git.remirepo.net/cgit/rpms/zephir-parser.git/commit/?h=v0.10&id=33d0a3eea172b82fe0d4f34ad76d293e5f76ddb2

[remicollet]

maybe I go back to stable land

IMHO, all "developers" should use the RC versions on their development environment.
https://twitter.com/RemiCollet/status/1142002686180646915

Because we need feedback "before" promoted to stable

And there is usually NO change between RC and GA version (excepted security fix, when needed)

P.S. and this will avoid some bad issues / regressions, such as 78106 or 78185