[BUG] EL9 3005~rc1-2 salt minion fails to run with FIPS enabled

[imjustvisiting]

Description
3005~rc1-2 salt minion fails to run on a Rocky 9 machine

Setup
Installed the RC1~2 RPMs on a baremetal workstation running Rocky 9.0:

# cat /etc/redhat-release 
Rocky Linux release 9.0 (Blue Onyx)

# rpm -qa | grep salt
salt-3005~rc1-2.el9.x86_64
salt-minion-3005~rc1-2.el9.x86_64

Steps to Reproduce the behavior
Any attempt to run the salt minion results in a trace:

# salt-call test.ping
Traceback (most recent call last):
  File "cffi/api.py", line 183, in _typeof
KeyError: 'const uint8_t*'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/event.py", line 63, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/channel/client.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/crypt.py", line 54, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/__init__.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/_mode_ecb.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Util/_raw_api.py", line 88, in <module>
  File "cffi/api.py", line 266, in new
  File "cffi/api.py", line 186, in _typeof
  File "cffi/api.py", line 171, in _typeof_locked
  File "cffi/cparser.py", line 552, in parse_type
  File "cffi/cparser.py", line 555, in parse_type_and_quals
  File "cffi/cparser.py", line 336, in _parse
  File "cffi/cparser.py", line 53, in _get_parser
  File "pycparser/c_parser.py", line 110, in __init__
  File "pycparser/ply/yacc.py", line 3256, in yacc
  File "pycparser/ply/yacc.py", line 2961, in signature
UnboundLocalError: local variable 'sig' referenced before assignment
[63524] Failed to execute script 'pyi_rth_subprocess' due to unhandled exception!
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
UnboundLocalError: local variable 'sig' referenced before assignment
Traceback (most recent call last):
  File "cffi/api.py", line 183, in _typeof
    result = self._parsed_types[cdecl]
KeyError: 'const uint8_t*'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
    from salt.utils.pyinstaller.rthooks._overrides import PyinstallerPopen
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
    import salt.utils.vt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
    import salt.utils.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
    import salt.loader
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/crypt.py", line 54, in <module>
    from Cryptodome.Cipher import AES, PKCS1_OAEP, PKCS1_v1_5 as PKCS1_v1_5_CIPHER
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/__init__.py", line 27, in <module>
    from Cryptodome.Cipher._mode_ecb import _create_ecb_cipher
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/_mode_ecb.py", line 29, in <module>
    from Cryptodome.Util._raw_api import (load_pycryptodome_raw_lib,
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Util/_raw_api.py", line 88, in <module>
    uint8_t_type = ffi.typeof(ffi.new("const uint8_t*"))
  File "cffi/api.py", line 266, in new
    cdecl = self._typeof(cdecl)
  File "cffi/api.py", line 186, in _typeof
    result = self._typeof_locked(cdecl)
  File "cffi/api.py", line 171, in _typeof_locked
    type = self._parser.parse_type(cdecl)
  File "cffi/cparser.py", line 552, in parse_type
    return self.parse_type_and_quals(cdecl)[0]
  File "cffi/cparser.py", line 555, in parse_type_and_quals
    ast, macros = self._parse('void __dummy(\n%s\n);' % cdecl)[:2]
  File "cffi/cparser.py", line 336, in _parse
    ast = _get_parser().parse(fullcsource)
  File "cffi/cparser.py", line 53, in _get_parser
    _parser_cache = pycparser.CParser()
  File "pycparser/c_parser.py", line 110, in __init__
    self.cparser = yacc.yacc(
  File "pycparser/ply/yacc.py", line 3256, in yacc
    signature = pinfo.signature()
  File "pycparser/ply/yacc.py", line 2961, in signature
    digest = base64.b16encode(sig.digest())
UnboundLocalError: local variable 'sig' referenced before assignment

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

That looks like a broken dependency. Does it do the same if you just run salt-call --versions?

[imjustvisiting]

Yes. salt-call returns the same trace with the --versions (or any) argument.

[imjustvisiting]

I dug into this a bit more...

The "crypto" messages clued me into it possibly being an issue with FIPS, and I have verified that this is the case. After disabling FIPS, the minion seems to run fine -- so really the bug to report is that the "salt-minion does not work with FIPS enabled".

I used fips-mode-setup --check to verify that FIPS was enabled, and then fips-mode-setup --disable to turn it off so I could continue testing 3005 on EL9.

[OrangeDog]

I assume FIPS mode was working before?

[imjustvisiting]

This was my first attempt at 3005 on EL9, so I can't say if it ever worked with FIPS enabled.

I'm okay relabeling this as a feature request if FIPS wasn't previously supported.

[dmurphy18]

@imjustvisiting Thanks for the issue but as per all issues, a versions report should be in the issue, esp. for the salt-master that you are using, understanding the salt-minion versions used, but wondering if we have and issue between the minion and the master, given the error is occurring in dependencies pulled in.

[OrangeDog]

@dmurphy18 A versions report is not possible.

salt-call returns the same trace with the --versions (or any) argument.

As this was the RC package, all the dependencies in question would be the bundled ones.

As the error occurs before any remote calls, the versions on the master should be irrelevant (but is also the same RC package).