[BUG] KeyError: '__file__' with setuptools 60.0.0

[LuisHenri]

setuptools version

setuptools==60.0.0

Python version

Python 3.8.12

OS

Windows 10

Additional environment information

I use Anaconda with Python 3.8.12.
I use PyArmor 7.0.3.
I user PyInstaller 4.7.

Description

I use PyArmor to pack my project (Obfuscation + Executable with PyInstaller).
It is used inside a Conda Environment.
I have a script that calls PyArmor.

    call_pyarmor(
        [
            "pack",
            "-s",
            "./main.spec",
            "-x",
            " --exclude .venv*"
            " --enable-suffix"
            " --with-license outer"
            " --plugin pyarmor_plugins/plugin_example.py",
            "--clean",
            "./main.py",
        ]
    )

Expected behavior

To the project's Executable be properly generated.

How to Reproduce

1. Set the environment:

conda create --yes --quiet --name .venv python=3.8
call activate .venv
python -m pip install --upgrade pip setuptools wheel
python -m pip install pyarmor==7.0.3 pyinstaller[encryption]==4.7

2. Create a sample default project (with main.spec and a plugin)
3. Run PyArmor from a script:

from PyInstaller.__main__ import run as call_pyinstaller
call_pyarmor(
        [
            "pack",
            "-s",
            "./main.spec",
            "-x",
            " --exclude .venv*"
            " --enable-suffix"
            " --with-license outer"
            " --plugin pyarmor_plugins/plugin_example.py",
            "--clean",
            "./main.py",
        ]
) 

Output

2021-12-20T12:51:26.2832859Z INFO     PyArmor Version 7.0.3
2021-12-20T12:51:26.2833655Z INFO     Python 3.8.12
2021-12-20T12:51:26.2837167Z INFO     Target platforms: Native
2021-12-20T12:51:26.2837966Z INFO     Source path is "D:\a\1\s"
2021-12-20T12:51:26.2838558Z INFO     Entry scripts are ['.\\main.py']
2021-12-20T12:51:26.2839403Z INFO     Use cached capsule C:\Users\VssAdministrator\.pyarmor\.pyarmor_capsule.zip
2021-12-20T12:51:26.2840362Z INFO     Search scripts mode: Recursive
2021-12-20T12:51:26.2841345Z INFO     Exclude path "dist"
2021-12-20T12:51:26.2844968Z INFO     Exclude path ".venv*"
2021-12-20T12:51:26.2851366Z INFO     Auto exclude output path "dist\obf"
2021-12-20T12:51:26.3066863Z no previously-included directories found matching 'dist'
2021-12-20T12:51:26.3073564Z no previously-included directories found matching '.venv*'
2021-12-20T12:51:26.3115156Z no previously-included directories found matching 'dist\obf'
2021-12-20T12:51:26.3119135Z INFO     Save obfuscated scripts to "dist\obf"
2021-12-20T12:51:26.3120364Z INFO     Read product key from capsule
2021-12-20T12:51:26.3151411Z INFO     Obfuscate module mode is 2
2021-12-20T12:51:26.3152052Z INFO     Obfuscate code mode is 1
2021-12-20T12:51:26.3152539Z INFO     Wrap mode is 1
2021-12-20T12:51:26.3153215Z INFO     Restrict mode is 1
2021-12-20T12:51:26.3153770Z INFO     Advanced value is 0
2021-12-20T12:51:26.3155708Z INFO     Super mode is False
2021-12-20T12:51:26.3156579Z INFO     Super plus mode is not enabled
2021-12-20T12:51:26.3163577Z INFO     Generating runtime files to dist\obf
2021-12-20T12:51:26.3169121Z INFO     Extract pytransform.key
2021-12-20T12:51:26.3171851Z INFO     Use outer license file
2021-12-20T12:51:26.3174756Z INFO     Copying C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\platforms\windows\x86_64\_pytransform.dll
2021-12-20T12:51:26.3175495Z INFO     Rename it to _pytransform_vax_0001.dll
2021-12-20T12:51:26.3210551Z INFO     Patch library dist\obf\_pytransform_vax_0001.dll
2021-12-20T12:51:26.6123821Z INFO     Patch library file OK
2021-12-20T12:51:27.2509613Z INFO     Copying C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\pytransform.py
2021-12-20T12:51:27.2511236Z INFO     Rename it to pytransform_vax_0001.py
2021-12-20T12:51:27.2521723Z INFO     Generate runtime files OK
2021-12-20T12:51:27.2527229Z INFO     Start obfuscating the scripts...
2021-12-20T12:51:27.7791265Z INFO     Found plugin check_processor_id at: pyarmor_plugins/example_plugin.py
2021-12-20T12:51:27.7800488Z INFO     Patch this script with plugins
2021-12-20T12:51:27.7803068Z INFO     Apply plugin example_plugin
2021-12-20T12:51:27.7807320Z INFO     Patch this entry script with protection code
2021-12-20T12:51:27.8445234Z INFO     Insert bootstrap code to entry script dist\obf\main.py
2021-12-20T12:51:27.8456836Z INFO     Obfuscate 9 scripts OK.
2021-12-20T12:51:30.6824262Z 2464 INFO: PyInstaller: 4.7
2021-12-20T12:51:30.6825312Z 2464 INFO: Python: 3.8.12 (conda)
2021-12-20T12:51:30.6996302Z 2483 INFO: Platform: Windows-10-10.0.17763-SP0
2021-12-20T12:51:30.7045110Z 2488 INFO: UPX is not available.
2021-12-20T12:51:30.7052400Z 2490 INFO: Removing temporary files and cleaning cache in C:\Users\VssAdministrator\AppData\Local\pyinstaller
2021-12-20T12:51:30.7111993Z Traceback (most recent call last):
2021-12-20T12:51:30.7112816Z   File "C:\Miniconda\envs\.venv\lib\runpy.py", line 194, in _run_module_as_main
2021-12-20T12:51:30.7113603Z     return _run_code(code, main_globals, None,
2021-12-20T12:51:30.7114264Z   File "C:\Miniconda\envs\.venv\lib\runpy.py", line 87, in _run_code
2021-12-20T12:51:30.7114913Z     exec(code, run_globals)
2021-12-20T12:51:30.7115338Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 134, in <module>
2021-12-20T12:51:30.7115776Z     run()
2021-12-20T12:51:30.7118603Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 124, in run
2021-12-20T12:51:30.7119564Z     run_build(pyi_config, spec_file, **vars(args))
2021-12-20T12:51:30.7120046Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
2021-12-20T12:51:30.7120607Z     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
2021-12-20T12:51:30.7121099Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\building\build_main.py", line 782, in main
2021-12-20T12:51:30.7122155Z     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
2021-12-20T12:51:30.7123140Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\building\build_main.py", line 714, in build
2021-12-20T12:51:30.7124574Z     exec(code, spec_namespace)
2021-12-20T12:51:30.7124905Z   File "main-patched.spec", line 5, in <module>
2021-12-20T12:51:30.7125488Z     from pip._internal.operations import freeze
2021-12-20T12:51:30.7126946Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2021-12-20T12:51:30.7128349Z   File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
2021-12-20T12:51:30.7128859Z   File "<frozen importlib._bootstrap>", line 914, in _find_spec
2021-12-20T12:51:30.7129415Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 83, in find_spec
2021-12-20T12:51:30.7129816Z     return method()
2021-12-20T12:51:30.7130701Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 104, in spec_for_pip
2021-12-20T12:51:30.7131192Z     if self.pip_imported_during_build():
2021-12-20T12:51:30.7131784Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 115, in pip_imported_during_build
2021-12-20T12:51:30.7132215Z     return any(
2021-12-20T12:51:30.7132809Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 116, in <genexpr>
2021-12-20T12:51:30.7133491Z     frame.f_globals['__file__'].endswith('setup.py')
2021-12-20T12:51:30.7133835Z KeyError: '__file__'
2021-12-20T12:51:30.8726971Z Traceback (most recent call last):
2021-12-20T12:51:30.8759812Z   File "deployment.py", line 70, in <module>
2021-12-20T12:51:30.8761995Z     main(sys.argv)
2021-12-20T12:51:30.8765357Z   File "deployment.py", line 27, in main
2021-12-20T12:51:30.8766912Z     run_pyarmor()
2021-12-20T12:51:30.8768238Z   File "deployment.py", line 40, in run_pyarmor
2021-12-20T12:51:30.8768744Z     call_pyarmor(
2021-12-20T12:51:30.8769402Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\pyarmor.py", line 1572, in main
2021-12-20T12:51:30.8769991Z     args.func(args)
2021-12-20T12:51:30.8770836Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 540, in packer
2021-12-20T12:51:30.8771583Z     _pyinstaller(src, entry, output, extra_options, xoptions, args)
2021-12-20T12:51:30.8772362Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 440, in _pyinstaller
2021-12-20T12:51:30.8773051Z     run_command([sys.executable] + packcmd + ['-y', '--clean', patched_spec])
2021-12-20T12:51:30.8773787Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 86, in run_command
2021-12-20T12:51:30.8774419Z     raise RuntimeError('Run command failed')
2021-12-20T12:51:30.8774985Z RuntimeError: Run command failed

Code of Conduct

• I agree to follow the PSF Code of Conduct

[ogrisel]

It seems that we have observed a similar crash on our CI: https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=36235&view=logs&jobId=aabdcdc3-bb64-5414-b357-ed024fe8659e&j=aabdcdc3-bb64-5414-b357-ed024fe8659e&t=08761b20-68b0-57e8-3399-a73e66658a1f

Traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/io/sklearn/utils/_show_versions.py", line 85, in show_versions
    deps_info = _get_deps_info()
  File "/io/sklearn/utils/_show_versions.py", line 69, in _get_deps_info
    mod = importlib.import_module(modname)
  File "/io/testvenv/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 906, in _find_spec
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in pip_imported_during_build
    for frame, line in traceback.walk_stack(None)
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in <genexpr>
    for frame, line in traceback.walk_stack(None)
KeyError: '__file__'

[lesteve]

A simple snippet reproducing the problem:

if [ -d /tmp/test-venv ]; then
    rm -rf /tmp/test-venv
fi
python -m venv /tmp/test-venv
source /tmp/test-venv/bin/activate
pip install 'setuptools>=60'

pip list
python -c 'import importlib; importlib.import_module("pip")'

A few comments:

• this fails with setuptools 60.0.0 with a similar error as shown above but succeeds with setuptools 59.8

  Full stack-trace

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/home/local/lesteve/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
    File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 925, in _find_spec
    File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
      return method()
    File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
      if self.pip_imported_during_build():
    File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 115, in pip_imported_during_build
      return any(
    File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 116, in <genexpr>
      frame.f_globals['__file__'].endswith('setup.py')
  KeyError: '__file__'
  

• this is only a problem with python -c, executing a .py file (with the same content) works fine

[AdmiralNemo]

This breaks python -m ensurepip:

python -m venv test-venv
test-venv/bin/python -m pip install 'setuptools>=60'
test-venv/bin/python -m ensurepip

Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/usr/lib64/python3.9/runpy.py", line 206, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib64/python3.9/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/usr/lib64/python3.9/importlib/util.py", line 103, in find_spec
    return _find_spec(fullname, parent_path)
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 115, in pip_imported_during_build
    return any(
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 116, in <genexpr>
    frame.f_globals['__file__'].endswith('setup.py')
KeyError: '__file__'
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.9/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 221, in _main
    return _bootstrap(
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 140, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 52, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/dhatch/test-venv/bin/python', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpijdjrgp5/setuptools-53.0.0-py3-none-any.whl\', \'/tmp/tmpijdjrgp5/pip-21.0.1-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpijdjrgp5\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

[ironstar94]

The bug is occuring because setuptools bundled distutils that's breaking stuff. It was apparently only tested on Fedora #2896

Temporary workaround is setting
SETUPTOOLS_USE_DISTUTILS=stdlib in your environment.

[mariocj89]

Changing to ignore __file__ if it is not present does seems to indeed fix ensurepip.
If the code there is just to check if this is being invoked from a setup.py it should be safe to just do something like:
frame.f_globals.get('__file__', '').endswith('setup.py')

Happy to send a PR.

[jaraco]

This issue is pretty much a duplicate of #2940.