Dependabot trying to download setup.py after move to pyproject.toml

[Panaetius]

Package ecosystem
poetry (pip)
Package manager version
1.1.11
Language version
Python 3.7
Manifest location and content prior to update
https://github.com/SwissDataScienceCenter/renku-python/blob/dependabot-updates/pyproject.toml

dependabot.yml content
https://github.com/SwissDataScienceCenter/renku-python/blob/dependabot-updates/.github/dependabot.yml

What you expected to see, versus what you actually saw
I expect PRs to be opened with version updates, but instead Dependabot errors with

Dependabot couldn't fetch all your path-based dependencies

The affected dependencies were setup.py.

To use path-based dependencies with Dependabot the paths must be relative and resolve to a directory in this project's source code.

  proxy | time="2021-12-03T09:16:26Z" level=info msg="proxy starting" commit=d5f262668736016da1a91e42cb4fba36a081bddf
  proxy | 2021/12/03 09:16:26 Listening (:1080)
updater | 2021-12-03T09:16:26.805063597 [anonymous-instance:main:WARN:src/firecracker/src/main.rs:370] You are using a deprecated parameter: --seccomp-level 2, that will be removed in a future version.
updater | 2021-12-03T09:16:26.904453390 [239653965:main:WARN:src/devices/src/legacy/serial.rs:432] Detached the serial input due to peer close/error.
updater | time="2021-12-03T09:16:30Z" level=info msg="guest starting" commit=8e918e4cf121d74a5b43e170ec4a717c1df98819
updater | time="2021-12-03T09:16:30Z" level=info msg="starting job..." fetcher_timeout=5m0s job_id=239653965 updater_timeout=45m0s updater_version=0.169.3-388839f8e94e2e09f180f781975462a82ae23d04
updater | I, [2021-12-03T09:16:33.533694 #6]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | warning: parser/current is loading parser/ruby27, which recognizes
updater | warning: 2.7.5-compliant syntax, but you are running 2.7.1.
updater | warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
updater | INFO <job_239653965> Starting job processing
  proxy | 2021/12/03 09:16:39 [002] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/git/refs/heads/dependabot-updates
  proxy | 2021/12/03 09:16:39 [002] * authenticating github api request
  proxy | 2021/12/03 09:16:39 [002] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/git/refs/heads/dependabot-updates
  proxy | 2021/12/03 09:16:39 [004] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [004] * authenticating github api request
  proxy | 2021/12/03 09:16:39 [004] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [006] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/pyproject.toml?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [006] * authenticating github api request
  proxy | 2021/12/03 09:16:39 [006] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/pyproject.toml?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [008] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/poetry.lock?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [008] * authenticating github api request
  proxy | 2021/12/03 09:16:39 [008] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/poetry.lock?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [010] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/.github?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [010] * authenticating github api request
  proxy | 2021/12/03 09:16:39 [010] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/.github?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [012] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/design?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:39 [012] * authenticating github api request
  proxy | 2021/12/03 09:16:40 [012] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/design?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [014] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [014] * authenticating github api request
  proxy | 2021/12/03 09:16:40 [014] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [016] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs/requirements.txt?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [016] * authenticating github api request
  proxy | 2021/12/03 09:16:40 [016] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs/requirements.txt?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [018] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs/spelling_wordlist.txt?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [018] * authenticating github api request
  proxy | 2021/12/03 09:16:40 [018] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/docs/spelling_wordlist.txt?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [020] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/helm-chart?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:40 [020] * authenticating github api request
  proxy | 2021/12/03 09:16:41 [020] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/helm-chart?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [022] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/renku?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [022] * authenticating github api request
  proxy | 2021/12/03 09:16:41 [022] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/renku?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [024] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/tests?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [024] * authenticating github api request
  proxy | 2021/12/03 09:16:41 [024] 200 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/tests?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [026] GET https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/setup.py?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
  proxy | 2021/12/03 09:16:41 [026] * authenticating github api request
  proxy | 2021/12/03 09:16:41 [026] 404 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/setup.py?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
updater | ERROR <job_239653965> Error during file fetching; aborting
updater | INFO <job_239653965> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2021-12-03T09:16:41Z" level=info msg="task complete" container_id=job-239653965-file-fetcher exit_code=0 job_id=239653965 step=fetcher
updater | time="2021-12-03T09:16:42Z" level=warning msg="failed during fetch, skipping updater" job_id=239653965

We previously used setuptools with setup.py but switched to using pyproject.toml and poetry. Dependabot still tries to download setup.py, even though the files is deleted.

Is setup.py mandatory or why does this fail?

[jku]

We previously used setup.py + requirements*.txt in https://github.com/theupdateframework/python-tuf, but have moved to setup.cfg + requirements*.txt (in both cases we're really only interested in dependabot updates to requirements files):

• dependabot started to silently fail when we did that change -- we have now been without dependency updates without knowing it for some time. I would not be surprised if there are many more projects modernising their build systems in the same way
• adding a completely empty "setup.py" file makes dependabot work again so dependabot clearly can handle the dependencies in the project: it just decides to not do it because it fails to download setup.py

I've tried to look into the dependabot sources but lack of ruby skills has prevented progress. Still, the issue seems to be this:
Even though code comments imply that setup.py is not required, the file fetcher fails if setup.py does not exist

Example log: https://github.com/theupdateframework/python-tuf/network/updates/276409663

[DanielNoord]

It's weird as the parser for pip and Python seem to check if the file exists. See:

dependabot-core/python/helpers/lib/parser.py

Line 97 in f5599d3

if os.path.isfile(setup_py_path):

I guess somebody with more knowledge of dependabot internals will need to take a look at this.

[Panaetius]

From the error in my logs:

404 https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/setup.py?ref=c29a723e458b7184d18bedeb4e31fac50122bed1
updater | ERROR <job_239653965> Error during file fetching; aborting

It looks like it tries to download the file and fails. So I guess this happens before the check you linked.

I'm not really familiar with Ruby and couldn't find where it executes this part.

But in the logs it first does https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/?ref=c29a723e458b7184d18bedeb4e31fac50122bed1, which returns all files/folders in the root of the repository. But it does NOT return setup.py as that doesn't exist at this point. But then later it does try to fetch https://api.github.com:443/repos/SwissDataScienceCenter/renku-python/contents/setup.py?ref=c29a723e458b7184d18bedeb4e31fac50122bed1 for some reason and fails. So I'm confused why it tries to get setup.py even though it's not included in the listing.

[Panaetius]

On a side note, I just updated our dependency branch (We have Dependabot running against a separate branch so we can do batched updates) and now it works again. I guess it had just cached that it needs setup.py and since I didn't update that branch, it just never tried to rerun on that one, even once the cache expired?

Maybe the code could be changed to retry and bypass the cache if downloading of a file fails? If this is indeed the reason.

[jurre]

Thanks for raising this, so it seems like we do expect a setup.py present, unless you're using pyproject.toml, the code where this fails is:

dependabot-core/python/lib/dependabot/python/file_fetcher.rb

Lines 278 to 293 in 316c2d1

	def fetch_path_setup_file(path, allow_pyproject: false)
	path_setup_files = []
	unless path.end_with?(".tar.gz", ".whl", ".zip")
	path = Pathname.new(File.join(path, "setup.py")).cleanpath.to_path
	end
	return [] if path == "setup.py" && setup_file
	path_setup_files <<
	begin
	fetch_file_from_host(
	path,
	fetch_submodules: true
	).tap { |f| f.support_file = true }
	rescue Dependabot::DependencyFileNotFound
	raise unless allow_pyproject

. I am however not sure why that assumption is made, I have a change locally that suppresses that error when a setup.cfg file is present and the update seems to succeed, however, I'd like to dig into why we assume a setup.py should be present (maybe oversight, but would like to confirm).

---

Fwiw (for future self) the change is:

 % g diff
diff --git a/python/lib/dependabot/python/file_fetcher.rb b/python/lib/dependabot/python/file_fetcher.rb
index d813bdeb1..d24f69b2b 100644
--- a/python/lib/dependabot/python/file_fetcher.rb
+++ b/python/lib/dependabot/python/file_fetcher.rb
@@ -290,7 +290,7 @@ module Dependabot
               fetch_submodules: true
             ).tap { |f| f.support_file = true }
           rescue Dependabot::DependencyFileNotFound
-            raise unless allow_pyproject
+            raise unless allow_pyproject || setup_cfg_file

             fetch_file_from_host(
               path.gsub("setup.py", "pyproject.toml"),

[DanielNoord]

@jurre Not sure if this is relevant, but setup.py is no longer required since setuptools 40.9.0, see https://setuptools.pypa.io/en/latest/setuptools.html. That version was released on April 3, 2019. I have no idea how old Dependabot is, but it might have been written when this change wasn't yet released or relatively unknown?
(I feel that this behaviour is still relatively unknown as I got multiple comments when I "forgot" the file in one of my packages)

[jurre]

@jurre Not sure if this is relevant, but setup.py is no longer required since setuptools 40.9.0, see https://setuptools.pypa.io/en/latest/setuptools.html. That version was released on April 3, 2019. I have no idea how old Dependabot is, but it might have been written when this change wasn't yet released or relatively unknown? (I feel that this behaviour is still relatively unknown as I got multiple comments when I "forgot" the file in one of my packages)

Yep, we do predate that, that's super useful context thanks!

[jku]

-            raise unless allow_pyproject
+            raise unless allow_pyproject || setup_cfg_file

Just to be clear: We had a pyproject.toml and setup.cfg when the issue was present: Dependabot would still fail hard when setup.py was not found.