support v1 recipe format via rattler-build

[beckermr]

This issue is to track progress on supporting the v1 recipe format via rattler build.

Minimal Work Items for Uploads:

• ensure conda-forge-feedstock-check-solvable works for the new format (feat: solvability implementation for rattler-build regro/conda-forge-feedstock-check-solvable#58)
• ensure the other migrators in the bot don't choke (feat: skip all migrators on schema v1 for now regro/cf-scripts#3006 misc: disable recipe.yaml for migrators regro/cf-scripts#2918)
• ensure ABI migrations in the bot can go through
  • enable build number bumps (feat: add build number editing for v1 recipes regro/cf-scripts#3009)
  • add some tests in the bot for ABI+v1 recipe migrations (replicate the tests of yaml-based migrations in this file: https://github.com/regro/cf-scripts/blob/main/tests/test_migrators.py, feat: test v1 recipe.yaml migrations regro/cf-scripts#3011)
• fix v1 recipe linting (v1 recipe linting broken conda-smithy#2067)
• fully turn on the linter (it still hints) (feat: enable v1 lint for PRs conda-forge-webservices#685, feat: enable v1 linting conda-forge-feedstock-ops#14)
• turn on uploads (last one! feat: allow uploading on main channel for v1 recipes conda-forge-ci-setup-feedstock#337)

Next Steps + Bug Reports

• rerendering readme fails for v1 recipes and c stdlib conda-smithy#2085
• move variants.yaml to a backup file when rendering conda-smithy#2082
• Support variants.yml staged-recipes#27750
• examples break build-locally.py staged-recipes#27695
• fix bug with test: None regro/cf-scripts#3018
• Support conda_build_config_vars in rattler-build conda-forge-ci-setup-feedstock#359
• feat: add version bumping for v1 recipes regro/cf-scripts#3525
• enable selected minimigrators as they are needed (Migrators for v1 recipes regro/cf-scripts#3642)
• v1 recipe format solvable tests are failing regro/conda-forge-feedstock-check-solvable#61
• add linter lint+hint for recipe being able to be parsed so we can run automatic conversion (feat: add lint+hint for recipe parsing conda-smithy#2141)

Running List of Blockers for Automatic Migration PRs from the Bot

• rattler-build produces bad package version if 0 involved prefix-dev/rattler-build#1455
• Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451
• ignore_run_exports doesn't work with ${{ compiler('cxx') }} prefix-dev/rattler-build#1445
• rattler-build doesn't include licenses if it's a directory prefix-dev/rattler-build#1443
• test environment not activated when running the python test type prefix-dev/rattler-build#1437
• test environment not recognized by conda as an environment prefix-dev/rattler-build#1439
• test env not properly activated in another case prefix-dev/rattler-build#1440
• cannot properly capture stderr and stdout in commands run in subshells via backticks for tests prefix-dev/rattler-build#1438
• variant env vars not set for test scripts prefix-dev/rattler-build#1317
• enable more flexible patching like conda-build prefix-dev/rattler-build#1474
• BUG: mac relinking doesn't rewrite default install_name in dependencies like conda-build prefix-dev/rattler-build#1476
• test.sources requires trailing slash for directories prefix-dev/rattler-build#1480
• Check applying patches with tabulations prefix-dev/rattler-build#1491
• rattler-build build panics when test.python.python_version match spec is invalid prefix-dev/rattler-build#1493
• [bug] rpath subdirectories on linux prefix-dev/rattler-build#1498
• Missing env vars in build script prefix-dev/rattler-build#889
• How to reference relative to $PREFIX in license_file? prefix-dev/rattler-build#1521
• bug: too many open files again and RATTLER_IO_CONCURRENCY_LIMIT env var not helping prefix-dev/rattler-build#1638
• Credential leaks in about.json when sending HTTP credentials via CLI prefix-dev/rattler-build#1636
• test env is in tempdir and so cannot debug by hand prefix-dev/rattler-build#1436
• turn off set -x for test scripts prefix-dev/rattler-build#1639
• test environment not recognized by conda prefix-dev/rattler-build#1643
• CONDA_BUILD_CROSS_COMPILATION is always defined and set to 1 for noarch: python builds, while it is not defined in conda-build prefix-dev/rattler-build#1784
• executed tests as part of the build are copied to /etc prefix-dev/rattler-build#1750
• bug(variants): empty string variants not processed correctly prefix-dev/rattler-build#1748
• SP_DIR not set during build and test time prefix-dev/rattler-build#1733
• make filter works with tarball/zip source prefix-dev/rattler-build#1671
• variant env vars not set for test scripts prefix-dev/rattler-build#1317
• CONDA_BUILD_STATE=test not set during test time prefix-dev/rattler-build#1897

supersedes #1612

[wolfv]

I think for the conda-forge-feedstock-check solvable I will have a good grip on what needs to be done.

For the second point, ensure that ABI migrations in the bot can go through, I actually don't really know where to start. Do you have additional pointers?

[beckermr]

Right so for that, once the solver checks work, we should add a test to the bot repo that does an abi migration and make sure it all runs.

[wolfv]

regro/conda-forge-feedstock-check-solvable#58 this is the PR for adding solvability checks with rattler-build.

[wolfv]

conda-forge/conda-forge-webservices#685 opened to enable lints on PRs etc.

[wolfv]

@beckermr - I think I need some help / guidance in order to understand what's needed for the cf-graph items. How can I ensure that?

[wolfv]

Thanks @beckermr for updating the todo items. I can make the build number PR right away (it's a subset of the version bump PR) so we can get that one in straight away! If you let me know how / where to add tests for migrations I can do it tomorrow, too :)

[beckermr]

I updated the issue list above with a pointer to the correct test file. We should replicate all of the tests there for yaml migrations to ensure they work on v1 recipes.

[wolfv]

Added PR for build number bumps: regro/cf-scripts#3009

[wolfv]

Only the tests to go! :)

[wolfv]

Ticked off the last one (uploading packages) :)

There are two more on staged-recipes:

• Support variants.yml staged-recipes#27750
• examples break build-locally.py staged-recipes#27695

[isuruf]

Opened

• Support conda_build_config_vars in rattler-build conda-forge-ci-setup-feedstock#359