Provide classifiers in python-inspector output

[mnonnenmacher]

ORT uses the classifiers as a fallback to get license information for a package. For example, this package does not provide a declared license but has the classifier License :: OSI Approved :: BSD License:
https://pypi.org/pypi/click/6.7/json

This is missing the version of the BSD license, but it is still much better than having no license information at all.

[pombredanne]

@mnonnenmacher actually the story is a slightly different: we collect the classifiers but sort them in two:

• https://github.com/nexB/python-inspector/blob/64d84269004e2ca5e6e6772db090092879f51f72/src/_packagedcode/pypi.py#L1118
• classifiers about licenses go into the "declared_license" package attribute https://github.com/nexB/python-inspector/blob/64d84269004e2ca5e6e6772db090092879f51f72/src/_packagedcode/pypi.py#L1100
  and we also combine classifiers there with the "License" field if any.
• other classifiers end up with other keywords https://github.com/nexB/python-inspector/blob/64d84269004e2ca5e6e6772db090092879f51f72/src/_packagedcode/pypi.py#L1142

Then (using code that's for now only in ScanCode toolkit and calls the license detection engine) we can compute a proper normalized license from that license "declaration" in https://github.com/nexB/python-inspector/blob/64d84269004e2ca5e6e6772db090092879f51f72/src/_packagedcode/pypi.py#L1142

That's actually the approach for all packages and not only Python packages.

Here we have these issues though:

1. we have not wired in completely collecting the licenses. We should at least get the declared licenses and other fields we already parse including keywords
2. if you need all classifiers verbatim, then could add them possibly as extra_data

Here is what we get with a full scancode for click-6.7:

packages:
    -   type: pypi
        namespace:
        name: click
        version: '6.7'
        qualifiers: {}
        subpath:
        primary_language: Python
        description: |
            A simple wrapper around optparse for powerful command line utilities.
            UNKNOWN
        release_date:
        parties:
            -   type: person
                role: author
                name: Armin Ronacher
                email: armin.ronacher@active-4.com
                url:
        keywords:
            - 'Programming Language :: Python'
            - 'Programming Language :: Python :: 3'
        homepage_url: http://github.com/mitsuhiko/click
        download_url:
        size:
        sha1:
        md5:
        sha256:
        sha512:
        bug_tracking_url:
        code_view_url:
        vcs_url:
        copyright:
        license_expression: bsd-new
        declared_license:
            classifiers:
                - 'License :: OSI Approved :: BSD License'
        notice_text:
        source_packages: []
        extra_data: {}
        repository_homepage_url: https://pypi.org/project/click
        repository_download_url: https://pypi.org/packages/source/c/click/click-6.7.tar.gz
        api_data_url: https://pypi.org/pypi/click/6.7/json
        package_uid: pkg:pypi/click@6.7?uuid=100d8043-6a50-4704-ac97-8fec63b45213
        datafile_paths:
            - click-6.7/click.egg-info/PKG-INFO
        datasource_ids:
            - pypi_editable_egg_pkginfo
        purl: pkg:pypi/click@6.7

[mnonnenmacher]

Thanks for the explanation @pombredanne, but with python-inspector I only get the classifiers in the files section, the entries in the packages section only have a plain string as declared license:

{
  ...
  "files": [
    {
      ...
      "package_data": [
        {
          "type": "pypi",
          "namespace": null,
          "name": "spdx-tools",
          "version": "0.5.4",
          ...
          "declared_license": {
            "license": "Apache-2.0",
            "classifiers": [
              "License :: OSI Approved :: Apache Software License"
            ]
          },
          ...
        }
      ]
    }
  ],
  ...
  "packages": [
    {
      "type": "pypi",
      "namespace": null,
      "name": "isodate",
      "version": "0.6.1",
      ...
      "declared_license": "BSD",
      ...
    },
    ...
  ]
}

The package above should also have the classifier "License :: OSI Approved :: BSD License" in the declared license according to your description: https://pypi.org/pypi/isodate/0.6.1/json

[sschuberth]

we can compute a proper normalized license from that license "declaration"

Note that, however, you cannot normalize e.g. "BSD License" to BSD-3-Clause as it theoretically could be BSD-2-Clause as well.

[pombredanne]

@mnonnenmacher

The package above should also have the classifier "License :: OSI Approved :: BSD License" in the declared license according to your description: https://pypi.org/pypi/isodate/0.6.1/json

Correct: this the bug I mentioned

[pombredanne]

@sschuberth re:

Note that, however, you cannot normalize e.g. "BSD License" to BSD-3-Clause as it theoretically could be BSD-2-Clause as well.

ScanCode now reports a BSD-3-Clause with a lowered relevance in these cases. Please see aboutcode-org/scancode-toolkit#1901 for an extensive discussion including some FOSS-savvy lawyers comments for the rationale.

[TG1999]

Solved in https://github.com/nexB/python-inspector/tree/v0.8.5

[mnonnenmacher]

Thanks @TG1999 , I can confirm that this is fixed now (also see oss-review-toolkit/ort#6021).