maven-heaven: Design and implement Scancode.io pipeline for a single Maven package

[pombredanne]

A new pipeline designed specifically for Java packages, that is aware of the conventions, and unwritten community practices in license and origin documentation. It will combine code matching, binary analysis, and license detection

[chinyeungli]

I think we have many area covered. The major gap is mapping the dependency packages listed in pom.xml against the deploy codebase #1907

[chinyeungli]

To create a specific pipeline for maven, we first need to understand the content or what can be in maven.
Maven packages are primarily Java-based, but it can also contains Koltin, Scala and Groovy. In addition, some packages may bundle native code (e.g. .so, .dll, .exe), so we need to take these into consideration when creating a maven pipeline.

[chinyeungli]

I'm revisiting this issue and I'm thinking #1907 may not be directly related, as it focuses on D2D. The core objective here should be to extract metadata from the Maven manifest specifically from the pom.xml.

There are two key scenarios:

1. No pom.xml

Sample Package: JUnit

Ran "scan_single_package" pipeline, many fields are not filled as the package doesn't have pom.xml, and therefore no download url, license, dependency data and other information.
See
https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2-sources.jar
https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.jar

We should somehow try to get the pom.xml and extract information from it:
https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.pom

Fields that worth of an attention are:

• <license>
• <scm>
• <organization>
• <developers>

2. Missing data in pom.xml

Sample Package: Guava (https://repo1.maven.org/maven2/com/google/guava/guava/33.5.0-jre/)

Ran "scan_single_package" pipeline, many fields are filled in correctly, but the "declared_license_expression" field is empty.
Although the package has pom.xml, it doesn't have any license information. (See https://repo1.maven.org/maven2/com/google/guava/guava/33.5.0-jre/guava-33.5.0-jre.pom)
perhaps we can use the license detected in "RESOURCES" such as META-INF/LICENSE that share the same "package_uid" in "for_package" and copy the license information over to the "PACKAGE" worksheet.

@pombredanne / @TG1999 Is this the main goal, or is there additional data we want to extract that isn't currently captured by the scan_single_package pipeline? If not. can you give me concrete examples of what's the main goal here? Thanks.

[chinyeungli]

For the first scenario, it will be something like the following:
From

purl	type	namespace	name	version	qualifiers	subpath	tag	primary_language	description	notes	release_date	parties	keywords	homepage_url	download_url	bug_tracking_url	code_view_url	vcs_url	repository_homepage_url	repository_download_url	api_data_url	size	md5	sha1	sha256	sha512	copyright	holder	declared_license_expression	declared_license_expression_spdx	other_license_expression	other_license_expression_spdx	extracted_license_statement	notice_text	source_packages	package_uid	is_private	is_virtual	datasource_ids	datafile_paths	file_references	missing_resources	modified_resources	xlsx_errors
pkg:maven/junit/JUnit@4.13.2	maven	junit	JUnit	4.13.2								{'url': None, 'name': 'JUnit', 'role': 'vendor', 'type': None, 'email': None}		http://junit.org																						pkg:maven/junit/JUnit@4.13.2?uuid=8ae4eaf6-4f7c-45de-964d-d44f27f1080c			java_jar_manifest	META-INF/MANIFEST.MF

To

purl	type	namespace	name	version	qualifiers	subpath	tag	primary_language	description	notes	release_date	parties	keywords	homepage_url	download_url	bug_tracking_url	code_view_url	vcs_url	repository_homepage_url	repository_download_url	api_data_url	size	md5	sha1	sha256	sha512	copyright	holder	declared_license_expression	declared_license_expression_spdx	other_license_expression	other_license_expression_spdx	extracted_license_statement	notice_text	source_packages	package_uid	is_private	is_virtual	datasource_ids	datafile_paths	file_references	missing_resources	modified_resources	xlsx_errors
pkg:maven/junit/junit@4.13.2	maven	junit	junit	4.13.2				Java	JUnit JUnit is a unit testing framework for Java, created by Erich Gamma and Kent Beck.			{'url': None, 'name': 'David Saff', 'role': 'developer', 'type': 'person', 'email': 'david@saff.net'} {'url': None, 'name': 'Kevin Cooney', 'role': 'developer', 'type': 'person', 'email': 'kcooney@google.com'} {'url': None, 'name': 'Stefan Birkner', 'role': 'developer', 'type': 'person', 'email': 'mail@stefan-birkner.de'} {'url': None, 'name': 'Marc Philipp', 'role': 'developer', 'type': 'person', 'email': 'mail@marcphilipp.de'} {'url': 'https://github.com/junit-team/junit4/graphs/contributors', 'name': 'JUnit contributors', 'role': 'contributor', 'type': 'person', 'email': 'team@junit.org'} {'url': 'http://www.junit.org', 'name': 'JUnit', 'role': 'owner', 'type': 'organization', 'email': None}		http://junit.org		https://github.com/junit-team/junit4/issues	https://github.com/junit-team/junit4	git://github.com/junit-team/junit4.git	https://repo1.maven.org/maven2/junit/junit/4.13.2/	https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.jar	https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.pom								epl-1.0	EPL-1.0			- license: name: Eclipse Public License 1.0 url: http://www.eclipse.org/legal/epl-v10.html		pkg:maven/junit/junit@4.13.2?classifier=sources	pkg:maven/junit/junit@4.13.2?uuid=0e471be3-b35d-42ca-ab4f-424c6bdd9ae0			maven_pom	https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.pom
pkg:maven/junit/JUnit@4.13.2	maven	junit	JUnit	4.13.2								{'url': None, 'name': 'JUnit', 'role': 'vendor', 'type': None, 'email': None}		http://junit.org																						pkg:maven/junit/JUnit@4.13.2?uuid=dd81497c-25e6-4897-bdd8-f51a3f65c160			java_jar_manifest	META-INF/MANIFEST.MF

Note that for the "datafile_paths", I use the direct URL to the pom file.

[chinyeungli]

@pombredanne / @TG1999 / @tdruez
I’ve successfully implemented the logic to fetch and scan a remote pom.xml, and I can update the existing scanned package data using the results from that remote file.
However, I'm running into an issue when trying to update the Dependencies data. During the load_inventory_from_toolkit_scan() step, the code attempts to create DiscoveredDependency instances and expects a datafile_path that maps to a CodebaseResource in the project (https://github.com/aboutcode-org/scancode.io/blob/1763_pipeline_for_maven/scanpipe/models.py#L4187).
Since the original project doesn't contain the pom.xml, and I'm fetching it remotely, I get the following error:

Cannot assign "....": "DiscoveredDependency.datafile_resource" must be a "CodebaseResource" instance.

Using the temporary location of the fetched pom.xml doesn’t help either, because that file isn't registered as a CodebaseResource in the project.
Any suggestions on how to resolve this?

[tdruez]

@chinyeungli DiscoveredDependency.datafile_resource so the first option is to keep it null (and maybe use another field to keep a loose relation/information about the pom.xml)

The second approach would be to create the fetched pom.xml as a project CodebaseResource, but this will break the integrity of the "codebase" as it is not really part of the package files. This may be a source of more issues.

Is the datafile_resource value critical in this analysis context?

[chinyeungli]

@tdruez the datafile_resource is not critical. It offers a clue about where the information comes from, but this can actually be determined indirectly by mapping the for_package_uid field in dependencies to the package_uid field in packages to identify which package the dependencies originate from. So, I think keeping it null will make things easier.