Add glibc advisories

[armijnhemel]

glibc has started publishing its own advisories:

https://sourceware.org/git/?p=glibc.git;a=tree;f=advisories;h=a0872e990274aee4d881508dad1bce3ea49d4d07;hb=HEAD

[harsh098]

May I work on this issue I have gotten a rough Idea about how advisories are being imported.
Please see @ziadhany and @TG1999

[pombredanne]

@harsh098 re:

May I work on this issue I have gotten a rough Idea about how advisories are being imported.

sure, please do: do not hesitate to ask questions as you go.

[pombredanne]

@harsh098 See this commit by @siddhesh for some details:
https://sourceware.org/git/?p=glibc.git;a=commit;h=60c57b8467f11e334e7c7fd07d588c248e93d952

Move CVE information into advisories directory

One of the requirements to becoming a CVE Numbering Authority (CNA) is
to publish advisories. Do this by maintaining a file for each CVE fixed
in the advisories directory in the source tree. Links to the advisories
can then be shared as:

https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=advisories/GLIBC-SA-YYYY-NNNN

The file format at the moment is rudimentary and derives from the git
commit format, i.e. a subject line and a potentially multi-paragraph
description and then tags to describe some meta information. This is a
loose format at the moment and could change as we evolve this.

Also add a script process-fixed-cves.sh that processes these advisories
and generates a list to add to NEWS at release time.

An example of the format is: https://sourceware.org/git/?p=glibc.git;a=blob;f=advisories/GLIBC-SA-2023-0003;h=0950dc0792d94b28703837dce353a8ca5be0d96b;hb=HEAD

@siddhesh just a FYI, we will start consuming these, which means writing a parser for these... If you like, we could help you with adopting some format that makes it easy to consume your advisories!

[harsh098]

Sure @pombredanne I'll be asking wherever I get stuck

[siddhesh]

@siddhesh just a FYI, we will start consuming these, which means writing a parser for these... If you like, we could help you with adopting some format that makes it easy to consume your advisories!

Thanks, I'd love to hear feedback. We started to but then stayed away from format discussions because we agreed on getting the information right first. Whatever feedback you give should help us clean up on both fronts, format as well as data, based on what you typically publish for advisories.

[harsh098]

Hi @pombredanne ,
I had problem figuring out the purl for glibc from glibc.
I had a look at
https://github.com/package-url/purl-spec
I came up with
pkg:generic/glibc@version?commit=commit-id
may I proceed with it ?

[harsh098]

@siddhesh and @pombredanne
I wrote a sample parser to parse advisories at https://sourceware.org/git/?p=glibc.git;a=tree;f=advisories;h=a0872e990274aee4d881508dad1bce3ea49d4d07;hb=HEAD

Please check the format, this will be adapted to AdvisoryData in final importer

def parse_advisory(file):
    """
        THE Given Function Scrapes the following data from the given advisory files at
        https://sourceware.org/git/?p=glibc.git;a=tree;f=advisories

            1. CVE_ID
                    DataType: str

            2. Vulnerable Versions 
                    Schema :
                        "version": "None if no Vulnerable-Backport Exists else report Commit_ID of Vulnerable-Backport"
            
            3. Fixed Versions
                    Schema :
                        "version": "Commit_ID that implements the fix"

            4. Title of CVE
                    DataType: str
            
            5. Summary of CVE
                    DataType: str

        
        The function returns the following data
                title: str #Title of CVE
                date: str  # Date Reported
                summary: str #Description of Advisory
                cve_desc: List[str]   # CVE_ID determined
                vuln_versions: Dict[str,str] #List of Vulnerable Versions
                fixed_versions: Dict[str,str] #List of Fixed Versions
                    
    """
    with open(file) as f:
        lines = f.readlines()
        title = lines[0]
        date: str
        cve_desc: str
        vuln_versions = dict()
        fixed_versions = dict()
        summary=""
        for i in range(1, len(lines)):
            if lines[i]=='\n':
                continue
            if "CVE-Id:" in lines[i]:
                cve_desc = lines[i].split(":")[1].strip()
            elif "Public-Date" in lines[i]:
                date = lines[i].split(":")[1].strip()
            elif "Vulnerable-Commit" in lines[i]:
                vulnerable_version = lines[i].split(":")[1].strip().split("(")[1].split(")")[0].strip()
                commit = lines[i].split(":")[1].strip().split("(")[0].strip()
                vuln_versions[vulnerable_version] = commit
            elif "Vulnerable-Backport" in lines[i]:
                vulnerable_version = lines[i].split(":")[1].strip().split("(")[1].split(")")[0].strip()
                commit = lines[i].split(":")[1].strip().split("(")[0].strip()
                vuln_versions[vulnerable_version] = commit
            elif "Fix-Commit" in lines[i] or "Fix-Backport" in lines[i]:
                fixed_version = lines[i].split(":")[1].strip().split("(")[1].split(")")[0].strip()
                commit = lines[i].split(":")[1].strip().split("(")[0].strip()
                if fixed_version not in vuln_versions:
                    vuln_versions[fixed_version]=None
                fixed_versions[fixed_version] = commit
            else: 
                summary+=lines[i].strip()+" "
        
        return title, date, summary, cve_desc, vuln_versions, fixed_versions



if __name__ == '__main__' :
    for field in parse_advisory("GLIBC-SA-2023-0005"):
        print(field)
        

Please suggest any edge cases I'm missing out on.

Also @pombredanne please help me with purl for the glibc official sources as asked above.
Also, please let me know if a version can be specified as pre-2.39 in purl.
As, this advisory https://sourceware.org/git/?p=glibc.git;a=blob;f=advisories/GLIBC-SA-2023-0005;h=a518af803a7121038e912fae7fdcf4fe7260d362;hb=HEAD
reports a version as pre-2.39

[pombredanne]

@harsh098 re:

I wrote a sample parser to parse advisories at https://sourceware.org/git/?p=glibc.git;a=tree;f=advisories;h=a0872e990274aee4d881508dad1bce3ea49d4d07;hb=HEAD

Can you push this in a branch with some tests?
Since there is a small number of advisories for now, you should be able to parse them all in the tests. It will be simpler to post comments on the code in a PR.

As for the code, my knee jerk reaction is that you should avoid using indexes for lines as this makes things hard to read and understand.... But please write the unit tests first!

Also I wonder if the format is not something more or less like a Debian/RFC822 format or vaguely YAML'ish. If Debian like, we have a robust parser for Debian control files in https://github.com/nexB/debian-inspector/blob/main/src/debian_inspector/deb822.py and https://github.com/nexB/debian-inspector/blob/main/src/debian_inspector/

[pombredanne]

@siddhesh re:

Hi @pombredanne ,
I had problem figuring out the purl for glibc from glibc.
I had a look at
https://github.com/package-url/purl-spec
I came up with
pkg:generic/glibc@version?commit=commit-id
may I proceed with it ?

There are multiple ways to handle this:

1. A generic PURL like you suggested but this will need a download_url qualifier

2. A new PURL type for glibc ... which IMHO makes the most sense since this is now a CNA and is a rather important package ;) ... we can start using it here and then submit a PR to the purl-spec to add this as a supported type.

3. Another option is to instead create a sourceware type and have glibc as a name under this. sourceware is "forge-like" with https://sourceware.org/glibc/ as a homepage URL as https://www.gnu.org/software/libc/ as a secondary (or the otherway around), http://ftp.gnu.org/gnu/glibc/ as the root for the downloads, the git repo https://sourceware.org/git/?p=glibc.git for the code and advisories. There is not much structured metadata otherwise, but this a super popular package so it does not matter, this is well known enough.

4. yet another option is to have a gnu type and a libc as a name.

I think I would like to go with 3. sourceware but since this would be the way glibc may start to be referenced in the many many tools and specs that use PURL, it would be great to have @siddhesh and some glibc community input. pkg:gnu/libc@<someversion> vs. pkg:sourceware/glibc@<someversion> and we can use glibc or libc as a name???

[harsh098]

@harsh098 re:

I wrote a sample parser to parse advisories at https://sourceware.org/git/?p=glibc.git;a=tree;f=advisories;h=a0872e990274aee4d881508dad1bce3ea49d4d07;hb=HEAD

Can you push this in a branch with some tests? Since there is a small number of advisories for now, you should be able to parse them all in the tests. It will be simpler to post comments on the code in a PR.

As for the code, my knee jerk reaction is that you should avoid using indexes for lines as this makes things hard to read and understand.... But please write the unit tests first!

Also I wonder if the format is not something more or less like a Debian/RFC822 format or vaguely YAML'ish. If Debian like, we have a robust parser for Debian control files in https://github.com/nexB/debian-inspector/blob/main/src/debian_inspector/deb822.py and https://github.com/nexB/debian-inspector/blob/main/src/debian_inspector/

I haven't started work on the importer yet. My first goal is to get the parser right.
So, may I write these advisories and tests in a separate repo and add you(@pombredanne ) and @siddhesh as contributor.
My Goal is to first get the parser to cover all edge cases.

The format is not vary YAML ish and is much more like Debian/RFC822.
But the grammar for the format used by GLIBC is not very well defined unlike RFC822 and the fields are entirely different.
That calls for a parser of its own.

A few recommendations would be to get the versioning in the advisories standardised instead of using versioning like pre-2.39 use concrete versions or use <= better would be to refer to version tags

[harsh098]

Also, I wanted to ask @pombredanne how do you fetch the tags in the VCSResponse class as this would be very valuable in determining affected versions for versions like pre-2.39 and so on ?

[pombredanne]

@harsh098 Please start a PR early, even if this is only for the parser. We cannot review code otherwise. That's the FOSS way!
If anything that will help us understand any issues with the format.

And wrt. the Git tags and in general the glibc versions there are two ways to get versions:

1. Parse http://ftp.gnu.org/gnu/glibc/ as HTML. We have code in PurlDB to do this in https://github.com/search?q=repo%3AnexB%2Fpurldb href&type=code way too many places... we should have a shared function instead. An example is this https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/etc/scripts/utils_thirdparty.py#L1739 or OpenSSL releases https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/minecode/visitors/openssl.py#L38
2. Collect Git tags either from a clone. There is code in FetchCode for this. And also in PurlDB https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/packagedb/find_source_repo.py#L427 We should have only one way in FetchCode instead. It boils down to running https://git-scm.com/docs/git-ls-remote.html (or similar through a library call)

@keshav-space @TG1999 do you have some hints on which function/code to call for this?

Note that ideally glibc would expose a simple way to list ALL the relevant released versions. There is a long tail of legacy release tags from CVS, Fedora tags, various heads and more that we should likely exclude, going back 30 years and to the CVS days. See https://sourceware.org/git/?p=glibc.git;a=tags

[pombredanne]

@siddhesh The meaning of these two tags is obvious:

Vulnerable-Commit: 973fe93a5675c42798b2161c6f29c01b0e243994 (pre-2.39)
Fix-Commit: ec6b95c3303c700eb89eebeda2d7264cc184a796 (2.39)

What about these other two?

Vulnerable-Backport: e09ee267c03e3150c2c9ba28625ab130705a485e (2.34)
Fix-Backport: 8006457ab7e1cd556b919f477348a96fe88f2e49 (2.34)

A search either leads to the glibc advisories or -- in a funny tautological way -- to this issue ;)

[harsh098]

@harsh098 Please start a PR early, even if this is only for the parser. We cannot review code otherwise. That's the FOSS way! If anything that will help us understand any issues with the format.

And wrt. the Git tags and in general the glibc versions there are two ways to get versions:

1. Parse http://ftp.gnu.org/gnu/glibc/ as HTML. We have code in PurlDB to do this in https://github.com/search?q=repo%3AnexB%2Fpurldb href&type=code way too many places... we should have a shared function instead. An example is this https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/etc/scripts/utils_thirdparty.py#L1739 or OpenSSL releases https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/minecode/visitors/openssl.py#L38
2. Collect Git tags either from a clone. There is code in FetchCode for this. And also in PurlDB https://github.com/nexB/purldb/blob/e1a20006b513a0379e69a69624da4e614a4f27dd/packagedb/find_source_repo.py#L427 We should have only one way in FetchCode instead. It boils down to running https://git-scm.com/docs/git-ls-remote.html (or similar through a library call)

@keshav-space @TG1999 do you have some hints on which function/code to call for this?

Note that ideally glibc would expose a simple way to list ALL the relevant released versions. There is a long tail of legacy release tags from CVS, Fedora tags, various heads and more that we should likely exclude, going back 30 years and to the CVS days. See https://sourceware.org/git/?p=glibc.git;a=tags

@pombredanne
Turns out I need the output of
git describe --tags <commit-id>
to find the first tag that contains the commit