Triage Buckets and Prioritization

[lirantal]

Background

We are thinking that the current bucket order helps and a step in a good direction but we want to further improve it. Right now we have 2 buckets: Low priority <100 d/l weekly and High priority >100 d/l weekly and current stats are 30 vs 36 respectively. So it helps, but we'd like to also make sure we don't starve the reports in >100 that are affecting a good part of the ecosystem (i.e: >100000 downloads).

Bucket Segmentation

Segmenting the buckets further so that we have:

1. Unmaintained - bucket for <100 but also for packages that didn't have a release/commit in the last 12 months. Possibly for this we could consider automatic disclosure once triage has confirmed the vulnerability. Requires discussion/agreement
2. Low (and active) - bucket for <10000 downloads
3. High - TBD based on stats
4. Critical - TBD based on stats but generally speaking this would be where we want to have significantly impactful modules, probably the list of top 1000 modules on npm or so that would have a serious impact in terms of their security issues and reach.

Bucket SLA and Leads

For each of the proposed buckets above we'd like to have defined SLAs so e.g. for the Critical bucket we'd have an SLA of triage of 24 hours so it can be addressed in a timely manner.

Moreover, we'd like to have defined Leads that would be the contact person for handling those and then we can assign members to each of these bucket groups too to focus their work there.

[sam-github]

While I don't object even a little to any of the above, I think an issue with just having different SLAs for different levels is it doesn't decrease the overall amount of work to be done by sec-wg volunteers, it just allows more time... but the work still needs to be done. And there is more work than people, so the result might be that bucket 3 and 4 meet SLAs, but buckets 1 and 2 will essentially go on a queue and never get off it, because no one will have had time.

I'd suggest that for buckets 1-3 above, that we make it clear that it is the reporter's responsibility to contact the package maintainers, and get a response from them, or to post a comment explaining that they have attempted to contact the maintainer, and got no response.

X days after the maintainers have been notified (or failed to respond), the vuln can be published, and the process no longer blocks on sec-wg activity.

The value of X could vary depending on the bucket, bucket 1 packages are essentially unused, the amount of energy the sec-wg should put into them should tend towards zero.

[lirantal]

it doesn't decrease the overall amount of work to be done by sec-wg volunteers, it just allows more time

Not only that. It allows us to focus on real impacting reports, which otherwise would get lost in the clutter of tens or hundreds of reports. The work will always remain, but also notice that if we have a bucket that we agree we will disclose right after triage then that is still helpful in terms of shortening the queue and the time to resolution.

I'd suggest that for buckets 1-3 above, that we make it clear that it is the reporter's responsibility to contact the package maintainers

Respectfully disagree here, and definitely for all of those buckets. We want to make it accessible for researchers to report, not harder. Plus, they may not even be familiar with the ecosystem so it would make their lives much harder.

X days after the maintainers have been notified (or failed to respond), the vuln can be published, and the process no longer blocks on sec-wg activity.

This is the current process anyway.

[sam-github]

I'll respond more fully, but just to make sure we agree on the current state of things:

I see Low Priority (< 100) vulns in H1 from over 6 months ago that have not been disclosed. This leads me to think that the vulnerabilities are being reported faster than the WG has capacity to handle the reports to completion.

Am I misunderstanding the state? I could be not understanding the flow, and maybe the vulns are slow through the process not because of WG capacity, but because the reporters aren't responding, or something like that.

[lirantal]

the vulnerabilities are being reported faster than the WG has capacity to handle the reports to completion

It means that we aren't doing a good job in catching up with them as we indeed don't seem to have the resource capacity to handle all of those reports. The issue isn't with reporters or maintainers not responding (as in, it may be, but we would still go forward with disclosure after 45 days of triage), so bottom line it's just us not being attentive to these reports sadly.

[MarcinHoppe]

I agree the bottleneck is definitely on the WG side right now.

[sam-github]

I don't at all think that anyone is "not doing a good job" or "not being attentive". I think the entire triage is being done by only 2 or 3 volunteers, on pretty much their own time, and are effective with the time they spend.

I want it abundantly clear that I have absolutely no criticisms of how people spend their time working on triage, and if I could arrange it, you'd all get bottles of champagne quarterly (too bad that's not covered by the bug bounty).

That said, if the time WG members have available is a bottleneck, then something should be done. I don't think its helpful to anybody to for the WG to commit to doing things that it doesn't have the capacity to get done. Its frustrating to researchers (witness the pinging in the H1 issues for status updates), and having a multi-month backlog of known to some but not yet publicized vulnerabilities isn't helping ecosystem security.

So, back to #604 (comment)

Bucketing things so issues are handled in priority order is great, no objections, but if there is too much work, then either more triagers have to be onboarded (would be great, but forcing volunteers to appear isn't possible!), or the commitments have to be scaled back so there are no bottlenecks.

Respectfully disagree here, and definitely for all of those buckets. We want to make it accessible for researchers to report, not harder. Plus, they may not even be familiar with the ecosystem so it would make their lives much harder.

@lirantal That is exactly the point, to make things harder, or to put it more positively, to distribute the work to the people who are motivated to do it. Reporters are motivated by H1 rankings, so they will be motivated to contact the projects. Its standard practice, IMO, for vulnerability researchers to contact the projects they have found vulnerabilities in. I think this program setup is unusual, in that the sec WG has committed to doing this legwork for the researchers, even though we don't have the capacity.

To be clear, for high priority projects, I think it is within the WGs capacity to contact the projects and help move things along.

Other, more radical, proposals for decreasing time commitments:

If the project is of low enough priority by downloads, I am personally OK with simply verifying that the vulnerability is valid, and publishing. The publish on H1 will trigger scraping by npm, inc, and npm audit will start to warn the (very, very few) users of the package, and those users can then move away from it or contact the package maintainer to get it resolved. Since triage and reproduction is handled by the H1 team, the amount of time required for WG members is pretty much a couple button pushes to promote to publish.

An even more radical proposal would be to stop maintaining the JSON database. It predates H1. H1 itself is a DB of vulnerabilities, so reformatting as JSON isn't required, though we assume it makes it easier to consume. Do we know if there are any users of the JSON DB? IBM doesn't use it directly (to my knowlege). npm pulls directly from H1. Any other known users? If no WG member is supported by a company or org pulling the JSON, perhaps we should stop updating the JSON... this would encourage people who actually pull the JSON and find value in it to get involved in keeping it up to date (or perhaps we'd find there isn't that much interest).

[lirantal]

I want it abundantly clear that I have absolutely no criticisms of how people spend their time working on triage, and if I could arrange it, you'd all get bottles of champagne quarterly (too bad that's not covered by the bug bounty).

cracked me up @sam-github 😆
❤️🤗

Reporters are motivated by H1 rankings, so they will be motivated to contact the projects.

I understand that but AFAIK even if they wanted to contact the maintainers and add them to the triage they couldn't. Instead, what we could here is ask for the H1 support team to reach out for us for those low buckets.

If the project is of low enough priority by downloads, I am personally OK with simply verifying that the vulnerability is valid, and publishing.

this is what we indeed want to do with bucket number 1 (see Unmaintained here #604 (comment))

An even more radical proposal would be to stop maintaining the JSON database.

that's an interesting take which I hadn't considered but unrelated directly to this so if you want to further see that we can talk about this in a new issue and over an agenda call.

[sam-github]

This is on the sec WG agenda, and next meeting is Monday, Dec 30th. I can make that, I wonder how many other people can?

I guess the WG agenda will be auto-published next week and we'll see.

[lirantal]

yeah let's skip that due to holidays and timeoff for everyone.

[sam-github]

ok, then we'll discuss in 6 weeks time. no desperate hurry. And all your prioritization suggestions look OK to me, in the immediate term. If you and the other people doing Triage agree and a bucking/SLA system, I'm good with whatever you want to do.