Mark `extra-phrase` position in rules

[alok1304]

Hey @AyanSinhaMahapatra, I want more details about this, like what we want to do, like mark the named entity i.e replace with extra-phrase marker like [[6]]]. Or what we want to do. How do we proceed with these issues?
How can I start?

Also, i am doing research about this problem, what if there 3-4 named entity in rules, we can't replace this all with extra-phrase it may impact the matching speed that we talked about in one of the meeting.

What are your thoughts on this? How do we tackle this problem?

[alok1304]

I am trying to find a named entity in rules. Creating a new repo is a great idea for NER. Like, Write scripts in the new repo perform this on src/licensedcode/data/rules after cloning the new repo.

But there are a lot of edge cases found for determining the named entity.

• There are various words that are considered a label of ORG i.e DATA, LIABILITY, POSSIBILITY, CONTRIBUTORS BE LIABLE etc that are present in rules file.
• Similarly PERSON label for word like AGPL in the text of license https://www.gnu.org/licenses/agpl.html GNU AGPL v3.0+, Cube License this is also consider as PERSON label. And there are many case where some word are not person name but consider a person's name.
• And some copyright and holder name i.e, present in many rules.

[alok1304]

I am working on marking extra-phrases in rules. Specifically, I am inserting [[n]] between the text Neither the name of and nor the names of its in rules commonly associated with the BSD license.

If there are words present between these two phrases, they are replaced with an extra-phrase marker (e.g., [[6]]). Once this process is complete, duplicate rules are deprecated.

I have started implementing this in a new repository dedicated to marking extra-phrase positions. While testing, I found that this change results in the deprecation of approximately 500+ rules.

I plan to raise a PR soon with a detailed explanation of the changes.

Previously, I tried using Named Entity Recognition (NER) for this task, but I encountered various edge cases that were difficult to handle (as I have already mentioned earlier). Because of these challenges, I decided to adopt this new approach of inserting extra-phrase markers where extra-words may be present.

Also, I got various eg. where extra-words are present between the text Neither the name of and nor the names of its.

1. https://github.com/pombredanne/DotNetPELib/blob/747df7035ff056cae4f81009dde1567025b00bef/Value.cpp#L20
2. https://github.com/bitmapdata/MSTranslateVendor/blob/f71780b519527e2ff77dd3f8e502759983c7f573/README.md?plain=1#L178

For this, there are no rules that why we got extra-words in detection-log.

[AyanSinhaMahapatra]

like mark the named entity i.e replace with extra-phrase marker like [[6]]]

Not at all, we want to avoid replacing named entities with extra phrases, as mentioned before because:

1. we are no longer matching those rules exactly, but with 3-seq, which takes a significantly longer time. So this is a performance issue.
2. we are losing data here where we tracked exact rules and their without named words versions which you're deprecating.

See for example the comment in #4059 (comment) and #4059 (comment) to understand the impact this has on performance, the license detection runs approximately 10-12 times faster for an exact match rather than a sequence match.

what if there 3-4 named entity in rules, we can't replace this all with extra-phrase it may impact the matching speed that we talked about in one of the meeting

Even if you replace one entity it impacts the performance as explained above, the performance hit for removing one entity is similar to removing 3-4 entities.

This is why we always want to have the exact rules, while also retaining/creating a named-entitites stripped rule with extra words markers so we detect other instances of named entities correctly.

Instead you need to do the following:

• check for named entities in rules, handling edge cases
  • use copyright detection lines to avoid detecting named entities in copyrights
• if there is a named entity in a rule then consider the following, is there another rule which is exactly this without the named entity?
  • if YES, then add a extra words marker(s) to the rule without the named entity, in the place which can have a named entity
  • if NO, then create a rule which is the exactly same as the previous rule, except the extra words, and has extra work marker(s) in the place where there can be named entities.

Can you also create a mapping (of identifiers) for these above rules from:

• the rule without the named entities (with extra words marker) -> the rule with the named entities in that extra word place (and no markers) [This is a one-to-many mapping]
• with data on if the rule without named entities was present before, or you added this

And also maintain a list of common named entities across our rules likewise.

This is so we have a record of exact changes made through the PR and for stats on the changes made.

To summarize:

• do not remove named entities for extra words markers
• do not deprecate rules.

[AyanSinhaMahapatra]

Creating a new repo is a great idea for NER.

Do you have this repo already, if so please link it here/on your PR. We need to understand/review the logic first before reviewing the rule changes in detail.

But there are a lot of edge cases found for determining the named entity.

Can you give examples and explain in detail?

There are various words that are considered a label of ORG i.e DATA, LIABILITY, POSSIBILITY, CONTRIBUTORS BE LIABLE etc that are present in rules file.

we can have a list of words which are common in legalese and not consider these as named entities. for example if these are in the legalese list at https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/legalese.py, these can be safely ignored as these are present in many rules, and are not really named entities or extra-words for the purpose of adding extra words markers.

Similarly PERSON label for word like AGPL in the text of license https://www.gnu.org/licenses/agpl.html GNU AGPL v3.0+, Cube License this is also consider as PERSON label. And there are many case where some word are not person name but consider a person's name.

Not able to understand what you mean, can you provide examples and details

[alok1304]

Hey @AyanSinhaMahapatra
I am performing NER using spacy library to find out the named entity. I ignored the named entity, which is in legalese. I have various problems in finding out the named entity in rules because in rules, there are various words and sentences are consider as ORG or PERSON label but they are not actually a person or organization name. I think this is happening because the pre-trained model which is used that is not well-trained to find out the actual named entity in rules.

I got various problems where some of the words or sentences/phrases which is considered as PERSON or ORGANIZATION.
Also, I attached the repo that is performing NER using spacy library as of now, this is not fully completed. I am trying to removed named entity in rules. Repo: https://github.com/alok1304/named-entity-utils

Also i give eg. like where we consider some of the words/sentences as named entity but they not actuall name of any PERSON or ORG name in rules.

These are the rules text.

• * Licensed under the ADI BSD license or the GPL-2 (or later) here ADI and BSD consider as ORG. [('ADI', 'ORG'), ('BSD', 'ORG')]
• license: Amazon Digital Services License [('Amazon Digital Services License', 'ORG')]
• Licensed under the Academic Free License version 1.1. [('the Academic Free License', 'ORG')]
• https://spdx.org/licenses/afl-1.1 [('https://spdx.org/licenses/afl-1.1', 'PERSON')]
• # Copyright: Licensed under the Academic Free License version 1.2 [('Copyright', 'PERSON'), ('the Academic Free License', 'ORG')]
• The text of the AFL and BSD licenses are reproduced below. [('AFL', 'ORG'), ('BSD', 'ORG')]
• released under the Aladdin Free Public License [('the Aladdin Free Public License', 'ORG')]
• available under a dual AFPL/GPL license [('AFPL/GPL', 'ORG')]
• # @TAG(NICTA_AGPL) [('@tag(NICTA_AGPL', 'PERSON')]
• license: AMD's plpa_map.c License [('AMD', 'ORG')]
• Pick your favourite OSI approved license :) [('OSI', 'ORG')]
• GPL (v2 or later), with Autoconf exception [('Autoconf', 'ORG')]
• Software unter AGLPv3) [('Software', 'ORG')]

These are some eg where this tag wrong named entity, in some case url is consider as PERSON name and there many more eg like this , so how to deal with this as we have 30k+ rules we got various problem to deal with this because a lots rules of where the words/sentences that is consider as named entity but actually this are not named entity. This is only happend due to pre-trained model in sapcy i.e not well trained, we always got a lots of false postitve detection of named entity.