NER and Relation Extraction pipeline: Could not determine any instances in doc. #8499
-
|
Hello SpaCy community, I, a freshly converted SpaCy newbie, am currently trying to plug a pretrained NER model into the relation extraction pipeline, I think I have implemented all the changes recommended in the similar posts on this forum, namely the change in the def score_relations(examples: Iterable[Example], threshold: float) -> Dict[str, Any]:
...
for key, pred_dict in pred.items():
if key not in gold.keys():
for k, v in pred_dict.items():
if v >= threshold:
micro_prf.fp += 1
continue
gold_labels = [k for (k, v) in gold[key].items() if v == 1.0]
...And during training phase i get this: And of course the loss is 0 because the def convert_rel(json_loc: Path, output_path: Path, rel_labels: str):
REL_LABELS = [label.strip() for label in rel_labels.split(",")]
Doc.set_extension("rel", default={})
vocab = Vocab()
docs = []
count_sample, count_pos, count_all = 0, 0, 0
for example in srsly.read_jsonl(json_loc):
if example["answer"] != "accept":
continue
span_starts = set()
pos, neg = 0, 0
try:
# Parse tokens
words = [t["text"] for t in example["tokens"]]
spaces = [t["ws"] for t in example["tokens"]]
doc = Doc(vocab, words=words, spaces=spaces)
# Parse entities
spans = example["spans"]
ents = []
span_end_to_start = {}
for span in spans:
start, end, label = span["start"], span["end"], span["label"]
entity = doc.char_span(start, end, label=label)
span_end_to_start[span["token_end"]] = span["token_start"]
ents.append(entity)
span_starts.add(span["token_start"])
doc.set_ents(ents)
# Parse relations
rels = {}
for x1 in span_starts:
for x2 in span_starts:
rels[(x1, x2)] = {}
relations = example["relations"]
for relation in relations:
# the 'head' and 'child' annotations refer to the end token in the span
# but we want the first token
start = span_end_to_start[relation["head"]]
end = span_end_to_start[relation["child"]]
label = relation["label"]
if label not in rels[(start, end)]:
rels[(start, end)][label] = 1.0
pos += 1
# The annotation is complete, so fill in zero's where the data is missing
for x1 in span_starts:
for x2 in span_starts:
for label in REL_LABELS:
if label not in rels[(x1, x2)]:
rels[(x1, x2)][label] = 0.0
neg += 1
doc._.rel = rels
# only keeping documents with at least 1 positive case
if pos > 0:
docs.append(doc)
count_pos += pos
count_all += pos + neg
count_sample += 1
except Exception as e:
msg.fail(f"Parse error: {e}")
db = DocBin(docs=docs, store_user_data=True)
db.to_disk(output_path)And finally here is my pipeline config where I initialize the NER component with the vectors of another pretrained NER model: I hope it's not too messy, thank you in advance for any help you can provide. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
This pipeline component currently doesn't support training on NER and REL at the same time - instead it assumes that gold entities are already present when training the REL component. The underlying reason is that in spaCy 3.0, components are updated in isolation during training, which means that they don't see the predictions of any earlier components in the pipeline. In the upcoming 3.1, this mechanism will become more transparent & extensible, and you'll be able to specify components that should set annotations on the predicted docs during training, so that downstream components can use those predictions as features or to build on them. In that case, you'd be able to use the predicted entities to train the REL. For now, my advice would be to have a two-step training procedure: first create & train an NER that identifies all candidate mentions for your relations, then in a next step train a REL component that decides whether a link is correct or not. For the latter, you'll want to create a custom reader that already sets the |
Beta Was this translation helpful? Give feedback.
-
|
@svlandeg So with the release of v3.1 I just have to specify |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I would hope that'd work! I haven't tested it yet though. Let us know how it goes! |
Beta Was this translation helpful? Give feedback.
This pipeline component currently doesn't support training on NER and REL at the same time - instead it assumes that gold entities are already present when training the REL component.
The underlying reason is that in spaCy 3.0, components are updated in isolation during training, which means that they don't see the predictions of any earlier components in the pipeline.
In the upcoming 3.1, this mechanism will become more transparent & extensible, and you'll be able to specify components that should set annotations on the predicted docs during training, so that downstream components can use those predictions as features or to build on them. In that case, you'd be able to use the predicted …