DisplaCy UserWarning: [W006] when evaluating NER model

[matteobrv]

I put together a simple NER model following the Project Templates. Everything seems to be running fine except for the evaluate command which doesn't generate any displaCy visualizations when it's executed on my test.spacy dataset.

This is the command I'm executing from my .yml file: python -m spacy evaluate training/model-best corpus/test.spacy -dp training/, and this is the error I get:

/cryptocurrency_NER_local_env/lib/python3.8/site-packages/spacy/displacy/__init__.py:189: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.
  warnings.warn(Warnings.W006)
✔ Generated 25 parses as HTML

As suggested, I checked test.spacy using the best-model obtained from the training, but I see no issues. The documents in the serialized DocBin contain the entity I'm interested in and the displacy.serve command works flawlessly:

nlp = spacy.load("/training/model-best")
doc_bin = DocBin().from_disk("/corpus/test.spacy")
docs = list(doc_bin.get_docs(nlp.vocab))
displacy.serve(docs, style="ent")

That being the case, I decided to have a look at my data which was annotated with Label Studio and exported in the CoNLL 2003 NER format:

Bitcoin -X- _ B-CRYPTOCURRENCY
to -X- _ O
the -X- _ O
moon -X- _ O

I used the convert command to get the files into spaCy’s binary training data format: python -m spacy convert assets/iob_data/test.iob corpus -c ner -b en_core_web_sm -n 5. Maybe this is where things went south... If not, I really don't know where else to look. If anyone can offer any input I would really appreciate it!

[adrianeboyd]

What do the evaluation results look like for NER and NER per type before you run into this warning? It sounds like your model isn't predicting any entities for these texts.

What do the evaluation steps look like while training? Here's what it looks like for a very simple toy example where in step 3 you can see the score going up as the model improves: https://github.com/explosion/spaCy/tree/de4f4c9b8a46395b5f2be5fa80afd746c4a7f3cb/examples#-get-started-with-a-demo-project

Your check with displacy is only loading the reference docs (with your manual annotation), not the predicted annotation using the model. Instead, try something like:

docs = list(nlp.pipe([doc.text for doc in doc_bin.get_docs(nlp.vocab)]))

There is currently a bug in that conversion related to -s, but it doesn't look like you're not using that. Do you have blank lines in between sentences in your .iob file? Does the conversion report a sensible number of documents (something close to what you expect for # sents / 5)?

[matteobrv]

Thank you for taking the time to get back to me,

What do the evaluation results look like for NER and NER per type before you run into this warning? It sounds like your model isn't predicting any entities for these texts.
What do the evaluation steps look like while training? Here's what it looks like for a very simple toy example where in step 3 you can see the score going up as the model improves: https://github.com/explosion/spaCy/tree/de4f4c9b8a46395b5f2be5fa80afd746c4a7f3cb/examples#-get-started-with-a-demo-project

The model I'm working on has only one type CRYPTOCURRENCY. The evaluation step while training looks promising and so do the overall and per type results:

E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     47.67    0.00    0.00    0.00    0.00
  1     200         61.80  13562.83   81.02   83.54   78.65    0.81
  3     400        100.23    442.99   82.93   84.97   80.99    0.83
  7     600        112.03    147.50   83.38   86.25   80.70    0.83
 13     800        148.68     71.33   84.57   89.54   80.12    0.85
 18    1000        140.80     47.32   83.01   85.45   80.70    0.83
 23    1200         91.47     24.50   82.87   86.86   79.24    0.83
 28    1400         97.79     35.16   83.49   89.33   78.36    0.83
 33    1600         94.33     29.24   82.46   84.62   80.41    0.82
 38    1800        110.92     17.58   84.09   91.13   78.07    0.84
 43    2000         82.85     21.37   83.84   87.58   80.41    0.84
 49    2200         51.40     10.39   83.54   89.07   78.65    0.84
 54    2400         91.01     25.42   83.98   89.70   78.95    0.84
✔ Saved pipeline to output directory
training/model-last

================================== evaluate ==================================
Running command: /home/matteo/Documents/cryptocurrency_NER_local_env/bin/python -m spacy evaluate training/model-best corpus/test.spacy -dp training/
ℹ Using CPU

================================== Results ==================================
TOK     -    
NER P   89.55
NER R   70.18
NER F   78.69
SPEED   32281

=============================== NER (per type) ===============================
                     P       R       F
CRYPTOCURRENCY   89.55   70.18   78.69

Your check with displacy is only loading the reference docs (with your manual annotation), not the predicted annotation using the model. Instead, try something like:
docs = list(nlp.pipe([doc.text for doc in doc_bin.get_docs(nlp.vocab)]))

Using the snippet you provided I see that test.spacy contains 16 documents, which is the result I was expecting (77 sentences / 5 ≈ 16). I also checked the entities in each document and everything seems to be in order:

docs = list(nlp.pipe([doc.text for doc in doc_bin.get_docs(nlp.vocab)]))
print(len(docs))
for doc in docs:
    print(doc.ents)

16
(bitcoin, BTC, BTC, Nano, BTC)
(bitcoin, Nano, BTC, ECC, Bitcoin, Bitgreen, PIVX)
(Xtrabytes, BTC, XRP, XRP, XRP, XRP, XRP, USDT, XRP, bitcoin, BTC)
(Bitcoin, xrp, NEO, Footsteps)
(BCH, Monero, Monero, monero, monero, Bitcoin, Bitcoin, Cash, Monero, Bitcoin, Nano, Bitcoin, Travala, Algorand, Cosmos, Algorand, Cosmos, Decred, Monero)
(Vechain, bitcoin, bitcoin, Nano, VET, VeChain, VET)
(ETH, Libra, Libra, tether, tether, XRP, btc)
(kucoin, ETH, ETH, Namecoin, Dash, Cardano)
(PIVX, PIVX, PIVX, ALL, ALL, PIVX, Bitgreen, PIVX, Bitgreen, PIVX, PIVX, Bitgreen, PIVX, Bitgreen, PIVX, Bitgreen, PIVX, PIVX, PIVX, LTC, BTC, LTC, BTC, Nano, LTC, BTC, BTC, LTC, SYS, ETH, ETH, Bitcoin, Bitcoin, Bitcoin, LN, Bitcoin)
(XMR, Bitcoin, Bitcoin, Nano, BTC)
(BTC, Monero, nano, PIVX, Bitcoin, Bitcoin, Ver, Nano)
(btc, KBC, ALL, doge, doge, LTC, LTC, Litecoin, XMR, Bitcoin, Bitcoin, Dash, BCH)
(btc, Dusk, Dusk, nxt, doge, doge, Ethereum, nano, bitcoin, bitcoin)
(Doge, doge, BTC, tether, btc, bitcoin, Large)
(Bitcoin, tether, bitcoin, bitcoin, Eth, NANO)
(DAG, bitcoin)

Lastly, I tried to run a check with displacy using the following snippet and I got the result I was expecting:

nlp = spacy.load("training/model-best")
doc_bin = DocBin().from_disk("corpus/test.spacy")
docs = list(nlp.pipe([doc.text for doc in doc_bin.get_docs(nlp.vocab)]))
displacy.serve(docs, style="ent")

There is currently a bug in that conversion related to -s, but it doesn't look like you're not using that. Do you have blank lines in between sentences in your .iob file? Does the conversion report a sensible number of documents (something close to what you expect for # sents / 5)?

Indeed, I have blank lines separating sentences but the convert command seems to be handling it just fine and the number of resulting documents is consistent with # sents / 5.

For completeness' sake, I include a copy of the trained best-model as well as of the test_set files .iob and .spacy.

[adrianeboyd]

Hmm, I can't reproduce this with spacy v3.0.5. I don't see any warnings and I see entities in the generated HTML file. Can you check that you have the latest versions of spacy and its dependencies? There was a bug fix related to this in v3.0.4.

[matteobrv]

Updating to v3.0.5 and installing spaCy in a clean environment solved the issue, thank you!

[sethidden]

For people using the rel_component tutorial that have this issue, see those discussions:
#8499 (comment)
#8627