Spacy entity detection ner with non printable characters

[bavithra-s]

How do we handle text with non printable characters so that Spacy ner identifies all occurrences.

In the below example
text2 has 2 occurrences of NY but ner identifies only one.

o/p:
(NY, NY)
(NY,) --> only one
(NY,)

import spacy

def print_ents():

    text1 = "NY is a city.NY is a state"
    text2 = "NY is a city.•NY is a state"
    text3 = "• NY is a state"
    nlp = spacy.load("en_core_web_trf")
    doc1 = nlp(text1)
    doc2 = nlp(text2)
    doc3 = nlp(text3)
    print(doc1.ents)
    print(doc2.ents)
    print(doc3.ents)

if __name__ == '__main__':
    print_ents()

[polm]

I'm a little confused about what you're asking here. The character you've included seems to be U+2022 BULLET, which is a normal printable character. A "non printable character" is like an ASCII bell or something.

It's not too surprising these characters confused the models, they wouldn't be in the training data. If you don't like the way spaCy is behaving around these characters, you can remove them, preprocess them to surround them with spaces, or add a tokenizer exception.

Also keep in mind the models are not perfect: #3052.