Lemmatizer problem migrating from spaCy v2

[joaneeet7]

I have been using Spacy with the version 2.1.3. Therefore, I would like to update to the current version. However, I have the next problem:

[E1004] Missing lemmatizer table(s) found for lemmatizer mode 'rule'. Required tables: ['lemma_rules']. Found: []. Maybe you forgot to call nlp.initialize() to load in the data?

I have tried:

config={"model": None, "mode": "rule", "overwrite": False}
self.lemmatizer = self.nlp.add_pipe("lemmatizer", config=config)
self.lemmatizer.initialize()

But I'm not able to create de lemmatizer.

With the old version of Spacy we did it in the following way:
**Word** -> Fails, **Lemma** -> Fail (**Version 2.3.1: with token.lemma_**)

What is the way in Spacy v3?

[polm]

Sorry to hear you're having trouble with this.

Can you clarify what you did that caused the E1004 to show up? Were you trying to load an old model, or trying to make a model in v3 with the same structure, or something else?

Note that at some point after 2.1 the lemmatizer tables were moved to a package called spacy-lookups-data, so you'll need to install that if you're making a Lemmatizer directly and want the lookup tables. Normally that's not necessary if you're using a pretrained pipeline.

With the old version of Spacy we did it in the following way:
**Word** -> Fails, **Lemma** -> Fail (**Version 2.3.1: with token.lemma_**)

Sorry, but I don't understand what you mean here. The way the lemma_ attribute works is the same in v3 as it was in v2.

---

Adding a lemmatizer like the one in the pretrained pipelines to a custom pipeline is slightly complex, and the process differs depending on the language. Generally you need POS tags from either a Morphologizer or a Tagger + Attribute Ruler (see #8341).

Can you clarify what language you're using and what your overall goal is?

[adrianeboyd]

The lookup lemmatizer worked differently in v2. In v3 you have to explicitly add a lemmatizer to your pipeline. The English default mode is rule and requires tagger+attribute_ruler, but the lookup lemmatizer does not and should be fast.

[polm]

Also note that if you do this:

nlp = spacy.load('en_core_web_sm')

You're going to be running components like the parser that you don't need, and turning them off will also help. See the speed FAQ: #8402

[joaneeet7]

Thanks for your support!

I have changed the code and I'm using NLP to do the tokenization.

[joaneeet7]

I have had a small problem with the issue of extracting a lemma. For example the word installed is passed to the word instal (with one 'l'), I do not understand why it does not return install (with two 'l'). Could you explain to me the reason for changing the word 'installed' to 'instal'?

[polm]

"instal" is a variant spelling and happens to be the first lemma listed (probably for sort order reasons). The lemma data is from WordNet originally, you can check there if you're curious about something.