spacy-transformers with GPT-2

[Btibert3]

I apologize if this question is obvious and/or naive, but I am really struggling to identify (or even work through, for that matter) a basic example of getting GPT-2 loaded into spacy. I am aware that we can use the default Roberta model with spacy-transformers and en_core_web_trf, but beyond that, I can't pin anything down in the documentation that highlights how we can load the other models. In my case, I am interesting in playing around with GPT-2 inside spacy.

The resources I am aware of:

• https://explosion.ai/blog/spacy-transformers#spacy-transformers
• https://spacy.io/universe/project/spacy-transformers

Worth noting, the first link above refers to an article from 2019 and itemizes LMs that I can't identify in the documentation. Speaking of documentation, I have seen examples of changing the configuration for the transfomers to specify a model in v3. However that is paired with adding the pipe to the existing nlp object, whereas the basics for en_core_web_trf identify creating a transformer object trf, which makes sense in my mental model.

In summary, what is the best way to load other other transformer models other than Roberta in v3?

Cheers.

[polm]

You can use other models by passing the name (from HuggingFace hub) to the Transformer model, as described in the docs here. In code that looks something like:

from spacy_transformers import Transformer, TransformerModel
from spacy_transformers.annotation_setters import null_annotation_setter
from spacy_transformers.span_getters import get_doc_spans

trf = Transformer(
    nlp.vocab,
    TransformerModel(
        "bert-base-cased",
        get_spans=get_doc_spans,
        tokenizer_config={"use_fast": True},
    ),
    set_extra_annotations=null_annotation_setter,
    max_batch_items=4096,
)

You can also change this in the config.

[adrianeboyd]

Hi, there are some known bugs with gpt2 and spacy-transformers v1.0.x.

For now, the easiest option is to try out a prerelease of the next version with pip install "spacy-transformers>=1.1.0.dev4" --pre. The final release should be coming soon. If you run into errors, do let us know!

You will probably need to set the padding token manually in the config:

[components.transformer.model.tokenizer_config]
use_fast = true
pad_token = "<|endoftext|>"

[adrianeboyd]

We decided to just go ahead with the official release, so:

pip install "spacy-transformers>=1.1.0"