When HuggaFace transformers model file has tokenizer_config.json and tokenizer.json，how config the config.cfg file?

[baiziyuandyufei]

How to reproduce the behaviour

I use the model https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main，

the training config like below

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "./chinese-roberta-wwm-ext"

[components.transformer.model.tokenizer_config]
use_fast = true

but the model files contain the file

• tokenizer_config.json
• tokenizer.json
• added_tokens.json

these files seems to not be used by the SpaCy.

Your Environment

Info about spaCy

• spaCy version: 3.1.1
• Platform: Linux-5.10.25-linuxkit-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• Pipelines: zh_core_web_sm (3.1.0), xx_ent_wiki_sm (3.1.0)

[polm]

When you load a Transformer/HuggingFace model with spaCy, it uses the HuggingFace code, so even if the config file doesn't mention it the model can do other stuff, which might include loading json config files.

I'm a little unclear about what you're trying to do. Are you trying to customize the json config files, or does loading that model not work, or...? If you have tried something and it didn't work it'd be helpful to have any error messages you've gotten.

[baiziyuandyufei]

I loaded chinese-roberta-wwm-ext model work.
My question is:

I think these file
"added_tokens.json",
"special_tokens_map.json",
"tokenizer.json",
"tokenizer_config.json",
in the chinese-roberta-wwm-ext, not be used by SpaCy, but these files are part of the model.

SpaCy's transformer model use the internal tokenizer.

Is there something I can do, customize the json config files, to use the files up.

[polm]

When spaCy uses Transformers, it actually uses the spaCy tokenizer and the HuggingFace tokenizer. It then creates an alignment between the tokens to share the embeddings properly. So I think those files are being used.

Is there something I can do, customize the json config files, to use the files up.

I don't understand what you mean here. I think if you edit them the settings should just be applied, but if the files change between training and inference the model may not give useful output.

[Einsteinder]

I'm facing the same issue, the tokenizer.json is not save in spaCy model. so I need to manually copy it to model-best/transformer/model to make inference work

[polm]

That's interesting, that sounds like it shouldn't be happening. Can you give us more details about your setup, like what Transformer model you're using, or your config?

I think that you have a different issue from the OP here - I'm still not sure what they wanted, but I think they saw that the model included various json files they thought were unused and they wanted to delete them.

[polm]

Actually, if you could open a different discussion about this that'd be best I think.