ValueError: Mismatched IDs! 6936129771849058806471 vs 3499314791125498363823

[moyid]

Hi, I'm getting this error of mismatched IDs for my entity linking model.

Here is my config file:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
kb = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","sentencizer","ner","entity_linker"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.entity_linker]
factory = "entity_linker"
entity_vector_length = 64
get_candidates = {"@misc":"spacy.CandidateGenerator.v1"}
incl_context = true
incl_prior = true
labels_discard = []
n_sents = 0

[components.entity_linker.model]
@architectures = "spacy.EntityLinker.v1"
nO = null

[components.entity_linker.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.ner]
source = "${paths.base_nlp}"
component = "ner"

[components.sentencizer]
factory = "sentencizer"
punct_chars = null

[components.transformer]
source = "${paths.base_nlp}"
component = "transformer"

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
sents_f = 0.33
sents_p = 0.0
sents_r = 0.0
ents_f = 0.33
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
nel_micro_f = 0.33
nel_micro_r = null
nel_micro_p = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.entity_linker]

[initialize.components.entity_linker.kb_loader]
@misc = "spacy.KBFromFile.v1"
kb_path = ${paths.kb}

[initialize.tokenizer]

Operating System::
spacy==3.1.3
python 3.8.10
transformers==4.9.2

[polm]

Couple of minor points:

When you have a multi-line code block, you need to use a backtick fence (``` on a line), not single backticks. You can check that it works with the preview function.

Is this E953? That has a similar but slightly different error message, but it's in tok2vec, not the Entity Linker specifically. I don't see an error message exactly like your title anywhere. If you could post the stack trace that'd help.

[moyid]

@polm Here is the stack trace:

spacy project run train

=================================== train ===================================
Running command: /home/user/env/bin/python3 -m spacy train configs/transformers_nel.cfg --output training --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --paths.kb temp/my_kb --paths.base_nlp temp/ner -c scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-09-28 13:05:58,993] [INFO] Set up nlp object from config
[2021-09-28 13:05:59,003] [INFO] Pipeline: ['transformer', 'sentencizer', 'ner', 'entity_linker']
[2021-09-28 13:05:59,003] [INFO] Resuming training for: ['ner', 'transformer']
[2021-09-28 13:05:59,008] [INFO] Created vocabulary
[2021-09-28 13:05:59,008] [INFO] Finished initializing nlp object
[2021-09-28 13:06:00,292] [INFO] Initialized pipeline components: ['sentencizer', 'entity_linker']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'sentencizer', 'ner', 'entity_linker']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS NER  LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  SCORE 
---  ------  -------------  --------  -------------  -------  -------  -------  ------  ------  ------  -----------  ------
  0       0          12.73     26.62           3.75    91.30    88.40    94.40   91.93   89.02   95.03         0.27    0.61
⚠ Aborting and saving the final best model. Encountered exception:
ValueError('Mismatched IDs! 6936129771849058806471 vs
3499314791125498363823')
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/env/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/user/env/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/env/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/user/env/lib/python3.8/site-packages/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/user/env/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/user/env/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/user/env/lib/python3.8/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/home/user/env/lib/python3.8/site-packages/spacy/language.py", line 1122, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "/home/user/env/lib/python3.8/site-packages/spacy/pipeline/entity_linker.py", line 254, in update
    sentence_encodings, bp_context = self.model.begin_update(sentence_docs)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 309, in begin_update
    return self._func(self, X, is_train=True)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/user/env/lib/python3.8/site-packages/spacy_transformers/layers/listener.py", line 58, in forward
    model.verify_inputs(docs)
  File "/home/user/env/lib/python3.8/site-packages/spacy_transformers/layers/listener.py", line 51, in verify_inputs
    raise ValueError(f"Mismatched IDs! {batch_id} vs {self._batch_id}")
ValueError: Mismatched IDs! 6936129771849058806471 vs 3499314791125498363823

[polm]

Hm, I'm not sure what's causing this. What is the value of vectors_model in your project.yml? That's the source of the transformer component, but in the default project it's using the medium model. I would expect there to be an error if you source a component that's not there, but maybe something is weird with your base model?

[moyid]

The vectors_model is a custom trained transformers based NER model, and I used the configs based on the training page here: https://spacy.io/usage/training

[moyid]

I think my setup is quite similar to the one in this discussion: #7884

[svlandeg]

This could be related to #9291 and it's likely a bug. I'll look into it!

[moyid]

Hi @svlandeg - I took a look at #9291 - I'm able to train an NEL model with the non-transformers set up, I only get an error when I switch to transformers, so this can be different.

[moyid]

After cleaning up my knowledge base a bit (I had some entities in the data that was not in the knowledge base), I am no longer getting the Mismatched IDs error. Now I'm getting TypeError: 'NoneType' object is not callable Here is the full stack trace:

spacy project run train

=================================== train ===================================
Running command: /home/user/env/bin/python3 -m spacy train configs/nel.cfg --output training --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --paths.kb temp/my_kb --paths.base_nlp temp/ner -c scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-09-30 15:57:08,934] [INFO] Set up nlp object from config
[2021-09-30 15:57:08,942] [INFO] Pipeline: ['transformer', 'ner', 'entity_linker']
[2021-09-30 15:57:08,946] [INFO] Created vocabulary
[2021-09-30 15:57:08,947] [INFO] Finished initializing nlp object
[2021-09-30 15:57:10,033] [INFO] Initialized pipeline components: ['entity_linker']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner', 'entity_linker']
ℹ Frozen components: ['transformer', 'ner']
ℹ Set annotations on update for: ['transformer']
ℹ Initial learn rate: 0.001
E    #       LOSS ENTIT...  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  ------  ------  ------  -----------  -----------  -----------  ------
⚠ Aborting and saving the final best model. Encountered exception:
TypeError("'NoneType' object is not callable")
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/spaCy/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/user/spaCy/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/env/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/env/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/user/spaCy/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/user/spaCy/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/user/spaCy/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/user/spaCy/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/home/user/spaCy/spacy/language.py", line 1122, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "/home/user/spaCy/spacy/pipeline/entity_linker.py", line 258, in update
    bp_context(d_scores)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  File "/home/user/env/lib/python3.8/site-packages/thinc/layers/chain.py", line 60, in backprop
    dX = callback(dY)
  [Previous line repeated 2 more times]
  File "/home/user/env/lib/python3.8/site-packages/spacy_transformers/layers/listener.py", line 39, in backprop_and_clear
    result = self._backprop(*args, **kwargs)
TypeError: 'NoneType' object is not callable

[moyid]

Hi, following up on this issue!

[svlandeg]

Hi, there's no need to artificially bump your thread (and it's only been a week...)

[polm]

Sorry for the significant delay getting back to you on this, this is in fact a bug, and NEL doesn't work with listeners.

We are working on a fix and will have a PR open soon, though not all details of the fix are clear so it might take a little while for it to be in a release. Once the PR is started I'll link it here.

[moyid]

thanks for the update, look forward to the PR!

…

On Tue, Nov 9, 2021 at 1:45 AM polm ***@***.***> wrote: Sorry for the significant delay getting back to you on this, this is in fact a bug, and NEL doesn't work with listeners. We are working on a fix and will have a PR open soon, though not all details of the fix are clear so it might take a little while for it to be in a release. Once the PR is started I'll link it here. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9310 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALDSLJB7GFWCEWLCX4AP4J3ULDURZANCNFSM5E3WZKKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[polm]

PR is up: #9669

Still needs some work, but updates will be posted there.

[moyid]

thanks for the update!

…

On Fri, Nov 12, 2021 at 10:33 PM polm ***@***.***> wrote: PR is up: #9669 <#9669> Still needs some work, but updates will be posted there. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9310 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALDSLJCY6YH6M3K5FCBONALULYBDJANCNFSM5E3WZKKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.