How to annotate a corpus to train a SpanFinder

[ddenz]

Hi,

I was wondering how to train a SpanFinder model on its own. I don't yet know all the classes I will assign to spans, but wish to train a model that will suggest spans to annotate in Prodigy. There is no available dataset for my specific task.

I've found the existing example config and corpus referenced in this post , but it's unclear to me how to go about annotating a corpus with unlabelled spans using Prodigy. I know how to train a span_cat model, but, as I mentioned, at this stage I'd like to train a span_finder to suggest spans that I will classify later.

My solution so far has been to annotate spans with a single label (SPAN). But this doesn't separate the two tasks as I would like. Hopefully, I can make use of the data I have annotated thus far.

Thanks!

PS I posted this on the Prodigy forum but was redirected here (see reply).

[rmitsch]

Hi @ddenz!

My solution so far has been to annotate spans with a single label (SPAN). But this doesn't separate the two tasks as I would like.

Can you elaborate on "But this doesn't separate the two tasks as I would like."?

[ddenz]

Hi yes sure. I meant the tasks of span identification and span classification, which I would like to perform separately. I’d like to annotate a corpus with unlabelled spans which I would then use to train a SpanFinder. Once that model is satisfactory and I have an idea of the different span classes I need, I will go on to annotate labelled spans and train a SpanCategorizer using spans suggested by the SpanFinder model.

Maybe my current strategy is fine as ultimately what I want is a corpus to train a SpanCategorizer model. I would still like to know if it’s possible to annotate unlabelled spans in Prodigy and how.

[rmitsch]

Annotating spans without assigning any label isn't possible. In your case it's probably easiest to do one of these two things:

• Remove the label from all spans in your corpus after you're done annotating and before you use them in training.
• Write a before_db callback that removes the label from your span before saving your annotations in the database.

[ddenz]

Thanks very much for your response @rmitsch - that's good to know. Seeing as I had existing annotated data, I have tried your first solution - removing all the labels from the dataset. However, I'm a little confused. Can the SpanFinder be trained directly on spans with labels? i.e. as a trainable component, does it not simply ignore the "label" feature?

Also, when generating a spacy dataset from annotations, there appears to be no option to specify span_finder:

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m prodigy data-to-spacy spans_unlabelled_3
✔ Created output directory
ℹ Using language 'en'

============================== Generating data ==============================
✘ You need to specify at least one training dataset using one of the
CLI options
--ner, --tagger, --senter, --parser, --spancat, --experimental_coref, --textcat,
--textcat_multilabel

How can I produce a spacy binary file without labels (is this really what I need to do?)? Or can I just use my existing data that contains labelled spans, specify the --spancat option to create the dataset and then modify the config file accordingly to train the span_finder?

More generally, would it not make sense to allow for the annotation of unlabelled spans in prodigy, seeing as the span_finder is a trainable component? The generation of a default config for span_finder training would be helpful as well.

I understand it is still an "experimental" feature, so maybe such things are on the todo list...

[rmitsch]

Can the SpanFinder be trained directly on spans with labels? i.e. as a trainable component, does it not simply ignore the "label" feature?

AFAIK it does. It was my understanding that you preferred having your dataset without labels.

Or can I just use my existing data that contains labelled spans, specify the --spancat option to create the dataset and then modify the config file accordingly to train the span_finder?

That should work.

More generally, would it not make sense to allow for the annotation of unlabelled spans in prodigy, seeing as the span_finder is a trainable component?

As training works also with labeled spans, this would amount to the same result.

[ddenz]

Can the SpanFinder be trained directly on spans with labels? i.e. as a trainable component, does it not simply ignore the "label" feature?
AFAIK it does. It was my understanding that you preferred having your dataset without labels.

I suppose I found a "workaround" by annotating spans using a single label. Perhaps my question now is "how do I train a span_finder without training a span_cat?" The case being that I don't know what my labels are going to be.

Or can I just use my existing data that contains labelled spans, specify the --spancat option to create the dataset and then modify the config file accordingly to train the span_finder?
That should work.

I tried this but got all zero scores for the span_finder model during training...something wrong with the data or config I guess.

More generally, would it not make sense to allow for the annotation of unlabelled spans in prodigy, seeing as the span_finder is a trainable component?
As training works also with labeled spans, this would amount to the same result.

Does this not imply an unnecessary training overhead due to training two models (span_finder + span_cat) instead of just one?

[adrianeboyd]

Be sure that the score_weights include the right scores for the span_finder.

In practice, especially if you have longer texts, I've found that it doesn't work that well to train span_finder+spancat together from scratch. The span_finder needs to be trained for long enough to start giving reasonable suggestions before it makes sense to add spancat on top. And spancat can run out of memory if the untrained span_finder starts trying to suggest every single possible span in the text.

What you can do instead is train span_finder on its own until its performance is reasonable and then continue training span_finder+spancat by sourcing the span_finder in the combined config. You don't need to freeze span_finder, you can continue training it in combination with spancat.

An example config is here (note the annotating_components):

https://github.com/adrianeboyd/workshop-dh2023/blob/main/litbank/configs/spancat_span_finder_lg.cfg

The two training steps look like this:

https://github.com/adrianeboyd/workshop-dh2023/blob/52ee57dba2e92ff5aa4cc71b1e32e4b8abdb84f4/litbank/project.yml#L132-L147

In the context of a larger demo project showing different types of suggesters with spancat:

https://github.com/adrianeboyd/workshop-dh2023/tree/main/litbank

[ddenz]

Thanks so much for this!

Be sure that the score_weights include the right scores for the span_finder.

What are the right scores? I am using the config in your reply which has this:

spans_sc_f = null
spans_sc_p = null
spans_sc_r = null
spans_entities_f = 0.0
spans_entities_p = 0.2
spans_entities_r = 0.8

I have tried your config with my data, but am getting 0.0 scores throughout all epochs of training:

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m spacy train models/spancat_difficulty_v2/config.cfg --output models/span_finder_lg --paths.train models/spancat_difficulty_v2/train.spacy --paths.dev models/spancat_difficulty_v2/dev.spacy
ℹ Saving to output directory: models\span_finder_lg
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  SPANS_ENTI...  SPANS_ENTI...  SPANS_ENTI...  SCORE
---  ------  ------------  -------------  -------------  -------------  -------------  ------
  0       0          1.21          64.38           0.00           0.00           0.00    0.00
  0     200          1.08          30.64           0.00           0.00           0.00    0.00
  0     400          0.00           0.00           0.00           0.00           0.00    0.00
  1     600          0.00           0.00           0.00           0.00           0.00    0.00
  2     800          0.00           0.00           0.00           0.00           0.00    0.00
...

When I do debug:

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m spacy debug data models\spancat_difficulty_v2\config.cfg --paths.train models/spancat_difficulty_v2/train.spacy --paths.dev models/spancat_difficulty_v2/dev.spacy

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, span_finder
1428 training docs
357 evaluation docs
✔ No overlap between training and evaluation data
⚠ Low number of examples to train a new pipeline (1428)

============================== Vocab & Vectors ==============================
ℹ 51158 total word(s) in the data (6616 unique)
ℹ 514157 vectors (514157 unique keys, 300 dimensions)
⚠ 3275 words in training data without vectors (6%)

================================== Summary ==================================
✔ 3 checks passed
⚠ 2 warnings

Seems odd that it says 1428 is a low number of examples doesn't it? (edit; I see in debug.py it uses 2000 as the BLANK_MODEL_THRESHOLD). Any idea why the model might not be training at all?

edit: I have annotated additional spans to get to over 2000, but still training with spacy train gives zero scores. However, training with the same dataset using prodigy train does work. Why might this be?

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m prodigy train --spancat difficulty_spans_v2 --base-model en_core_web_lg models\spancat_difficulty_v2
ℹ Using CPU

========================= Generating Prodigy config =========================
ℹ Auto-generating config with spaCy
Using 'spacy.ngram_range_suggester.v1' for 'spancat' with sizes 2 to 16 (inferred from data)
ℹ Using config from base model
✔ Generated training config

=========================== Initializing pipeline ===========================
[2023-11-15 16:05:31,405] [INFO] Set up nlp object from config
Components: spancat
Merging training and evaluation data for 1 components
  - [spancat] Training: 1628 | Evaluation: 406 (20% split)
Training: 1628 | Evaluation: 406
Labels: spancat (16)
[2023-11-15 16:05:32,233] [INFO] Pipeline: ['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner', 'spancat']
[2023-11-15 16:05:32,234] [INFO] Resuming training for: ['tok2vec']
[2023-11-15 16:05:32,248] [INFO] Created vocabulary
[2023-11-15 16:05:36,246] [INFO] Added vectors: en_core_web_lg
[2023-11-15 16:05:38,627] [INFO] Finished initializing nlp object
[2023-11-15 16:05:40,134] [INFO] Initialized pipeline components: ['spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
Components: spancat
Merging training and evaluation data for 1 components
  - [spancat] Training: 1628 | Evaluation: 406 (20% split)
Training: 1628 | Evaluation: 406
Labels: spancat (16)
ℹ Pipeline: ['tok2vec', 'tagger', 'parser', 'attribute_ruler',
'lemmatizer', 'ner', 'spancat']
ℹ Frozen components: ['tagger', 'parser', 'attribute_ruler',
'lemmatizer', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SPEED   SCORE
---  ------  ------------  ------------  ----------  ----------  ----------  ------  ------
  0       0       6855.52       4496.97        0.01        0.00       29.69  941.72    0.00
  2    1000      17942.61      12487.21       33.33       92.86       20.31  1447.44    0.33
  9    2000         29.20       1897.28       62.38       85.14       49.22  1446.39    0.62
 25    3000         26.48       2868.12       61.90       79.27       50.78  1439.41    0.62
 42    4000         19.53       2844.79       62.38       85.14       49.22  1419.27    0.62
 58    5000         15.11       2747.75       62.38       85.14       49.22  1441.56    0.62
 75    6000         23.49       2679.58       63.37       86.49       50.00  1415.33    0.63
 91    7000         21.77       2174.06       64.08       84.62       51.56  1428.83    0.64
107    8000         19.31       1981.29       63.46       82.50       51.56  1444.46    0.63
124    9000          4.97       1932.87       63.11       83.33       50.78  1452.37    0.63
140   10000         27.60       2007.67       64.08       84.62       51.56  1447.23    0.64
157   11000         14.88       1925.67       65.73       82.35       54.69  1415.12    0.66
173   12000         12.63       1901.42       65.42       81.40       54.69  1440.35    0.65
190   13000         18.88       1883.95       66.36       82.56       55.47  1450.45    0.66
206   14000          8.63       1878.16       65.74       80.68       55.47  1430.21    0.66
223   15000          9.21       1894.75       64.52       78.65       54.69  1434.36    0.65
239   16000         14.56       1892.30       65.12       80.46       54.69  1448.10    0.65
255   17000         12.96       1903.28       64.22       77.78       54.69  1448.66    0.64
272   18000          5.92       1867.61       63.89       78.41       53.91  1448.12    0.64
✔ Saved pipeline to output directory

[adrianeboyd]

The thresholds in spacy debug data are just rough guidelines, so ignore the warnings if it doesn't make sense for your data.

To see the right scores while training, you need to make sure the same spans_key is used throughout, e.g.:

https://github.com/adrianeboyd/workshop-dh2023/blob/52ee57dba2e92ff5aa4cc71b1e32e4b8abdb84f4/litbank/configs/span_finder_lg.cfg#L28

https://github.com/adrianeboyd/workshop-dh2023/blob/52ee57dba2e92ff5aa4cc71b1e32e4b8abdb84f4/litbank/configs/spancat_span_finder_lg.cfg#L30

https://github.com/adrianeboyd/workshop-dh2023/blob/52ee57dba2e92ff5aa4cc71b1e32e4b8abdb84f4/litbank/configs/spancat_span_finder_lg.cfg#L52

https://github.com/adrianeboyd/workshop-dh2023/blob/52ee57dba2e92ff5aa4cc71b1e32e4b8abdb84f4/litbank/configs/spancat_span_finder_lg.cfg#L123-L125

Unfortunately, the config files don't support variable interpolation in variable names like spans_${components.spancat.spans_key}_f = 0.8, so you have to make sure everything lines up manually. Also spacy init config only supports sc by default, so a custom spans_key always requires some manual editing.

This example intentionally uses a non-default spans_key to show how to configure a custom key, but in practice if you only have one spancat component, I'd recommend using sc since the default settings will be easier to get working.

[ddenz]

Thanks - I have tried again making sure the spans_key is consistent throughout the config file, and using the default "sc". But I am still getting zero scores when training with spacy train:

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m spacy train configs/span_finder_lg.cfg --output training/span_finder_lg --paths.train corpora/difficulty_spans_v2/train.spacy --paths.dev corpora/difficulty_spans_v2/dev.spacy
ℹ Saving to output directory: training\span_finder_lg
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE
---  ------  ------------  -------------  ----------  ----------  ----------  ------
  0       0          1.47          73.51        0.00        0.00        0.00    0.00
  0     200          1.40         362.50        0.00        0.00        0.00    0.00
  0     400          0.00         379.00        0.00        0.00        0.00    0.00
  1     600          0.00         525.00        0.00        0.00        0.00    0.00
  1     800          0.00         602.00        0.00        0.00        0.00    0.00
  2    1000          0.00         794.00        0.00        0.00        0.00    0.00
  3    1200          0.00         941.00        0.00        0.00        0.00    0.00
  4    1400          0.00        1182.00        0.00        0.00        0.00    0.00
  6    1600          0.00        1477.00        0.00        0.00        0.00    0.00
✔ Saved pipeline to output directory
training\span_finder_lg\model-last

I am getting the same zero scores with spacy train even when I use the default configs generated by prodigy data-to-spacy for span-annotated datasets using the --spancat option. Even the tok2vec loss is zeroing out (which is a bit too good to be true right?). Does this suggest there is something "wrong" with my data?

[adrianeboyd]

Can you double-check that your training data also uses sc as the spans_key?

What kinds of spans are you annotating?

[ddenz]

Can you double-check that your training data also uses sc as the spans_key?

Not sure I understand what you suggest I check but the data split was done from a single dataset using prodigy data-to-spacy so I assume dev and train sets are identical in this respect.

What kinds of spans are you annotating?

I am annotating spans that indicate someone has difficulty with a particular task or activity around their home. For example, "has trouble walking up stairs", "finds it difficult to get into the shower", "has a hard time getting to the bathroom". There is a large-ish lexical variety, but there is a fairly limited set of syntactic configurations.

I have managed to get the span_finder to train. I used a config generated by prodigy data-to-spacy and adapted it copying from the config you posted previously. However, I had to omit the tok2vec component settings for training to work. I also had to set the components.span_finder.model.tok2vec width manually to 96. Any idea what the problem was? I am not sure I understand this well enough and would appreciate any insights!

The config I used is below:

[paths]
train = null
dev = null
vectors = "en_core_web_lg"
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","span_finder"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 256
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.span_finder]
factory = "span_finder"
max_length = 30
min_length = null
scorer = {"@scorers":"spacy.span_finder_scorer.v1"}
spans_key = "sc"
threshold = 0.3

[components.span_finder.model]
@architectures = "spacy.SpanFinder.v1"

[components.span_finder.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = 2
nI = null

[components.span_finder.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 96
upstream = "*"

[components.tok2vec]
source = "en_core_web_lg"

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
train_corpus = "corpora.train"
dev_corpus = "corpora.dev"
seed = ${system:seed}
gpu_allocator = ${system:gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 5000
max_epochs = 0
max_steps = 20000
eval_frequency = 200
before_to_disk = null
annotating_components = []
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Training outputs:

(prodigy) U:\workspace\DS027_disability_data\prodigy>python -m spacy train configs\spans_v2_config.cfg --paths.train corpora\difficulty_spans_v2\train.spacy --paths.dev corpora\difficulty_spans_v2\dev.spacy --output training\difficulty_span_finder_lg
✔ Created output directory: training\difficulty_span_finder_lg
ℹ Saving to output directory: training\difficulty_span_finder_lg
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE
---  ------  ------------  -------------  ----------  ----------  ----------  ------
  0       0          4.18          60.45        0.05        0.02       39.06    0.00
  0     200         21.94         563.83        0.00        0.00        0.00    0.00
  0     400          2.12         360.48        0.00        0.00        0.00    0.00
  1     600          7.39         340.90       39.76       86.84       25.78    0.40
  1     800          8.58         346.15       49.44       88.00       34.38    0.49
  2    1000         10.61         418.97       56.38       88.33       41.41    0.56
  3    1200         12.38         407.10       58.76       86.36       44.53    0.59
  4    1400         15.94         427.26       64.00       88.89       50.00    0.64
  6    1600         17.76         434.76       65.40       83.13       53.91    0.65
  7    1800         14.36         359.16       66.36       82.56       55.47    0.66
  9    2000         15.37         356.40       64.19       79.31       53.91    0.64
 12    2200         10.71         283.79       62.86       80.49       51.56    0.63
 15    2400          9.73         273.63       62.56       79.52       51.56    0.63
 19    2600          6.84         207.80       62.26       78.57       51.56    0.62
 22    2800          5.26         186.47       62.50       81.25       50.78    0.62
 25    3000          5.07         161.49       63.41       84.42       50.78    0.63
 28    3200          4.40         151.76       64.39       85.71       51.56    0.64
 32    3400          3.72         133.18       63.05       85.33       50.00    0.63
 35    3600          3.93         136.98       61.76       82.89       49.22    0.62
 38    3800          2.31         118.01       62.80       82.28       50.78    0.63
 42    4000          3.51         125.42       62.20       80.25       50.78    0.62
 45    4200          1.54         111.12       62.14       82.05       50.00    0.62
 48    4400          1.95         118.41       63.73       85.53       50.78    0.64
 51    4600          2.45         108.37       64.04       86.67       50.78    0.64
 55    4800          2.59         108.94       63.81       81.71       52.34    0.64
 58    5000          1.96         111.08       63.46       82.50       51.56    0.63
 61    5200          3.76         128.62       64.11       82.72       52.34    0.64
 65    5400          2.85         111.67       64.73       84.81       52.34    0.65
 68    5600          3.20         118.50       66.34       88.31       53.12    0.66
 71    5800          3.53         120.71       65.05       85.90       52.34    0.65
 74    6000          2.71         117.17       65.70       86.08       53.12    0.66
 78    6200          2.90         112.39       66.02       87.18       53.12    0.66
 81    6400          4.01         117.40       66.02       87.18       53.12    0.66
 84    6600          3.09         115.48       65.69       88.16       52.34    0.66
 88    6800          1.94         100.59       64.00       88.89       50.00    0.64
Epoch 89:   0%|                                                                                                                                                                                       | 0/200 [00:00<?, ?it/s]✔ Saved pipeline to output directory
training\difficulty_span_finder_lg\model-last

I also had to use width = 96 in [components.spancat.model.tok2vec] when training span_finder+spancat together.

[adrianeboyd]

I meant to load train.spacy and inspect doc.spans to see which spans_key the annotation is saved under. I think if you use data-to-spacy then it's always under sc.

The tok2vec listener config does need to line up with the sourced tok2vec from en_core_web_lg or you will run into errors.

I'd try to lower the threshold here, since the recall is too low for further use. You want span_finder to overgenerate a bit so that spancat gets most of the intended spans as suggestions. If you start out at ~50% recall, the spancat performance will never get over this threshold. (I'll say that in my experiments for the workshop linked above, it was difficult to configure span_finder to get the precision/recall balance that I would have liked, and I had a few cases with transformer where I couldn't get it to train reasonably at all.)