custom text classification using spacy v 3.1.3

[hiteshpara]

Hi,
I am trying to solve a text classification problem using Spacy v 3.1.1. I have two datasets of size 2,30,000 samples.
Dataset 1 ==> train=1,90,000 test=40,000 and number of unique labels are 120.
Dataset 2 ==> train=1,90,000 test=40,000 and number of unique labels are 2000.
I can able to train the spacy Textcat pipeline with dataset 1 but not with dataset2.
when I am training the model with command(python -m spacy train config.cfg --verbose --output ./output) with Dataset2 no scores are printed like LOSS TEXTCAT CATS_SCORE SCORE , and in the output folder, there are no subfolders like model best and model last.
simple command got executed with no errors.
In spacy are there any limitations with the number of labels?
and for dataset1 the .Spacy files are less in size compared to dataset2 .Spacy files .so is anything to be done when we are using large size spacy file

Your Environment

• Operating System: tried on both Linux and windows
• Python Version Used:3.7.11
• spaCy Version Used:3.1.1
• Environment Information:

[polm]

when I am training the model with command(python -m spacy train config.cfg --verbose --output ./output) with Dataset2 no scores are printed like LOSS TEXTCAT CATS_SCORE SCORE , and in the output folder, there are no subfolders like model best and model last.

If no scores are being printed what's probably happening is there's some issue initializing the model. spaCy has no limit on the number of labels, but adding more labels would require more memory, and it's possible you're out of memory.

It sounds like the command exits on its own, is that correct? If so it may be killed due to OOM errors. On Linux you should be able to check the dmesg log for evidence of that.

Some other things to check:

1. How much RAM (or GPU RAM if training on GPU) do you have?
2. how big are your .spacy files?
3. Are your datasets identical except for the labels, or are they just the same size?

If this is a memory issue, you should look at using streaming corpora.

[hiteshpara]

RAM 20GB
5.18 GB is the size of .spacy file
yes datasets are identical except for the labels

[hiteshpara]

when I am training the model with command(python -m spacy train config.cfg --verbose --output ./output) with Dataset2 no scores are printed like LOSS TEXTCAT CATS_SCORE SCORE , and in the output folder, there are no subfolders like model best and model last.

If no scores are being printed what's probably happening is there's some issue initializing the model. spaCy has no limit on the number of labels, but adding more labels would require more memory, and it's possible you're out of memory.

It sounds like the command exits on its own, is that correct? If so it may be killed due to OOM errors. On Linux you should be able to check the dmesg log for evidence of that.

Some other things to check:

1. How much RAM (or GPU RAM if training on GPU) do you have?
2. how big are your .spacy files?
3. Are your datasets identical except for the labels, or are they just the same size?

If this is a memory issue, you should look at using streaming corpora.
can you please provide some other resources on how to
use streaming corpora

[polm]

can you please provide some other resources on how to use streaming corpora

I think the link is our only official documentation on the subject. Was there something in particular that you wanted to know that wasn't there?

Also, were you able to check dmesg to see if it was an OOM error or not? If it didn't show up there it'd be something else, and we'd have to do some more debugging.

Another thing you can try is using a smaller set of the data and see if that works, which would also suggest a memory issue.

[hiteshpara]

can you please provide some other resources on how to use streaming corpora

I think the link is our only official documentation on the subject. Was there something in particular that you wanted to know that wasn't there?

Also, were you able to check dmesg to see if it was an OOM error or not? If it didn't show up there it'd be something else, and we'd have to do some more debugging.

Another thing you can try is using a smaller set of the data and see if that works, which would also suggest a memory issue.

I can able to train models with small datasets and also with large datasets. the problem is when we have more number labels in the large datasets the size of the .spacy file is high.
There is no OOM error because I can successfully train the model with less dataset in linux.
I think training is not happening because of memory when I am using high size. spacy file.
can you please how to overcome memory errors?
I tried by keeping max_epochs=-1 in config.cfg file which was mentioned in the documentation but no use.

[polm]

There is no OOM error because I can successfully train the model with less dataset in linux.
I think training is not happening because of memory when I am using high size. spacy file.
can you please how to overcome memory errors?

An OOM error is an "out of memory" error. You can check if your process is being killed due to running out of memory by checking the output of dmesg on Linux.

If you can't fit all your training data in memory, you need to use a streaming corpus reader. The linked part of the docs explains how to do that. In particular, since you have a lot of labels, keep in mind you'll want to explicitly provide them. (How to do this is explained in the docs.)

I tried by keeping max_epochs=-1 in config.cfg file which was mentioned in the documentation but no use.

Do you mean that setting max_epochs = -1 didn't change anything? Or did you get an error?

[hiteshpara]

There is no OOM error because I can successfully train the model with less dataset in linux.
I think training is not happening because of memory when I am using high size. spacy file.
can you please how to overcome memory errors?

An OOM error is an "out of memory" error. You can check if your process is being killed due to running out of memory by checking the output of dmesg on Linux.

If you can't fit all your training data in memory, you need to use a streaming corpus reader. The linked part of the docs explains how to do that. In particular, since you have a lot of labels, keep in mind you'll want to explicitly provide them. (How to do this is explained in the docs.)

I tried by keeping max_epochs=-1 in config.cfg file which was mentioned in the documentation but no use.

Do you mean that setting max_epochs = -1 didn't change anything? Or did you get an error?

didn't change anything

[polm]

Just to be sure, were you able to confirm this is an OOM error using dmesg or other methods?

If using a streaming corpus didn't fix the issue, then I guess an individual batch is also causing OOM errors. In that case you'll need to reduce batch size. You should try reducing nlp.batch_size and training.batcher.size - I would reduce them to 1 or something to confirm that works and then raise them up to more reasonable values. You can see this comment for more details about batch size.

[hiteshpara]

start = 100
stop = 1000
compound = 1.001
t = 0.0
above parameters i can see in training.batcher.size
you mean to say that reduce all those number?

[polm]

You should reduce the start and stop values, which actually control the batch size.

Since it seems like this is not a bug but a configuration issue, I'm moving it to Discussions.