Bert models from Spacy

[balachander1964]

Hi, I request you to let me know how to load BERT clinical NER and TEXTCAT models into SPACY 3 and use them as a pipeline.

[polm]

If you want to load a specific Transformer model and train an NER or textcat pipeline on it you should be able to just follow the instructions here.

Is there a particular pretrained NER/textcat model you wanted to use? If so could you link to it?

[balachander1964]

Hi, Thank you for your mail. I am trying to use clinical BERT NER and TEXTCAT models from my SPACY application. This may reduce the time I spend annotating the text, which is killing me. I am currently fine-tuning the SCISPACY BIONLP, SCISPACY CDR models for detecting anatomic entities, and diseases. Also, I am fine-tuning the Core-LG English model for identifying the drug related information, and events. I am using Multi-Label TextCats for identifying if an event is open or closed. Bala Sent from Mail for Windows 10 From: polm Sent: Monday, May 24, 2021 9:53 AM To: explosion/spaCy Cc: balachander1964; Author Subject: Re: [explosion/spaCy] Bert models from Spacy (#8181) If you want to load a specific Transformer model and train an NER or textcat pipeline on it you should be able to just follow the instructions here. Is there a particular pretrained NER/textcat model you wanted to use? If so could you link to it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[polm]

OK, thanks for the extra details. I'm not sure I understand what you're having trouble with though. It sounds like you already have a working spaCy pipeline but you'd like to use BERT? It looks like ScispaCy provides a scibert model, have you tried using that as your base model?

If you change from a non-Transformers model to a Transformers model you'll have to alter the training pipeline in your config slightly, but it shouldn't be that much of a change.

[balachander1964]

Hi polm, I am trying to load other clinical BERT NER models and check if any model predicts better than SCISPACY. I am forced to manually annotate 50000 paragraphs for 5 different models, different concept annotation for each. We are working on application that would provide full details of a report to a doctor, which would provide details like: 1. What disease / condition / sign / symptom 2. Current state of the illness 3. Medications administered, 4. What are the medical events with details about time / date as well. 5. Which site is affected etc. My application should cover the morbidities as well as co-morbidities. Sent from Mail for Windows 10 From: polm Sent: Monday, May 24, 2021 12:27 PM To: explosion/spaCy Cc: balachander1964; Author Subject: Re: [explosion/spaCy] Bert models from Spacy (#8181) OK, thanks for the extra details. I'm not sure I understand what you're having trouble with though. It sounds like you already have a working spaCy pipeline but you'd like to use BERT? It looks like ScispaCy provides a scibert model, have you tried using that as your base model? If you change from a non-Transformers model to a Transformers model you'll have to alter the training pipeline in your config slightly, but it shouldn't be that much of a change. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[balachander1964]

Hi polm,, I am requesting you to let me know how to load the BERT models as a Spacy pipeline using the Spacy.Load() method because, BERT models may not define a configuration file. I request for your help with this. Bala Sent from Mail for Windows 10 From: Balachander Agoramurthy Sent: Monday, May 24, 2021 1:12 PM To: explosion/spaCy; explosion/spaCy Cc: Author Subject: RE: [explosion/spaCy] Bert models from Spacy (#8181) Hi polm, I am trying to load other clinical BERT NER models and check if any model predicts better than SCISPACY. I am forced to manually annotate 50000 paragraphs for 5 different models, different concept annotation for each. We are working on application that would provide full details of a report to a doctor, which would provide details like: 1. What disease / condition / sign / symptom 2. Current state of the illness 3. Medications administered, 4. What are the medical events with details about time / date as well. 5. Which site is affected etc. My application should cover the morbidities as well as co-morbidities. Sent from Mail for Windows 10 From: polm Sent: Monday, May 24, 2021 12:27 PM To: explosion/spaCy Cc: balachander1964; Author Subject: Re: [explosion/spaCy] Bert models from Spacy (#8181) OK, thanks for the extra details. I'm not sure I understand what you're having trouble with though. It sounds like you already have a working spaCy pipeline but you'd like to use BERT? It looks like ScispaCy provides a scibert model, have you tried using that as your base model? If you change from a non-Transformers model to a Transformers model you'll have to alter the training pipeline in your config slightly, but it shouldn't be that much of a change. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[polm]

You should be able to follow the docs here to load any Huggingface Transformers model.

https://spacy.io/usage/embeddings-transformers#transformers-training

Just change the "name" to the name on the Huggingface Hub.

Note that spacy-transformers only uses Transformers models as a feature source, so if a model is pretrained for NER you can't use it directly, you need to train a spaCy model using it as input. You can use the quickstart NER GPU config as a starting point.

[Lolologist]

Just noting here that after plenty of looking around I came across this. I had no idea it was that simple! It might behoove the company to make it more explicit in the documentation. I was checking, like, https://towardsdatascience.com/how-to-fine-tune-bert-transformer-with-spacy-3-6a90bfe57647 and not feeling very confident about it. But you're right, I'm training a model that's plug-and-play just as you said, right now. Thanks!

[balachander1964]

Thank you polm. I will try it out. I appreciate your kind help. Bala Sent from Mail for Windows 10 From: polm Sent: Wednesday, May 26, 2021 10:01 AM To: explosion/spaCy Cc: balachander1964; Author Subject: Re: [explosion/spaCy] Bert models from Spacy (#8181) You should be able to follow the docs here to load any Huggingface Transformers model. https://spacy.io/usage/embeddings-transformers#transformers-training Just change the "name" to the name on the Huggingface Hub. Note that spacy-transformers only uses Transformers models as a feature source, so if a model is pretrained for NER you can't use it directly, you need to train a spaCy model using it as input. You can use the quickstart NER GPU config as a starting point. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.