Trained NER Model Not Labeling Custom Labels

[matthewbriancoleman]

I am trying to train a blank NER (SpaCy 2.3.1) model to recognize PERSON, ORG, and PHONE entities. I have about 1000 observations of data with ~1300 PHONE entities, and less than 100 of both PERSON and ORG entities (I am POC'ing the phone number aspect, so I have a bigger dataset with more PERSON and ORG entities which I will train on after I figure out this issue.)

For some weird reason, when I train the model, it does not predict a single PHONE entity when trying to predict on ANY data (even when trying to predict the training dataset), but it predicts PERSON and ORG entities, even though they are in a way smaller concentration than the PHONE entities.

It seems to be related to the PHONE entities specifically, because when I train on the bigger dataset with only PERSON and ORG entities, the model works fine. If I train only on PHONE entities (removing PERSON and ORG entities), then the model does not predict PHONE, PERSON, or ORG entities.

My training function is below:

import random
from spacy.util import minibatch, compounding
from pathlib import Path

nlp = spacy.blank('en')
ner = nlp.create_pipe('ner')
ner.add_label('PHONE')
ner.add_label('ORG')
ner.add_label('PERSON')
nlp.add_pipe(ner)

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']

# TRAINING THE MODEL
with nlp.disable_pipes(other_pipes):
    optimizer = nlp.begin_training()

  # Training for 30 iterations
    for iteration in range(10):

    # shuffling examples  before every iteration
        random.shuffle(train_data)
        losses = {}
        # batch up the examples using spaCy's minibatch
        batches = minibatch(train_data, size=compounding(4.0, 32.0, 1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(
                        texts,  # batch of texts
                        annotations,  # batch of annotations
                        drop=0.5,  # dropout - make it harder to memorise data
                        losses=losses,
                    )
        print("Losses", losses)

Some training data

[('CPR (713) 284-4175 [Pager]', {'entities': [(4, 18, 'PHONE')]}),
 ('SAP/ISC (713) 345-4727', {'entities': [(8, 22, 'PHONE')]}),
 ('Fax: (403) 974-6707', {'entities': [(5, 19, 'PHONE')]}),
 ('> Fax: 412-268-4146', {'entities': [(7, 19, 'PHONE')]}),
 ('Also, he wants you to call him at 202-371-7120.',
  {'entities': [(34, 46, 'PHONE')]}),
 ('Fax 713/646-8498', {'entities': [(4, 16, 'PHONE')]}),
 ('FAX: 610-765-7649', {'entities': [(5, 17, 'PHONE')]}),
 ('(713) 207-0161 (fax)', {'entities': [(0, 14, 'PHONE')]}),
 ('Phone: 713-345-8897', {'entities': [(7, 19, 'PHONE')]}),
 ('If you have any questions or need any additional information, please contact Harry Grubbs at 713-853-5417.',
  {'entities': [(77, 89, 'PERSON'), (93, 105, 'PHONE')]}),
 ('212-333-6401 fax', {'entities': [(0, 12, 'PHONE')]}),
 ('have any questions please call Don Black at (713) 853-4750.',
  {'entities': [(31, 40, 'PERSON'), (44, 58, 'PHONE')]}),
 ('OSS/UA4/TARP (713) 285-3165 [Pager]', {'entities': [(13, 27, 'PHONE')]}),
 ('Phone: (403) 974-6923', {'entities': [(7, 21, 'PHONE')]}),
 ('713/853-3233.', {'entities': [(0, 12, 'PHONE')]}),
 ('Phone: (713) 853 3848', {'entities': [(7, 21, 'PHONE')]}),
 ('Future conference enquiries should go to Paul Bristow (US - 212 925 6990) or',
  {'entities': [(41, 53, 'PERSON'), (60, 72, 'PHONE')]}),
 ('City of Los Angeles Detrich B. Allen (213) 978-0840',
  {'entities': [(20, 36, 'PERSON'), (37, 51, 'PHONE')]}),
 ('OSS/UA4/TARP (713) 285-3165 [Pager]', {'entities': [(13, 27, 'PHONE')]}),
 ('713-853-0596', {'entities': [(0, 12, 'PHONE')]}),
 ('IT Helpdesk on +44 (0) 20 7936 4000 Ext.2000 or email [email protected]',
  {'entities': [(15, 44, 'PHONE')]}),
 ('-7598.15119', {'entities': [(0, 15, 'PHONE')]}),
 ('973-622-5000', {'entities': [(0, 12, 'PHONE')]}),
 ('503-464-3803', {'entities': [(0, 12, 'PHONE')]}),
 ('fax (713) 681-9186', {'entities': [(4, 18, 'PHONE')]}),
 ('Phone: 202-637-4407 Home: 202-882-3385',
  {'entities': [(7, 19, 'PHONE'), (26, 38, 'PHONE')]}),
 ('My home phone is 713/528-3763.', {'entities': [(17, 29, 'PHONE')]})]

The meta.json file

{
  "lang":"en",
  "name":"model",
  "version":"0.0.0",
  "spacy_version":">=2.3.1",
  "description":"",
  "author":"",
  "email":"",
  "url":"",
  "license":"",
  "spacy_git_version":"554291558",
  "vectors":{
    "width":0,
    "vectors":0,
    "keys":0,
    "name":"spacy_pretrained_vectors"
  },
  "pipeline":[
    "ner"
  ],
  "factories":{
    "ner":"ner"
  },
  "labels":{
    "ner":[
      "ORG",
      "PERSON",
      "PHONE"
    ]
  }
}

[polm]

Sorry you're having trouble with this. It's been a while since I trained a v2 model, but your code doesn't have any obvious issues to me - for example, you're calling add_label properly. Could you provide a complete script with example data that reproduces the problem we could check?

Besides your problem with training, a couple of other things...

1. Is there a particular reason you're using v2? v3 has been out for most of this year, and training in v3 is designed to avoid subtle problems like whatever's going on here.
2. For phone numbers in particular, I usually see people use a regex to match them rather than an NER model because the format is very constrained. It looks like you're trying to match phone numbers from arbitrary countries here, which makes using a model more palatable, but just wanted to mention the alternative.

[matthewbriancoleman]

Hi @polm, here is a script of the code I have been working with. The outputted .csv (in my case) only has predictions on PERSON and ORG entities, but not PHONE ones.

Script:

# Import step
import pandas as pd
import random
from spacy.util import minibatch, compounding
from pathlib import Path
import ast
import spacy

# Function Definintions

def retype(df):
    """
    This function is for changing the stringed columns to lists.
    """
    print('-----------\nColumns changed:')
    for col in df.columns:
        if '[' in str(df[col].iloc[1]):
            print(col)
            df[col] = df[col].apply(lambda x: ast.literal_eval(x))

def spacy_data(df, fuzzified = True, augmented = True):
    '''
    This function turn the dataframes to spacy training ready data
    '''
    output = []
    if augmented == False:
        df = df[df.name_lan == 'English']
        
    for row in df.iterrows():
        ents = []
        if fuzzified == True:
            sent = row[1].sentence_fuzzified
            tuples = row[1].true_tuples_fuzzified
        else:
            sent = row[1].sentence
            tuples = row[1].true_tuples
            
        for i, each_tup in enumerate(tuples):
            ents.append((each_tup[0], each_tup[1], row[1].true_entities[i]))
        
        output.append((sent, {"entities": ents}))
    return output

def spacy_pred(df, model, colname, fuzzified = False):
    """
    The function will predict the data frame with the given spacy model and generate the corresponding columns
    (tuples, text, entity), fuzzified data will be used if fuzzified == True.
    :param df: data after all preprocessing steps
    :param model: spacy model which will be used for prediction
    :param colname: column prefix which will be generated
    :param fuzzified: If True, will predict against the fuzzified version of the data.
    :return: None, the new columns with the prediction will be added to the original df.
    """

    result_dic = {'tuples':[], 'texts':[], 'entities':[]}
    for row in df.iterrows():
        if fuzzified == True:
            sent = row[1].sentence_fuzzified
        else:
            sent = row[1].sentence
        sent_idx = row[1].sent_idx
        
        doc = model(sent)
        starts = []
        ends = []
        names = []
        entities = []
        
        for ent in doc.ents:
            if ent.label_ in ['PERSON', 'LOC', 'ORG']:
                starts.append(ent.start_char)
                ends.append(ent.end_char)
                names.append(ent.text)
                entities.append(ent.label_)
                
        result_dic['tuples'].append([(starts[i], ends[i]) for i in range(len(ends))])
        result_dic['texts'].append(names)
        result_dic['entities'].append(entities)
    
    df['spacy_{}_tuples'.format(colname)] = result_dic['tuples']
    df['spacy_{}_texts'.format(colname)] = result_dic['texts']
    df['spacy_{}_entities'.format(colname)] = result_dic['entities']

# Reading in the data and converting to SpaCy v2 format

if __name__ == '__main__': 
    
    df_train = pd.read_csv('train_phone_enron_total.csv')
    df_val = pd.read_csv('val_phone_enron_total.csv')

    for df in [df_train, df_val]:
        retype(df)

    train_data = spacy_data(df_train, fuzzified = True, augmented = True)
    val_data = spacy_data(df_val, fuzzified = True, augmented = True)

    # Model Training

    nlp = spacy.blank('en')
    ner = nlp.create_pipe('ner')
    ner.add_label('PHONE')
    ner.add_label('ORG')
    ner.add_label('PERSON')
    nlp.add_pipe(ner)

    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']

    # TRAINING THE MODEL
    with nlp.disable_pipes(other_pipes):
        optimizer = nlp.begin_training()

    # Training for 30 iterations
        for iteration in range(10):

        # shuffling examples  before every iteration
            random.shuffle(train_data)
            losses = {}
            # batch up the examples using spaCy's minibatch
            batches = minibatch(train_data, size=compounding(4.0, 32.0, 1.001))
            for batch in batches:
                texts, annotations = zip(*batch)
                nlp.update(
                            texts,  # batch of texts
                            annotations,  # batch of annotations
                            drop=0.5,  # dropout - make it harder to memorise data
                            losses=losses,
                        )
            print("Losses", losses)


    spacy_pred(df_train, nlp, 'core_lg_val', fuzzified = True)

    df_train.to_csv('train_data_predicted.csv')

Data Used:

train_phone_enron_total.csv
val_phone_enron_total.csv

To answer your question:

1. At the moment we are sticking with v2 for a couple of reasons.

• There was not much(if any) improvement in the accuracy of our NER model after converting from v2 to v3.
• We already had all the infrastructure set up for a v2 model, and trying to get everyone up to date with v3 when v2 was working just as fine did not seem to be worth the time.

2. We have considered using rule-based matching for phone numbers, but we wanted to POC a PHONE entity model first to see if it was possible and what the accuracy of that model would be. The thought was that having a single model with multiple capabilities would be preferable over having multiple different plugins for NER.

[polm]

I ran your code with v2.3.5 and it works fine, including detecting phone numbers.

In spacy_pred you are ignoring PHONE entities, is that your problem? This line:

            if ent.label_ in ['PERSON', 'LOC', 'ORG']:

[matthewbriancoleman]

You are right, I should have triple checked that function as well. Thank you for the help and second pair of eyes! :)