Entity extraction

Machine-learning-based extraction for general entities

Copy link to section

The machine-learning-based extraction model entity-mentions_bert_multi_stock is trained on labeled data for the more complex entity types such as person, organization and location.

Capabilities

The entity block extract entities from the input text. The following types of entities are recognized:

• Date
• Duration
• Facility
• Geographic feature
• Job title
• Location
• Measure
• Money
• Ordinal
• Organization
• Person
• Time

Supported languages

Entity extraction is available for the following languages. For a list of the language codes and the corresponding language, see Language codes.

ar, cs, da, de, en, es, fi, fr, he, hi, it, ja, ko, nb, nl, nn, pt, ro, ru, sk, sv, tr, zh-cn

Dependencies on other blocks

The following block must run before you can run the Entity extraction block:

• syntax_izumo_<language>_stock

Code sample

import watson_nlp

# Load Syntax Model for English, and the multilingual BERT Entity model
syntax_model = watson_nlp.load(watson_nlp.download('syntax_izumo_en_stock'))
bert_entity_model = watson_nlp.load(watson_nlp.download('entity-mentions_bert_multi_stock'))

# Run the syntax model on the input text
syntax_prediction = syntax_model.run('IBM\'s CEO Arvind Krishna is based in the US')

# Run the entity mention model on the result of syntax model
bert_entity_mentions = bert_entity_model.run(syntax_prediction)
print(bert_entity_mentions)
Copy to clipboardCopied to clipboard

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 0,
        "end": 3,
        "text": "IBM"
      },
      "type": "Organization",
      "producer_id": {
        "name": "BERT Entity Mentions",
        "version": "0.0.1"
      },
      "confidence": 0.9944692850112915,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 6,
        "end": 9,
        "text": "CEO"
      },
      "type": "JobTitle",
      "producer_id": {
        "name": "BERT Entity Mentions",
        "version": "0.0.1"
      },
      "confidence": 0.9871304631233215,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 10,
        "end": 24,
        "text": "Arvind Krishna"
      },
      "type": "Person",
      "producer_id": {
        "name": "BERT Entity Mentions",
        "version": "0.0.1"
      },
      "confidence": 0.9988446235656738,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 41,
        "end": 43,
        "text": "US"
      },
      "type": "Location",
      "producer_id": {
        "name": "BERT Entity Mentions",
        "version": "0.0.1"
      },
      "confidence": 0.9911670088768005,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "BERT Entity Mentions",
    "version": "0.0.1"
  }
}
Copy to clipboardCopied to clipboardShow more

Machine-learning-based extraction for PII entities

Copy link to section

The machine-learning-based extraction model entity-mentions_bilstm_en_pii is trained on labeled data for types where labeled data can be obtained, namely person and location.

Capabilities

The entity block entity-mentions_bilstm_en_pii recognizes the following types of entities:

Dependencies on other blocks

The following block must run before you can run the entity-mentions_bilstm_en_pii block:

• syntax_izumo_en_stock

Code sample

import os
import watson_nlp

text = 'Denver is the capital of Colorado. The total estimated government spending in Colorado in fiscal year 2016 was $36.0 billion. IBM office is located in downtown Denver. Michael Hancock is the mayor of Denver.'

# Load rbr model in WatsonNLP
rbr_model = watson_nlp.load(watson_nlp.download('entity-mentions_rbr_en_stock', parent_dir=parent_dir))

# Run rbr model in WatsonNLP
rbr_result = rbr_model.run(text)

print(type(rbr_result))
rbr_result
Copy to clipboardCopied to clipboard

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 102,
        "end": 106,
        "text": "2016"
      },
      "type": "Number",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    },
    {
      "span": {
        "begin": 112,
        "end": 124,
        "text": "36.0 billion"
      },
      "type": "Number",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}
Copy to clipboardCopied to clipboardShow more

Rule-based extraction for general entities

Copy link to section

The rule-based model entity-mentions_rbr_xx_stock identifies syntactically regular entities.

Capabilities

Rule-based extraction handles syntactically regular entity types. The entity block extract entities from the input text. The following types of entities are recognized:

• PhoneNumber
• EmailAddress
• Number
• Percent
• IPAddress
• HashTag
• TwitterHandle
• URLDate

Supported languages

Entity extraction is available for the following languages. For a list of the language codes and the corresponding language, see Language codes.

ar, cs, da, de, en, es, fi, fr, he, hi, it, ja, ko, nb, nl, nn, pt, ro, ru, sk, sv, tr, zh-cn, zh-tw

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load a rule-based Entity Mention model for English
rbr_entity_model = watson_nlp.load(watson_nlp.download('entity-mentions_rbr_en_stock'))

# Run the entity model on the input text
rbr_entity_mentions = rbr_entity_model.run('My email is john@us.ibm.com')
print(rbr_entity_mentions)
Copy to clipboardCopied to clipboard

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 12,
        "end": 27,
        "text": "john@us.ibm.com"
      },
      "type": "EmailAddress",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}
Copy to clipboardCopied to clipboardShow more

Rule-based extraction for PII entities

Copy link to section

The rule-based model entity-mentions_rbr_multi_pii handles the majority of the types by identifying common formats of PII entities and performing possible checksum or validations as appropriate for each entity type. For example, credit card number candidates are validated using the Luhn algorithm.

Capabilities

The entity block entity-mentions_rbr_multi_pii recognizes the following types of entities:

Some PII entity type names are country-specific. The _ in the following entity types is a placeholder for a country code.

• BankAccountNumber.BBAN._ : These are more variable national bank account numbers and the extraction is mostly language-specific without a general checksum algorithm.
• BankAccountNumber.IBAN._ : Highly standardized IBANs are supported in a language-independent way and with a checksum algorithm.
• NationalNumber.NationalID._: These national IDs don’t have a (published) checksum algorithm, and are being extracted on a language-specific basis.
• NationalNumber.Passport._ : Checksums are implemented only for the countries where a checksum algorithm exists. These are specifically extracted language with additional context restrictions.
• NationalNumber.TaxID._ : These IDs don't have a (published) checksum algorithm, and are being extracted on a language-specific basis.

Which entity types are available for which languages and which country code to use is listed in the following table.

Dependencies on other blocks

None

Code sample

import watson_nlp

# Load the RBR PII model. Note that this is a multilingual model supporting multiple languages.
rbr_entity_model = watson_nlp.load(watson_nlp.download('entity-mentions_rbr_multi_pii'))

# Run the RBR model. Note that language code of the input text is passed as a parameter to the run method.
rbr_entity_mentions = rbr_entity_model.run('Please find my credit card number here: 378282246310005. Thanks for the payment.', language_code='en')
print(rbr_entity_mentions)

Copy to clipboardCopied to clipboard

Output of the code sample:

{
  "mentions": [
    {
      "span": {
        "begin": 40,
        "end": 55,
        "text": "378282246310005"
      },
      "type": "BankAccountNumber.CreditCardNumber.Amex",
      "producer_id": {
        "name": "RBR mentions",
        "version": "0.0.1"
      },
      "confidence": 0.8,
      "mention_type": "MENTT_UNSET",
      "mention_class": "MENTC_UNSET",
      "role": ""
    }
  ],
  "producer_id": {
    "name": "RBR mentions",
    "version": "0.0.1"
  }
}
Copy to clipboardCopied to clipboardShow more

Parent topic: Watson Natural Language Processing block catalog