Creating a metadata enrichment asset

Configure and run metadata enrichment to add several layers of metadata to your data assets.

You can create a data profile to classify a data asset and compile statistics about the values that it contains. Use predefined data quality checks for an initial quality assessment of your data. Enrich assets with business vocabulary that describes the semantic meaning of the data for your organization. Identify relationships between data assets.

Required services

IBM Knowledge Catalog
DataStage for advanced key or relationship analysis and advanced profiling

Required permissions

To create and run a metadata enrichment, you must have the Admin or the Editor role in the project, and you must have at least view access to the categories that you want to use in the enrichment. Also, you must be authorized to access the connections to the data sources of the data assets to be enriched.

All operations that are run as part of a metadata enrichment require credentials for secure authorization. Typically, your user API key is used to execute such long-running operations without disruption. If credentials are not available when you create a metadata enrichment or try to run any type of enrichment, you are prompted to create an API key. That API key is then saved as your task credentials. See Managing the user API key.

You can also create metadata enrichments with APIs instead of the user interface. The links to these APIs are listed in the Learn more section.

To create a metadata enrichment asset and a job for enriching data:

1. Open a project and click New asset > Metadata Enrichment. After you create the first metadata enrichment in this way, you can add new metadata enrichment assets from the project's Asset page.

2. Define details:

   • Specify a name for the metadata enrichment.
   • Optional: Provide a description.
   • Optional: Select or create tags to be assigned to the metadata enrichment asset to simplify searching. You can create new tags by entering the tag name and pressing Enter.

3. Set the initial data scope.

   Select the data assets that you want enrich from Data assets. See Initial data scope. Review the selected scope before you proceed. You can directly delete assets from the data scope or you can rework the entire scope by clicking Edit data scope.

   You can skip this step to create an empty metadata enrichment asset, and set the scope later.

4. Define the objective of this metadata enrichment asset. You can add several layers of metadata to a data asset:

   • Profile the data to classify it and compile statistics about the values.
   • Enrich assets with business vocabulary that describe the semantic meaning of the data for your organization.
   • Run predefined data quality checks for an initial quality assessment.
   • Identify primary keys and key relationships. See Enrichment objectives.

5. Select categories to determine the business vocabulary that can be applied during the enrichment. See Category selection.

6. Select a sampling type. See Sampling.

7. Define whether you want to run scheduled enrichment jobs. If you don't set a schedule, you run the enrichment when you save the metadata enrichment asset. You can rerun the enrichment manually at any time. See Scheduling options

8. Select the data scope for the reruns of the enrichment, whether scheduled or run manually. See Scope of reruns of the enrichment

9. Review the metadata enrichment configuration. To make changes, click the edit icon on the tile and update the settings.

10. Click Create. The metadata enrichment asset is added to the project, and several jobs are created:

    • A metadata enrichment job
    • A job for deep primary key analysis named metadata-enrichment-name (PK Detection)
    • A job for deep relationship analysis named metadata-enrichment-name (Relationship Detection)

    If you didn't configure a schedule, the enrichment is run immediately. If you configured a schedule, the enrichment will run on the defined schedule.

After the enrichment is complete, you can access a high-level overview of the enrichment results by viewing the metadata enrichment asset. From there, you can drill down into and work with the results for each asset. See Working with the enrichment results.

Metadata enrichment is run on assets that are available in the project. Thus, the list of enriched assets might not correspond to the configured scope of included metadata import assets in these cases:

• Metadata import was not yet complete when the enrichment started.
• Metadata import failed for a set of assets or failed completely.

For information about how to update, rerun, or delete a metadata enrichment, see Managing an existing metadata enrichment.