Data governance tutorial: Govern virtualized data

There are two required steps to enabling governance of virtualized data:

• Enforce data protection rules in Watson Query.
• Set up authorization between IBM Knowledge Catalog and Watson Query.

To preview this task, watch the video beginning at 02:20.

With data protection rules in place, virtual tables are governed by those rules. Follow these steps to run an SQL query on a governed virtual table:

1. From the Watson Query service menu, click Run SQL.
   

2. Copy and paste the following SELECT statement for the new query. Replace <your schema> with the schema name that you noted earlier.

   SELECT * FROM <your-schema>.MORTGAGE_APPLICANT WHERE STATE_CODE LIKE 'CA'
   

   Your query looks similar to SELECT * FROM DV_IBMID_663002GN1Q.MORTGAGE_APPLICANT WHERE STATE_CODE LIKE 'CA'
   

3. Click Run all.

4. After the query completes, select the query on the History tab. On the Results tab, you can see that the table is filter to only applicants from the state of California. The data protection rules apply in the Watson Query, catalog preview, catalog download, Data Refinery, and Project Asset preview. The rule doesn’t apply to the asset owner. Watch the video at 02:47 to see what other users see when they run the SQL query.

To preview this task, watch the video beginning at 03:02.

In the Virtualize external data tutorial, you created virtual tables and virtual join views, and copied them to your Data integration project. If you would like to use that project to complete this tutorial, then skip to Task 3. If you would like to use your Data governance project to complete this tutorial, then follow these steps:

1. From the service menu, click Virtualization > Virtualized data.
   

2. Select the following tables:

   • MORTGAGE_APPLICATION
   • MORTGAGE_APPLICANT
   • CREDIT_SCORE
   • APPLICANTS_APPLICATIONS_JOINED
   • APPLICANTS_APPLICATIONS_CREDIT_SCORE_JOINED

3. Click Assign.

4. For the Project, select Data governance.

5. Click Assign.

6. When the virtual objects are successfully assigned, navigate to your project.

7. In the Data governance project, click the Assets tab. The virtual data tables begin with <your schema>.

8. Open any of the virtual data tables. For example, click the APPLICANTS_APPLICATIONS_CREDIT_SCORE_JOINED virtual table to view it.

9. Provide your credentials to access the data asset.

   1. For the Authentication method, select API Key.

   2. Paste the same API key that you created in the Virtualize external data tutorial.

   3. Click Connect. The data protection rules apply in the catalog preview, catalog download, Data Refinery, and Project Asset preview. The rule doesn’t apply to the asset owner. Watch the video at 04:09 to see what other users see trying to access the virtual data table.

To preview this task, watch the video beginning at 04:21.

You can enrich data assets with information that helps users to find data faster. Users can use the enrichments to decide whether the data is appropriate for the task at hand, whether they can trust the data, and how to work with the data. Such information includes, for example, terms that define the meaning of the data, rules that document ownership or determine quality standards, or reviews. Follow these steps to enrich the virtual data tables:

1. Click Data governance in the navigation trail to return to the project.
   

2. On the Assets tab, click New asset > Enrich data assets with metadata.

3. For the name, copy and paste the following text:

   Virtual mortgage data - metadata enrichment
   

4. Click Next to continue.

5. Click Select data from project.

   1. Select Data asset.

   2. Click the checkbox next to the following assets:

      • <your schema>.MORTGAGE_APPLICATION
      • <your schema>.MORTGAGE_APPLICANT
      • <your schema>.CREDIT_SCORE
      • <your schema>.APPLICANTS_APPLICATIONS_JOINED
      • <your schema>.APPLICANTS_APPLICATIONS_CREDIT_SCORE_JOINED

   3. Click Select.

6. Click Next to continue to the enrichment objective.

7. Select all enrichment objectives:

   • Profile data
   • Assign terms
   • Run basic quality analysis

8. For Categories, click Select categories.

   1. Select only [uncategorized] and Banking.

   2. Click Select.

9. For the Sampling, select Basic.

10. Click Next to continue to the schedule.

11. Click Next to continue to the review.

12. Click Create.

13. The metadata enrichment asset displays, but the job might take several minutes to complete. Click the Refresh icon to watch the status change from Queued to In progress to Finished. When the job run is complete, you see the five assets listed.

To preview this task, watch the video beginning at 05:48.

After Metadata enrichment run is completed, follow these steps to view the enriched data:

1. From the Virtual mortgage data - metadata enrichment screen, click the Columns tab.

2. Search for mortgage_applicant.

3. In the list of Columns, locate the EMAIL_ADDRESS column for your-schema.MORTGAGE_APPLICANT asset.

   1. Click the Overflow menu at the end of the EMAIL_ADDRESS for your your_schema.MORTGAGE_APPLICANT row, and choose View column details.

   2. In the side panel on the Details tab, you see profiling information such as: Format, Frequency distribution, Statistics.

   3. In the side panel, click the Governance tab. This tab includes the data classes and business terms that were auto-assigned during the metadata enrichment. You might also see suggested business terms and data classes, and manually assign them.

   4. Review any suggested business terms or data classes and manually assign them. For example, you may see Address as a suggested business term.

      1. Click Suggested business terms.

      2. For Address, click Assign.

      3. Click Suggested data classes.

      4. For Text, click Assign.

4. At the end of the EMAIL_ADDRESS column for your your_schema.MORTGAGE_APPLICANT asset row, click the Overflow menu , and choose View data quality details.

   1. View the data quality score. IBM Knowledge Catalog automatically generates a data quality score for each column and data asset by analyzing every value in every record according to pre-built dimensions.

   2. Click the X to close the Data quality window.

5. Search for credit_score.

6. For the CITY column for your_schema.CREDIT_SCORE asset, click the Overflow menu , and choose Mark as reviewed.

7. Click the Assets tab.

8. In the list of Assets, for your your_schema.MORTGAGE_APPLICANT asset, click the Overflow menu , and choose View asset details.

   1. In the side panel, click the Governance tab to see any business term that were auto-assigned.

   2. Click the Add icon (or you might see the Edit icon ) to manually assign business terms.

   3. Search for social. If you don't see any results, then make sure that the drop-down list is set to All terms instead of Suggested terms.

   4. Select Social Security Number.

   5. Click Assign.

To preview this task, watch the video beginning at 7:18.

Now that the virtualized data is enriched with business terms, follow these steps to publish the virtual tables it to a catalog:

1. Click Data governance in the navigation trail to return to the project.
   

2. Click the Assets tab.

3. Navigate to Data > Data assets.

4. Click the checkbox next to the following assets:

   • <your schema>.MORTGAGE_APPLICATION
   • <your schema>.MORTGAGE_APPLICANT
   • <your schema>.CREDIT_SCORE
   • <your schema>.APPLICANTS_APPLICATIONS_JOINED
   • <your schema>.APPLICANTS_APPLICATIONS_CREDIT_SCORE_JOINED

5. Click Publish to catalog.

   1. Select the Mortgage Approval Catalog (or your catalog name) from the list, and click Next.

   2. Select the option to Go to the catalog after publishing it, and click Next.

   3. Review the assets, and click Publish.

6. In the Mortgage Approval Catalog, search for <your-schema>.

7. Open one of the virtual tables. If prompted, provide your credentials:

   1. For the Authentication method, select API Key.

   2. Paste the same API key that you created in the Virtualize external data tutorial.

8. Click Asset tab to view the data. The data protection rules apply in the catalog preview, catalog download, Data Refinery, and Project Asset preview. The rule doesn’t apply to the asset owner. Watch the video at 08:17 to see what other users see trying to access the virtual data table in the catalog.