AI governance tutorial: Test and validate the model

Preview the tutorial

Watch this video to preview the steps in this tutorial. There might be slight differences in the user interface shown in the video. The video is intended to be a companion to the written tutorial.

This video provides a visual method as an alternative to following the written steps in this documentation.

Prerequisites

Task 1: Run the notebook to set up the monitors

Run the second notebook included in the sample project to:

• Fetch the model and deployments.
• Configure Watson OpenScale.
• Create the service provider and subscription for your machine learning service.
• Configure the quality monitor.
• Configure the fairness monitor.
• Configure explainability.

Follow these steps to run the notebook included in the sample project. This notebook sets up monitors for your model, which can also be configured through the user interface. However, it is quicker and less error prone to set them up with a notebook. Take some time to read through the comments in the notebook, which explain the code in each cell.

1. From the Cloud Pak for Data navigation menu , choose Projects > All projects.

2. Open the AI governance project.

3. Click the Assets tab, and then navigate to Notebooks.
   

4. Click the Overflow menu for the 2-monitor-wml-model-with-watson-openscale notebook, and choose Edit.

5. Click Cell > Run All to run all of the cells in the notebook. Alternatively, you can run the notebook cell by cell if you want to explore each cell and its output.

6. The third cell requires your input.

   1. At the Enter host name prompt, type your Cloud Pak for Data hostname beginning with https://, and press Enter. For example, https://mycpdcluster.mycompany.com.

   2. At the Username prompt, type your Cloud Pak for Data username, and press Enter.

   3. At the Password prompt, type your Cloud Pak for Data password, and press Enter.

7. The notebook takes 1 - 3 minutes to complete. You can monitor the progress cell by cell noticing the asterisk "In [*]" changing to a number, for example, "In [1]".

8. If you encounter any errors during the notebook run, try these troubleshooting tips:

   • Click Kernel > Restart & Clear Output to restart the kernel, and then run the notebook again.
   • Verify that you created the model use case, deployment space, and deployment name in the Build and deploy a model tutorial by copying and pasting the specified artifact name exactly with no leading or trailing spaces.

Task 2: Evaluate the model

Follow these steps to download holdout data, and use that data to evaluate the model in Watson OpenScale:

1. Click the AI governance project in the navigation trail.
   

2. On the Assets tab, click Data > Data assets.

3. Click the Overflow menu for the GoldenBank_HoldoutData.csv data asset, and choose Download. To validate that the model is working as required, you need a set of labeled data, which was held out from model training. This CSV file contains that holdout data.

4. Launch Watson OpenScale. From the Cloud Pak for Data navigation menu , choose Services > Instances.

5. On the Instances page, click the Overflow menu at the end of the row for your Watson OpenScale instance, and choose Open.

6. On the Insights dashboard, click the Mortgage Approval Model Deployment tile.

7. From the Actions menu, select Evaluate now.

8. From the list of import options, select from CSV file.

9. Drag the Golden Bank_HoldoutData.csv data file you downloaded from the project into the side panel.

10. Click Upload and evaluate.

Task 3: Observe the model monitors for quality

The Watson OpenScale quality monitor generates a set of metrics to evaluate the quality of your model. You can use these quality metrics to determine how well your model predicts outcomes. When the evaluation that uses the holdout data completes, follow these steps to observe the model quality or accuracy:

1. In the left navigation panel, click the Insights dashboard icon.

2. Locate the Mortgage Approval Model Deployment tile. Notice that the deployment has 0 issues, and that both Quality and Fairness tests passed, meaning that the model met the thresholds that are required of it.

3. Click the Mortgage Approval Model Deployment tile to see more detail.

4. In the Quality section, click the Configure icon. Here you can see that the quality threshold that is configured for this monitor is 70% and that the measurement of quality being used is area under the ROC curve.

5. Click Go to model summary to return to the model details screen.

6. In the Quality section, click the right arrow icon to see the model quality detailed results. Here you see a number of quality metric calculations and a confusion matrix showing correct model decisions along with false positives and false negatives. The calculated area under the ROC curve is 0.9 or higher, which exceeds the 0.7 threshold, so the model is meeting its quality requirement.

7. Click Mortgage Approval Model Deployment in the navigation trail to return to the model details screen.

Task 4: Observe the model monitors for fairness

The Watson OpenScale fairness monitor generates a set of metrics to evaluate the fairness of your model. You can use the fairness metrics to determine if your model produces biased outcomes. Follow these steps to observe the model fairness:

1. In the Fairness section, click the Configure icon. Here you see that the model is being reviewed to ensure that applicants are being treated fairly regardless of their gender. Women are identified as the monitored group for whom fairness is being measured and the threshold for fairness is to be at least 80%. The fairness monitor uses the disparate impact method to determine fairness. Disparate impact compares the percentage of favorable outcomes for a monitored group to the percentage of favorable outcomes for a reference group.

2. Click Go to model summary to return to the model details screen.

3. In the Fairness section, click the right arrow icon to see the model fairness detailed results. Here you see the percentage of male and female applicants who are being automatically approved, along with a fairness score of over 100%, so the model performance far exceeds the 80% fairness threshold required.

4. Note the identified data sets. To ensure that the fairness metrics are most accurate, Watson OpenScale uses perturbation to determine the results where only the protected attributes and related model inputs are changed while other features remain the same. The perturbation changes the values of the feature from the reference group to the monitored group, or vice-versa. These additional guardrails are used to calculate fairness when the "balanced" data set is used, but you can also view the fairness results using only payload or model training data. Since the model is behaving fairly, you don't need to go into additional detail for this metric.

   

5. Click the Mortgage Approval Model Deployment navigation trail to return to the model details screen.

Task 5: Observe the model monitors for explainability

It is also important to understand how the model came to its decision. This understanding is required both to explain decisions to people involved in the loan approval and to ensure model owners that the decisions are valid. To understand these decisions, follow these steps to observe the model explainability:

1. In the left navigation panel, click the Explain a transaction icon.

2. Select Mortgage Approval Model Deployment to see a list of transactions.

3. For any transaction, click Explain under the Actions column. Here you see the detailed explanation of this decision. You will see the most important inputs to the model along with how important each was to the end result. Blue bars represent inputs that tended to support the model's decision while red bars show inputs that might have led to another decision. For example, an applicant might have enough income to otherwise be approved but their poor credit history and high debt together lead the model to reject the application. Review this explanation to become satisfied about the basis for the model decision.

4. (Optional) If you want to delve further into how the model made its decision, click the Inspect tab. Use the Inspect feature to analyze the decision to find areas of sensitivity where a small changes to a few inputs would result in a different decision, and you can test the sensitivity yourself by overriding some of the actual inputs with alternatives to see whether these would impact the result.

Task 6: Promote the model to pre-production and approve the model

Follow these steps to change the status of the model use case in the model inventory and approve the model:

Task 7: Share the model

You can generate a report from a factsheet or model use case in PDF, HTML, and DOCX format so you can share or print the details about a model being tracked in a model inventory.

1. In OpenPages, switch to the model use case.

2. Click the link in the Third Party Link field to open the use case in the model inventory.

3. From the Asset tab in the model use case, click Export report.

4. For the Format options, choose a format.

5. For the Report template, select a template:

   • MOdel use case full report: contains all data from the Basic report and details about the models and deployments in the model use case.

   • MOdel use case basic report: contains the set of facts visible on the Overview and Assets tabs.

6. Click Export. The report displays in a new window.

Cleanup (Optional)

If you would like to retake the tutorials in the AI governance use case, delete the following artifacts.

Next steps

• Try the Orchestrate an AI pipeline with model monitoring tutorial.

• Sign up for another Data fabric use case.

• Try more building, deploying, and trusting models tutorials.

• Build a model using AutoAI.

• Quick start tutorials

Learn more

• Notebooks

• Train and deploy machine learning models in notebooks

• Watson OpenScale

• Model inventory

Parent topic: Data fabric tutorials