Quick start: Evaluate a deployment in spaces

To preview this task, watch the video beginning at 00:06.

You need a project to store the assets used to build the model. For example, the training data, the AutoAI experiement or Jupyter notebook, and the saved model. Follow these steps to create a project based on a sample:

1. Access the Evaluate an ML model sample project in the Resource hub.

2. Click Create project.

3. If prompted to associate the project to a Cloud Object Storage instance, select a Cloud Object Storage instance from the list.

4. Click Create.

5. Wait for the project import to complete, and then click View new project to verify that the project and assets were created successfully.

6. Click the Assets tab to view the assets in the sample project.

7. View the Mortgage Approval Prediction Model. This model predicts if a mortgage applicant should be approved based on several factors, such as their credit history, total debt, income, loan amount, and employment history.

8. Return to the project's Assets tab.

Before you can deploy the model, you need to promote the model to a new deployment space. Deployment spaces help you to organize supporting resources such as input data and environments; deploy models or functions to generate predictions or solutions; and view or edit deployment details. The next task is to promote the evaluation data and the model to a deployment space, and then create an online deployment.

To preview this task, watch the video beginning at 01:19.

Follow these steps to configure evaluations in this deployment space:

To preview this task, watch the video beginning at 06:22.

Follow these steps to use holdout data to evaluate the model:

1. From the Actions menu, select Evaluate now.

2. From the list of import options, select from CSV file.

3. Drag the Golden Bank_HoldoutData.csv data file you downloaded from the project into the side panel.

4. Click Upload and evaluate and wait for the evaluation to complete.

5. When the evaluation completes, you will see the number of tests run, tests passed, and tests failed. Scroll to see the results for Fairness, Quality, Drift, and Global explanation.

To preview this task, watch the video beginning at 06:36.

The Watson OpenScale quality monitor generates a set of metrics to evaluate the quality of your model. You can use these quality metrics to determine how well your model predicts outcomes. When the evaluation that uses the holdout data completes, follow these steps to observe the model quality or accuracy:

1. In the Quality section, click the Configure icon . Here you can see that the quality threshold that is configured for this monitor is 70% and that the measurement of quality being used is area under the ROC curve.

2. Click the X to close the window and return to the model evalution screen.

3. In the Quality section, click the Details icon to see the model quality detailed results. Here you see a number of quality metric calculations and a confusion matrix showing correct model decisions along with false positives and false negatives. The calculated area under the ROC curve is 0.9 or higher, which exceeds the 0.7 threshold, so the model is meeting its quality requirement.

4. Click Back to summary to return to the model details screen.

To preview this task, watch the video beginning at 07:20.

The Watson OpenScale fairness monitor generates a set of metrics to evaluate the fairness of your model. You can use the fairness metrics to determine if your model produces biased outcomes. Follow these steps to observe the model fairness:

1. In the Fairness section, click the Configure icon . Here you see that the model is being reviewed to ensure that applicants are being treated fairly regardless of their gender. Women are identified as the monitored group for whom fairness is being measured and the threshold for fairness is to be at least 80%. The fairness monitor uses the disparate impact method to determine fairness. Disparate impact compares the percentage of favorable outcomes for a monitored group to the percentage of favorable outcomes for a reference group.

2. Click the X to close the window and return to the model evalution screen.

3. In the Fairness section, click the Details icon to see the model fairness detailed results. Here you see the percentage of male and female applicants who are being automatically approved, along with a fairness score of over 100%, so the model performance far exceeds the 80% fairness threshold required.

4. Note the identified data sets. To ensure that the fairness metrics are most accurate, Watson OpenScale uses perturbation to determine the results where only the protected attributes and related model inputs are changed while other features remain the same. The perturbation changes the values of the feature from the reference group to the monitored group, or vice-versa. These additional guardrails are used to calculate fairness when the "balanced" data set is used, but you can also view the fairness results using only payload or model training data. Since the model is behaving fairly, you don't need to go into additional detail for this metric.

   

5. Click Back to summary to return to the model details screen.

To preview this task, watch the video beginning at 08:25.

The Watson OpenScale drift monitor measures changes in your data over time to ensure consistent outcomes for your model. Use drift evaluations to identify changes in your model output, the accuracy of your predictions, and the distribution of your input data. Follow these steps to observe the model drift:

1. In the Drift section, click the Configure icon . Here you see the drift thresholds. Output drift measures the change in the model confidence distribution. Model quality drift measures the drop in accuracy by comparing the estimated runtime accuracy to the training accuracy. Feature drift measures the change in value distribution for important features. The configuration also shows the number of selected features and the most important features.

2. Click the X to close the window and return to the model evalution screen.

3. In the Drift section, click the Details icon to see the model drift detailed results. You can view the history of how each metric score changes over time with a time series chart. Lower values are better, so in this case, the results are above the upper thresholds that are set in the configuration. Then view details about how the scores output and feature drifts are calculated. You can also view details about each feature to understand how they contribute to the scores that Watson OpenScale generates.

4. Click Back to summary to return to the model details screen.

To preview this task, watch the video beginning at 09:10.

It is also important to understand how the model came to its decision. This understanding is required both to explain decisions to people involved in the loan approval and to ensure model owners that the decisions are valid. To understand these decisions, follow these steps to observe the model explainability:

1. Click the Transactions tab.

2. On the graph, select a timeframe to see a list of transactions during that period.

3. For any transaction, click Explain under the Actions column. Here you see the detailed explanation of this decision. You will see the most important inputs to the model along with how important each was to the end result. Blue bars represent inputs that tended to support the model's decision while red bars show inputs that might have led to another decision. For example, an applicant might have enough income to otherwise be approved but their poor credit history and high debt together lead the model to reject the application. Review this explanation to become satisfied about the basis for the model decision.

4. Optional: If you want to delve further into how the model made its decision, click the Inspect tab. Use the Inspect feature to analyze the decision to find areas of sensitivity where a small changes to a few inputs would result in a different decision, and you can test the sensitivity yourself by overriding some of the actual inputs with alternatives to see whether these would impact the result. Click Run analysis to have Watson OpenScale reveal the minimum changes required to deliver a different outcome. Select Analyze controllable features only to limit changes to the controllable features only.