AI governance tutorial: Build and deploy a model

Preview the tutorial

Watch this video to preview the steps in this tutorial. There might be slight differences in the user interface shown in the video. The video is intended to be a companion to the written tutorial.

This video provides a visual method as an alternative to following the written steps in this documentation.

Prerequisites

The following prerequisites are required to complete this tutorial.

Follow these steps to verify your roles and permissions. If your Cloud Pak for Data account does not meet all of the prerequisites, contact your administrator.

1. Click your profile image in the toolbar.

2. Click Profile and settings.

3. Select the Roles tab.

The permissions that are associated with your role (or roles) are listed in the Enabled permissions column. If you are a member of any user groups, you inherit the roles that are assigned to that group. These roles are also displayed on the Roles tab, and the group from which you inherit the role is specified in the User groups column. If the User groups column shows a dash, that means the role is assigned directly to you.

Task 1: Set up tracking for your model

You track models by adding model use cases to a catalog. You can create a new catalog, if you have access to create catalogs, or you can use the Default Catalog.

Task 2: Configure IBM OpenPages Model Risk Governance

This tutorial uses a model risk governance sample. Follow these steps to download the sample files and configure IBM OpenPages Model Risk Governance:

1. Download the following files:

   • MRG-CP4D-trial-contents-op-config.xml

   • MRG-Users-op-config.xml

   • Golden_Bank_trial_content.xlsx

2. In OpenPages, click the settings icon to select System Migration > Import Configuration > add file from the zip. Select MRG-Users-op-config.xml.

3. Import the Golden_Bank_trial_content.xlsx file from the zip using FastMap Import. Click the settings icon to select FastMap Import.

4. Click the settings icon to select System Migration > Import Configuration > add file from the zip. Select MRG-CP4D-trial-contents-op-config.xml.

Task 3: Create the model use case in the model inventory

For this type of project, it is best to create the model use case when a project commences. A model use case can reference multiple machine learning models that you can use to solve business problems. Then, data engineers and model evaluators can add models to the model use case and track the model as it progresses through its lifecycle.

Task 4: Run the notebook to create the model

Now you are ready to run the first notebook included in the sample project. The notebook includes the code to:

• Set up AI Factsheets used to track the lifecycle of the model.
• Load the training data, which is stored in the Db2 Warehouse connection in the sample project.
• Specify the target, categorical, and numerical columns along with the thresholds used to build the model.
• Build data pipelines.
• Build machine learning models.
• View the model results.
• Save the model.

Follow these steps to run the notebook included in the sample project. Take some time to read through the comments in the notebook, which explain the code in each cell.

1. From the Cloud Pak for Data navigation menu , choose Projects > All projects.

2. Click the AI governance project name.

3. Click the Assets tab, and then navigate to Notebooks.
   

4. Click the Overflow menu for the 1-model-training-with-factsheets notebook, and choose Edit.

5. Click Cell > Run All to run all of the cells in the notebook. Alternatively, you can run the notebook cell by cell if you want to explore each cell and its output.

6. The first cell requires your input.

   1. At the Enter host name prompt, type your Cloud Pak for Data hostname beginning with https://, and press Enter. For example, https://mycpdcluster.mycompany.com.

   2. At the Username prompt, type your Cloud Pak for Data username, and press Enter.

   3. At the Password prompt, type your Cloud Pak for Data password, and press Enter.

7. The notebook takes 1 - 3 minutes to complete. You can monitor the progress cell by cell, noticing the asterisk "In [*]" changing to a number, for example, "In [1]".

8. If you encounter any errors during the notebook run, try these tips:

   • Click Kernel > Restart & Clear Output to restart the kernel, and then run the notebook again.
   • Verify that you created the model use case by copying and pasting the specified artifact name exactly with no leading or trailing spaces.

Task 5: View the model's factsheet and associate it with a model use case

After running all the cells in notebook, follow these steps to view the model's factsheet in the project and then associate that model with a model use case in the model inventory:

1. Click the AI governance project name in the navigation trail.
   

2. Click the Assets tab, and then navigate to Models.

3. Click the Mortgage Approval Prediction Model asset that was created by the notebook.

4. Review the AI Factsheet for your model. AI Factsheets capture model metadata across the model development lifecycle, facilitating subsequent enterprise validation or external regulation. AI Factsheets enables model validators and approvers to get an accurate, always up-to-date view of the model lifecycle details.
   In the last task, you ran a notebook containing the AI Factsheets Python client code in the notebook that captured training metadata. Scroll to the Training metrics and Training tags sections to review the captured training metadata.
   The following image shows the AI Factsheet for the model:
   
   

5. Scroll up on the model page, and click Track this model.

   1. From the list of model use cases, select Mortgage Approval Model Use Case.

   2. Select Create a new record.

   3. Click Track.

6. Back on the model page, click Open in model inventory.

7. On the model use case page, click the Asset tab.

8. Review the Model tracking. AI Factsheets track models through their lifecycle. This model is still in the Develop stage as it has not been deployed yet.

Task 6: Deploy the model

Before you can deploy the model, you need to promote the model to a new deployment space. Deployment spaces help you to organize supporting resources such as input data and environments; deploy models or functions to generate predictions or solutions; and view or edit deployment details.

Task 7: Perform model risk assessment

The Model Owner can leverage captured metadata to consider carefully when approving this model as the implications can be large for the end customers and Golden Bank’s reputation. When the data scientist created a test deployment, all of the training time facts were captured. Now you perform a model risk assessment to determine if this model should move to a pre-production environment, and then eventually a production environment.

1. On the model tracking page, click the Mortgage Approval Model Use Case link to open the same use case in IBM OpenPages Model Risk Governance.

2. Under Associated Models, click the Mortgage Approval Prediction Model name to view the model.

3. Retrieve the training accuracy score:

   1. Scroll down to the Associations section, and click the Metrics node in the tree.

   2. In the side panel, click the training_accuracy_score metric.

      

   3. Copy the value for the training accuracy score for the model. You need this value for the risk assessment.

      Tip: If you have trouble locating the training_accuracy_score metric, click the Metrics tab to locate the metric.

4. Return to the model tab to create the model risk assessment:

   1. In the Model Risk Assessment section, click New.

   2. For the Description, type Initial risk assessment for Mortgage Approval model.

   3. Answer the following questions to assess the risk of the model.

      • Model uses other models outputs or feeds downstream models: Choose No since this model makes a decision on the loan application and does not depend on any other models.

      • Model Training Accuracy score: Paste the value for the training accuracy score metric that you copied earlier.

      • Is this model used in granting loans or mortgages?: Choose Yes since this is a mortgage approval model.

      • Is there information on protected groups in the training data?: Choose No since this model is trained with mortgage data that has been anonymized.

   4. After you provide the information for all of the required fields, click Save.

5. When the risk assessment is complete, review the Computed tier. The risk assessment generates a computed risk tier for the model (Tier 3 for low risk, Tier 2 for medium risk, and Tier 1 for high risk). Note that you can override the computed tier if that’s appropriate.
   

6. Edit the Status field:

   1. Click the edit icon next to the Status field.

   2. Choose Confirmed to confirm the assessment.

   3. Click Save.

7. Return to the model page to assign an owner and submit the model for pre implementation review.

   1. Click the edit icon next to the Model Owner field.

   2. Select your user name from the list.

   3. Click Save.

   4. Click Action > Submit for Pre Implementation Review, and then click Continue to send the model for validation.

Next steps

You are now ready to validate and monitor your deployed machine learning model to ensure it is working accurately and fairly. For this task, you will use Watson OpenScale. See the Test and validate the model tutorial.

Learn more

• Notebooks

• Train and deploy machine learning models in notebooks

• Watson OpenScale

• Model inventory

Parent topic: Data fabric tutorials