About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Last updated: Mar 07, 2025
The ibm-watsonx-gov
Use the ibm-watsonx-gov
Modules
The Python SDK supports the following modules that can help you automate tasks for model evaluations and generate insights:
Metrics
The Python SDK supports metrics that help you evaluate traditional machine learning model evaluations and prompt template evaluations for generative AI assets. For more information, see Evaluation metrics.
The following metrics are currently available only with the Python SDK:
| Metric | Description |
|---|---|
| Adversarial robustness | Measures the robustness of your model and prompt template against adversarial attacks such as prompt injections and jailbreaks |
| Keyword inclusion | Measures the similarity of nouns and pronouns between the foundation model output and the reference or ground truth |
| Prompt leakage risk | Measures the risk of leaking the prompt template by calculating the similarity between the leaked prompt template and original prompt template |
| Question robustness | Detects the English-language spelling errors in the model input questions |
The following metric category is also available only with the Python SDK:
Content validation metrics
Content validation metrics use string-based functions to analyze and validate generated LLM output text. The input must contain a list of generated text from your LLM to generate content validation metrics.
If the input does not contain transaction records, the metrics measure the ratio of successful content validations and compares the ratio to the total number of validations. If the input contains transaction records, the metrics measure the
ratio of successful content validations when compared to the total number of validations and calculate validation results with the specified record_id
You can calculate the following content validation metrics:
| Metric | Description |
|---|---|
| Contains all | Measures whether the rows in the prediction contain all of the specified keywords |
| Contains any | Measures whether the rows in the prediction contain any of the specified keywords |
| Contains email | Measures whether each row in the prediction contains emails |
| Contains_JSON | Measures if the rows in the prediction contains JSON syntax |
| Contains link | Measures whether the rows in the prediction contain any links |
| Contains none | Measues whether the rows in the prediction do not contain any of the specified keywords |
| Contains string | Measures whether each row in the prediction contains the specified string |
| Contains valid link | Measures whether the rows in the prediction contain valid links |
| Ends with | Measures whether the rows in the prediction end with the specified substring |
| Equals to | Measures whether the rows in the prediction are equal to the specified substring |
| Fuzzy match | Measures if the prediction fuzzy matches the keyword |
| Is email | Measures whether the rows in the prediction contain valid emails |
| Is JSON | Measures whether the rows in the prediction contains valid JSON syntax |
| Length greater than | Measures whether the length of each row in the prediction is greater than a specified maximum value |
| Length less than | Measures whether the length of each row in the prediction is less than a specified maximum value |
| No invalid links | Measures whether the rows in the prediction have no invalid links |
| Regex | Measures whether the rows in the prediction contain the specified regex expression |
| Starts with | Measures whether the rows in the prediction start with the specified substring |
Was the topic helpful?
0/1000