Customizing RAG experiment settings

Optimization metric

Choose the metric to maximize when searching for the optimal RAG patterns. For more information about optimization metrics and their implementation details, see RAG metrics.

• Answer faithfulness measures how closely the generated response aligns to the context retrieved from the vector store. The score is calculated using a lexical metric which counts how many of the generated response tokens are included in the context retrieved from the vector store. A high score indicates the response represents the retrieved context well. Note that a high faithfulness score does not necessarily indicate correctness of the response. For more information on how the metric is implemented, see Faithfulness.
• Answer correctness measures the correctness of the generated answer compared to the correct answer provided in the benchmark files. This includes the relevance of the retrieved context and the quality of the generated response. The score is calculated using a lexical metric which counts how many of the ground-truth response tokens are included in the generated response. For more information on how the metric is implemented, see Correctness.
• Context correctness indicates to what extent the context retrieved from the vector store aligns with the ground truth context provided in the benchmark. The score is calcuated based on the rank of the ground truth context among the retrieved chunks. The closer the ground truth context is to the top of the list, the higher the score. For more information on how the metric is implemented, see Context correctness.

Retrieval methods

You can select configurations automatically for retrieving relevant data or edit the configuration settings. Retrieval methods differ in the ways that they filter and rank documents.

• Choose the window retrieval method or the simple retrieval method.

  • Window retrieval method surrounds retrieved chunks with additional chunks before and after the chunks, based on what was in the original document. This method is useful for including more context that might be missing in the originally retrieved chunk. Window retrieval works as follows:
    • Search: Finds the most relevant document chunks in the vector store.
    • Expand: For each found chunk, retrieves surrounding chunks to provide context.
    • Each chunk stores its sequence number in its metadata.
    • After retrieving a chunk, the chunk metadata is used to fetch neighboring chunks from the same document. For example, if window_size is 2, it adds 2 chunks before it and 2 chunks after it.
    • Merge: Combines overlapping text within the window to remove redundancy.
    • Metadata handling: Merges metadata dictionaries by keeping the same keys and grouping values into lists.
    • Return: Outputs the merged window as a new chunk, replacing the original one.
  • Simple retrieval method finds the most relevant chunks in the vector store.

• Select the number of chunks from 1 to 10. The number of retrieval chunks determines the number of smaller chunks into which a retrieved text passage is split.

• If you select the window retrieval method, you can set the window size from 1 to 4. The window size is the number of adjacent chunks considered by the model when retrieving information from the indexed documents.

• You can optionally select a hybrid strategy to improve the output quality. A hybrid strategy combines sparse and dense embeddings vectors to perform a similarity search in the vector database. Sparse embeddings prioritize exact keyword matches, and dense embeddings prioritize outputs that have semantic similarity. Combining sparse and dense embeddings improves search accuracy and relevance, resulting in more comprehensive information retrieval from the database. This setting is not available for the in-memory Chroma vector database. If you're using the Elasticsearch vector database, you need to install the ELSER model.

  Choose from one of these hybrid strategy options:

  • RRF (Reciprocal Rank Fusion): Combines rankings from multiple sources into a single, more relevant list. To use RRF with the Elasticsearch vector database, you need Elasticsearch version 8.8 or later.
  • Weighted: Assigns importance to outputs and prioritizes the most reliable one for the final output.
  • None: Uses only dense embeddings, without a hybrid strategy.

Foundation models to include

You can choose between using provided foundation models or custom foundation models.

By default, all available provided foundation models that support AutoAI for RAG are selected for experimentation. You can manually edit the list of provided foundation models that AutoAI can consider for generating RAG patterns. For each model, you can click Model details to view or export details about the model.

For the list of available provided foundation models along with descriptions, see Foundation models by task.

To use custom foundation models, click Custom models and select the models you want AutoAI to consider for generating RAG patterns. The list of custom models includes deploy on demand models and the custom models that are deployed in the project where you are running the experiment and all spaces where you are a member.

To add a new custom foundation model, see Deploying custom foundation models.

For information on how to code an experiment with a custom foundation model, see Coding an AutoAI RAG experiment with a custom foundation model.

Max RAG patterns to complete

You can specify the number of RAG patterns to complete in the experimentation phase, up to a maximum of 20. A higher number compares more patterns, and might result in higher-scoring patterns, but consumes more compute resources.

Match input language

By default, AutoAI automatically detects the language used in prompts and instructs models to respond in the same language. Models that do not support the input language are given lower priority in the search for the RAG pattern. Turn off this setting to consider all available models and generate responses in English only.

Chunking

You can select configurations automatically for chunking your data or edit the configuration settings. Chunking settings determine how indexed documents are broken down into smaller pieces before ingestion into a vector store. Chunking data allows search and retrieval of those chunks in a document most relevant to a query. This allows the generation model to process only the most relevant data.

AutoAI RAG uses LangChain’s recursive text splitter to break down the documents into chunks. This method has the effect of decomposing the document in a hierarchical fashion, trying to keep all paragraphs (and then sentences, and then words) together as long as possible, until the chunk is smaller than the requested chunk size. For more information about the recursive chunking method, see Retrieval recursively split by character in the Langchain documentation.

How to best chunk your data depends on your use case. Smaller chunks provide a more granular interaction with text, enabling more focused search for relevant content, whereas larger chunks can provide more context. For your chunking use case, specify one or more options for:

• The number of characters to include in each chunk of data.
• The number of characters to overlap for chunking data. The number must be smaller than the chunking size.

The selected options are explored and compared in the experimentation phase.

Embedding models

Embedding models are used in retrieval-augmented generation solutions for encoding chunks and queries as vectors to capture their semantic meaning. The vectorized input data chunks are ingested into a vector store. Given a query, the vectorized representation is used to search the vector store for relevant chunks.

For a list of embedding models available for use with AutoAI RAG experiments, see Supported encoder models available with watsonx.ai.