CRAG (Corrective Retrieval Augmented Generation) is a RAG technique proposed around February 2024 that offers the advantage of reducing hallucinations compared to traditional RAG (Retrieval Augmented Generation).
The key characteristic of CRAG is the use of a Retrieval Evaluator to assess the relevance between retrieved documents and the user’s question.
This retrieval evaluation classifies documents into three categories: “Correct (highly relevant),” “Incorrect (low relevance),” and “Ambiguous (difficult to determine),” and takes the following actions for each:
Correct (highly relevant): Generate responses using the documents
Incorrect (low relevance): Generate responses from web searches without using the retrieved documents
Ambiguous (difficult to determine): Generate responses using both the retrieved documents and web search results
Building CRAG in Copilot Studio
Here’s the flow we’ll be implementing. We’ll use GPT4o for search evaluation and SerpAPI for web searches.
Implementation
Since our goal is simply to demonstrate how to build CRAG in Copilot Studio, we won’t focus on optimizing accuracy (such as refining prompts).
SharePoint Search
First, we’ll search for documents from SharePoint.
We’ll use a prompt action to convert the user’s question into search terms.
The conversion prompt is reused from the previous example.
*Note: “Excluded Keywords” remains here only because we reused the prompt from the previous (Self-RAG) implementation. It’s not actually necessary for this purpose.
Here are examples of user questions and their corresponding search queries. Use these examples to guide your transformation of the input question.
Example Input 1: "How to connect to a database with Python?"
Example Output: "Python database connection method"
Example Input 2: "What impact does climate change have on ecosystems?"
Example Output: "Climate change ecosystem impact"
Additionally, if certain words are specified as "excluded keywords," ensure that these words are NOT included in the generated search query.
For example:
Excluded Keywords: ["Python connection"]
Example Input: "How to connect to a database with Python?"
Example Output: "Database connection method"
Now, based on these examples and the excluded keywords, transform the following user question into an effective search query, avoiding the specified excluded keywords.
Input: {input}
Excluded Keywords: {excluded_keywords}
Using the created search terms, we’ll search for SharePoint documents from AI Search.
For information on how to set up AI Search, see here.
Next, we’ll score the relevance between the searched documents and the user’s question.
First, execute a Foreach loop for all documents in the search results,
Within the loop, use a prompt action to determine the relevance score (0.0 to 1.0) for each document.
Here’s what the prompt looks like.
# Task
Evaluate relevance between user question and retrieved document on a 0.0-1.0 scale
# Evaluation Criteria
## 1.0 - Document provides complete & direct answer
(Example: Contains specific numbers/dates/names matching query)
## 0.7-0.9 - Directly relevant but requires:
- Context synthesis OR
- Partial information extraction OR
- Terminology clarification
## 0.4-0.6 - Partial relevance through:
- Shared domain knowledge
- Indirect supporting evidence
- Related concepts without direct answer
## 0.1-0.3 - Barely relevant with only:
- Common keywords
- Generic domain overlap
- Indirect conceptual connections
## 0.0 - No semantic/contextual connection
# Output Format
- Relevance Score: [Strictly 0.0-1.0 numeric value only]
# Input Data
Query: {question}
Retrieved Document:
{document}
Finally, create an object array “Documents” with a new column “Score” added to the searched documents.
Checking Relevance Scores
Once the relevance score evaluation is complete, get the maximum score and determine whether to perform a web search or generate an answer directly.
Use the Max function to get the maximum value from the “score” column in the Documents we created earlier,
And use a condition to branch subsequent processing.
Case 1: Maximum Score is 0.8 or Higher →Generate Response from Highly Relevant Documents
If the maximum score is 0.8 or higher, extract only the highly relevant documents (those with scores of 0.8 or higher) and generate the response.
From the Documents object array, use the Filter function to extract documents with scores of 0.8 or higher, and assign them to the variable “k_in”,
Then use this “k_in” as an argument when generating the response.
Case 2: Maximum Score is Below 0.8 →Generate Response from Web Search and Moderately Relevant Documents
If the maximum score is less than 0.8, perform either “Ambiguous” (documents with scores of 0.4 or higher + Web search) or “Incorrect” (Web search only) processing.
First, extract documents with scores of 0.4 or higher and assign them to the variable “k_in”.
*Note: In the “Incorrect” case (Web search only), k_in will be empty.
Next, perform a Web search (SerpAPI call) via Power Automate.
SerpAPI is an API that can perform Web searches. In this case, we’ll use “related_question” and “organic_results” from the obtained information as RAG targets.
*Note: Scraping the sites obtained from web searches would increase accuracy, but we’ll skip that step here.
*Note: Ideally, we would regenerate search terms specifically for web searches, but we’ll skip that as well.
Use both k_in and the web search results as arguments for response generation.
Answer Generation
At the end, use a prompt action to run RAG with GPT-4o, using the information obtained so far as arguments.
If the query cannot be answered based on the provided context, respond with 'The provided information is not sufficient to answer this question.'
Otherwise, generate a complete and accurate answer, and as an annotation, output the name of the referenced files at the end.
# query
{query}
# information
{information}
Testing the Implementation
First, let’s try asking a question that even simple RAG (Naive RAG) could answer,
A document with a relevance score of 1 (maximum) is found,
And an answer is generated from this document.
Next, let’s try asking a question that simple RAG couldn’t answer,
Although the maximum score is 0, it still generates a correct answer.
This is because the web search results contain the correct information, demonstrating the benefit of CRAG in reducing hallucinations.
So we’ve confirmed that CRAG successfully improves accuracy.
Since this solution relies on “web search,” it may not improve response accuracy for RAG systems searching internal documents, but it’s worth remembering this approach as it might be useful in certain scenarios.
コメント