[Question]: How to effectively use agentworkflow to build a specialized corrective rag

[Akash-Kumar-Sen]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hello everyone,
Recently I went through this documentation for Agentworkflow - https://docs.llamaindex.ai/en/stable/examples/agent/agent_workflow_multi/

And have been trying to build a multi agent corrective RAG workflow using that,

    # Set up the retrieval agent
    retrieval_agent = FunctionAgentExt(
        name="RetrievalAgent",
        description="Useful for retrieving relevant documents and extracting key information.",
        system_prompt=(
            "You are an AI assistant responsible for searching the knowledge base for information relevant to the user's query. "
            "Your job is to retrieve the most relevant documents that might help answer the query and extract key information. "
            "IMPORTANT: NEVER communicate directly with the user. Track your process internally and summarize findings. "
            "Always hand off to ResponseSynthesisAgent when ready to present information to the user.\n\n"
            "For effective retrieval:\n"
            "1. For direct questions about document content, retrieve relevant information\n"
            "2. Hand off to ToolCallAgent for questions related to metadata or requiring metadata comparison\n"
            "3. Hand off to ToolCallAgent for any questions requiring statistical analysis or complex calculations\n"
            "4. NEVER include meta-commentary or function calls in user-facing responses"
        ),
        tools=[retrieve],
        llm=llm,
        can_handoff_to=["ToolCallAgent", "ResponseSynthesisAgent"],
    )

    tool_call_agent = FunctionAgentExt(
        name="ToolCallAgent",
        description="Useful for calling tools to retrieve information.",
        system_prompt=(
            "You are an advanced AI assistant specializing in using various tools to solve complex queries. "
            "Your expertise includes handling metadata analysis, document statistics, and complex comparisons. "
            "IMPORTANT: NEVER communicate directly with the user. Track your findings internally. "
            "Always hand off to next agent when ready to present information to the user.\n\n"
            "Guidelines:\n"
            "1. For metadata-related queries, use appropriate tools to extract and analyze document properties\n"
            "2. For statistical queries, use tools to calculate document metrics and generate summaries\n"
            "3. NEVER include function names or meta-commentary in user-facing responses\n"
            "4. If you are satisfied with the results of a tool call, directly provide the results"
        ),
        tools=tools,
        llm=llm,
        can_handoff_to=["ResponseSynthesisAgent"],
    )

    # Set up the response synthesis agent
    response_synthesis_agent = FunctionAgentExt(
        name="ResponseSynthesisAgent",
        description="Useful for synthesizing comprehensive responses with proper citations.",
        system_prompt=(
            "You are an AI assistant responsible for synthesizing comprehensive responses with proper citations. "
            "Your job is to synthesize the response from the retrieved information or the results of tool calling agents and the analysis.\n\n"
            "Response formatting guidelines:\n"
            "1. Structure complex information hierarchically using Markdown formatting\n"
            "2. Use headers (##) for major sections and subheaders (###) for subsections\n"
            "3. Include summaries for complex responses\n"
            "4. Employ bullet points for lists and code blocks with language tags\n"
            "5. Format all mathematical expressions using LaTeX syntax\n"
            "6. For data or tabular information, use standardized CSV format:\n"
            "   ```csv\n"
            "   <Descriptive Title>\n"
            '   "column1","column2","column3"\n'
            '   "value1","value2","value3"\n'
            "   ```\n"
            "7. Cite sources when providing factual information\n"
            "8. For analysis requiring calculations or data interpretation:\n"
            "   - First use perform_analysis to create detailed analysis with calculations and insights\n"
            "   - Then use generate_response to create your final response\n"
            "9. Focus on accuracy over speculation and acknowledge limitations when present"
        ),
        tools=[generate_response, perform_analysis],
        llm=llm,
        can_handoff_to=[],
    )

    # Create and return the agent workflow
    return AgentWorkflowExt(
        agents=[retrieval_agent, tool_call_agent, response_synthesis_agent],
        root_agent="RetrievalAgent" if index else "ToolCallAgent",
        initial_state={
            "index": index,
            "metadata_filters": metadata_filters,
            "nodes": [],
            "document_notes": {},
            "response": "",
            "analysis": "",
        },
        timeout=kwargs.get("timeout", 120.0),
        verbose=kwargs.get("verbose", False),
        num_concurrent_runs=kwargs.get("num_concurrent_runs", 1),
    )

The code responsible for streaming -

async for event in handler.stream_events():
    if not stream_started:
        # Start the stream with an empty message
        stream_started = True
        yield self.convert_text("")

    # Handle different types of events
    if isinstance(event, (AgentStream, StopEvent)):
        async for chunk in self._stream_text(event):
            await self.handler.accumulate_text(chunk)
            yield self.convert_text(chunk)
    elif isinstance(event, dict):
        yield self.convert_data(event)
    elif hasattr(event, "to_response"):
        event_response = event.to_response()
        yield self.convert_data(event_response)

Now there are few questions I have been having,

1. The response very often includes meta commentary about the agents like this - I will hand off to the ResponseSynthesisAgent to respond to the user

2. What is the use of these three arguments, can we modify the description and add more documentation for all these arguments and their necessary usage -

handoff_prompt: Optional[Union[str, BasePromptTemplate]] = None,
handoff_output_prompt: Optional[Union[str, BasePromptTemplate]] = None,
state_prompt: Optional[Union[str, BasePromptTemplate]] = None,

3. Is there any example implementation of RAG using AgentWorkflow, checked this concierge and deep research workflow example:

• https://github.com/run-llama/multi-agent-concierge/blob/main/workflow.py
• https://github.com/run-llama/create-llama/blob/main/templates/components/agents/python/deep_research/app/workflows/deep_research.py

Both seems to be using the Workflow class instead of Agentworkflow.

[dosubot]

Hello @Akash-Kumar-Sen! I'm Dosu, a friendly bot here to help you solve bugs, answer your questions, and guide you in becoming a contributor. I'm ready to assist while we wait for a human maintainer. How can I help you today?

To address your questions:

1. Meta Commentary in Responses: The issue of meta commentary, such as "I will hand off to the ResponseSynthesisAgent," likely arises from the internal prompts of the agents. To eliminate this, you should adjust the system prompts of each agent to avoid including such commentary. Ensure that the prompts focus on the task at hand without mentioning the internal workings or transitions between agents.

2. Use of handoff_prompt, handoff_output_prompt, and state_prompt:

   • handoff_prompt: This is used when transitioning control from one agent to another. It should include the {agent_info} placeholder to provide necessary information about the agent involved in the handoff.
   • handoff_output_prompt: This generates output during an agent handoff and must include {to_agent} and {reason} placeholders to specify the target agent and the reason for the handoff.
   • state_prompt: This describes the current state of the workflow and should contain {state} and {msg} placeholders to represent the current state and any associated message. These prompts help manage agent transitions and maintain clear communication about the workflow's status ^[1].

3. Examples of RAG Implementations Using AgentWorkflow: It seems that the examples you found use the Workflow class instead of AgentWorkflow. Unfortunately, I couldn't find specific examples of RAG implementations using AgentWorkflow in the available documentation. You might need to explore further or adapt the existing examples to fit the AgentWorkflow class.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[logan-markewich]

Corrective RAG is a very specific workflow -- why not implement it explicitly ?
https://docs.llamaindex.ai/en/stable/examples/workflow/corrective_rag_pack/

Generally its easier to implement complex flows from scratch like the workflow examples you use

Either that, or use query engines or other workflows as tools for your agent

As to receiving text you dont want in the output, definitely modifying some of those params would help. Dosu explained these params well

[dosubot]

Hi, @Akash-Kumar-Sen. I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You sought guidance on setting up a specialized corrective RAG workflow using the Agentworkflow.
• I provided detailed advice on adjusting system prompts to eliminate unwanted meta commentary.
• Logan-markewich suggested implementing the corrective RAG workflow explicitly or using query engines as tools.
• The issue was resolved with actionable advice from both contributors on optimizing the multi-agent system.

Next Steps:

• Please confirm if this issue is still relevant to the latest version of the LlamaIndex repository. If so, you can keep the discussion open by commenting here.
• Otherwise, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!