When the professor asked us to decide on a project that will take the length of the semester and needs to be in the realm of machine learning, my first thought was this could be either be the easiest or the hardest project I will do during this semester. He gave us a list of projects we could choose from and said implement these ideas—not too big, not too small, but enough to cover a lot of ground.

So after some deliberation, I decided to go for a topic I have never explored. I have heard of benchmarking but never really dived deep into it; I never even ventured in that direction because I focused mainly on using the agents, not testing them. So I decided to go for Benchmarking Multi-Tool LLM Agents and add my own twist by adding multiple configurations to see how they perform under different settings.

Understanding the Landscape

Before writing any code, I spent some time researching how the industry currently evaluates AI. The most common benchmarks, like MMLU (Massive Multitask Language Understanding), focus heavily on knowledge—testing if a model can answer multiple-choice questions about chemistry, history, or math.

However, Agents are different. They aren't just knowledge bases; they are systems that use tools to perform actions. I looked into specialized frameworks like AgentBench and DeepEval, which are excellent, comprehensive tools for this purpose.

While these frameworks are powerful, for this project, I wanted to build the evaluation harness from scratch. My goal was to understand the internal mechanics of an agentic workflow—how the prompt is structured, how the tool output is parsed, and where the failures actually happen. I realized that evaluating an agent is less like grading a written exam and more like observing a process. I needed a custom setup that would allow me to track that specific behavior—tool selection, routing, and execution—in a controlled environment.

The Architecture

Auto Bench is in a deterministic sandbox environment. Here's how it works:

1. The Dataset (40 Curated Scenarios)

I focused on Public Health & Pandemics because:

It's fact-dense (requires precise retrieval)
It has verifiable ground truth
It demands multi-step reasoning

Each task is stored as a structured JSON object:

{
  "id": 21,
  "task_text": "List the vaccine-development phases and create a quiz...",
  "category": "Vaccinology",
  "task_type": "Multi-Tool",
  "gold_end_state": {
    "summary": "Phase I: Safety testing...",
    "quiz": [{"question": "...", "answer": "Phase III"}]
  },
  "oracle_plan": [
    {"tool_name": "content_extractor", "arguments": {...}},
    {"tool_name": "quiz_generator", "arguments": {...}}
  ]
}

Why this matters: The oracle_plan defines the optimal path. If an agent takes 5 steps to do what should take 2, I know it's inefficient.

2. The Sandbox (Controlled Tool Environment)

Instead of live web search (which changes daily), I built a local knowledge base:

def web_search(query: str) -> str:
    """Simulated web search that queries local documents."""
    kb_path = "knowledge_base/"

    # Match query to relevant document
    if "WHO" in query.upper():
        return load_file(f"{kb_path}who_covid19.txt")
    elif "CDC" in query.upper():
        return load_file(f"{kb_path}cdc_pandemic_flu.txt")

    return "No relevant document found."

This guarantees reproducibility—every test run sees identical data.

3. Three Agent Configurations

I tested three levels of complexity:

| Agent Type | Tools Available | Tests | |------:|:------|:------:|:------:| | No-Tool | None (pure reasoning) | Can the model answer from memory? | | Single-Tool | Web search only | Can it retrieve and synthesize? | | Multi-Tool | Search + Summarizer + Quiz Gen | Can it route tasks correctly? |

The ReAct Prompting Strategy

I used the ReAct (Reason + Act) framework to make the agent's reasoning transparent:

REACT_PROMPT = """
You are an intelligent agent. For each task, follow this format:

Thought: [Analyze what you need to do]
Action: [tool_name]
Action Input: [arguments for the tool]
Observation: [tool output will appear here]
... (repeat Thought/Action/Observation as needed)
Final Answer: [your complete response]

Available Tools:
- web_search: Search the knowledge base
- content_extractor: Summarize documents
- quiz_generator: Create quiz questions
"""

This forces the model to "show its work"—every decision is logged.

The Development Process: 4 Critical Phases

Phase 1: Logic Validation (Stubbed Testing)

Before spending money on API calls, I tested with stubbed agents that return fake responses:

class StubbedAgent:
    def run(self, task):
        return {
            "thought": "I need to search for information",
            "action": "web_search",
            "action_input": "CDC pandemic guidelines"
        }

This proved my parser logic worked before involving real LLMs.

Phase 2: The Initial Live Tests

When I first ran live tests, Multi-Tool agents got 0% success. Both Gemini and GPT failed every single task.

I dug into the execution logs and found the issue:

{
  "thought": "I need to summarize the ACP paper",
  "action": "content_extractor",
  "error": "No text provided to extract from"
}

The diagnosis: The agents had tools but no context. They were being asked to summarize documents they'd never seen.

Phase 3: The RAG Solution

I implemented Retrieval-Augmented Generation to inject context automatically:

def retrieve_relevant_context(task):
    """Autonomous librarian that finds the right document."""
    keywords = {
        "Virology": "who_covid19.txt",
        "Vaccinology": "mdpi_vaccine_development.txt",
        "Policy": "cdc_pandemic_flu.txt"
    }

    # Scan task text for category
    for category, filename in keywords.items():
        if category.lower() in task["task_text"].lower():
            doc_path = f"knowledge_base/{filename}"
            with open(doc_path, 'r') as f:
                return f.read()

    return ""

# Inject into prompt
context = retrieve_relevant_context(task)
full_prompt = f"{REACT_PROMPT}\n\n=== SOURCE DOCUMENT ===\n{context}\n\n{task['task_text']}"

Result: Success rate jumped from 0% → 80% for Gemini.

Phase 4: Comprehensive Evaluation

I tracked five metrics for every run:

def evaluate_task(agent_output, gold_standard):
    return {
        "success": check_keyword_overlap(agent_output, gold_standard),
        "precision": correct_tools / total_tools_used,
        "recall": correct_tools / required_tools,
        "efficiency": oracle_steps / agent_steps,
        "latency": end_time - start_time,
        "cost": (input_tokens * rate_in + output_tokens * rate_out)
    }

Every experiment produced structured logs:

{
  "task_id": 6,
  "model": "gemini-2.5-flash",
  "agent_type": "Single-Tool",
  "success": true,
  "latency_seconds": 9.41,
  "run_log": {
    "steps": [
      {"type": "llm_call", "output": "Thought: I need WHO data..."},
      {"type": "tool_call", "tool": "web_search", "output": "[WHO Document]"},
      {"type": "llm_call", "output": "Final Answer: The WHO strategy..."}
    ]
  }
}

Key Results

Testing Google Gemini 2.5 Flash vs OpenAI GPT-4o-mini across 40 scenarios:

Metric	Gemini 2.5 Flash	GPT-4o-mini
Multi-Tool Success	80%	45%
Single-Tool Success	90%	50%
Cost per Success	$0.0276	$0.0178
Average Latency	69.29s	10.50s
Adversarial Refusal Rate	100%	100%

The Trade-Off: Gemini is nearly 2x more accurate but 6x slower and 55% more expensive. The "best" model depends on your priorities.

Conclusions and Learnings

System Design Matters: Many failures were due to missing context or poor tool routing, not model limitations. A well-architected agent can outperform a more powerful model with a weak setup.
Reproducibility is Key: Using a static knowledge base allowed for consistent, comparable results across runs. Live web search would have introduced too much variability.
Transparent Reasoning Aids Debugging: The ReAct framework made it easy to identify where agents went wrong—whether in tool selection or interpretation.
Cost vs Performance Trade-Offs: Depending on the application, a cheaper, faster model with lower accuracy may be preferable to a more expensive, slower one.

Future improvements

Expand the dataset to cover more domains and task types.
Integrate more sophisticated tools (e.g., calculators, code interpreters).
Explore hybrid models that combine strengths of different LLMs.

Tech Stack

Languages: Python
APIs: Google Gemini 2.5 Flash, OpenAI GPT-4o-mini
Frameworks: ReAct prompting, RAG
Testing: pytest, stubbed agents
Analysis: Statistical validation (95% CI), paired comparative analysis

Skills

LLM Prompt Engineering
Agentic System Design
Benchmarking Methodologies
Data Analysis & Visualization

Resources

📂 View Code on GitHub
📄 Read Full Technical Report

Built for DTSC 5505 (Applied Machine Learning) at UNT | Fall 2025