Tool Agents: Empowering LLMs to Use Tools and Explore Environments

Relying solely on the parametric knowledge acquired during pre-training, Large Language Models (LLMs) struggle to answer real-time queries, perform complex mathematical calculations, or interact with external systems. This is where we need to upgrade LLMs into Tool Agents—giving them the ability to call search engines, execute code, query APIs, and even interact with graphical user interfaces.

This post reviews the core paradigms of modern Tool Agents in extreme detail, from how tools are executed and how models teach themselves to use tools, to how agents understand, represent, and explore complex environments.

1. Tool Execution Paradigms

How exactly does an LLM execute a tool? The community has explored several highly innovative pathways:

1.1 Tool Tokens (ToolkenGPT)

Conventional tool usage (like OpenAI’s Function Calling) requires the model to generate a JSON-formatted string. ToolkenGPT introduced a much more foundational approach: treating each tool as a special vocabulary token (a “Toolken”).

• Embedding Representation: The embedding vectors of tools are inserted directly into the LLM’s vocabulary head, acting just like ordinary text tokens.
• Reasoning Mode: The model generates text normally, evaluating both standard tokens and toolkens in its probability distribution.
• Tool Mode: Once a toolken is predicted, the LLM switches to a special “tool mode” to generate the specific arguments required for that tool.
• Result Injection: After the tool executes, the result is returned to the text context, and reasoning continues until the final answer is reached.
• Advantages: It only requires learning lightweight toolken embeddings without fine-tuning the entire LLM, and it supports dynamically adding massive amounts of new tools simply by expanding the vocabulary.

1.2 Code Tags (CodeAct)

Pure text formats like JSON have severe limitations in expressiveness: they natively lack support for loops, conditionals, exception handling, and state persistence. Therefore, CodeAct proposes a bold shift: Python code itself is the most general, Turing-complete “universal tool.” Instead of calling discrete, isolated APIs, the model writes Python scripts and executes them in a secure sandbox (with access to libraries, files, and computations). Upon receiving errors or results (Observation), the model can self-reflect, debug, and revise the code, forming a robust closed loop of “Generate -> Run -> Observe -> Revise.”

2. How Do Models Learn to Use Tools?

2.1 Prompting: DocPrompting

Given the constantly growing and evolving landscape of open-source repositories and APIs, no model can memorize all functions during training. How can it generalize to unseen libraries? DocPrompting offers a general-purpose Natural Language to Code (NL→Code) framework:

1. Retriever: Given a user’s natural language intent, it retrieves relevant API documentation snippets from a massive document library.
2. Generator: It feeds these documents, along with the user’s intent, into the prompt to guide the model in generating the correct tool-calling code. This is essentially a highly specialized Retrieval-Augmented Generation (RAG) pipeline designed for API documentation.

2.2 Self-Learning: Toolformer

If we want to avoid writing lengthy prompts every time, can the model “teach itself” to use tools? Toolformer introduces an elegant self-supervised learning method consisting of three stages:

1. Sampling: Provide the model with a few-shot prompt of tool usage. The model traverses a massive text corpus and generates candidate API calls at positions where tools might be helpful (e.g., <API>Calculator(2+2)</API>).
2. Execution: Actually call the API to obtain the result, appending it to the text (e.g., <API>Calculator(2+2) -> 4</API>).
3. Filtering (Core): Compare the next-token prediction loss in two scenarios:
   • Loss without the tool result ($L^-$)
   • Loss with the tool result ($L^+$)
   • Only keep the call if $L^- - L^+ \ge \text{threshold}$, meaning the tool’s output significantly and genuinely improved the model’s prediction for the next tokens. Finally, the model is fine-tuned on this strictly filtered, high-quality augmented data, allowing it to autonomously master when and how to trigger tools.

2.3 Tool Induction: TroVE

What if no ready-made toolset is provided in a specific environment? The TroVE framework enables LLMs to automatically induce reusable high-level functions from a stream of programmatic tasks. It builds a compact, efficient, and automatically verifiable toolbox from scratch, completely eliminating the need for additional training or human supervision.

3. Environment Representation and Exploration

Before an agent can use tools effectively, it must first understand the “environment” it inhabits.

3.1 Representing the Environment (Representation)

• Text: e.g., ALFWorld translates embodied environments into pure text descriptions.
• Images: e.g., Touchdown performs natural language navigation and spatial reasoning in visual street environments.
• Textual Web: e.g., WebArena abstracts complex DOM trees of web pages into text sequences for the language model to read.
• Set of Marks (SoM): Multi-modal models (like GPT-4V) can understand images but often struggle to click specific web elements accurately. SoM overlays numbered tags (Marks) on interactive elements in an image (such as a webpage or screen). The model simply outputs the corresponding number to click the target precisely, unleashing extraordinary Visual Grounding capabilities in multi-modal LLMs.

3.2 Environment Exploration and Memory (Exploration)

Models are not omniscient; even with vast world knowledge, specific environmental rules must be learned on the fly.

• Environment-specific Prompts (SteP): Manually crafting stacked LLM policies that provide explicit navigation guidelines specific to certain web actions.
• Unsupervised Workflow Memory: The agent automatically remembers successful trajectories during task execution and extracts them to prompt itself in future, similar tasks.
• Curiosity-driven Exploration: Introducing curiosity mechanisms from Reinforcement Learning. When the model enters “unpredictable” parts of the state space, it receives higher rewards, encouraging the exploration of the unknown.
• Exploration-based Trajectory Memorization (BAGEL): The model bootstraps itself by sampling initial instructions and broadly exploring the environment. It then re-labels these successful exploration trajectories with newly generated, more accurate instructions, significantly enhancing the agent’s capabilities through self-play.

4. Conclusion

Tool Agents are undergoing a massive paradigm shift from “rigid JSON API filling” to “writing Turing-complete code to interact with systems.” Whether it’s synthesizing data via self-supervised contrastive loss like Toolformer, or interpreting GUIs using visual grounding like Set-of-Mark, the boundaries of what an “armed” agent can do are constantly expanding. Coupled with Memory, Reasoning, and iterative mechanisms for environmental exploration, proper Tools are what finally allow Language Agents to break out of the chatbox and become active, autonomous executors in the digital world.