Lessons in Conversational AI: What I Learned from Using Tool Calls
low temperature for LLM using tools if possible
split responsible to fulfil both good temperature and using tools
It's essential to offer tools to agents handling various tasks, especially when we aim to provide them with the latest and additional information. Initially, we believed the tool would be used naturally when needed. However, many developers find that the tool's usage doesn't always meet our expectations.
The biggest issue is when it skips using the tools because it thinks it already "knows" the answer, even though I want it to get the latest and most accurate information to ensure correctness. Even when I emphasize using the tools in the prompt, it doesn't always follow. Here's how I handle it: I set a very low temperature (0.0-0.2) to ensure it doesn't pretend to know the answer. This reduces the randomness of the LLM and improves accuracy. However, be aware that this might also reduce creativity.
To strike a balance, I sometimes separate the LLM's tasks. One focuses on using the tools, while the other handles the creative aspect. For instance, in a multi-agent project with LangGraph for writing a novel, the author needs to be creative yet realistic. So, I make two calls: one for the author and one for the information searcher. The information searcher operates at a low temperature to ensure logical analysis, while the author can use a higher temperature for creativity.
Another approach is to use the review method to ensure tools are utilised. By using the state to track tool calls, outputting with JSON, and reviewing with a review LLM agent, you can make sure enough tools are used before moving to the next step. However, if you can limit tool usage, why not do it without the agent? This could work well. Not everything needs an agent, and in production, relying on agents could significantly increase token costs. If logical decision-making by the LLM isn't necessary, we can handle it ourselves, similar to RAG.
