Temperature Settings Across Loop Steps

An agent I reviewed had been set to temperature 0.9 across the board, on the theory that a higher temperature made it more "creative and capable." On a routine step, it invented a tool called

fetch_customer_history

This is a teardown of the single-temperature mistake, why one setting can't serve every step, and how to vary temperature across the loop so each step gets the sampling behavior it needs.

Before: The Weak Prompt

Here's the setup — a loop with one temperature hardcoded for every model call.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], _ ,[object Object], ,[object Object],(max_steps):
        action = model_decide(state, tools, temperature=temperature)
        ,[object Object], action.name == ,[object Object],:
            ,[object Object], action.args[,[object Object],]
        state = update_state(state, action, run_tool(action))
    ,[object Object], force_answer(state)

What this does: It applies a single temperature to every model call in the loop regardless of the step's purpose — the same randomness for choosing a tool, reasoning about a result, and writing the final answer, even though those steps have opposite needs.

Why It Fails

One temperature can't serve every step because different steps want opposite things from the sampler. Tool selection wants determinism: given the state, there's usually one right tool, and randomness there means occasionally picking the wrong one — or, at high enough temperature, hallucinating a tool that doesn't exist, exactly what crashed the agent above. Structured decisions are where high temperature does its worst damage.

Final-answer synthesis is more forgiving of some variation, and a little temperature can make prose read less robotically. But even there, high temperature raises the odds of a fabricated detail slipping in. And intermediate reasoning sits in between — you want enough flexibility to consider alternatives but not so much that the agent's logic wanders.

The failure of a fixed high temperature is hallucination and instability on the structured steps. The failure of a fixed low temperature is the opposite and subtler: the agent gets rigid, takes the same unproductive action every time it hits a certain state, and can't explore its way out of a rut because there's no variation to explore with. One value can't avoid both failure modes at once, which is why agent loop temperature should not be a single constant.

⚡ Pro tip: If your agent occasionally calls tools that don't exist or invents parameter names, check your temperature before you touch anything else. High temperature on tool-selection steps is the most common cause of hallucinated tool calls, and dropping it to near zero for those steps often fixes the problem outright.

There's a subtle second failure too. A fixed temperature interacts badly with retries. When an agent retries a failed step at the same temperature that produced the failure, a low temperature makes it repeat the identical failing action, while the right move is often to raise temperature slightly on retry specifically to shake loose a different attempt. A single global value can't express "be deterministic normally but exploratory on retry."

It's worth being precise about what temperature does mechanically, because the "creativity dial" framing is what leads teams astray. Temperature reshapes the probability distribution the model samples its next token from. Low temperature sharpens the distribution toward the single most likely token; high temperature flattens it so less likely tokens get a real chance. On a tool-selection step where one tool is clearly correct, flattening the distribution just gives probability mass to wrong tools — and at the extreme, to token sequences that spell out a tool name the model never saw in its toolset. That's the mechanism behind the hallucinated

fetch_customer_history

⚡ Pro tip: If you can only change one temperature in your whole loop, make tool selection deterministic and leave the rest alone. The structured steps are where randomness does concrete damage — wrong tools, invented parameters, hallucinated calls — while the prose steps degrade gracefully. Cold tool selection is the single highest-value temperature change you can make.

After: The Improved Prompt

The rewrite sets temperature per step type, matching the sampler to what each step needs.

TEMPS = {
    ,[object Object],: ,[object Object],,   ,[object Object],
    ,[object Object],: ,[object Object],,        ,[object Object],
    ,[object Object],: ,[object Object],,     ,[object Object],
}

,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], _ ,[object Object], ,[object Object],(max_steps):
        phase = classify_step(state)             ,[object Object],
        temp = TEMPS.get(phase, ,[object Object],)
        action = model_decide(state, tools, temperature=temp)
        ,[object Object], action.name == ,[object Object],:
            ,[object Object], synthesize(state, temperature=TEMPS[,[object Object],])
        state = update_state(state, action, run_tool(action))
    ,[object Object], force_answer(state)

What this does: It classifies each step's purpose and applies a temperature suited to it — near-zero for tool selection to stop hallucinated calls, moderate for reasoning, higher for the final answer's readability — so every step samples the way it should instead of inheriting one blanket value.

For retries specifically, nudge temperature up so a repeated attempt differs from the one that failed.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], ,[object Object],(base_temp + ,[object Object], * attempt, ,[object Object],)

What this does: It increases temperature with each retry so a second or third attempt at a failed step explores a genuinely different action rather than deterministically repeating the failure, turning retries into real alternatives instead of copies.

Breaking Down Each Element

The per-phase temperature map is the core idea. Tool selection at 0.0 makes the agent pick the obvious right tool every time and eliminates hallucinated tool calls, because there's no randomness to invent one. This alone fixes the most common temperature-driven failure.

The moderate reasoning temperature keeps the agent flexible enough to consider more than one interpretation without letting its logic drift. Zero here can make an agent brittle — locked into one line of reasoning — so a little warmth helps it stay adaptive.

The retry escalation is the clever part. Deterministic-by-default plus warmer-on-retry gives you the best of both: stable, correct behavior in the normal case, and genuine exploration exactly when the normal case has failed and repeating it is pointless.

⚠️ Common mistake: Setting a high global temperature to make the agent seem smarter or more creative. Temperature is not an intelligence dial — it's a randomness dial. On the structured steps that make up most of a loop, randomness is a liability, not an asset, and a high global temperature mostly buys you instability, hallucinated tool calls, and irreproducible runs that are miserable to debug. Reserve warmth for the steps that actually benefit from it.

⚡ Pro tip: Run your tool-selection steps at temperature 0 during development specifically to make runs reproducible. Debugging a nondeterministic agent is brutal because you can't reproduce the failure; deterministic tool selection means the same input gives the same trace, so you can actually chase a bug down. You can reintroduce a little variation later if you need it.

Variations for Different Contexts

For a creative-writing agent, the final-answer temperature runs high because varied, surprising prose is the goal — but tool selection and retrieval still run near zero, because even a creative agent shouldn't hallucinate its sources.

For a financial or compliance agent, every step runs cold, including the final answer, because reproducibility and precision matter more than natural-sounding prose. A risk engineer at a bank told me they run their entire agent at temperature 0 specifically so any output can be exactly reproduced in an audit.

For an exploratory brainstorming agent, reasoning steps run warmer than usual to encourage divergent thinking, while any step that touches real tools or data stays cold to keep the exploration grounded in fact rather than invention.

For a data-analysis agent that writes and runs queries, split the temperature by risk of the operation: query generation can run slightly warm to consider different analytical angles, but any step that interprets results and states a number to the user runs cold, because a fabricated statistic is far more damaging than a slightly rigid query plan. This "warm to explore, cold to conclude" split maps cleanly onto most analytical agents — you want creativity in how the agent approaches a problem and precision in what it reports as fact.

The unifying principle across every variation is to match agent loop temperature to the cost of a wrong sample at each step. Where a wrong sample is cheap and variety is valuable, run warm. Where a wrong sample is expensive — a hallucinated tool, a fabricated figure, an unreproducible audit trail — run cold. Once you think in those terms, the right temperature for each step usually picks itself.

⚡ Pro tip: Log the temperature used on each step alongside the trace. When you're debugging a weird action weeks later, knowing whether that step ran at 0.0 or 0.8 is often the first clue — a bizarre tool call at high temperature is a sampling problem, while the same call at 0.0 is a genuine reasoning or prompt problem. Without the logged temperature, you can't tell those apart.

Save and Reuse This

A per-phase temperature map and a retry-escalation rule are small, stable pieces of configuration that nearly every agent benefits from, and they're easy to forget when you're building a new loop from scratch. Getting them right once and reusing them saves you from rediscovering the hallucinated-tool-call problem on every project.

A prompt library like PromptABCD is a handy place to keep your temperature maps and retry rules versioned alongside the prompts they pair with, so your next agent starts with sensible per-step sampling instead of one blanket value that's wrong for most of the loop.