Hierarchical Multi-Agent Systems

A benchmark study on multi-agent reasoning found something counterintuitive: adding a third level of hierarchy to an already two-level agent system improved task accuracy by only 3% while increasing average task completion time by 180%. The accuracy gain was real but marginal. The latency penalty was severe and compounding.

This is the hidden cost of hierarchical agent systems that most tutorials ignore: every layer of hierarchy multiplies your latency budget. A hierarchical agent system with three tiers doesn't take three times as long as a flat system — it often takes ten times as long, because agents at each level are waiting for the level below them to complete.

Understanding when hierarchy earns that cost and when it doesn't is the difference between a system that scales and a system that's impressively complex but impractical.

Before: The Flat System That Breaks Under Complexity

A flat multi-agent system — where one orchestrator coordinates all workers directly — works well up to a certain complexity threshold. That threshold is roughly 4–6 workers with clearly defined, non-overlapping responsibilities.

[object Object],
,[object Object], anthropic ,[object Object], Anthropic

client = Anthropic()

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],
    results = {}

    results[,[object Object],] = run_worker(,[object Object],, task)
    results[,[object Object],] = run_worker(,[object Object],, ,[object Object],)
    results[,[object Object],] = run_worker(,[object Object],, ,[object Object],)
    results[,[object Object],] = run_worker(,[object Object],, ,[object Object],)

    ,[object Object], results

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    system_prompts = {
        ,[object Object],: ,[object Object],,
        ,[object Object],: ,[object Object],,
        ,[object Object],: ,[object Object],,
        ,[object Object],: ,[object Object],
    }
    response = client.messages.create(
        model=,[object Object],,
        max_tokens=,[object Object],,
        system=system_prompts[role],
        messages=[{,[object Object],: ,[object Object],, ,[object Object],: message}]
    )
    ,[object Object], response.content[,[object Object],].text

What this does: Clean flat orchestration that works excellently for 4 agents. The orchestrator is pure Python — no reasoning required. Each worker knows its job. The problem: add 8 more workers and the orchestrator's routing logic becomes unmanageable, and a single human-readable task might genuinely require different subsets of those 12 workers depending on what the task involves.

Why Flat Systems Break at Scale

A flat system with 12 workers has three structural problems that don't exist at 4 workers.

Routing complexity explodes: With 12 workers and dependencies between their outputs, the orchestration logic for "which workers to run in what order for what inputs" becomes extremely difficult to maintain as code. You start wanting an LLM to make routing decisions — which introduces unpredictability.

Responsibility confusion: When 12 workers all report to one orchestrator, it's unclear which worker handles tasks that fall between clear categories. A research-adjacent but analysis-heavy task goes to the researcher? The analyst? Some combination? Flat systems handle ambiguity poorly.

Debugging becomes archaeological: When a 12-agent flat system produces a wrong answer, you're checking 12 possible sources of error. There's no natural structure to guide the investigation.

⚠️ Common mistake: Building a hierarchical agent system to solve a complexity problem that actually has a simpler solution. Before adding hierarchy, ask whether your flat system's complexity is a function of the number of agents (hierarchy helps) or a function of poorly scoped agent responsibilities (better system prompts help). Hierarchy doesn't fix bad agent design.

After: Hierarchical Structure That Earns Its Overhead

A hierarchical agent system adds intermediate supervisors between the top-level orchestrator and the front-line workers. Each supervisor manages a logically related cluster of workers.

[object Object], concurrent.futures
,[object Object], anthropic ,[object Object], Anthropic

client = Anthropic()

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    response = client.messages.create(
        model=,[object Object],,
        max_tokens=max_tokens,
        system=system,
        messages=[{,[object Object],: ,[object Object],, ,[object Object],: message}]
    )
    ,[object Object], response.content[,[object Object],].text

,[object Object],
,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],
    web_result = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    data_result = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    synthesis = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    ,[object Object], {,[object Object],: web_result, ,[object Object],: data_result, ,[object Object],: synthesis}

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],
    outline = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    draft = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    edit = run_agent(
        ,[object Object],,
        ,[object Object],
    )
    ,[object Object], {,[object Object],: outline, ,[object Object],: draft, ,[object Object],: edit}

,[object Object],
,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],

    ,[object Object],
    ,[object Object], concurrent.futures.ThreadPoolExecutor() ,[object Object], executor:
        research_future = executor.submit(research_department, task)
        research_results = research_future.result()

    ,[object Object],
    production_results = production_department(task, research_results[,[object Object],])

    ,[object Object], {
        ,[object Object],: research_results,
        ,[object Object],: production_results,
        ,[object Object],: production_results[,[object Object],]
    }

What this does: The top-level orchestrator only knows about two departments — research and production. It doesn't know about web researchers, data researchers, content strategists, or editors. Each department supervisor manages its own workers. Adding a new worker to the research department only requires changing the research_department function, not the top-level orchestrator.

⚡ Pro tip: Run department-level agents in parallel wherever their outputs are independent. In the example above, a quality assurance department could run in parallel with production if given access to the research output directly. Parallel departments dramatically reduce the latency multiplication effect of deep hierarchies.

Breaking Down the Architecture

The hierarchical agent system structure has three distinct tiers, each with specific responsibilities:

Tier 1 — Strategic (top-level orchestrator): Understands the overall task goal, knows which departments exist, coordinates department-level activities, and handles cross-department conflicts or decisions. Should be a simple code-based coordinator whenever possible. Makes no domain-level decisions.

Tier 2 — Tactical (department supervisors): Each supervisor manages a logically related cluster of workers, decomposes department-level tasks into worker-level tasks, evaluates worker output for domain accuracy, and communicates results upward. This is where LLM-based coordination is most justified — routing within a domain is complex enough to benefit from reasoning.

Tier 3 — Operational (workers): Individual specialists that execute narrow, specific tasks. Each worker has a highly focused system prompt, a well-defined output format, and no awareness of the broader system. Workers are the easiest agents to test, replace, and improve.

Variations for Different Contexts

Two-tier hierarchy: Most real systems need only two tiers, not three. A top-level code orchestrator and a set of 3–5 domain supervisor agents that each manage 2–4 workers covers the vast majority of production use cases without the latency penalty of a third tier.

Dynamic department activation: Instead of always running all departments, have the top-level orchestrator classify the task and activate only the relevant departments. A short factual question might only activate the research department. A complex analysis might activate research, production, and quality assurance.

Cross-department communication: Sometimes workers in different departments need to share information without going through the full hierarchy. A research worker's finding might be directly relevant to a production worker. Design explicit information-sharing channels for these cross-department dependencies rather than routing everything upward through supervisors.

⚡ Pro tip: The depth of your hierarchy should match the complexity of the task domain, not the ambition of your architecture. Two tiers handle most production requirements well. Add a third tier only when department supervisors themselves become overloaded — when each supervisor is managing more than 6 workers with complex interdependencies.

Save and Reuse This Pattern

The hierarchical agent system pattern is particularly valuable because the department supervisor prompts and worker prompts can be reused across many different top-level tasks. Once you have a well-tuned research department or a production department, those components can be assembled into different hierarchies for different use cases.

The Three-Tier Hierarchy in Code

For complex domains, a three-tier hierarchy — executive orchestrator, department supervisors, specialist workers — distributes both the routing decisions and the domain reasoning across dedicated agents:

[object Object], asyncio
,[object Object], anthropic ,[object Object], Anthropic

client = Anthropic()

EXECUTIVE_PROMPT = (
    ,[object Object],
    ,[object Object],
    ,[object Object],
    ,[object Object],
    ,[object Object],
)

RESEARCH_SUPERVISOR_PROMPT = (
    ,[object Object],
    ,[object Object],
    ,[object Object],
    ,[object Object],
)

EDITORIAL_SUPERVISOR_PROMPT = (
    ,[object Object],
    ,[object Object],
    ,[object Object],
    ,[object Object],
)

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    response = client.messages.create(
        model=,[object Object],,
        max_tokens=,[object Object],,
        system=system_prompt,
        messages=[{,[object Object],: ,[object Object],, ,[object Object],: user_content}]
    )
    ,[object Object], response.content[,[object Object],].text

,[object Object], json

,[object Object], ,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],
    executive_output = run_agent(EXECUTIVE_PROMPT, ,[object Object],)
    assignment = json.loads(executive_output)
    
    ,[object Object],
    research_task = asyncio.to_thread(
        run_agent, RESEARCH_SUPERVISOR_PROMPT,
        ,[object Object],
    )
    editorial_assignment = assignment[,[object Object],]
    
    research_output_raw = ,[object Object], research_task
    research_output = json.loads(research_output_raw)
    
    ,[object Object],
    editorial_input = (
        ,[object Object],
        ,[object Object],
    )
    editorial_output_raw = run_agent(EDITORIAL_SUPERVISOR_PROMPT, editorial_input)
    editorial_output = json.loads(editorial_output_raw)
    
    ,[object Object], {
        ,[object Object],: assignment,
        ,[object Object],: research_output,
        ,[object Object],: editorial_output,
        ,[object Object],: editorial_output[,[object Object],]
    }

What this does: The executive orchestrator receives the full request and decomposes it into department-level assignments without executing any work itself. Department supervisors receive their bounded assignments and coordinate their specialist workers (not shown here for brevity). The executive never needs to know how research or editorial accomplish their work — it only needs to know what to assign them. This is the core benefit of the hierarchical agent system: each tier knows only what it needs to know to do its own coordination job.

⚡ Pro tip: Add a "summary only" mode for each tier when debugging. Before running the full hierarchical pipeline, run the executive orchestrator alone and print its decomposition plan. Then run each department supervisor alone with its decomposed assignment. Isolating each tier's behavior makes it much easier to identify whether a quality problem originates at the executive level (bad decomposition), the supervisor level (bad coordination), or the worker level (bad execution).

When Three Tiers Is Too Many

The appeal of hierarchy is that it mirrors how human organizations manage complexity. The danger is that it mirrors human organizations too faithfully — adding management overhead without adding execution capability.

Two-tier hierarchies (one supervisor, multiple workers) handle most production multi-agent system requirements. The supervisor routes tasks to specialists; specialists execute. This is sufficient for customer support routing, content production, research synthesis, and most domain-specific processing pipelines.

Three-tier hierarchies earn their additional complexity when: the executive's task decomposition is genuinely different from supervisor-level coordination (requiring a distinct reasoning mode), or when you have multiple departments that each need their own supervisor-level coordination before their workers can run. If your third tier is doing the same kind of routing as the second tier, you've added a management layer without a management distinction.

Store your department supervisor prompts and worker prompts in PromptABCD so you can assemble new hierarchies quickly without re-engineering the foundational agents each time. The reusability of well-designed agent components is one of the most underappreciated productivity multipliers in multi-agent development.

Avoiding the Coordination Bottleneck

The most common failure mode in hierarchical agent systems is the coordination bottleneck: the executive or department supervisor becomes the performance constraint because every decision in the system routes through it.

Hierarchical systems avoid this bottleneck by two mechanisms. First, the hierarchy should parallelize wherever possible — department supervisors at the same tier can run concurrently since they operate on independent assignments from the executive. If your implementation runs department supervisors sequentially, you're not getting the latency benefit the pattern is designed to provide.

Second, the executive's job should be decomposition and assignment, not decision-making. When the executive starts making substantive judgments about how departments should do their work, it's accumulated responsibilities that belong in the departments. An overloaded executive produces a bottleneck that defeats the purpose of delegation. Keep the executive's system prompt focused on decomposition criteria and department assignment rules, not on operational details.

Practically, audit your hierarchical system by timing each tier in isolation. If the executive is taking as long as any department supervisor, it's doing too much. If a department supervisor is taking as long as the workers it coordinates, the supervisor's job has been scoped too broadly. Time-to-output by tier is the most direct signal of where coordination overhead has accumulated and where the next architectural improvement should be targeted.