Agents don’t fail in demos, they fail in orchestration: what production teams use Haystack for to control routing, memory, and retrieval

I’m Denis Shokhirev, an Agentic AI Systems Architect based in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship production AI systems on Claude, Supabase, n8n, Doppler, and self-hosted Postgres for DACH B2B clients. In demos, agents usually look fine; in production, they fail one layer earlier than most teams expect: not in reasoning, but in orchestration, where routing picks the wrong tool, memory carries stale facts, and retrieval floods context with near-duplicates.

That is why production teams reach for Haystack less as a “chatbot framework” and more as an orchestration discipline for retrieval, routing, and memory. If your agent has to survive audits, partial outages, changing source data, and human escalation paths, the hard part is not getting an answer. The hard part is getting a controlled answer.

The production failure is usually between components

Most agent incidents I see are not model-quality incidents. They are system-design incidents. A good model inside a weak orchestration layer still produces unstable behavior because three control surfaces drift first:

• routing: which tool, data source, or workflow gets selected;
• memory: which facts persist and which should expire;
• retrieval: which documents enter context and why.

Anthropic’s documentation on tool use and structured outputs keeps pointing in the same direction: define the contract clearly, constrain the action space, and keep the system state inspectable. See the Anthropic docs on tool use: https://docs.anthropic.com/ . In practice, this matters more than another round of prompt tuning once the agent moves into real operational traffic.

What Haystack gives production teams that demos usually hide

Explicit pipelines instead of hidden agent behavior

The useful part of Haystack is not that it makes agents sound intelligent. It is that it encourages you to model the system as explicit components: retriever, router, ranker, prompt builder, generator, validator. When a response goes wrong, you can ask a concrete question: which node misbehaved?

That sounds mundane, but it is the difference between debugging and storytelling. If your only artifact is a final answer plus a vague “agent trace,” you cannot tell whether the problem came from wrong routing, weak ranking, stale memory, or a prompt-level instruction conflict.

Routing becomes a governed layer, not a model impulse

Demo systems often let the LLM decide everything: pick the tool, pick the index, pick the next action. Production systems need more structure. Haystack is useful here because it makes routing a first-class concern.

A practical routing policy often looks like this:

• if the request contains a case ID or contract ID, query Postgres first;
• if a fresh structured record exists, skip document retrieval;
• if retrieval confidence is below threshold, ask for clarification or escalate;
• if the query touches a regulated action, require human review before write access.

This is not just architecture preference. OWASP lists prompt injection and excessive agency in the OWASP Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/ . If routing is unconstrained, the model can be manipulated into using the wrong tool or the wrong source of truth.

Memory is a policy problem, not a chat-history feature

Teams get into trouble when they treat memory as one long conversation buffer. In production, memory should be split at least into session memory, task memory, and verified system memory. Haystack is helpful because it nudges teams toward separable components instead of one opaque blob of context.

That separation matters. Otherwise, intermediate guesses become persistent facts, old ticket states resurface as current truth, and the model starts arguing from its own past output. I have seen this pattern more often than outright model hallucination on recent B2B agent deployments.

Retrieval in production is narrower, not broader

A common mistake is to answer retrieval issues by adding more context. In production, the winning move is usually the opposite: narrower retrieval, stronger metadata filters, better reranking, and clearer confidence handling. The OpenAI Cookbook examples on RAG repeatedly show that chunking strategy and retrieval design matter as much as embeddings: https://cookbook.openai.com/ .

If I am auditing an agent stack, I want three numbers before I want another prompt experiment:

• retrieval precision on a labeled sample;
• percentage of requests hitting fallback or escalation;
• number of routes where the agent selected a non-authoritative source.

Without those, teams tend to confuse “sounds plausible” with “is controlled.”

A practical pattern with Postgres, Supabase, and n8n

You do not need to build every part inside Haystack to apply the same orchestration logic. The key is explicit routing and visible control flow. Here is a minimal Python pattern that checks structured data first, then falls back to model generation.

import os
import psycopg
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

def fetch_order(order_id: str):
    with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
        with conn.cursor() as cur:
            cur.execute(
                "select status, summary from orders where id = %s",
                (order_id,)
            )
            return cur.fetchone()

def handle_query(query: str, order_id: str | None = None):
    if order_id:
        row = fetch_order(order_id)
        if row:
            status, summary = row
            return f"Order {order_id}: {status}. {summary}"
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=300,
        messages=[{"role": "user", "content": query}],
    )
    return msg.content[0].text

And this is the same idea represented as an n8n-style flow, where route decisions and escalation remain inspectable:

steps:
  - name: classify_request
    type: llm
  - name: lookup_postgres
    type: postgres
    run_if: "{{ $json.route == 'structured_lookup' }}"
  - name: retrieve_docs
    type: http
    run_if: "{{ $json.route == 'retrieval' }}"
  - name: human_review
    type: manual
    run_if: "{{ $json.confidence < 0.75 }}"

This is the production shift: not “make the model smarter,” but “make the system easier to constrain, inspect, and recover.”

Why this matters more in regulated markets

In regulated European environments, orchestration is not only a quality topic. It is a governance topic. If an agent touches customer records, internal SOPs, contract state, or payment workflows, you need to explain why a source was chosen, why a fact entered the context window, and why the system took a given branch.

That is why teams often borrow from Haystack even when they do not standardize on it end to end. They want pipeline-first discipline:

• explicit processing nodes;
• separate verified data from generated text;
• logged routing decisions;
• controlled fallback and human review.

Once an incident happens, “the model seemed confident” is not an operational explanation.

FAQ

Is Haystack only useful for RAG?

No. Its real value is the way it pushes teams to think in explicit components and controlled flow. RAG is just one common use case.

Can I apply the same pattern without Haystack?

Yes. I often use Anthropic SDK, Postgres, Supabase, and n8n directly. The point is not the brand of framework; it is the orchestration discipline.

What breaks agents more often in production: memory or retrieval?

Usually the interaction between them. Bad retrieval injects weak context, and bad memory turns that weak context into a persistent assumption.

Should the model choose its own tools?

Sometimes, but inside narrow boundaries. In sensitive workflows, full autonomy is less useful than constrained choice plus visible fallback.

Where should an orchestration audit start?

With route maps and node-level logs. If you cannot reconstruct how the system selected a source or action, you do not really control the agent.

Which part of your LLM pipeline is causing more real production pain right now: routing, memory policy, retrieval quality, or missing fallback to human review? I would genuinely like to compare that with what I am seeing across DACH B2B systems.
I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.