Optimizing AI Agent Orchestration: Reducing LLM Latency

Engineering question: How do you make an AI assistant feel responsive when the underlying model's response time is highly variable — sometimes 800ms, sometimes 6 seconds for the same kind of query?

Latency in an LLM-backed feature is rarely about the model alone. It's about how many round-trips you make, how much you re-compute work you've already done, and whether the user is staring at a spinner or watching text appear. Three techniques do most of the heavy lifting.

1. Semantic caching — don't ask the model what you already know

In a business assistant, a large share of questions are near-duplicates: "What's our refund policy?", "How do refunds work?", "Can I get my money back?" These are three strings but one intent. An exact-match cache misses all three; a semantic cache catches them.

The idea: embed each incoming query into a vector, and before calling the LLM, check whether a semantically similar query was already answered.

# embed the query, then search prior answers by cosine similarity
qvec = embed(user_query)
hit = redis.execute_command(
    "FT.SEARCH", "qcache",
    f"*=>[KNN 1 @vec $q AS score]",
    "PARAMS", "2", "q", qvec.tobytes(),
    "DIALECT", "2",
)

if hit and hit.score >= 0.95:        # tunable similarity threshold
    return hit.cached_answer          # skip the LLM entirely

With Redis acting as a vector store, a cache hit returns in single-digit milliseconds instead of seconds. The threshold matters: too low and you serve answers to questions the user didn't ask; 0.93–0.96 is a safe starting band for domain Q&A, tuned against real traffic.

Cache the answer keyed by the meaning, set a sensible TTL, and invalidate on the events that change the underlying facts (policy updated, price changed). For a domain assistant, hit rates of 30–50% are realistic — and every hit is a full LLM call avoided.

2. Streaming — change the perception of latency

Total generation time may be 4 seconds, but a user who sees the first words at 400ms perceives the system as fast. Stream tokens as they're produced instead of buffering the whole completion:

async for chunk in model.stream(prompt):
    await ws.send(chunk.delta)   # push each token to the client as it arrives

This doesn't reduce total compute, but time-to-first-token is what the human actually feels. Pair it with an immediate typing indicator so the interface never looks frozen.

3. Cut the round-trips in the agent loop

Agentic flows are slow when they're chatty — plan, call tool, observe, plan again, each step a full model round-trip. Reduce the count:

• Parallelize independent tool calls instead of awaiting them one by one. If the agent needs an account balance and recent orders, fetch both concurrently.
• Let the model request multiple tools in a single turn where the API supports it, rather than one per turn.
• Route by difficulty. Send trivial or cached-pattern requests to a small, fast model and reserve the large model for genuinely hard reasoning. Most traffic is not hard.

Putting it together

The improvement does not come from one trick — it comes from the stack. Semantic caching removes repeat model calls, parallel tool execution collapses multi-step loops, and streaming makes the remaining latency feel shorter than it is. None of this requires a faster model; it requires not wasting the model you have.

The principle underneath all three: an AI feature is a software system, not a prompt. It earns its responsiveness through caching, concurrency, and honest UX — the same engineering you'd apply to any latency-sensitive service.