Agentic AI Platforms
Service: Agentic AI Platforms · Typical shape: 6–12 week build after a PoC, 3–5 engineers, integrated with the customer's internal systems through secured tool APIs
An agentic system lets an LLM choose and sequence actions — calling tools, reading results, deciding the next step — to finish a multi-step job. The engineering work is making that loop reliable, observable, safe, and affordable: the right amount of autonomy, well-designed tools, durable state, human checkpoints for consequential actions, and evals that measure outcomes.
What we deliver
- A use-case design that places each step on the workflow ↔ agent spectrum, with the reasoning written down.
- Tool layer: internal systems exposed as narrowly-scoped tools, usually as MCP servers behind an authenticating gateway.
- Agent runtime on a durable execution engine, with human-in-the-loop approval for consequential actions.
- Guardrails and security controls mapped to the OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications.
- Observability with OpenTelemetry GenAI spans (agent, LLM call, tool call), cost and latency dashboards.
- Eval suite with outcome-graded tasks and trajectory checks, gating releases in CI.
Workflows or agents?
Anthropic's distinction (Building effective agents): workflows orchestrate LLMs and tools through predefined code paths; agents let the LLM dynamically direct its own process and tool use. Use the simplest thing that hits the eval bar — agents trade latency, cost, and predictability for flexibility, and errors can compound.
| Pattern | Shape | Use when | Example |
|---|---|---|---|
| Prompt chaining | Fixed sequence of LLM steps with code checks between | Task decomposes cleanly into known steps | Extract → validate → draft reply |
| Routing | Classifier sends input to a specialised handler | Distinct input categories handled better separately | Ticket triage to billing / technical / legal flows |
| Parallelisation | Sectioning or voting | Independent subtasks, or confidence via multiple attempts | Check a contract against 12 policy clauses in parallel |
| Orchestrator–workers | LLM plans subtasks at runtime and delegates | Subtasks not known in advance | Research across several internal systems |
| Evaluator–optimizer | Generator + critic loop | Clear evaluation criteria; iteration adds measurable value | Drafting a tender response against a checklist |
| Autonomous agent | LLM + tools in a loop until done or budget hit | Open-ended tasks, unpredictable step count, verifiable outcome | Resolve an IT incident using runbooks and diagnostics tools |
In practice most production systems are workflows with agentic steps: a deterministic graph where one or two nodes run a bounded agent loop.
Reference architecture
Implementation steps
1. Choose the framework
Status as of 2026-10 (versions from PyPI):
| Framework | Version | Model coverage | Strengths | Pick when |
|---|---|---|---|---|
| LangGraph | 1.2 | Any (via LangChain chat models / OpenAI-compatible) | Explicit state graphs, checkpointers, interrupt() for human-in-the-loop, mature | Default for workflow-with-agentic-steps systems in Python |
| Pydantic AI | 2.57 | Any; first-class OpenAI-compatible | Typed tools and outputs, dependency injection, MCP client, durable execution via Temporal, DBOS, Prefect, Restate | Type-safe Python services; our default for single-agent services |
| OpenAI Agents SDK | 0.24 | OpenAI plus other providers | Handoffs, guardrails, tracing, sandboxes; Temporal integration | Lightweight multi-agent handoffs |
| Claude Agent SDK | 0.2.168 | Claude (Anthropic API, Bedrock, Vertex) | Claude Code's agent loop as a library: built-in file/shell/web tools, hooks, permissions, subagents, sessions, MCP | Coding/ops agents that work on files and shells, where Claude is approved |
| Google ADK | 2.12 | Gemini plus others | Python/TS/Java/Go, strong A2A support | Google Cloud customers, A2A-heavy designs |
| CrewAI | 1.15 | Any | Role-based crews, quick prototyping | Fast prototypes; prefer graph frameworks for production control |
Guidance: frameworks are optional. Many patterns are a few dozen lines on a plain chat-completions API; if you adopt a framework, make sure the team understands what it sends to the model (Anthropic). For sovereign deployments, require that the framework works against an OpenAI-compatible self-hosted endpoint.
2. Design the tools
Tools are the agent's interface to the world; invest in them as much as in a UI (Anthropic: Writing effective tools for agents).
- Task-shaped, not API-shaped.
find_customer_open_invoices(customer_name)beats exposingGET /invoiceswith 30 query parameters. - Mistake-proof arguments (poka-yoke): enums instead of free strings, absolute IDs instead of positional references, validated date formats.
- Return what the model needs: concise, human-readable fields, stable IDs for follow-up calls, pagination and truncation with a hint ("42 more results; refine by date").
- Errors that teach: "customer_name matched 3 customers: … pass customer_id instead" rather than a stack trace.
- Separate read and write tools; write tools carry explicit side-effect descriptions and require approval where consequential.
- Idempotency keys on every write tool so retries never double-book.
from typing import Literal
from mcp.server import MCPServer # mcp 2.x; FastMCP was renamed to MCPServer
mcp = MCPServer("erp-invoices")
@mcp.tool()
def find_open_invoices(customer_id: str, status: Literal["open", "overdue"] = "open", limit: int = 20) -> dict:
"""List a customer's unpaid invoices. Use find_customer first to resolve a name to customer_id.
Returns at most `limit` invoices, newest first, amounts in EUR and original currency."""
rows = erp.invoices(customer_id=customer_id, status=status, limit=limit + 1)
return {"invoices": [r.summary() for r in rows[:limit]], "truncated": len(rows) > limit}
if __name__ == "__main__":
mcp.run(transport="streamable-http")
Written against the Python SDK mcp 2.3 (as of 2026-10). In 2.x FastMCP became MCPServer; see the v2 migration guide when porting v1 servers.
3. Connect tools with the Model Context Protocol
MCP standardises how agents discover and call tools, resources, and prompts. The current specification is 2026-07-28 (as of 2026-10), now hosted by the Linux Foundation's Agentic AI Foundation. Key points for implementers:
| Topic | 2026-07-28 behaviour |
|---|---|
| Core | Stateless: the initialize handshake and Mcp-Session-Id are retired; each request carries protocol version, client identity, and capabilities in _meta; optional server/discover |
| Transport | Streamable HTTP with Mcp-Method / Mcp-Name headers for gateway routing; legacy HTTP+SSE deprecated (one-year transition); stdio for local servers |
| Server-initiated input | Multi Round-Trip Requests: the server returns input_required; the client retries with inputResponses |
| Long-running work | io.modelcontextprotocol/tasks extension (tasks/get, tasks/update) |
| Deprecated | Sampling, Roots, Logging (supported ≥ 12 months; do not adopt in new code) |
| Authorization | OAuth 2.1 resource server model; Protected Resource Metadata (RFC 9728) required; Resource Indicators (RFC 8707) required; PKCE; issuer validation (RFC 9207); Client ID Metadata Documents preferred, Dynamic Client Registration deprecated |
Authorization rules that matter in reviews (from the spec):
- Servers MUST validate that access tokens were issued for them as audience, and MUST NOT accept or pass through other tokens — no token passthrough to downstream APIs; the server obtains its own downstream credentials.
- Clients send the
resourceparameter (canonical server URI) in both authorization and token requests. - Scope minimisation: advertise a minimal
scopes_supported; use403 insufficient_scopechallenges for step-up. - stdio servers take credentials from the environment, not OAuth.
Production pattern: run remote MCP servers behind a gateway that terminates auth, enforces per-tool policy (who may call create_payment), rate-limits, and writes an audit log. Pin MCP server versions and review third-party servers like any dependency (see the MCP security best practices).
Agent2Agent (A2A) is the complementary protocol for agent-to-agent delegation across vendors: agents publish Agent Cards, exchange tasks, and stay opaque to each other. A2A v1.0 shipped in March 2026 and is now hosted by the Linux Foundation's Agentic AI Foundation (as of 2026-10). Use it when agents owned by different teams or vendors must collaborate; inside one codebase, plain function calls or framework handoffs are simpler. Python SDK: a2a-sdk 1.2.
4. Build the agent loop with budgets
import os
from pydantic import BaseModel
from pydantic_ai import Agent, UsageLimits
from pydantic_ai.mcp import MCPToolset
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIChatModel(
os.environ["LLM_MODEL"],
provider=OpenAIProvider(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"]),
)
erp = MCPToolset(os.environ["ERP_MCP_URL"], auth="oauth") # OAuth flow per the MCP authorization spec
class Resolution(BaseModel):
summary: str
actions_taken: list[str]
needs_human: bool
agent = Agent(
model,
toolsets=[erp],
output_type=Resolution,
instructions="Resolve dunning questions using ERP tools. Never promise payment terms; set needs_human if unsure.",
)
result = await agent.run(
ticket_text,
usage_limits=UsageLimits(request_limit=15, tool_calls_limit=25, total_tokens_limit=200_000),
)
Every agent run has hard limits: max steps, max tool calls, max tokens, wall-clock timeout, and a cost ceiling at the gateway (LiteLLM per-key budgets). Snippet checked against pydantic-ai 2.57 (as of 2026-10); the library releases frequently, so re-check names on upgrade.
5. Make it durable
Agents that run for minutes to days, wait for approvals, or call flaky APIs need durable execution: state is persisted after each step, retries are automatic, and a crashed worker resumes without repeating completed LLM calls or side effects.
| Option | Fit |
|---|---|
| Temporal (server 1.32, Python SDK 1.34) | Default for business-critical, long-running agents. Official integrations: OpenAI Agents SDK (model calls as Activities), Pydantic AI (TemporalDurability capability; the older TemporalAgent wrapper is deprecated) |
| LangGraph checkpointers (Postgres) | LangGraph-based systems; pause/resume and interrupt() with persisted state |
| DBOS, Restate, Prefect | Lighter-weight durable execution; Pydantic AI supports them |
Rules: LLM calls and tool calls run as activities (retryable, recorded); the orchestration logic is deterministic; write tools are idempotent.
6. Human-in-the-loop
- Classify every tool: read (auto), reversible write (auto with audit), consequential write (approval), forbidden.
- Approval requests show the proposed action, its arguments, the agent's justification, and the evidence (trace link). Approvers can approve, edit arguments, or reject with a reason that is fed back to the agent.
- Implement with LangGraph
interrupt()+ resume, a Temporal signal, or the framework's approval hooks; the workflow waits durably for hours or days. - Also escalate on low confidence, budget exhaustion, guardrail hits, or repeated tool errors.
from langgraph.types import interrupt, Command
def execute_payment_node(state):
decision = interrupt({"action": "create_payment", "args": state["proposed_payment"], "trace": state["trace_url"]})
if decision["approved"]:
return {"payment": create_payment(**decision.get("args", state["proposed_payment"]))}
return Command(goto="replan", update={"feedback": decision["reason"]})
7. Guardrails and security
Map controls to the OWASP Top 10 for LLM Applications 2025 and the OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10).
| Threat | Control we implement |
|---|---|
| Prompt injection, incl. indirect via documents, emails, web pages, tool outputs (LLM01, ASI01 Agent Goal Hijack) | Treat all tool output and retrieved content as untrusted data; isolate it in clearly delimited sections; no tool may escalate privileges based on content; approval for consequential actions; injection test cases in evals |
| Excessive agency, tool misuse (LLM06, ASI02) | Minimal tool set per agent; narrow scopes; allow-listed arguments; rate limits; approval tiers |
| Identity and privilege abuse (ASI03) | Per-agent service identity; short-lived, audience-bound tokens; act on behalf of the user with the user's permissions, never with a superuser key |
| Supply chain (LLM03, ASI04) | Pin models, MCP servers, and packages; review third-party MCP servers; SBOM/AIBOM |
| Unexpected code execution (ASI05) | Generated code runs only in a sandbox: gVisor or Firecracker microVMs, no network by default, CPU/memory/time limits, ephemeral filesystem |
| Memory and context poisoning (ASI06) | Provenance on stored memories; write-access controls; expiry; never store tool output as instructions |
| Insecure inter-agent communication (ASI07) | mTLS between agents; authenticated A2A; signed Agent Cards where available |
| Cascading failures (ASI08) | Step and cost budgets, circuit breakers on tools, timeouts, kill switch per agent |
| Sensitive information disclosure (LLM02) | PII redaction in logs/traces; output filters; data minimisation in tool responses |
| Unbounded consumption (LLM10) | Gateway budgets per key/tenant; max tokens; concurrency limits |
Guardrail components: input/output classifiers (e.g. Llama Guard 4 self-hosted), policy rails (NeMo Guardrails), the framework's own guardrail hooks (OpenAI Agents SDK guardrails, Claude Agent SDK hooks and permissions), and — most important — deterministic checks in the tool layer (authorisation, argument validation, limits) that no prompt can bypass.
Assume the model can be convinced to call any tool it has with any arguments. Security comes from what the tools allow for this user, not from what the prompt says.
8. Observability with OpenTelemetry
Emit spans following the OpenTelemetry GenAI semantic conventions. They are still in Development status and moved to the dedicated semantic-conventions-genai repository in 2026 (as of 2026-10), so pin the instrumentation version and expect attribute renames.
Span (gen_ai.operation.name) | Name pattern | Key attributes |
|---|---|---|
invoke_agent | invoke_agent {gen_ai.agent.name} | gen_ai.agent.name, gen_ai.agent.id, gen_ai.provider.name, gen_ai.request.model |
chat | chat {gen_ai.request.model} | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons |
execute_tool | execute_tool {gen_ai.tool.name} | gen_ai.tool.name, gen_ai.tool.call.id, error.type |
invoke_workflow, plan, retrieval | — | Workflow and planning steps |
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
batch: {}
attributes/redact:
actions:
- { key: gen_ai.input.messages, action: delete } # keep content capture opt-in
exporters:
otlphttp/langfuse:
endpoint: https://langfuse.internal/api/public/otel
headers: { Authorization: "Basic ${env:LANGFUSE_OTLP_AUTH}" }
service:
pipelines:
traces: { receivers: [otlp], processors: [attributes/redact, batch], exporters: [otlphttp/langfuse] }
Most frameworks emit OTel natively or via OpenInference instrumentors; Langfuse and Phoenix both ingest OTLP. Dashboards: task success rate, steps per task, tool error rate per tool, approval rate and wait time, cost per task, p95 task duration.
9. Evaluate agents
Agents are evaluated on outcomes first, trajectories second (Anthropic: Demystifying evals for AI agents):
- Outcome graders check the final environment state (did the ticket get the right status, does the refund exist with the right amount) — not the agent's claim that it did.
- Trajectory checks are diagnostic and for hard constraints: forbidden tools never called, approval requested before
create_payment, no more than N steps. Avoid asserting an exact tool sequence; valid alternative paths should pass. - Multiple trials per task: report pass@k (at least one of k succeeds) for exploratory capability and pass^k (all k succeed) for customer-facing reliability — 75 % per trial is about 42 % pass^3.
- Capability vs. regression suites: capability tasks start with low pass rates; tasks that become reliable graduate into the regression suite that gates CI.
- Start with 20–50 tasks from real cases, run in an isolated environment (seeded test ERP, sandboxed mailboxes).
Tooling: Inspect for sandboxed agent tasks, RAGAS agent metrics (tool call accuracy, tool call F1, agent goal accuracy), DeepEval agent metrics, promptfoo for red-teaming against the OWASP agentic list.
import pytest
@pytest.mark.parametrize("case", load_cases("evals/dunning_tasks.jsonl"))
async def test_outcome_and_constraints(case, sandbox_erp):
sandbox_erp.seed(case["fixture"])
run = await run_agent(case["ticket"], erp_url=sandbox_erp.url)
assert sandbox_erp.ticket(case["ticket_id"]).status == case["expected_status"] # outcome
called = [s.tool for s in run.trace.tool_calls]
assert "create_payment" not in called or run.trace.approval_requested("create_payment") # constraint
assert len(run.trace.steps) <= case.get("max_steps", 15)
10. Control cost
- Measure cost per successful task, not per call; retries and failed runs count.
- Route by difficulty: a small model for classification, extraction, and routing; the large model only for planning and hard steps.
- Prompt and prefix caching: stable system prompts and tool definitions first in the prompt; vLLM prefix caching for self-hosted models.
- Trim context: summarise or drop old tool outputs; return compact tool results; load tool definitions on demand when the tool catalogue is large.
- Budgets everywhere: per-run step/token limits in the framework, per-key and per-tenant budgets at the LLM gateway, alerts on cost anomalies.
- Prefer workflows for the predictable parts; every deterministic step removed from the agent loop is cheaper and more reliable.
Best practices
- Start with a workflow; promote steps to agents only where the eval shows it pays off.
- Design tools for the model: few, task-shaped, mistake-proof, with instructive errors.
- Enforce security in the tool and identity layer; use prompts and classifiers as additional layers.
- Persist state durably and make every side effect idempotent.
- Trace with OTel GenAI conventions from the first prototype.
- Grade outcomes; use trajectories for constraints and debugging; report pass^k for reliability.
Pitfalls to design around
- Over-autonomy → explicit approval tiers per tool, reviewed with the business owner.
- Tool sprawl → each agent sees only the tools its task needs; split into routed sub-agents when the catalogue grows.
- Token passthrough to backends → the MCP server uses its own credentials; tokens are audience-bound.
- Lost progress on crashes → durable execution, not in-memory loops.
- Unpinned conventions → pin MCP spec version, SDK versions, and OTel instrumentation; schedule upgrades.
- Evals only on happy paths → include injection, ambiguous requests, tool failures, and permission-denied cases.
Definition of done
- Each step placed on the workflow ↔ agent spectrum with written justification
- Tools are task-shaped, validated, idempotent, and classified (read / reversible / approval / forbidden)
- MCP servers on the current spec, behind a gateway with OAuth audience validation, scopes, and audit logs
- Agent runs on durable execution; crash-and-resume tested
- Human approval flow live for consequential tools; approver UX reviewed with the business
- Controls mapped to OWASP LLM Top 10 and Agentic Top 10; prompt-injection tests pass
- Generated code (if any) runs only in an isolated sandbox
- Per-run budgets and gateway budgets configured; cost per successful task reported
- OTel GenAI traces for agent, LLM, and tool spans; dashboards and alerts live
- Outcome-graded eval suite with multiple trials; regression suite gates CI
- Runbook covers kill switch, budget exhaustion, tool outage, and model swap
Sources
- Anthropic — Building effective agents, Writing effective tools for agents, Demystifying evals for AI agents
- Model Context Protocol — 2026-07-28 release post, Specification 2026-07-28, Authorization, Security best practices, Python SDK on PyPI
- A2A — a2a-protocol.org, A2A joins the Agentic AI Foundation, a2a-sdk on PyPI
- Frameworks — LangGraph durable execution, LangGraph interrupts, Pydantic AI, OpenAI Agents SDK, Claude Agent SDK overview, Google ADK, CrewAI; versions from PyPI: langgraph, pydantic-ai, openai-agents, claude-agent-sdk, google-adk, crewai
- Temporal — OpenAI Agents SDK integration, Pydantic AI durable execution with Temporal, Temporal releases, temporalio on PyPI
- OWASP — Top 10 for LLM Applications 2025, Top 10 for Agentic Applications 2026
- OpenTelemetry — GenAI semantic conventions, GenAI agent spans (semantic-conventions-genai), OpenInference
- Guardrails and sandboxing — Llama Guard 4, NeMo Guardrails, OpenAI Agents SDK guardrails, gVisor
- Evals — Inspect agents, RAGAS metrics