The first time an AI feature breaks in production, it usually does not announce itself as a model problem. It shows up as a slow answer, a strange tool call, a sandbox that never came back, or a bill that suddenly looks wrong.
That is when the useful question becomes what happened before the answer came back. You need to know which agent ran, which tool was called, whether the gateway chose a different provider, whether the workflow retried, and whether the model failed or simply received bad input from the step before it.
This is why AI observability cannot stop at a single LLM request. The model call is one part of the story. The infrastructure around it is where the production behavior shows up, from retries and queues to tool failures, token spikes, runtime errors, container logs, workflow state, and provider latency.
We think about this as AI infrastructure observability. Not because it needs a bigger name, but because the thing being operated is now bigger than the model. You need to ask questions across the whole path and follow the evidence without rebuilding telemetry every time the architecture changes.
Why the categories are useful
The AI stack is easier to operate when you separate it by the kind of work each layer does. Otherwise everything turns into the AI is broken, which is not a diagnosis. It is a shrug.
Agent frameworks carry application intent. They know which agent ran, which task started the run, which tool was called, and where the agent spent time. Sandboxes and runtimes are where those decisions become execution. They run code, keep workflows durable, and expose the operational state around long-running jobs. LLMs and gateways sit closer to the model path. They tell you which model was called, how tokens were used, how long providers took, and where routing or fallback behavior changed.
These layers fail in different ways, so they should not all be flattened into one vague stream of events. A framework can show a tool call failure. A sandbox can show what the code printed before it exited. A runtime can show a workflow stuck in a queue. A gateway can show provider latency or model-level errors. When each layer is collected with the right signal and brought into the same place, the debugging path gets shorter.
That is how we at Parseable think about the AI infrastructure ecosystem across agent frameworks, sandboxes and runtimes, LLMs, and gateways. Each category answers a different kind of production question.
Agent frameworks
Agent frameworks are usually where the story begins. They sit close to the application logic, so they can tell you what the user asked for, which agent handled it, which model was selected, which tool was called, and how the run moved from one step to another.
That context is hard to recover later if you do not collect it while the run is happening. Agent failures are often not clean exceptions. The model may pick the wrong tool. The tool may return valid data that sends the agent down a bad path. The agent may retry and eventually succeed, but burn far more tokens than expected. The final answer may look fine while the execution underneath it was anything but fine.
Parseable supports agent framework telemetry through integrations such as Pydantic AI, CrewAI, Mastra, n8n, LangChain, LlamaIndex, AutoGen, and DSPy. The point is not to make every framework look the same. The point is to keep the useful context from each one and make the run inspectable through logs, metrics, and traces.
Pydantic AI is a natural fit when the application already uses Pydantic AI agents and wants agent, model, and tool-call telemetry through OpenTelemetry. CrewAI helps when the run is made of crews, agents, tasks, LLM calls, and tool execution. Mastra gives TypeScript teams a view into agent and workflow activity. n8n is useful when AI nodes and automation steps are part of the same operational path.
These are not just LLM logs. They are records of execution. They tell you how the application behaved, not only what the model returned.
Sandboxes and runtimes
Once an agent can call tools, run code, or coordinate long-running work, the interesting part often moves outside the framework. You need to see the environment where the work happened.
A code sandbox can tell you what ran, what failed, what was printed, and what resources were used. A durable workflow engine can keep state across retries and restarts, but you still need to know which step ran, which activity failed, and whether work is backing up. A serverless runtime can hide the machine from you, but it still produces function logs, spans, cold starts, retries, and resource signals.
Parseable has guides for systems such as Daytona, Temporal, DBOS, Restate, and Modal. Each one has a different telemetry path because each system exposes a different operational shape.
Daytona is a good example when agent work needs a sandbox where code can run. You want to see sandbox logs, traces, and metrics, not just that a tool call was attempted. Temporal, DBOS, and Restate fit the durable execution side of the stack, where work may retry, pause, resume, or survive process failure. Modal fits the serverless Python side, where teams need visibility into function execution without pretending the runtime is invisible.
This layer deserves attention because AI systems increasingly do work, not just generate text. They call APIs, run scripts, schedule workflows, wait on other systems, and retry. If that execution layer is invisible, the agent will look unreliable even when the real problem is lower in the stack.
LLMs
The LLM layer is where most teams start, and that makes sense. It is the obvious place to ask which model was used, how long the request took, how many tokens were consumed, and whether the provider returned an error.
That view is necessary. It is just not enough on its own.
Parseable supports direct model and SDK-level integrations such as LiteLLM SDK, OpenAI, Anthropic, and vLLM. These are useful when the model call itself is the primary object of investigation.
For hosted APIs, the questions are often about latency, token usage, model behavior, and error responses. For self-hosted inference with vLLM, the questions also include serving behavior, throughput, GPU-backed runtime signals, and the health of the inference layer.
The LLM layer gives you the model-facing view. It tells you whether the model request was slow, expensive, failed, or routed in a way you did not expect. When combined with agent and runtime telemetry, it becomes more useful because the model call can be placed inside the larger execution path.
Gateways
Gateways are becoming a control point for AI infrastructure. They sit between applications and providers, so they are a natural place to manage routing, keys, budgets, fallbacks, policies, and provider-level telemetry.
This is where integrations such as OpenRouter and LiteLLM Gateway fit. They help teams observe model traffic without forcing every application to carry the same instrumentation burden.
OpenRouter Broadcast can send traces for requests that pass through OpenRouter, which is useful when teams want model and provider visibility from the gateway layer. LiteLLM Gateway gives teams a way to observe gateway traces and metrics around routing and provider calls.
The gateway view is useful because it can sit above many applications. If ten services call models through the same gateway, the gateway becomes one place to see provider latency, routing behavior, failures, token usage, and traffic patterns. It does not replace application-level traces, but it gives platform teams a central view of model traffic.
What this gives you
Once these layers start sending telemetry into one place, teams get a clearer way to follow the path of an AI request. The next question is where that data lives and who owns it.
Data sovereignty in the AI era
AI telemetry can be sensitive. Prompts, completions, tool inputs, trace attributes, logs, user context, workflow metadata, and model payloads may all carry information that teams do not want scattered across disconnected systems.
This is where Parseable's architecture goes beyond dashboards. Parseable stores telemetry in an open, object-storage-backed architecture. Teams can use the object storage of their choice, including S3-compatible storage, and keep control over where the data lives. The data is compressed and stored for high-volume telemetry, instead of assuming that every byte must sit in an expensive proprietary backend forever.
For platform teams, this becomes a governance conversation as much as an observability one. You need to decide how long to keep sensitive telemetry, where it lives, who can access it, and what is allowed to leave your boundary. Data sovereignty is not a slogan when prompts, tool inputs, workflow logs, and model traffic start flowing through the same system. It becomes part of the operating model.
Controlling the storage layer gives teams more say in those decisions. It also makes long-retention observability more realistic, especially when agent traces, workflow logs, sandbox output, and model traffic all start growing together.
Where to start
The easiest way to start is to pick the layer you already depend on most.
If your application is built around an agent framework, begin with the framework guide. If agents run code or long workflows, look at the sandbox and runtime guides. If the biggest unknown is model traffic, start with the LLM or gateway path. You do not need to instrument the entire AI stack on day one.
But it helps to keep the full stack in mind. AI systems are becoming a chain of frameworks, runtimes, model calls, gateways, and infrastructure. Observability should follow that chain.
Parseable's AI infrastructure integrations are built around that idea. Each layer may emit telemetry differently through OTLP, Prometheus Remote Write, logs, or structured JSON, and that is fine. The useful part is bringing those signals into one place so teams can ask sharper questions across the system, keep the data under their control, and understand what happened when an AI system made a decision.
You can explore the full set of integrations in the AI Infrastructure documentation.
