MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

TL;DR

Model Context Protocol (MCP) is the standard Anthropic published at the end of 2024 and which by 2026 has become the dominant protocol for connecting AI agents to external tools and data. Its value, the reason the whole industry adopted it in under 18 months, is that it solves a combinatorial problem: before MCP, integrating M AI apps with N tools required M×N ad-hoc integrations; with MCP, M + N. It is the same move the Language Server Protocol made in 2016 for code editors. The architecture is three well-defined roles, Host (the AI app), Client (the connection, one per server) and Server (the piece that exposes capabilities); the primitives are six, three on the server side (Tools, Resources, Prompts) and three on the client side (Sampling, Roots, Elicitation); the protocol is JSON-RPC over two transports, stdio for local processes and Streamable HTTP for remote. The operational challenge appears when there are 10-20 MCP servers running simultaneously, each with several tools, connected to an agent that chains multistep calls: observing what happens, where things fail, how much each tool costs, which tenant invokes what becomes critical. The ecosystem’s answer in 2026: the new OpenTelemetry GenAI semantic conventions for MCP (already stable), trace context propagation via params._meta (because JSON-RPC does not bring it natively), FastMCP with built-in OTel instrumentation, MCP Gateways as a centralised layer (Traefik Hub, MintMCP, OpenObserve), and MCP Inspector for interactive debugging. This article walks the architecture from the outside in, puts each concept in its exact place, and goes down to the detail of observability: traces, RED metrics, real use cases and pitfalls.

This is the third post in the post-tracing series. Previous posts: Evals and Guardrails. Here we go down to the protocol that connects agents to tools, and how to see it in production.

The master analogy (in three versions)

MCP is a communication protocol. Like any protocol, it is best understood with the right analogy. I am going to give you three, because each one lights up a different facet and the combination leaves you understanding it better than any technical definition.

Version 1 — The USB-C of AI apps (the official one)

This is the analogy Anthropic adopted when presenting it. Before USB-C, every electronic device had its own connector. Your phone had microUSB or Lightning, your laptop a proprietary power port, your headphones a 3.5mm jack, your external disk USB-A at one end and mini-USB at the other. The result: three boxes full of specific cables that got lost, none of them good for two things, and buying a new device meant buying new accessories.

USB-C changed that. A single physical connector that many protocols cross: data (USB 3, USB 4, Thunderbolt), video (DisplayPort), power (Power Delivery), audio. You connect anything to anything and it works; the protocols negotiate above.

MCP plays the same role for AI apps. Before MCP, every application that wanted to integrate tools with an LLM, Claude Desktop, Cursor, Continue, in-house custom agents, invented its own way of doing it. Every tool vendor had to write N different integrations, one per app. The result: massive fragmentation, a lot of duplicated code, integrations that broke when an app changed its internal API.

With MCP, the connector is one: any app that speaks MCP can use any MCP tool. Just as your USB-C talks to printers, monitors and disks without the printer “knowing” whether the cable is connected to a Mac or to a Linux box.

Version 2 — The LSP of code editors (the most technically precise)

This is my favourite because the analogy is structurally identical, not just metaphorical.

Until 2016, if you wanted your code editor to support a new language, Rust, Go, TypeScript, somebody had to write a specific plugin for your particular editor. VSCode had its Rust plugin, IntelliJ a different one, Vim another, Emacs another. Every decent feature (go-to-definition, autocompletion, refactoring) was an implementation duplicated N times. M editors × N languages = M·N integrations.

In 2016 Microsoft proposed the Language Server Protocol (LSP): each language implements a single “language server” (a process that understands that language); each editor implements a single LSP client; when you work with Rust code in VSCode, VSCode launches rust-analyzer as a subprocess and talks LSP to it over stdio. Any LSP editor + any LSP server = it works. M + N.

MCP is literally this pattern, transferred from “editor + language server” to “AI app + tool provider”. And it shares even the technical detail: both pass JSON-RPC over stdio (among other transports). When Anthropic designed MCP, they looked at LSP. Anyone coming from the world of editors and IDEs will find MCP familiar.

Version 3 — The operating system driver (the operational one)

Finally, an analogy that helps you understand what a specific MCP server does.

An operating system does not know directly how to talk to your particular HP LaserJet printer. What it knows is a generic interface: “print document”, “query status”, “cancel job”. The printer driver is the piece that translates that generic interface into the proprietary commands of your specific printer.

An MCP server does exactly the same:

• Your AI agent knows a generic interface: invoke a tool with a defined schema, read a resource by URI, ask for a prompt template by name.
• The MCP server is the driver: it translates those generic operations into the concrete APIs of the underlying system, your PostgreSQL database, your filesystem, your GitHub API, your Stripe.

This leaves the AI agent free from knowing how it authenticates with GitHub, what exact SQL PostgreSQL uses, what endpoints Stripe has. It speaks MCP; the server takes care of the details.

With the three analogies combined: MCP is the layer between the LLM and the world, a standard USB-C implemented like LSP in JSON-RPC, with each server acting as a driver for one concrete underlying system.

What concrete problem MCP solves

Before going down to the architecture, it is worth pinning down the specific problem MCP solves, because without that many design decisions look arbitrary.

The problem is the quadratic cost of integrations.

Imagine you have M applications that use LLMs (Claude Desktop, Cursor, Continue, ChatGPT Desktop, your own custom agent, and so on) and N external tools those LLMs could use (filesystem, GitHub, Slack, PostgreSQL, Jira, Notion, and so on). Without a standard:

• Each (application, tool) pair requires a specific integration.
• Every time the application changes its internal API, N integrations have to be updated.
• Every time the tool changes its API, M have to be updated.
• For your new tool to be adopted, you have to write M integrations.
• For your new application to support the ecosystem, you have to write N.

The real result in 2023-2024: massive fragmentation. OpenAI’s function calling was not compatible with Anthropic’s tool use; every framework (LangChain, LlamaIndex, dspy) had its own wrapper; Claude Desktop plugins did not work in Cursor; and so on.

MCP breaks the quadratic. Each application implements the protocol once; each tool implements the protocol once; any pair works. M + N.

It is exactly what happened with USB-C, with LSP, with SQL (before there were proprietary APIs per database), with POSIX (before there were proprietary APIs per operating system). The pattern repeats because it always solves the same kind of problem.

The architecture: three roles, clearly placed

We are going to pin down where each thing lives, because mixing up the roles is the number one source of confusion in MCP.

Three roles. Let us pin down what each one does and where it physically lives.

Host: the AI application

The Host is the application the user opens. Claude Desktop, Cursor, Continue, ChatGPT Desktop, a custom agent your team builds, a VSCode extension. What the user perceives as “the product”.

The Host is responsible for:

• Deciding which MCP servers to connect (configured by the user in a file or via the UI).
• Launching or connecting to each MCP server.
• Creating one MCP Client per server (it is 1:1, they are not shared).
• Embedding the LLM (or calling it via API) that takes the decisions about which tools to use.
• Mediating the user’s authorisation for sensitive actions (showing the human “the agent wants to run tool X, do you allow it?”).

Important: the LLM lives inside the Host, not in the servers. The servers are dumb; they execute operations when asked. The reasoning (“should I call this tool now?”) lives in the host’s LLM.

Client: the connection, one per server

An MCP Client is a specific connection between the Host and a Server. If your Host has 5 MCP servers configured, it has 5 clients, not one shared. Each client:

• Maintains its socket or stdio pipe with the server.
• Negotiates capabilities in the initial handshake (which protocol version, which primitives both support).
• Serialises JSON-RPC requests to the server and deserialises responses.
• Is the point where the Host invokes operations on the server.

The 1:1 client-server separation matters because it lets each server have its own session state, its specific permissions and its independent authenticated context. There is no multiplexing in the client.

Server: the piece that exposes capabilities

The MCP Server is the piece that implements the tool-provider side of the protocol. It receives JSON-RPC from the client, processes it, executes the action against the underlying system and returns a response.

There are two flavours physically:

• Local server: it starts as a subprocess of the Host and communicates over stdio. Its lifecycle is the Host’s (when you close Claude Desktop, the local servers die). Typical model: your Host launches node filesystem-mcp-server.js as a child.
• Remote server: it runs as an independent service, reachable over HTTP. Multi-tenant, authenticated, scalable. Typical model: a company publishes https://mcp.acme.com/v1 and many hosts connect to it.

This difference has enormous consequences for observability (we will come back to it shortly).

Summary of where each thing lives

If you get confused in a discussion, come back to this table. The number one source of errors in MCP is saying “the server” when you mean “the host”.

The two layers of the protocol

MCP separates the data layer and the transport layer. This separation is what lets the protocol work over local stdio and over remote HTTP without changing anything in the primitives.

Data Layer: JSON-RPC with MCP extensions

The data layer defines the vocabulary of the messages. It is JSON-RPC 2.0. Each message is a JSON with jsonrpc: "2.0", a method (eg tools/call, resources/read), params, and an id to correlate request with response.

On top of JSON-RPC, MCP adds:

• Lifecycle: the initial handshake (initialize, initialized) that negotiates capabilities.
• The primitives (next section): tools/*, resources/*, prompts/*, sampling/* and so on.
• Notifications: messages without a response (eg notifications/cancelled to abort a tool in progress).
• Meta-information: the params._meta field carries cross-cutting metadata by convention (trace context, request IDs).

Transport Layer: how the messages move

The transport layer defines how the JSON-RPC messages travel. Two official transports:

stdio: the client launches the server as a subprocess and they communicate over its stdin/stdout/stderr with JSON-RPC. One message per line, separated by a newline. No network, no TLS handshake, no auth (trust is inherited from the operating system itself: if you launch the subprocess, you trust it). Minimal latency (~100 μs round-trip), maximum bandwidth (memcpy, not a socket).

Use case: local servers that live on the same machine as the host. Most of the MCP servers you see in public directories are stdio.

Streamable HTTP: the client sends a POST to an HTTP endpoint on the server; the server responds with JSON, optionally opening a Server-Sent Events stream to send asynchronous notifications or long responses. Auth by bearer token, API key or custom headers.

Introduced in the November 2025 spec, it replaces the pure SSE transport of earlier versions, which had bidirectionality limitations. Use case: remote servers serving many simultaneous clients, with authentication and multi-tenancy.

Important: the primitives are the same on both transports. A tools/call is identical over stdio and over HTTP. The transport is accidental, not fundamental.

The six primitives: placed in the architecture

Here is the meat. There are six primitives in MCP. They are often confused because several look like they do similar things. The key classification: three live on the server side (the server exposes, the client consumes) and three on the client side (the client exposes, the server consumes).

Server-side: what the server gives the host

Tools are actions the server exposes. Each tool has a schema (typed parameters, description) and an implementation. When the host’s LLM decides to invoke a tool, the client sends tools/call to the server, which executes it and returns a result.

• Example: the github-mcp server exposes create_issue(repo, title, body). The host’s LLM decides “I am going to create an issue”, calls this tool, github-mcp talks to the GitHub API, returns the issue ID to the LLM.
• Architectural place: the server exposes them, the LLM consumes them.

Resources are contextual data the server exposes, addressable by URI. They are not actions; they are content reads. A resource has a URI (file:///path/to/doc.md, postgres://table/users), metadata and an endpoint to read content.

• Example: the filesystem-mcp server exposes the files in the authorised directories as resources. The LLM asks for resources/read with the URI file:///docs/api.md and gets the text.
• Architectural place: the server exposes them, the host reads them (and optionally passes them to the LLM as context).

Key difference between Tools and Resources: Tools are verbs (they execute, they modify state, they have side effects); Resources are nouns (they exist, they are read, they are idempotent). If you have something that is “search for text in files” → probably a Tool (an action). If it is “this specific file” → a Resource. The distinction matters for auditing and permissions: tools require more control.

Prompts are parameterised prompt templates the server exposes. The user or the host can invoke them to inject a conversational pattern into the model.

• Example: a code-review-mcp server exposes a prompt review_diff(diff_text, style="strict") that returns a complete, well-written prompt asking the LLM to review code.
• Architectural place: the server exposes them, the user or the host invokes them, the LLM receives them as input.

Prompts are the least used of the three primitives; many servers do not even implement them. But they let a team publish good prompts as a reusable library, separate from the agent.

Client-side: what the host gives the server

This is where MCP differs from protocols like HTTP REST: the server can also ask things of the host, it is not a one-way street. Three primitives travel in that direction.

Sampling: the server asks the host to run a generation with its LLM. That is, the server borrows the host’s LLM in order to reason.

• Example: the search-mcp server receives a query from the agent, searches its corpus, finds 50 results and needs to summarise them before returning. Instead of having its own LLM, it sends a sampling/createMessage to the client; the host passes this to its LLM, runs the generation with the user’s permissions, and returns the summary to the server.
• Architectural place: the server asks for it, the host (with its LLM and the user’s authorisation) fulfils it.
• Why it matters: the user controls which model is used, what cost is paid, what permissions apply. The server does not need its own OpenAI API key.

Roots: the host tells the server where to look. Roots are URIs (directories, repositories, namespaces) the host authorises the server to explore.

• Example: your Claude Desktop starts filesystem-mcp with roots [file:///Users/me/projects]. The server knows it must only operate inside that folder, not in /etc/passwd.
• Architectural place: the host declares them in the handshake, the server respects them.

Elicitation: the server asks the host for additional information from the human user via a structured UI.

• Example: the stripe-mcp server is about to process a refund of €5000. Before executing, it sends an elicitation/createMessage to the client; the host shows the user “Confirm this €5000 refund” with a button; when the user confirms, it returns OK to the server, which then proceeds.
• Architectural place: the server asks, the host shows the user, the user decides, the answer goes back to the server.
• It is the key primitive for human-in-the-loop on sensitive actions.

Visualising the flow of the six primitives

                  HOST                                 SERVER
                    │                                     │
   Server-side ─────┼─────────────────────────────────────┤
                    │                                     │
   tools/list ──────┼─────── asks what tools exist ──────▶│
                    │◀────── returns the list ────────────│
                    │                                     │
   tools/call ──────┼────────── run this tool ───────────▶│
                    │◀────── result ──────────────────────│
                    │                                     │
   resources/read ──┼────────── read this URI ───────────▶│
                    │◀────── content ─────────────────────│
                    │                                     │
   prompts/get ─────┼─────── give me this prompt ────────▶│
                    │◀────── compiled prompt ─────────────│
                    │                                     │
   Client-side ─────┼─────────────────────────────────────┤
                    │                                     │
   sampling ────────│◀────── I need a generation ─────────│
                    │── use my LLM ───┐                   │
                    │── returns ──────▼──────────────────▶│
                    │                                     │
   roots ───────────┼───── declared in the handshake ────▶│
                    │                                     │
   elicitation ─────│◀────── ask the user X ──────────────│
                    │── shows UI ────┐                    │
                    │── confirms ────▼───────────────────▶│

JSON-RPC in action: a concrete example

To make the theory concrete, here is a real MCP conversation between a client and the filesystem-mcp server:

// 1. Initial handshake (client → server)
{
  "jsonrpc": "2.0", "id": 1, "method": "initialize",
  "params": {
    "protocolVersion": "2026-03-01",
    "capabilities": {
      "sampling": {},                     // this client supports sampling
      "roots": { "listChanged": true }
    },
    "clientInfo": { "name": "ClaudeDesktop", "version": "1.2.0" }
  }
}

// 2. The server responds with its capabilities
{
  "jsonrpc": "2.0", "id": 1, "result": {
    "protocolVersion": "2026-03-01",
    "capabilities": {
      "tools": { "listChanged": true },
      "resources": { "subscribe": true, "listChanged": true },
      "prompts": {}
    },
    "serverInfo": { "name": "filesystem-mcp", "version": "0.5.2" }
  }
}

// 3. The client asks for the list of tools
{
  "jsonrpc": "2.0", "id": 2, "method": "tools/list"
}

// 4. The server returns its tools with a schema
{
  "jsonrpc": "2.0", "id": 2, "result": {
    "tools": [
      {
        "name": "read_file",
        "description": "Read a file from the filesystem",
        "inputSchema": {
          "type": "object",
          "properties": { "path": { "type": "string" } },
          "required": ["path"]
        }
      },
      { "name": "write_file", "description": "...", "inputSchema": {} },
      { "name": "list_directory", "description": "...", "inputSchema": {} }
    ]
  }
}

// 5. The LLM decides to call read_file; the client sends tools/call
{
  "jsonrpc": "2.0", "id": 3, "method": "tools/call",
  "params": {
    "name": "read_file",
    "arguments": { "path": "/Users/me/projects/notes.md" },
    "_meta": {                            // ← the extension where trace context will go
      "traceparent": "00-abc123...-def456-01"
    }
  }
}

// 6. The server returns the file content
{
  "jsonrpc": "2.0", "id": 3, "result": {
    "content": [
      { "type": "text", "text": "# My notes\n\n..." }
    ]
  }
}

The important thing to note: params._meta. That is the bag where MCP conventionally passes cross-cutting metadata, including trace context. We will come back to it shortly.

The observability problem: why traditional tracing is not enough

So much for theory. Let us go down to the operational problem: in a 2026 production cluster, a typical agent has 5-15 MCP servers connected simultaneously, each with 5-20 tools, and each conversation with the agent can generate dozens of chained tool calls. Without observability, debugging incidents is impossible.

Why generic tracing (Hubble, OTel without MCP conventions) is not enough:

Stdio is not visible on the network. Local servers talk over OS pipes. Your Hubble or your Datadog APM see nothing; there are no packets to capture. AgentSight (seen in the previous post in the eBPF series) with stdiocap captures it but gives you the raw JSON-RPC, without semantic context (which tool it is, which resource, which prompt).

Generic HTTP does not understand MCP either. If you trace the HTTP to a remote MCP server without MCP conventions, you see a POST to /v1 with an opaque JSON-RPC body. You lose “which tool was invoked”, “what arguments”, “was it elicitation or sampling”. RED metrics per endpoint are no use to you; you need RED per tool.

JSON-RPC does not propagate trace context natively. Unlike HTTP (the W3C traceparent header) or gRPC (metadata), JSON-RPC has no standard field for trace context. If you do not propagate it, every call to the server starts a new trace disconnected from the agent’s trace.

Multistep multi-server is very hard to follow. A single user conversation can translate into: 1) a call to github-mcp get_pr; 2) a call to filesystem-mcp read_file for several files; 3) a call to the main LLM with all the context; 4) a call to postgres-mcp query; 5) a call to slack-mcp send_message. Without propagated trace context, those are five disconnected traces. With propagation, it is one tree.

The solution: OpenTelemetry semantic conventions for MCP, already stable in 2026.

OpenTelemetry semantic conventions for MCP

The GenAI MCP semantic conventions are the set of standardised attributes for MCP-related spans and metrics. They were published as part of the GenAI subgroup of the OpenTelemetry SIG and are the first part of the GenAI semantic conventions to reach stable.

Why MCP-specific semantic conventions

Before they existed, teams instrumented MCP with the generic RPC semantic conventions (the ones you would use for gRPC or XML-RPC). It half worked. The MCP-specific conventions add:

• Attributes to identify which primitive was executed (mcp.method.name = "tools/call").
• Attributes to identify which specific tool/resource/prompt was touched (mcp.tool.name, mcp.resource.uri, mcp.prompt.name).
• Attributes for the bidirectional flow (sampling/elicitation requests from the server to the client).
• Attributes for the handshake (mcp.protocol.version, mcp.client.name, mcp.server.name).
• Standardised RED metrics per tool (mcp.tool.call.duration, mcp.tool.call.errors).

The canonical attributes

The attributes any MCP-aware instrumentation should emit:

The last two come from the generic GenAI semantic conventions and apply when the MCP call involves sampling (the server using the client’s LLM).

RED metrics per tool

Beyond spans, the semantic conventions define three core metrics:

• mcp.tool.call.duration (histogram): the latency of each invocation.
• mcp.tool.call.count (counter): the total number of invocations.
• mcp.tool.call.errors (counter): errors per tool.

Labelled with mcp.tool.name, mcp.server.name, mcp.client.name. Pivotable in Grafana to answer “which tool is the slowest”, “which tool fails most”, “which client loads which server hardest”.

Trace context propagation: the params._meta trick

JSON-RPC has no headers like HTTP, so MCP cannot use the W3C traceparent header directly. The solution the ecosystem has converged on: propagate trace context in params._meta.

When the MCP client sends a tools/call, its OTel instrumentation does:

import json
from opentelemetry.propagate import inject

carrier = {}
inject(carrier)  # fills it with traceparent/tracestate from the active span

params = {
    "name": "read_file",
    "arguments": {"path": "/notes.md"},
    "_meta": carrier,                    # ← propagates trace context
}

When the server receives it, it does the symmetrical thing:

from opentelemetry.propagate import extract

ctx = extract(request.params.get("_meta", {}))
with tracer.start_as_current_span("tools/call", context=ctx):
    # this span is a child of the client's
    return execute_tool(request.params)

The result: the server’s span is a child of the client’s span in the trace tree. When you look at the trace in Tempo or Phoenix, you see the whole chain: user → host → client → server → execution → response → client → host → response to the user.

This requires both ends to instrument consistently. If the server does not extract the context, you see disconnected spans but at least you have traceability on the client side.

Instrumentation patterns

There are three routes to instrumenting MCP, in increasing order of effort:

1. FastMCP with built-in OpenTelemetry

FastMCP is one of the most used Python frameworks for building MCP servers. It ships built-in OpenTelemetry instrumentation: every tool, resource template and prompt operation generates spans automatically with the correct MCP conventions.

from fastmcp import FastMCP
from opentelemetry.sdk.trace.export import OTLPSpanExporter

mcp = FastMCP("my-server", otel_endpoint="https://otel-collector:4318")

@mcp.tool()
def search_docs(query: str) -> str:
    """Search the corpus for matching documents."""
    # this automatically generates a span with
    # mcp.tool.name=search_docs, mcp.method.name=tools/call, etc.
    return run_search(query)

Zero instrumentation code. Spans with the correct conventions. It is the recommended pattern if you are starting an MCP server in Python from scratch.

2. Manual OpenTelemetry SDK

For servers that already exist or are in other languages (TypeScript, Go), the option is to instrument manually with the standard OTel SDK and emit the conventional MCP attributes:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

async def handle_tools_call(req: JSONRPCRequest):
    ctx = extract_trace_context(req)
    with tracer.start_as_current_span("mcp.tools.call", context=ctx) as span:
        span.set_attribute("mcp.method.name", "tools/call")
        span.set_attribute("mcp.tool.name", req.params["name"])
        span.set_attribute("mcp.server.name", "filesystem-mcp")
        try:
            result = await execute_tool(req.params)
            return result
        except Exception as e:
            span.set_attribute("mcp.error.code", -32603)
            span.record_exception(e)
            raise

More boilerplate but it works with any existing server.

3. MCP Inspector for interactive debugging

MCP Inspector (official) is a tool for interactive protocol-level debugging. It launches a local proxy (port 6277) between your client and the server, and opens a web UI (port 6274) where you see every JSON-RPC message back and forth in real time.

It is not production observability, it is development and debugging. But it is irreplaceable during the bring-up of a new server: you see exactly which requests arrive, which responses are returned, which errors occur. It saves hours of ad-hoc logging.

MCP Gateways: the centralised piece for enterprise

When your organisation has many agents connecting to many MCP servers, managing the connection matrix becomes operationally serious. The natural question, “can there be a proxy in front of all the MCP servers that centralises auth, rate limiting, logging and observability?”, already has an answer: MCP Gateways.

An MCP Gateway is a proxy that:

• Accepts MCP connections from the hosts/agents.
• Routes them to the corresponding backend MCP servers.
• Applies centralised authentication and authorisation (which agent can call which tool).
• Applies rate limiting per agent, per tool, per tenant.
• Observes: it emits OTel metrics for every operation passing through.
• Propagates the identity of the agent to the backend server (with several models: token forwarding, token exchange, impersonation).

The options that have established themselves in 2026:

• Traefik Hub MCP Gateway — from the Traefik team. Declarative configuration, native integration with Traefik’s Kubernetes/Helm ecosystem.
• MintMCP — a gateway focused on observability and multi-tenancy. SaaS and self-host.
• OpenObserve MCP Gateway — integrated with the OpenObserve observability platform.

For small deployments (one team, few agents) a Gateway can be overkill. For enterprise (dozens of agents, dozens of servers, regulated compliance), it is practically mandatory.

Real use cases for MCP observability

Let us land this with five cases where properly instrumented MCP observability delivers immediate value:

1. Audit per tool, per tenant, per agent

Question: “who ran the delete_repo tool last month?”. Without MCP observability, impossible. With OTel conventions + identity propagation: a query in your trace backend filtering by mcp.tool.name="delete_repo", grouping by mcp.client.name or by the user_id propagated in _meta. Compliance happy.

2. Cost per tool and per tenant

Question: “how much does each tool cost?”. If the tools invoke external APIs (Stripe, OpenAI sampling) or consume significant resources (a GPU for an inference tool), knowing their aggregated cost matters. With mcp.tool.call.duration + gen_ai.usage.* aggregated by tool and tenant, you build cost accountability dashboards without instrumenting anything extra.

3. Debugging multistep chains that fail

Question: “the agent failed to complete this task, where did it go wrong?”. The propagated trace connects: the user’s span → the LLM’s span with its CoT → the spans of each tool invoked → the final LLM span. If the chain broke at the third tool, in Tempo you see the red span with the specific error message. Reproducing the failure is trivial.

4. Tool latency and degradation

Question: “which tool is degrading?”. RED metrics per tool in Grafana show p95/p99 latency over time. When a tool starts climbing from 200ms to 800ms (because the underlying service is collapsing), you see it before the users complain.

5. Detecting loops and agentic anomalies

Question: “is any agent stuck in a loop?”. If an agent calls tools/call read_file 80 times in 30 seconds for the same path, something is clearly wrong. An alert on mcp.tool.call.count grouped by (session_id, tool_name) detects this. Combined with loop detection at the reasoning level, it closes the circle.

Operational pitfalls

Lack of identity propagation

Your Gateway authenticates the agent, but passes requests to the backend without propagating identity. The result: the backend’s logs say “service-account” for everything, and it is impossible to audit who invoked what. Choose a propagation strategy early: token forwarding (simple, exposes tokens to the backend), token exchange (more secure), or impersonation with cross-logging.

Stdio servers that do not show up in your APM

This is the field’s number 1 pitfall. Your Cursor agent uses filesystem-mcp over stdio; you see nothing in Datadog because there is no network traffic. Solution: instrument the stdio server with an OTel SDK that exports over OTLP to your collector (via gRPC or HTTP; the OTel collector can receive even though the server talks stdio to its client). Or use AgentSight stdiocap to capture the raw JSON-RPC and process it offline.

Multiple protocol versions in production

Different clients use different MCP versions simultaneously. Your metrics dashboard mixes apples and oranges. ALWAYS label with mcp.protocol.version and filter/group by it.

_meta lost when passing through a proxy

Your Gateway accepts the client’s request, rewrites it for the backend, and forgets to copy params._meta. The result: the trace is broken at the Gateway, two disconnected traces. Make sure your Gateway preserves or re-injects trace context at each hop.

Trace volume with chatty servers

Some MCP servers emit many small operations (filesystem listings, partial reads). Without sampling, they fill your backend with useless traces. Apply tail-based sampling that keeps complete sessions or keeps only traces with errors or high latency.

Cardinality in metrics

mcp.tool.call.duration with mcp.session.id as a label blows up cardinality. Do not include unique per-session IDs in labels; keep cardinality under control with labels that take few discrete values (tool name, server name, client name, error code).

Confusing client and server spans

When you look at the tree, distinguish: the client sees the total latency from its perspective (network included); the server sees only its own work. If you look only at the server span to debug the latency perceived by the user, you miss the RTT. Use both.

What we have not covered

• Experimental WebSocket MCP transport: an alternative to Streamable HTTP, not yet standard.
• MCP servers in cloud-native deployments with sidecars: an emerging pattern of deploying MCP servers as pod sidecars.
• MCP federation: composing several servers as one (similar to GraphQL federation).
• eBPF + MCP: how AgentSight’s stdiocap and Cilium’s hooks complement native instrumentation.
• MCP testing and contract tests: how to validate that your server complies with the spec.

References

Specification and concepts:

• Model Context Protocol — sitio oficial — entrada canónica.
• MCP architecture overview.
• Transports — MCP docs.
• MCP Inspector (GitHub) — debugging interactivo.

OpenTelemetry GenAI MCP:

• Semantic conventions for Model Context Protocol — OpenTelemetry — referencia normativa.
• Adding OpenTelemetry Trace Support to MCP (Discussion #269) — historia de la propuesta.
• How to Instrument MCP Servers with OpenTelemetry (OneUptime).
• How to trace MCP server tool calls with OpenTelemetry and Elastic APM.
• MCP Observability with OpenTelemetry (SigNoz).
• Distributed tracing for agentic workflows (Red Hat Developer).
• OpenTelemetry for AI Agents in MCP Workflows (MintMCP).

Frameworks and gateways:

• FastMCP OpenTelemetry — instrumentación built-in.
• Traefik Hub MCP Gateway — gateway de Traefik.
• MintMCP — gateway con foco en observabilidad.
• OpenObserve MCP Gateway guide.
• What is an MCP Gateway (DEV Community).
• OpenTelemetry MCP Server (Traceloop) — el patrón inverso: usar MCP para que agentes consulten traces OTel.

Cross-references:

• Post anterior: Guardrails y safety.
• AgentSight y el nuevo tracing de LLMs — donde se introdujo stdiocap para capturar stdio de servidores MCP locales.
• Evals: la capa después del tracing.