Sessions, Traces & Spans in LLM Observability: One of These Isn't a Structural Level

Sessions, Traces & Spans in LLM Observability: One of These Isn't a Structural Level

Datadog's terms page, the #1 Google result for LLM observability sessions traces spans, defines two of those three words. Not three. The missing one maps to gen_ai.conversation.id, and the reason it's missing is that the OpenTelemetry spec never made it a structural level. If you need the case for observability itself, start here. This post picks up where that one stops: the data model.

Key Takeaways

• Spans nest inside traces; traces group into sessions. The nesting runs innermost-out: span, then trace, then session.
• A span is one timed operation. A trace is one end-to-end request. A session is one multi-turn conversation.
• OpenTelemetry's GenAI conventions define spans and the gen_ai.conversation.id attribute. They do not define a session level.
• Trace and span IDs propagate automatically through context. The session ID does not. You set it, every turn.

Sessions vs Traces vs Spans, at a Glance

In LLM observability, a span is one timed operation (a model call, a retrieval step), a trace is the tree of spans one request produces, and a session groups many traces from the same conversation. Nesting runs inward: spans inside traces, traces inside sessions. The third grouping is the one that isn't what it looks like.

Those count and lifetime figures are typical ranges you'd expect in a RAG chatbot or an agent loop, not measurements from a controlled test. Your numbers will differ. What won't differ: the Session row is the one that isn't a structural level in the spec, and the "Sessions: The Level Your Tool Probably Invented" section proves it.

What Is a Span, and What Is a Span Kind?

A span is one timed operation with a name, a start timestamp, an end timestamp, a status code, and a bag of key-value attributes. In LLM tracing, the attributes are where the useful data lives: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.request.model tell you what the operation cost and which model ran it.

A span is one operation, not one function call

Every span carries a parent span ID pointer (empty on the root span) that builds the tree. The attribute bag is open: you attach whatever context you need. The OpenTelemetry GenAI span conventions (status: Development) require gen_ai.operation.name and gen_ai.provider.name on every GenAI span, and recommend the token-usage attributes above.

One practical rule from Datadog's terms page: LLM, Workflow, and Agent spans may serve as a root span; Tool, Task, Embedding, and Retrieval spans may not. That's Datadog's rule, not a universal one, but it's the only vendor that states it, and it saves you from building a trace that starts on a tool call with no parent.

Span kinds: the same idea, five vocabularies

Every tool needs a way to say "this span is a model call" versus "this span is a retrieval." They just don't agree on the word:

The OpenInference spec lists ten kinds. Datadog lists seven. OTel takes a third route: its GenAI attribute registry publishes 15 well-known values for gen_ai.operation.name (chat, create_agent, create_memory, create_memory_store, delete_memory, delete_memory_store, embeddings, execute_tool, generate_content, invoke_agent, invoke_workflow, plan, retrieval, search_memory, text_completion) and states that if one of them applies, that value MUST be used; a custom value MAY be used only when none fits. So it's a semi-open enum, not the absence of one. Three lists, three lengths, and no alignment between them. If you're choosing a tool, this vocabulary gap matters more than the feature list, because it's what your dashboards and alert filters will be keyed on.

What Is a Trace, and Why Does the Tree Shape Matter?

A trace is the tree of spans produced by one request. One root span sits at the top; every other span hangs below it via parent-span-ID edges. The tree shape is the whole point: a flat log tells you something was slow, but the tree tells you which step was slow and which step produced the bad output.

chat_request (root)                         2,340ms
├── retrieval                                 410ms
│   └── rerank                                 85ms
├── chat gpt-4o                             1,720ms
└── tool_call: search_calendar                190ms

Read that tree and the diagnosis is immediate: 74% of latency sat in the model call, not the retrieval. A flat log of five timestamps gives you the same total but none of the attribution.

An agent loop makes this tree deeper and wider than a plain RAG request. Each tool call spawns its own sub-tree; a five-step agent turn can easily produce 30+ spans under one root. That's normal, and it's the reason the span-granularity question below exists.

The distinction between tracing and logging matters here too: logging records events, tracing records causality. If you're still deciding what to log versus what to trace, our LLM logging best practices post draws that line.

Sessions: The Level Your Tool Probably Invented

No. A session is not a structural level in the OpenTelemetry GenAI conventions. The spec defines spans and the gen_ai.conversation.id attribute (conditionally required, "when available," status: Development), described as the unique identifier for a conversation or thread used to correlate messages. Vendors then build their own session object on top of that attribute. Nobody else on this SERP states the spec status plainly, so here it is.

The consequence is the sentence this whole post exists to deliver:

A session is a grouping key, not a parent span. It doesn't propagate the way a trace ID does; you set it yourself on every turn.

Miss one turn and that turn falls out of the session. There is no automatic context propagation for it.

When does a session start and end?

Vendor-defined. Some tools open a session on the first trace carrying a new conversation ID and close it on an inactivity timeout (Langfuse defaults to a configurable window). Others require an explicit close call. The spec says nothing about lifecycle because the spec doesn't model a session as an object.

What carries across turns, and what doesn't?

The model's context window is not the session. The session is a grouping key over independent traces. Each turn gets its own trace, its own root span, its own token counts. What carries is the conversation ID attribute you stamped on each root span. What doesn't carry: latency, token usage, span structure. Those are per-trace.

What does a session-level metric measure?

Things a single trace cannot: resolution rate (did the conversation solve the user's problem?), turns-to-answer (how many traces before the user got what they needed?), and abandoned conversations (sessions with no closing signal). Running evals on live traces at the session level is how you catch multi-turn failures that look fine turn by turn.

The code, vendor-neutral

This snippet uses only stable OTel primitives. No vendor SDK. It creates a root span for one turn, a child span for retrieval, a child for the model call, and sets gen_ai.conversation.id so three turns land in one session:

from opentelemetry import trace

tracer = trace.get_tracer("my-llm-app")

SESSION_ID = "conv-8f3a2c"  # same value on every turn

def handle_turn(user_message: str):
    with tracer.start_as_current_span("chat_request") as root:
        # You set this. It does not propagate automatically.
        root.set_attribute("gen_ai.conversation.id", SESSION_ID)

        with tracer.start_as_current_span("retrieval") as ret:
            ret.set_attribute("gen_ai.operation.name", "retrieval")
            docs = retrieve(user_message)

        with tracer.start_as_current_span("chat gpt-4o") as llm:
            llm.set_attribute("gen_ai.operation.name", "chat")
            llm.set_attribute("gen_ai.provider.name", "openai")
            llm.set_attribute("gen_ai.request.model", "gpt-4o")
            response = call_model(user_message, docs)
            llm.set_attribute("gen_ai.usage.input_tokens", 1_204)
            llm.set_attribute("gen_ai.usage.output_tokens", 312)

    return response

Call handle_turn three times with the same SESSION_ID and all three traces group under one session in any backend that reads the attribute. Change the ID and you've started a new session. That's the whole mechanism.

We Read Five Vendors' Docs Side by Side. They Don't Agree.

On 2026-07-30 we read the current data-model docs for Langfuse, LangSmith, OpenInference / Phoenix, and Datadog side by side, plus the OpenTelemetry GenAI span spec. Four of the five call the same object something different. Only one treats a session as a first-class object rather than an attribute. Datadog's terms page, the #1 Google result for this query, doesn't define a session at all.

One source note on that first row: OpenInference's session.id is not in the traces spec linked above, which covers the ten span kinds. It's defined in the sibling OpenInference semantic conventions file as the unique identifier for a session. Two files, one spec.

We didn't invent the cross-vendor comparison; FutureAGI publishes an OTel-vs-vendor table too. Our two additions are the session row (FutureAGI skips it) and the same-word-different-meaning trap: Langfuse's "observation" and LangSmith's "run" are the same object as a span, while Datadog's and OpenInference's span kinds are different vocabularies for the same idea.

Langfuse calls it an observation, LangSmith calls it a run, Datadog calls it a span. Same object, three dashboards that break when you migrate.

That's our reading of the migration cost, not a vendor claim. But it's the reason saved filters, eval configs, and alert rules keyed on "observation" or "run" stop working the day you switch tools. You're not renaming a field. You're renaming a level. If you're weighing those two specific tools, our Langfuse vs LangSmith comparison goes deeper on the divergence.

Readers may also have Opik, PostHog, Sentry, or Weights & Biases in their stack already; Google associates all four with llm tracing, and each maps these concepts slightly differently. Picking the right one? Our observability platform roundup covers the field.

One freshness note: the GenAI conventions have moved into their own repository, out of the main semantic-conventions repo. The old opentelemetry.io/docs/specs/semconv/gen-ai/ path now carries only a pointer.

Which ID Goes Where?

A trace ID identifies one request and propagates through context automatically. A span ID identifies one operation within that trace, also automatic. A correlation ID (or request ID) comes from your web tier before tracing starts, and it's the one people most often confuse with the trace ID. The session ID is the odd one out: yours to set, manually, on every turn.

The practical rule: attach gen_ai.conversation.id as a span attribute on the root span of every turn, and stamp the user ID alongside it. Skip one turn and your session-level metrics silently lose that turn.

One warning on cardinality: user IDs and session IDs are high-cardinality values. That matters for your backend's indexing bill, which is the next section's problem.

How Granular Should a Span Be?

Two failure modes, both common:

Over-spanning. A span per function call gives you a 400-span trace nobody can read and a per-span bill nobody signed off on. Hosted backends (Datadog, Langfuse Cloud) price by span volume. A chatty agent loop that instruments every string concatenation will burn through a free tier in an afternoon.

Under-spanning. One span for "the whole chain" tells you it was slow but not where. You end up adding print statements back in, which is what tracing was supposed to replace.

The rule of thumb (and it is a rule of thumb, not a measurement): span the boundaries where a decision or an external call happens.

• Retrieval step: span it.
• Rerank call: span it.
• Each model call: span it.
• Each tool call: span it.
• Each guardrail check: span it.
• Pure in-process transformations (string formatting, JSON parsing, prompt assembly): attributes on the parent span, not their own spans.

On cardinality, sampling, and retention:

• High-cardinality attributes (user IDs, full prompts) inflate storage costs. Sample or truncate them.
• Most backends let you sample at the trace level. Keep 100% of error traces; sample the happy path.
• Retention windows vary: 7 days on free tiers, 30–90 days on paid. Decide before you need the data.

For the actual cost model behind span volume and per-span pricing, see our LLM cost monitoring guide. We won't rebuild it here.

How Techsy Approaches This

For client agent work, we standardise on three rules:

1. One trace per turn. Never merge two user turns into one trace, even if the agent loops internally.
2. A session ID stamped on every root span, set in application code, never assumed to propagate.
3. Span kinds kept to a small fixed set (retrieval, inference, tool, guardrail) so dashboards survive a vendor change.

That third rule is the one teams skip, and it's the one that saves a migration. If your span vocabulary is tied to one vendor's enum, every alert and saved view breaks the day you switch.

If you're building an agent system and want a second opinion on the tracing architecture, get a free consultation.

Frequently Asked Questions

What is a span in distributed tracing?

A span is one timed unit of work: it has a name, a start time, an end time, a status, and a set of attributes. Spans link to each other through parent-span-ID references, forming a tree. In LLM applications, a span typically wraps one model call, one retrieval, or one tool invocation.

What is a span in Datadog?

In Datadog's LLM Observability, a span is the same timed operation, but Datadog adds a span kind taxonomy: LLM, Workflow, Agent, Tool, Task, Embedding, and Retrieval. Only LLM, Workflow, and Agent kinds may serve as a root span. The taxonomy is Datadog-specific; it is not part of the OpenTelemetry standard.

What are the four pillars of observability?

The four pillars are logs, metrics, traces, and (depending on whose framing) profiles or events. Traces are the pillar this post lives in. The LLM case adds a wrinkle: token usage and model identity are attributes on trace spans, not separate metric streams, which collapses what would be two pillars into one query.

What are the four golden signals for observability?

Latency, traffic, errors, and saturation. For LLM systems, latency means time-to-first-token and total generation time; traffic means requests per second per model; errors mean failed spans (status code ERROR); saturation means token-budget exhaustion or queue depth. The signals are the same; the units differ.

Is a session part of the OpenTelemetry specification?

Not as a structural level. The OTel GenAI span conventions define gen_ai.conversation.id as a conditionally required attribute ("when available") for correlating messages in a conversation or thread. It sits on spans. Vendors like Langfuse and LangSmith build their own session or thread objects on top of it.

What's the difference between a trace ID, a span ID and a correlation ID?

A trace ID identifies one request and propagates automatically through all downstream services. A span ID identifies one operation within that trace. A correlation ID (or request ID) is generated by your web tier before tracing begins and is the value people most often mistake for the trace ID. They overlap in scope but originate differently.

How many spans should one trace have?

There's no fixed answer, but typical ranges are 3–30 for a RAG request and 10–50+ for an agent loop with multiple tool calls. The rule of thumb: span external calls and decision points, not in-process transformations. If your trace exceeds 100 spans, you're likely over-instrumenting.

Are Langfuse "observations" the same thing as spans?

Yes. A Langfuse observation is the same object as an OTel span: one timed operation with attributes. Langfuse splits observations into three types (generation, span, event) where OTel uses gen_ai.operation.name. If you're evaluating tools that read your traces, our LLM evaluation tools roundup covers which ones accept both vocabularies.

How do you group a multi-turn chatbot conversation into one session?

Set the same conversation identifier on the root span of every turn. In OTel terms, that's gen_ai.conversation.id. In Langfuse, you pass a session_id when creating traces. In LangSmith, you set session_id or thread_id metadata. Miss one turn and that turn falls out of the grouping.

Do I need sessions if I only ever handle single-turn requests?

Probably not. Sessions exist to correlate multiple traces into one conversation. If every request is independent (a classification API, a one-shot summariser), trace-level metrics are sufficient. Add sessions when you need cross-turn metrics: resolution rate, turns-to-answer, or conversation-level cost. Our LLM evaluation guide covers when session-level evals earn their keep.

The Short Version

Spans nest inside traces; traces group into sessions. The nesting is real, but the spec only structures two of the three levels. gen_ai.conversation.id is an attribute you set yourself, not a parent span that propagates. And the vendor you pick today names these objects differently from the vendor you'll switch to in 18 months, so keep your span vocabulary small and portable.

If you're choosing a platform, start with our observability platform comparison. If you're building evals on top of your traces, the LLM evaluation guide picks up from here.