LLM Logging Best Practices: 9 Rules We Follow in Production [2026]

LLM Logging Best Practices: 9 Rules We Follow in Production [2026]

These nine LLM logging best practices are the rules our production stack actually runs: we log 1.2M LLM requests a month across four services, and every one lands in Grafana Loki as a single JSON line with model, tokens, latency, cost_usd, and trace_id. structlog 25.4.0 writes the record, Presidio strips the PII first, and the whole pipeline is the logging half of our observability stack.

Key Takeaways

• Log every LLM request as structured JSON with 14+ named fields, never free text.
• Redact PII before the log write with Presidio or equivalent, not after.
• Attach OpenTelemetry GenAI semantic convention attributes to every trace.
• At 1M requests/day, the same 60GB costs $108/month in Datadog, $30 in Loki, $1.20 in ClickHouse.

What LLM Logging Actually Means (and Why "Just Log Everything" Fails)

LLM logging means capturing a structured record of every model request and response: the prompt, the completion, token counts, latency, cost, and the trace that ties it all to a user session. It is not infrastructure logging. CPU, memory, and pod restarts belong in your metrics stack; this post covers only the request-level record that lets you debug, cost, and audit model behavior.

The "just log everything" instinct dies hard, and it is expensive. Full prompts and completions at 1M requests a day produce roughly 60GB of text a month, and a chunk of that text is customer PII you are now storing indefinitely. GDPR's Article 5 data minimisation principle requires that personal data be "adequate, relevant and limited to what is necessary," and a raw prompt dump fails that test on day one. Logging everything is not a strategy; it is a liability with a monthly invoice.

What Are the 9 LLM Logging Rules?

Nine rules, in the order we would implement them: log full prompts and responses with hashed identifiers, emit structured JSON, capture tokens and cost per request, attach OpenTelemetry trace context, redact PII before the write, sample at high volume, set retention tiers, separate safety events, and make the result queryable. Each one below comes with the code or the table that enforces it.

Rule 1: Log the Full Prompt and Response (With Hashes, Not Raw PII)

Log the complete prompt and the complete completion for every request, because partial logs are how you end up staring at an incident with no record of what the model actually saw. The one exception is identity: never write raw user IDs, emails, or names into the record. Store a SHA-256 hash of the user ID instead. A hash still lets you reconstruct one user's full session history with an offline lookup, while the log line itself stays useless to anyone who should not read it. Same logic for system prompts: hash them, log the hash, and keep the plaintext in your prompt registry where it is already version-controlled.

Rule 2: Use Structured JSON: Every Field Named, Nothing Free-Text

For LLM logging best practices in Python or any other language, structured logging in JSON is the non-negotiable one: every field named, typed, and queryable, nothing dumped as a formatted string. A free-text line like INFO called gpt-4o, took 812ms can only be grepped. A JSON record can be aggregated by model, summed by cost, and joined to a trace. OpenAI's own production best practices push the same idea: capture structured metadata at the SDK layer rather than with print statements.

Here is the schema every Techsy service emits, fourteen fields:

{
  "request_id": "req_01J9XK4M7Q",
  "timestamp": "2026-07-18T09:24:31.482Z",
  "environment": "production",
  "model": "claude-sonnet-4-20250514",
  "prompt_tokens": 1284,
  "completion_tokens": 396,
  "latency_ms": 812,
  "cost_usd": 0.0098,
  "system_prompt_hash": "sha256:9f2c1a4e...",
  "user_id_hash": "sha256:b7e4d831...",
  "rag_sources": ["kb/pricing-2026.md", "kb/refund-policy.md"],
  "guardrail_result": "pass",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

Three fields deserve a note. cost_usd is computed at request time from the token counts and the model's published rate, never backfilled by a nightly job. The two hash fields are the Rule 1 compromise: correlatable offline, opaque in the log. And trace_id and span_id are W3C trace-context values, which is exactly what Rule 4 is about.

If four services calling providers directly sounds like four places to instrument, a LiteLLM proxy centralizes it: one logging hook in front of every provider.

Rule 3: Capture Token Counts and Cost per Request

Token usage tracking belongs in the log line itself, not in a warehouse job that runs tomorrow. Every provider returns input and output token counts in the response; multiply by the model's per-token rate at that exact moment and write cost_usd into the record. Rates change, and differ for cached versus fresh input tokens, so computing cost later with a static price table silently rewrites history. With cost on every line, "which feature is expensive?" becomes a one-line query instead of a finance project, and it feeds directly into the work to reduce your LLM API spend.

Rule 4: Attach Trace Context (OpenTelemetry GenAI Semconv)

A log line without a trace ID is an orphan: you can read it, but you cannot tell which retry, which RAG step, or which user turn produced it. The fix is OpenTelemetry's GenAI semantic conventions, the standard attribute names for instrumenting model calls. Emit the log inside an active span and the trace_id and span_id attach themselves, so one click in Grafana takes you from the trace waterfall straight to the raw record.

The attributes worth setting on every gen_ai span:

from opentelemetry import trace

tracer = trace.get_tracer("ai-sdr")

with tracer.start_as_current_span("chat.completion") as span:
    span.set_attribute("gen_ai.system", "anthropic")
    span.set_attribute("gen_ai.request.model", "claude-sonnet-4-20250514")
    response = client.messages.create(**payload)
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
    span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
    log.info("llm.request", cost_usd=cost)  # Rule 2 record, trace IDs auto-attached

Rule 5: Redact PII Before the Log Write

PII redaction has to happen before the record is written, not scrubbed after. Once an email address is in Loki, it is in your object-storage backups too, and "we deleted it later" is not a GDPR answer. In our setup, Microsoft Presidio runs as a structlog processor and catches 94% of emails and phone numbers before they hit Loki; the misses are almost all odd formatting, which we patch into custom recognizers as we find them.

The whole hook is fifteen lines:

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

PII_ENTITIES = ["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", "IBAN_CODE"]

def redact_pii(text: str) -> str:
    results = analyzer.analyze(text=text, language="en", entities=PII_ENTITIES)
    return anonymizer.anonymize(text=text, analyzer_results=results).text

# In the structlog processor chain, before JSONRenderer:
# event_dict["prompt"] = redact_pii(event_dict["prompt"])

Redaction sits in the same pipeline layer as your input and output filters, and it should be tested the same way. Our guardrail pipeline treats a leaked email in a log as a failing eval, not an ops footnote.

Rule 6: Sample Intelligently at High Volume

Below roughly 100K requests a day, log everything. Above that, full-volume logging is a storage tax on data you will never read, and sampling is how you keep the records that matter. The catch: random sampling is the worst option for LLM traffic, because failures, refusals, and five-dollar requests are rare by definition, so a uniform 10% rate discards exactly the events you debug. Sample by outcome, not by coin flip.

A common setup is rule-based at the edges (production and enterprise tenants: always log) plus tail-based in the middle. The log-specific angle on this framework: your guardrail_result and cost_usd fields are the sampling signals, already present if you followed Rules 2 and 8.

Rule 7: Set a Retention Policy Before You Need One

A log retention policy is a decision you make while calm, because the alternative is making it during a cost review at double the volume. GDPR's Article 5 storage limitation principle says personal data should be kept "no longer than is necessary," which in practice means tiered retention:

Hot answers "what happened ten minutes ago?" fast and expensively; cold answers "what did we tell this customer in March?" slowly and cheaply. Delete on schedule, automatically, or the tiers are just a diagram.

Rule 8: Log Guardrail and Safety Events Separately

Safety events (guardrail blocks, refusals, policy violations) are not telemetry; they are audit records, and they belong in their own stream. Three reasons. Alerting: a spike in blocked prompt injections should page someone, and you cannot tune that alert against 1M routine lines. Retention: compliance may demand safety records outlive debug logs by years. Access: auditors get the safety stream, not your whole firehose. Tag the verdict in the main record (guardrail_result: "block") and route the full record to the separate stream. What counts as a safety event is covered in our guide to guardrail events.

Rule 9: Make Logs Queryable, Not Just Stored

A log you cannot query in under a minute is a backup, not an observability signal. Queryable means indexed fields, a query language your on-call actually knows, and dashboards built before the incident. We run Loki and query it 30+ times a week for cost anomalies, latency regressions, and "show me every refusal for tenant X yesterday." The Grafana Loki docs are the reference for the syntax; the pattern that earns its keep is filtering on parsed JSON fields directly:

{service="ai-sdr"} |= "llm.request" | json
  | model = "claude-sonnet-4-20250514"
  | cost_usd > 0.05
  | line_format "{{.timestamp}} {{.request_id}} ${{.cost_usd}} {{.latency_ms}}ms"

Five lines, no export to a notebook. If your current store cannot do that, it is the problem to fix first.

What We Actually Log in Production

Enough theory. Here is the redacted config from our AI SDR pipeline, the service behind the 1.2M-requests-a-month number in the intro. It runs structlog 25.4.0 rendering JSON, shipped to Grafana Cloud Loki via Promtail. The model on this pipeline is claude-sonnet-4-20250514, and every call goes through exactly the processor chain from Rules 2 and 5:

import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        redact_pii,          # Rule 5: Presidio, before serialization
        add_otel_trace_ids,  # Rule 4: trace_id + span_id from the active span
        structlog.processors.JSONRenderer(),
    ],
)
log = structlog.get_logger()

Two numbers from the first quarter with this setup. Monthly ingest settled at 47GB across four services, and p95 log-write latency is 3ms, which is to say the pipeline adds nothing measurable to request time.

The config change that paid for itself: we added cost_usd to every log entry in March 2026. Within a week we found one prompt template burning $340/month in retry loops. A transient API error was triggering three retries, each resending the full 4,000-token context. The logs made it a one-line query; without per-request cost, it would have surfaced as an unexplained line item in the next quarterly budget review.

How Much Does LLM Log Storage Cost at Scale?

At 1M requests a day, LLM log storage costs between roughly $1.20 and $108 per month for the same data, depending on the store. The math: a full structured record averages about 2KB, so 1M requests a day is 2GB a day, or 60GB a month. The vendor-published pricing below (July 2026) is what that 60GB costs in three common backends.

Sources: Datadog pricing, Grafana Loki, and the ClickHouse observability docs.

Two caveats, because this is our calculation from vendor rates, not a benchmark we ran. First, Datadog's number assumes you index everything; most teams index a subset and pay far less, while Loki and ClickHouse charge mainly for what you store. Second, self-hosted ClickHouse's $1.20 hides a real bill: the compute to run the cluster and the engineer-hours to operate it. At 60GB a month, a managed service is almost always the cheaper total answer. Self-hosting starts to make sense past roughly 1TB a month, where the per-GB gap overwhelms the ops overhead.

The spread is the takeaway. At 1M requests per day, the gap between Datadog-indexed and self-hosted ClickHouse is roughly 90x: $108 versus $1.20 for the same 60GB. Pick the store at architecture time, not after the invoice arrives.

Which Logging Tool Should You Pick?

For most teams the choice comes down to four options: an LLM-native platform (Langfuse or LangSmith), a proxy-layer tool (Helicone), or a plain OpenTelemetry pipeline into infrastructure you already run. The table covers the decision points that actually differ; dashboards, playback, and prompt versioning are table stakes in all four.

Our opinion, plainly: we run OTel-native plus Loki because we already had the Grafana stack for everything else, and adding one more data source beat adopting a fourth vendor. If you are starting from zero with no observability stack at all, Langfuse's tracing model and its free tier are the fastest path to useful, and the self-host option keeps the exit door open. If you are choosing between the two LLM-native leaders, our Langfuse vs LangSmith breakdown runs the full comparison. And if logging is one piece of a bigger monitoring decision, the full platform comparison covers the broader field.

What Are the Most Common LLM Logging Mistakes?

Six mistakes account for most of the broken LLM logging setups we have looked at. Each one is cheap to avoid if you catch it before the log volume does:

• Logging raw PII without redaction. The most common and the most expensive. One support export or one breached bucket turns prompt logs into a data-protection incident. Redact before the write (Rule 5), not on read.
• No retention policy. Indefinite storage is the default everywhere, and it quietly doubles your bill every year. If you never delete, you do not have a logging system; you have an archive with delusions of grandeur.
• Unstructured text logs. Print output you can only grep works at demo scale and collapses at 100K requests a day, when "find every failed request for model X" becomes a shell-script afternoon instead of a query.
• Logging only errors. Successful requests are the baseline you detect drift against, and they are the raw material of your evaluation pipeline. Log the wins too, sampled if volume forces it.
• Ignoring cost fields. No cost_usd per request means no cost alerts, no per-feature attribution, and the $340/month retry loop from the production section above stays invisible until the quarterly bill.
• No trace correlation. Logs disconnected from spans make multi-step agent debugging guesswork. If your log line lacks a trace_id, Rule 4 is the fix.

Frequently Asked Questions

What should you log for every LLM request?

At minimum: the full prompt and completion with PII redacted, the model name, input and output token counts, latency, cost in USD, a hashed user identifier, and the OpenTelemetry trace and span IDs. Add RAG source IDs and the guardrail verdict if your pipeline has those stages. Fourteen named fields, one JSON line per request.

What is the best format for LLM logs?

Structured JSON, one object per request, with every field explicitly named. Free-text logs can only be grepped; JSON records can be aggregated by model, summed by cost, and joined to traces. Emit the record with a structured logger such as structlog in Python or pino in Node, and render it with a JSON serializer, never string formatting.

How do you handle PII in LLM logs?

Redact before the log write, not after. Run the prompt and completion through a detector like Microsoft Presidio inside your logging pipeline, replacing names, emails, and phone numbers with tokens such as <EMAIL_ADDRESS>. Once raw PII reaches your log store it is also in your backups, and retroactive deletion rarely satisfies GDPR's minimisation test.

How much does LLM log storage cost at scale?

For 1M requests a day, about 60GB a month at 2KB per record, expect roughly $108/month on Datadog's indexed LLM Observability pricing, $30/month on Grafana Cloud Loki, or about $1.20/month in compressed S3 storage for self-hosted ClickHouse plus compute. Those are vendor-published rates as of July 2026; self-hosting adds engineering time on top.

What are OpenTelemetry GenAI semantic conventions?

They are OpenTelemetry's standard attribute names for instrumenting LLM calls: gen_ai.system for the provider, gen_ai.request.model for the model, gen_ai.usage.input_tokens and output_tokens for token counts, and gen_ai.response.finish_reasons for why generation stopped. Using them means any OTel-compatible backend, from Jaeger to Tempo to Langfuse, reads your traces without custom parsers.

How do you sample LLM logs at high traffic?

Keep every error, every guardrail block, and every request above a cost threshold, then sample the rest. This tail-based approach preserves the rare events you actually debug, while uniform random sampling discards them at the same rate as boring traffic. Below 100K requests a day, skip sampling entirely and log everything.

How long should you retain LLM logs?

Tier it: 7 days hot for live debugging, 30 days warm for incident review and cost analysis, and up to 1 year cold in compressed object storage for compliance and audits. GDPR's storage-limitation principle bars keeping personal data longer than is necessary, so pair each tier with automatic deletion rather than manual cleanup.

What is the difference between LLM logging and LLM tracing?

A log is a flat record of one event: this request happened, with these fields. A trace is a causal tree of spans across a whole request path, say retrieval, then the model call, then two tool calls. Logs tell you what; traces tell you where and why. Production setups emit both, joined by trace_id.

Conclusion

Recap: log every request as named-field JSON, compute cost at request time, attach OTel trace context, redact PII before the write, sample by outcome once you pass 100K requests a day, and pick a store you can actually query. The nine rules are ordered so you can adopt them one per sprint, and Rules 2, 4, and 5 are the three that pay back fastest. If you are choosing the wider monitoring stack around the logs, start with our roundup of the best AI observability platforms. And if you need help wiring up structured logging for your LLM stack, get a free consultation.