Skip to content

What is LLMOps?

Sergey Almyashev

Definition

LLMOps is the practice of running applications built on large language models in production: versioning prompts and model IDs, testing every change with evals, tracing cost and quality on live traffic, and migrating before a provider retires a model. It adapts MLOps to apps where most teams call a hosted model instead of training one.

What does LLMOps stand for, and what does it cover?

LLMOps stands for large language model operations. Microsoft's AI playbook describes it as the tools and processes that manage developing, deploying and maintaining LLM-based applications, and splits the work into two loops (Microsoft Learn, last updated April 15, 2025, so an older source):

StageLoopWhat you manage in practice
Data curationInnerThe documents, examples and test cases the app depends on
ExperimentationInnerPrompt versions, retrieval settings, model choice, sometimes fine-tuning
EvaluationInnerMetrics and test sets that compare each variant
Validate and deployInner to outerRelease checks and A/B tests before a variant reaches all users
InferenceOuterLatency, throughput and batch processing
MonitorOuterResource use, alerts on anomalies or privacy breaches, review of responses
Feedback and data collectionOuter back to innerUser feedback that flows back into the test sets

The inner loop is where a team builds and refines. The outer loop is where it runs the result for real users and learns from them.

LLMOps vs MLOps: what changes?

Many explainers frame LLMOps as MLOps applied to bigger models, with training at the center; Google Cloud's best practices, for example, open with training data quality, then training algorithms and hyperparameter tuning. For most product teams today the model is someone else's API, and that moves the work elsewhere.

Classic MLOpsLLMOps for an app on a hosted model
What you trainYour own model, on your dataUsually nothing; you pick a model ID
What you versionDatasets, features, model weightsPrompts, model IDs, retrieval index, tool definitions, eval sets
Who changes the modelYour team, on your scheduleThe provider, including retiring old versions
Typical failureData drift degrades accuracyA prompt edit or model swap changes behavior on cases nobody retested
Main cost driverTraining and serving computeTokens per request, multiplied by traffic and agent steps
RollbackRedeploy the previous model artifactPoint back to the previous prompt version and model ID

What is the release unit in LLMOps?

In classic software you ship a build. In an LLM app, behavior depends on several parts that can change independently:

  1. Model ID, pinned to a dated snapshot rather than a moving alias where the provider offers one.
  2. Prompt version, stored outside the code so it can change and roll back on its own.
  3. Retrieval index and the documents in it.
  4. Tool definitions the model can call.
  5. Eval suite version that approved this combination.

Treat these five as one release. Registries help with the second item. In the MLflow Prompt Registry, prompt versions are immutable and commit-based, and aliases let you move production to a new version, run A/B tests and roll back without a code deploy.

Why is model retirement an LLMOps problem?

Because a model you depend on can stop answering. Anthropic's model deprecations page tracks each model as active, legacy, deprecated or retired. Once a model is retired, requests to it fail. For publicly released models, Anthropic promises at least 60 days' notice before retirement. A concrete case on that page: claude-sonnet-4-5-20250929 was deprecated on September 30, 2026, with retirement set for November 30, 2026 and claude-sonnet-5-5 as the recommended replacement. The page also notes that partner platforms such as Amazon Bedrock set their own retirement dates.

A migration is a release like any other: run your eval suite on the replacement, compare cost and latency, then switch. A small CI check can make the deadline visible before it becomes an outage:

from datetime import date

# Model IDs pinned in our release manifest.
PINNED = {"support-bot": "claude-sonnet-4-5-20250929", "summarizer": "claude-haiku-4-5-20251001"}

# Retirement dates from the provider's deprecations page.
RETIRES = {"claude-sonnet-4-5-20250929": date(2026, 11, 30)}

def check(today, warn_days=60):
    failures = []
    for app, model in PINNED.items():
        retire = RETIRES.get(model)
        if retire and (retire - today).days <= warn_days:
            failures.append(f"{app}: {model} retires {retire}, run evals on a replacement")
    return failures

for line in check(date(2026, 11, 19)):
    print(line)

Run with Python 3 on our machine on 2026-11-19, it prints support-bot: claude-sonnet-4-5-20250929 retires 2026-11-30, run evals on a replacement. In a real pipeline, fail the job when the list is not empty.

What should an LLMOps setup have before production?

  • Evals on every change to any of the five release parts, with a pass threshold. The AI evals entry shows a runnable CI gate.
  • Traces of real runs with tokens, cost and latency per request or agent run. See agent observability.
  • A budget per request and an alert when cost per task moves, because prompt and model changes shift it quietly.
  • A migration calendar fed by provider deprecation notices.
  • A rollback path that does not need a code deploy.

Many generative AI projects stall between pilot and production; our post on why generative AI fails after the demo looks at the evidence and at what the teams that ship do differently. Setting up this operating layer, from prompt versioning and tracing to cost tracking and release gates in CI, is part of our generative AI development work. Founded in 2011 by the people behind Codeception and CodeceptJS, SDCLabs brings test discipline to it: automated tests verify every change, and engineers decide what ships.

Sources

  1. LLMOps: Operational management of LLMs, Microsoft Learn (AI playbook), 2025-04-15
  2. LLMOps: What it is and how it works, Google Cloud, 2026-11
  3. Prompt Registry, MLflow, 2026-11
  4. Model deprecations, Anthropic (Claude Platform Docs), 2026-11

Frequently asked questions

What does LLMOps stand for?

Large language model operations: the processes and tools for building, deploying, monitoring and maintaining applications that use large language models.

Is MLOps outdated now that LLMOps exists?

No. Teams that train or fine-tune their own models still need MLOps for data, training runs and model registries. LLMOps adds what apps on hosted models need on top: prompt versions, evals on every change, token costs and provider migrations.

Do you need LLMOps if you only call a model API?

Yes, a lighter version of it. Even a single prompt behind an API needs a pinned model ID, an eval set to rerun when anything changes, cost tracking and a plan for the day the provider retires that model.

Who owns LLMOps in a team?

Usually the platform or DevOps engineers who already own CI and monitoring, working with QA on the eval suites and with product owners on quality thresholds. The work fails when nobody owns model migrations.

Sergey AlmyashevSergey AlmyashevChief Operating Officer14+ years in QA, automated, exploratory and performance testing, and continuous integration and delivery within Scrum, helping teams in product and outsourcing companies organize teams, processes and tools.LinkedIn

Building with AI agents? Talk to an engineer.