The llm-d project has released version 1.0 of its open source AI inference software. It was founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA.
The release is the first full production version of software designed to help companies run large language models more efficiently across shared computing infrastructure. The project is now governed by the Cloud Native Computing Foundation and has attracted contributions from more than 500 developers across more than 25 organisations.
llm-d focuses on inference, the stage at which trained AI models generate responses for users and applications. That part of the AI stack has become a growing cost centre for companies deploying chatbots, coding tools and so-called agentic systems that make repeated calls on memory, compute and stored context.
Version 1.0 adds multi-tenant controls to manage competing workloads on the same hardware, alongside high-availability routing and rolling upgrades intended to allow maintenance without service interruption. It also introduces a stable configuration interface for users seeking to standardise deployments across different clouds and accelerators.
The release also adds session-aware routing, which keeps multi-turn conversations close to cached context so systems can avoid recomputing the same information. The software supports a range of model servers, including vLLM and SGLang.
Cost pressure
The launch comes as businesses look for ways to cut the cost of serving AI models after the initial training phase. In one example cited by project backers, IBM Research used open models with llm-d in work on quantum error-correction hardware and reported inference costs 11.5 times lower than an all-proprietary setup.
That project recorded a prefix-cache hit rate of about 98% and a blended serving cost of roughly USD $0.11 per million tokens. Caching matters because many AI workloads repeatedly draw on the same context, and reusing that information can reduce the fresh computation needed for each request.
In another test, researchers used llm-d to deploy the open-weight GLM 5.2 model, described as having about 753 billion parameters, on NVIDIA GPUs already widely used in data centres. On benchmark agentic workloads with hundreds of parallel sessions, the system produced more than 6.6 million output tokens a minute at peak and served up to 3,000 concurrent coding agents without pre-emptions.
According to figures released by the project, that setup would have delivered a per-token cost five to 10 times lower than equivalent commercial API pricing at current cloud rental rates. The biggest savings were reported on input-heavy workloads, which are common in software agents that repeatedly submit long prompts and context windows.
Industry backing
The project's list of supporters points to broad interest in open alternatives to proprietary inference platforms. Alongside the founding companies, quoted backers include AMD, DigitalOcean, JetBrains, Mistral and Roblox, each describing live or planned use across their own AI services and infrastructure.
Chris Aniszczyk, Chief Technology Officer, CNCF, said: "AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organisations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It's an important step toward making high-performance open source inference more operationally mature and accessible across the cloud native ecosystem."
Several companies involved highlighted the operational challenge of handling unpredictable bursts in demand from AI agents while keeping latency stable for higher-priority users. Others pointed to the need to run across mixed fleets of cloud and on-premises GPUs rather than being tied to a single serving engine or hardware environment.
Brian Stevens, Senior Vice President and AI Chief Technology Officer, Red Hat, said: "As organisations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardised, hardened core for distributed inference to help enterprises optimise hardware performance, address operational costs and build high-density AI architectures on an open foundation."
Priya Nagpurkar, Vice President, AI Native Systems, IBM Research, said: "At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organisations get more value from their AI infrastructure. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs and accelerating innovation across the AI ecosystem."