Runpod launches batch inference for cheaper AI workloads

Runpod has launched Batch Inference on its serverless platform for non-urgent AI workloads.

The product lets developers send background jobs to the same inference endpoints and model weights they already use for live traffic, rather than building a separate setup for offline processing. The aim is to keep heavy tasks away from the interactive requests handled by real-time applications.

Background tasks can include scoring benchmark datasets, embedding large document collections and evaluating AI models. The jobs run on dedicated batch workers, so they do not use the concurrency limits reserved for live user requests.

Developers can submit batches of up to 5,000 JSON requests through an API call or by pasting raw JSON into a web console. Endpoints can support up to 10 active batches and 50,000 queued requests at once, while larger enterprise configurations can handle up to one million requests a day.

Runpod is also pitching the launch on cost. Batch compute is priced at a 30% discount to standard GPUs and at a 20% discount to higher-end Nvidia hardware, including the H100, H200, B200 and B300.

Jobs are scheduled on off-peak capacity, with multi-hour start times and a best-effort completion target within 24 hours. That makes the service suited to workloads that can tolerate delays in exchange for lower compute spending.

Inference demand

The launch comes as AI developers face growing pressure to manage inference costs as applications move from experimentation to wider deployment. Running models in production has become a larger share of AI infrastructure spending, particularly for businesses serving both real-time user interactions and large volumes of internal or back-office processing.

By using the same endpoint for both live and background jobs, Runpod is seeking to reduce the operational burden on developers who would otherwise need separate infrastructure, custom retry queues and additional orchestration for offline work.

Runpod said more than one million developers use its cloud platform. It offers infrastructure for different stages of AI development, including experimentation, training, fine-tuning, deployment and scaling.

Brennen Smith, Chief Technology Officer at Runpod, outlined the market backdrop for the launch.

"Inference is projected to take up roughly two-thirds of all AI compute this year, up from just a third in 2023. Real-time apps need low-latency concurrency, while bulk tasks demand raw throughput and lower cost per request. As developers scale inference, infrastructure efficiency requires smarter orchestration and isolating offline pipelines so that production systems stay responsive, cost-effective and reliable," Smith said.