Skip to content

GPU Inference Autoscaler on Kubernetes

A latency-aware GPU autoscaler for LLM and vision inference with batching, queueing, and spot fallback.

KubernetesPythonAWSCI/CD
Abstract near-black and electric cyan cover illustration for the GPU Inference Autoscaler project, showing pooled GPU blocks pulsing with cyan latency signals.

Highlights

  • Continuous batching and priority queueing tuned per model.
  • SLO-driven autoscaling with spot + on-demand blending.
  • Warm-pool preloading to hide cold-start on large models.

Outcomes

  • Cut inference cost per 1k tokens by 47% at matched p99 latency.
  • Sustained 99.95% availability across bursty production traffic.

Stack

  • Kubernetes
  • Python
  • AWS
  • Docker
  • CI/CD