Prompt & Model Cost Optimizer
A routing layer that sends each request to the cheapest sufficient LLM behind quality gates and A/B evals.
PythonFastAPIOpenAI APICI/CD
Highlights
- Request classifier that estimates required capability per call.
- Cascaded routing across small, mid, and frontier models.
- Continuous A/B eval to prevent quality regressions on downshift.
Outcomes
- Cut total LLM spend by 51% while holding quality within 1% of baseline.
- Reduced p95 latency by 28% via smaller-model fast paths.
Stack
- Python
- FastAPI
- OpenAI API
- PostgreSQL
- Redis