llm-client — Provider-Agnostic LLM Runtime
Writeup in progress. Full case study covering architecture decisions, provider adapter design, circuit breaker implementation, and lessons from six production deployments coming soon.
All projectsProject
llm-client — Provider-Agnostic LLM Runtime
Open-source LLM runtime that abstracts OpenAI, Anthropic, and Gemini behind a unified async client with streaming, structured output validation, retries, circuit breakers, and provider routing. Powers six production AI products including CanApply's Dana platform.
llmopen-sourceinfrastructureproduction
Problem
Production AI systems need provider-agnostic LLM access with consistent error handling, structured output validation, streaming support, and operational guardrails. Existing wrappers either lock you into one provider, lack production-grade retry/circuit-breaker patterns, or impose framework-level coupling (LangChain).
Solution
A minimal, framework-free async Python client with provider adapters for OpenAI, Anthropic, and Gemini. Unified message schema, streaming, JSON Schema validation, exponential-backoff retries with provider-aware error classification, circuit breakers, and pluggable routing/fallback policies.
Impact
Adopted as the LLM kernel for six internal production products. Replaces ad-hoc provider clients across CanApply and Trellion. Recently published as an open-source library.
Stack
PythonasynciohttpxpydanticJSON SchemaOpenAIAnthropicGemini