https://han8931.github.io/categories/llm-proxy-server/Recent content in LLM Proxy Server on Han's XYZHugoentabularasa8931@gmail.com (Han)tabularasa8931@gmail.com (Han)This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.Sat, 22 Nov 2025 20:30:54 +0900https://han8931.github.io/litellm/Thu, 18 Sep 2025 00:00:00 +0000tabularasa8931@gmail.com (Han)https://han8931.github.io/litellm/<h2 id="why-an-llm-proxy-at-all">Why an LLM proxy at all?</h2> <p>An LLM proxy sits between your app and model providers (OpenAI, Anthropic, Google, Ollama, etc.). It gives you a unified API (usually OpenAI-compatible), centralized auth, usage controls (budgets / rate-limits), routing and fallbacks, and caching—without changing your application code for each vendor.</p> <!-- There is a trade-off: adding a proxy introduces another moving piece (and potential single point of failure). For pure observability When you do want a proxy, Langfuse recommends LiteLLM, which is open source, self-hostable, and has first-class integration with Langfuse. --> <h2 id="what-is-litellm">What is LiteLLM?</h2> <p>LiteLLM is an OpenAI-compatible LLM Gateway that lets you call 100+ providers behind one API, plus adds budgets/rate-limits, model access control, caching, routing, admin UI, and more. You can run it as a single Docker container with a YAML config.</p>