The trade-offs of running your own inference service: hardware, model-serving frameworks, cost, and operations, weighed against just calling a cloud API.
The key question in self-hosted inference isn't how fast the engine is — it's your GPU utilization. A fully saturated A100 costs ~$0.70 per million output tokens; at 10% utilization that becomes $7, more than most cloud APIs. This overview maps seven tools across three layers to help you decide which layer you need.
vLLM uses PagedAttention to eliminate KV cache memory waste, combining continuous batching and prefix caching to become the most widely adopted open-source LLM inference engine today.
vLLM is the de facto standard for self-hosted LLM inference (89,470 GitHub stars, verified 2026-08-21), built on managing the KV cache the way an OS manages paged memory. But the selection question isn't how fast it is — it's your GPU utilization. Using Red Hat's measured 793 output tokens/second, a fully saturated A100 costs roughly $0.70 per million output tokens; at 10% utilization that becomes $7, more than most cloud APIs.
SGLang is an inference engine for generative models. RadixAttention reuses KV cache across shared prefixes, while OpenAI-compatible APIs, structured output, and multi-GPU parallelism support production LLM serving; it is not a complete product backend.
Triton Inference Server serves TensorRT, ONNX, PyTorch, and other models through consistent HTTP and gRPC APIs. Its defining tools are the model repository, dynamic batching, instance groups, and ensembles—not LLM-specific KV-cache scheduling.
Ray Serve is a distributed serving layer on Ray. Deployments and handles compose Python service graphs, while replicas, CPU/GPU scheduling, autoscaling, and model multiplexing handle orchestration; it complements rather than replaces vLLM or SGLang.
Text Generation Inference (TGI) is HuggingFace's own LLM inference server, built in Rust and Python. It pioneered continuous batching and Flash Attention in open-source inference engines. The GitHub repository was archived on March 21, 2026, and HuggingFace recommends migrating to vLLM or SGLang. TGI still matters: it defined the architectural baseline that successor engines inherited, and many HuggingFace Inference Endpoints still run it.
TensorRT-LLM is NVIDIA's open-source LLM inference library (Apache 2.0). It offline-compiles model weights and compute graphs into optimized TensorRT engines, then serves them with custom CUDA kernels, in-flight batching, and multi-dimensional parallelism. The cost: NVIDIA GPUs only, compilation takes tens of minutes, and switching models or quantization means rebuilding.
A Mac has no NVIDIA GPU, so it can't really run vLLM — but Colab's free T4 (compute capability 7.5, exactly vLLM's minimum) can. Serve the 0.6B Qwen3 as an OpenAI-compatible API with vllm serve, fire 8 requests at once, read /metrics, and you can watch continuous batching happen.