LocalAI

What it is

Open source AI engine that runs language models, vision, voice, image and video workloads on any hardware without GPU requirements.

Quick answers

What APIs does LocalAI support?

LocalAI speaks the OpenAI, Anthropic, Ollama and ElevenLabs APIs. Most tools need only a URL change and nothing else to point at LocalAI instead of the cloud service.

How does model selection and engine switching work?

The engine behind the API is swappable. One model can run on llama.cpp while the next loads on vLLM, SGLang or MLX, and the client never notices the difference. Switching is one line in the model's config.

Does LocalAI require a GPU?

No. Every feature ships a CPU path first that is tested in CI on the hardware most people already have. A GPU makes it faster but is not required.

How does distributed clustering work in LocalAI?

When you add a second machine, it reports what hardware it has and which backends it can run, then joins the pool. Requests land on the replica that already holds the model and the matching prefix cache, and if a node fails, in-flight work reschedules automatically.

What is APEX quantization?

APEX assigns a different precision to every tensor and every layer, allowing large models to fit on consumer GPUs with faster inference. A 35B mixture-of-experts model goes from 64.6 GB at 30.4 tokens per second to 12.2 GB at 74.4 tokens per second, with minimal quality loss.