Local LLM
A language model that runs on your own hardware instead of in a provider's cloud.
Explanation
A local LLM is a large language model that runs on your own hardware, such as a laptop, a workstation or a server in your own data center. Prompts and data never leave your own infrastructure.
How it works
You download an open-weight model and run it with a runtime such as Ollama, LM Studio, llama.cpp or vLLM. Quantization, a compressed way of storing the model weights, lets even larger models run with less memory. Applications access the model through a local API.
Example
A machinery manufacturer uses a locally hosted model to search design documents and service manuals without sending confidential engineering data to an external provider.
Why it matters
Local models give full control over data, costs and availability. For sensitive information and strict compliance requirements, they are often the precondition for using AI at all.
Local AI: which models can you run?
You can run so-called open-weight models locally, whose weights have been published. Well-known model families include Llama from Meta, Mistral, Qwen from Alibaba, Gemma from Google, DeepSeek and gpt-oss from OpenAI. They come in different sizes, measured in billions of parameters.
As a rough guide, models with about 7 to 20 billion parameters run quantized on a current laptop or a single graphics card and handle summaries, classification and simple assistant tasks. Models with 70 billion parameters or more need considerably more GPU memory, usually server GPUs, but come close to cloud models on many tasks. Only a test with real tasks shows which size is enough.
Local LLM vs. cloud AI: pros and cons
Pros: data stays in-house, there is no dependence on a provider's pricing or terms of use, and at high, steady utilization the cost per request is often lower. Local models also work without an internet connection.
Cons: hardware and operations cost money and require know-how. The strongest models for complex reasoning, long contexts and agents are mostly available only in the cloud. Updates, security and scaling are your own responsibility. Many companies therefore combine both: local models for sensitive data and routine tasks, cloud models under appropriate contracts for demanding tasks with non-critical data. See also on-premise hosting.
Frequently asked questions
What is a local LLM?
A local LLM is a language model that runs on your own hardware instead of being used through a provider's cloud. Prompts and data therefore stay within your own infrastructure.
What hardware do I need to run an LLM locally?
For smaller models, a current computer with 16 to 32 GB of memory or a graphics card with enough VRAM is sufficient. Large models for team use need a server with one or more powerful GPUs.
What is Ollama?
Ollama is a free, open-source tool that lets you download and run open-weight models locally with a few commands. It provides an API that other applications can use to access the model.