A model download can look promising until the machine tries to load it. The file fits on disk, the name says “7B,” and the chat still runs out of memory. Those details describe different parts of the same system.
Running a local model becomes much easier to reason about once the name, the weight storage, and the working memory are separated. The goal here is to read a model listing, choose a sensible first experiment, and understand what changes when the conversation grows.
The numbers learned during training
Parameters are the numerical values learned during training. During inference, the model uses those values to transform input tokens into predictions for the next token. A token can be a word, part of a word, or punctuation. Repeating that process produces an answer.
The “B” in a name such as 7B means billions of parameters. It describes size, not a promise of correctness. Training data, architecture, instruction tuning, and the task all affect capability. A newer small model can outperform an older large model on a particular job.
Mixture-of-experts models add another wrinkle: total parameters and the parameters active for each token can differ. The downloaded weights may still need substantial storage even when only part of the model participates in each step. Read the model card instead of treating a size suffix as the entire specification.
A weight has a value, not just a name.
W[1,3] = +0.9One coefficient in this small example. A model can store billions of
them.
Read the label before the download
Qwen2.5-Coder-7B-Instruct is a concrete example with a published model card. A converted GGUF download may add a quantization suffix such as Q4_K_M. The exact filename depends on the distributor.
“Coder” indicates a training emphasis, not a compiler. Generated code still needs inspection and tests. “Instruct” indicates preparation for following instructions, while the base model is primarily a continuation model. A chat application also needs the correct prompt template to present messages in the format the model expects.
Choose the behavior as well as the size
An instruction-tuned variant is a practical starting point for chat and coding questions. A base model is useful for experiments that need raw text continuation or further tuning, but may continue a question instead of answering it in the expected style.
Reasoning or thinking variants can spend more generated tokens working through a problem before giving their final answer. That can help on some tasks and add latency on others. Support and controls vary by model and runtime; a switch in the application cannot add a capability the model doesn't have.
Compare variants with a task whose answer can be checked. For example, ask for a function that totals a list, then check empty input and a small known total. A convincing explanation and a passing test are different kinds of evidence.
Check what inputs and outputs it supports
A vision model needs support for processing images, not just an interface with an upload button. Tool use needs a model and runtime that agree on how calls are represented. Structured output may use constrained decoding to match a schema, but the resulting values still need validation. Check these capabilities separately in the model card and runtime documentation. A code-oriented name does not imply all three.
Store the weights with fewer bits
Quantization represents numerical values with reduced precision, usually alongside information needed to reconstruct approximate values. That can shrink the weights enough to make a model practical on a smaller machine. It can also change output quality.
The simplest estimate is parameter count multiplied by bits per parameter, divided by eight. For an illustrative model with exactly nine billion parameters, 16-bit storage needs 18 GB of raw weight values. Eight-bit storage needs 9 GB; four-bit storage needs 4.5 GB. These are decimal gigabytes and arithmetic estimates, not download or VRAM measurements.
Four values, rather than every value.
2 bits
llama.cpp's quantization documentation describes the available recipes. Q4_K_M uses a mixture of quantization choices; it doesn't mean every stored value occupies exactly four bits. The M distinguishes a recipe from other variants, rather than specifying a universal quality score.
A four-bit download is a reasonable experiment when a higher-precision copy won't fit, but quality loss depends on the model and task. Check the actual file size and evaluate the output. Quantization preserves the parameter count while changing how its values are represented.
A conversation needs working memory too
The weights are only one part of the allocation. Many transformer runtimes retain attention keys and values for previous tokens in a KV cache, avoiding some repeated computation during generation. Cache size depends on architecture, sequence length, precision, and the number of concurrent sequences.
Longer context can therefore consume more memory even when the model file stays the same size. Runtime buffers, vision components, and the rest of the operating system also need space. A fixed “add one gigabyte” rule cannot predict every setup.
For a 10 GB GPU such as an RTX 3080, a model may load comfortably with a short context and fail with a much longer one. Watch the runtime's memory estimate, set a modest context first, and increase it gradually. Partial CPU offload can make a larger allocation possible, but transfers and CPU work can reduce speed.
The count stays. The bytes change.
parameters × bits ÷ 8Ideal bytes for the weights
Every bar represents the same parameter count. Reducing bit width lowers ideal weight storage, but it does not account for the KV cache or runtime buffers. A smaller download can help the model load; a longer conversation can still consume the headroom left over. Choose precision and context length together.
Match the download to the runtime
GGUF is a model container used by llama.cpp-based software. It is available across several platforms, including Apple Silicon; it is not limited to Windows or NVIDIA hardware.
MLX is an Apple Silicon machine-learning framework, not simply another file extension. An MLX model distribution contains weights and configuration that its runtime can load. Apple Silicon's unified memory is shared with the operating system and other applications, so the entire installed capacity is not a private model budget.
Choose a compatible model package for the runtime and architecture the machine supports. Neither format nor framework guarantees faster generation in every case. The model, quantization, context, and backend all matter.
Make the first local run small
LM Studio provides a desktop interface for finding, downloading, loading, and chatting with supported local models. Its offline documentation distinguishes the initial downloads from local operation once the required files are present.
- Choose a supported instruction-tuned model and read its model card.
- Pick a quantized download that leaves room for the runtime, not just the weights.
- Load it with a modest context length and inspect the memory usage.
- Try a small coding question with known test cases.
- Change one setting at a time and compare the result.
Temperature changes sampling variability, not knowledge or correctness. A lower value can make outputs less varied, but it doesn't replace tests. Use structured-output features when the runtime and model support the required schema, then validate the returned data in the consuming program.
Use the same model from a script
LM Studio can expose OpenAI-compatible endpoints on a local server. Start it in the Developer interface and inspect the available model identifiers:
curl http://localhost:1234/v1/modelsUse a returned identifier in a chat request, replacing the placeholder below:
curl http://localhost:1234/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"replace-with-loaded-model-id","messages":[{"role":"user","content":"Write a function that totals a list. Explain the empty-input case."}],"temperature":0.2}'Compatibility helps reuse clients that support this endpoint shape. It doesn't guarantee identical features or behavior to a hosted API. A tool-capable model can propose calls, but an application still has to execute those calls and return results.
The terminal route
Ollama offers a command-line path to a local chat. Its Qwen2.5 Coder listing documents available sizes; this is a concrete starting example rather than a claim that it is the newest model:
ollama run qwen2.5-coder:7bThe first run downloads the required model files. A graphical browser may be easier for comparing packages; the terminal may fit repeated experiments better. Choose the interface that makes loading, inspecting, and checking the model straightforward.
The same questions apply to hosted models
A hosted assistant also works within choices about architecture, precision, context, and runtime. Without published specifications, its parameter count cannot be inferred from a convincing answer. Compare the behavior on relevant tasks instead. A local experiment makes the resource trade-offs visible, while a hosted service manages those allocations elsewhere.
Choose an experiment the machine can finish
Local inference can keep model computation on the machine and remove per-request provider billing. Downloads, remote tools, and external API calls are separate network activity, while hardware and electricity still have a cost.
The useful starting point is a model that fits with headroom, a bounded task, and a way to check the answer. Read the card, understand the weights, watch the working memory, and test the result. That turns a confusing download label into a practical decision about what the machine can actually do.