Discussion of local LLMs often starts with hardware: how much memory is needed, which GPU will work, and what quantization to choose. For a business, that is the second question. The first is which process we are improving and what counts as a successful result. The same model may be cost-effective for classifying inquiries and useless for preparing a legally significant response.

It helps to divide the choice into four axes: data sensitivity, acceptable latency, load profile, and the team's ability to operate the service. If even one axis is undefined, comparing models becomes a specification demo rather than a business decision.

Data: local does not automatically mean secure

A local model reduces external transfers, but does not eliminate access control, logging, backups, or workstation security. A document can remain inside the network and still end up in a shared log, cache, vector store, or backup with incorrect permissions. First, map the complete data journey, from upload to deletion.

Ask the same questions about the cloud: where the request is processed, whether inputs are retained, whether training on the data can be disabled, and what retention periods and contractual guarantees apply. Compare the actual flow and each party's responsibilities, rather than marketing terms such as “private” and “enterprise.”

Latency, context, and real load

A local model is useful for short, frequent operations close to the data: classification, field extraction, drafts, and internal search. For occasional peak requests, the cloud is often more sensible: you do not have to keep an accelerator idle or scale the queue yourself.

Context is particularly expensive for agents and coding tasks. Ollama explicitly links a larger context window with higher memory consumption. So a test where “the model answered one question” says nothing about a process that needs to read dozens of documents, call tools, and serve multiple users simultaneously.

Hardware and quantization without worshipping specifications

llama.cpp lets you run quantized models on different CPUs and GPUs, including split placement between processor memory and graphics memory. This makes a pilot accessible, but does not guarantee the required quality. Stronger compression saves memory at the cost of potentially reduced accuracy, especially for complex instructions and structured output.

Test three model sizes on your own task set. A small model may confidently handle email routing, a medium one extraction and summarization, while complex exceptions remain with a cloud model or a person. A hybrid architecture is often cheaper than trying to make one system do everything.

The cost of a successful operation

For an on-premises option, costs include hardware or rental, electricity, redundancy, updates, monitoring, engineer time, and downtime. For the cloud, they include tokens, network latency, limits, storage, enterprise features, and the cost of switching providers. The common unit of comparison is an operation accepted by the business without manual rework, rather than a request.

  • The share of answers accepted without corrections.
  • Mean and 95th-percentile time to a result.
  • The cost of retries and manual review.
  • The number of access incidents, failures, and format violations.
  • The team's weekly maintenance time.

A pilot you can finish

Choose one process with a clear owner and 50–100 anonymized examples. Define a reference baseline, quality criteria, and an error budget. Compare local, cloud, and hybrid setups on the same data. The pilot lasts two weeks and ends with a decision: implement it, change the task, or abandon it.

A local LLM is a hosting option, rather than an ideology. It makes sense when data constraints and the load profile translate into a measurable advantage. If the team is not ready to maintain the service, or quality on real exceptions falls below the threshold, the cloud or a hybrid setup will be more manageable.

An example calculation without self-deception

Imagine a service that classifies 40,000 inquiries a month. A cloud model costs 90,000 rubles including retries, while a local server costs 420,000 upfront and another 70,000 per month for operations and part of an engineer's time. Comparing 90,000 with 70,000 is not enough: 82% of the local model's answers need no edits, compared with 94% for the cloud model, and manually handling an exception costs 25 rubles. That quality gap adds around 120,000 rubles of manual work to the local option. In this calculation, “our own tokens are cheaper” ceases to be an argument.

Then run a sensitivity analysis. What happens if volume doubles, cloud pricing falls, a backup server is needed, or a new inquiry format reduces quality? Separately define the cost of an hour of downtime and the deadline by which the team must install a security update. This moves the conversation from GPU specifications to service commitments.

A hosting matrix by data and task

  • Public text and irregular load: the cloud is usually faster to get started.
  • Internal documents that may be shared with a provider: a cloud service with enterprise terms, or a hybrid setup.
  • Regulated data with no permitted external environment: on-premises hosting after checking the entire chain.
  • High-volume, narrow classification: a small local model if its quality is confirmed on a sample.
  • Rare, complex exceptions: route to a stronger cloud model or a person.

Hybrid does not mean choosing models chaotically for each request. Routing rules must be deterministic: data type, task class, acceptable latency, and confidence threshold. Logs retain the reason for the route without unnecessary request contents. If the local model fails to follow the format or scores below the threshold, the system uses a predefined safe path rather than improvising.