Local is a workload property
“Run AI locally” can mean several different architectures: a model embedded in a desktop application, an inference server inside a data center, a Kubernetes service on private infrastructure, or an edge system operating near machines and sensors.
Foundry Local addresses a specific shape: model inference shipped with an application and executed on the user’s device. Microsoft describes SDKs for several languages, a curated catalog of optimized models, automatic hardware acceleration, model lifecycle handling, and an OpenAI-compatible interface.
That is valuable—but it should not be confused with a multi-user server inference platform.
When on-device inference fits
Foundry Local is a credible candidate when the mission needs one or more of these conditions:
- prompts and outputs should remain on the device during inference;
- the application must continue without network connectivity;
- response startup cannot depend on a cloud round trip;
- a single user interacts with the model on their own hardware;
- the supported local model meets the measured task requirement;
- distributing and updating model assets is operationally acceptable.
Examples include private document assistance on a managed laptop, offline field support, local transcription, or an interactive application that uses a small model for immediate classification and drafting.
Local execution does not automatically make the whole application private. Model downloads, application updates, diagnostics, synchronization, and any cloud fallback still require a documented data flow.
When it does not fit
Do not use a client runtime as a server platform for many concurrent users. Shared inference needs request queuing, batching, capacity management, isolation, and GPU utilization patterns designed for a service.
Local inference can also be the wrong choice when the workflow requires a model too large for target hardware, centrally governed retrieval across frequently changing sources, consistent performance across diverse devices, or operational controls that the endpoint fleet cannot support.
“No per-token cloud charge” is not the same as no cost. Account for endpoint hardware, model distribution, storage, energy, support, update failures, and the engineering needed to test across the device estate.
Design the hybrid route explicitly
A hybrid application can use different inference paths for different work:
- a local model handles sensitive, routine, or latency-critical tasks;
- a managed cloud model handles complex cases that need greater capability;
- private retrieval prepares a minimized context;
- a policy layer decides whether a request may leave the device or boundary;
- queued work synchronizes when connectivity returns.
The router should be deterministic where possible. Base it on data classification, task type, connectivity, measured model capability, latency budget, and user consent—not on an unconstrained model deciding where to send data.
Always make fallback behavior visible. If the local model is unavailable, the application should not silently send the request to a cloud endpoint.
Evaluate per target device
A local model variant can behave differently from the cloud model used during development. Build evaluation against the models, quantization, prompts, runtime, and hardware classes that will actually ship.
Measure:
- task success and critical failure cases;
- startup time and response latency;
- memory, storage, and power impact;
- behavior across CPU, GPU, and NPU paths;
- offline and interrupted-download recovery;
- model-version compatibility and rollback;
- output differences between local and cloud routes.
Pin versions when reproducibility matters. Treat model and runtime updates as application changes that pass the release gate.
Operate the fleet
On-device AI moves part of platform operation to the endpoint. Define how models are approved, distributed, cached, removed, and supported. Know which versions are active and how a vulnerable or defective model package is withdrawn.
Collect only the telemetry needed to operate the product, within consent and privacy boundaries. Local inference should not become a reason to copy sensitive prompts into centralized logs.
Choose the smallest sufficient environment
The right deployment is the least complex environment that satisfies quality, privacy, latency, continuity, cost, and operational ownership.
Use on-device inference when the device is genuinely the best boundary. Use managed or private server inference when centralized capacity and control are required. Use hybrid only when the mission has distinct classes of work that justify the additional routing and observability complexity.
Official reference
Microsoft explains the runtime’s scope, supported scenarios, local data processing, model handling, and server limitations in What is Foundry Local?.