Local inference, explained by layer

Map the local AI stack layer by layer.

An educational map of model engines, inference servers, API contracts, agent harnesses, and control planes—and where Ollama, oMLX, LM Studio, llama.cpp, vLLM, MLX, and Hermes Agent fit.

See the recommendation Understand the stack

Example: MacBook Pro Apple M5 Pro 48GB unified memory 20–35B quantized models Apple silicon

05Control planecontrol-plane agent · tmux · Slack

04Agent harnessClaude · Codex · Hermes

03API contractAnthropic · OpenAI

02Inference hostOllama · oMLX · LM Studio

01Engine + weightsMLX · llama.cpp · GGUF

The category error

LM Studio≠Hermes Agent

Two products, two different layers.

LM Studio is primarily a model workbench and inference server. It downloads and runs model weights, then exposes an API. Hermes is an agent harness. It plans, uses tools, remembers, delegates, and calls a model provider. Hermes can use LM Studio; it does not replace the same layer.

01 / Taxonomy

The five layers of a local agent system

Products often span two or three layers, which makes broad vendor comparisons misleading. The useful first question is which job each tool performs.

Control plane

Coordinates identities, work ownership, handoffs, liveness, and human visibility across agents.

tmuxcontrol-plane agentSlackHarness DB

Agent harness

Turns model responses into a loop: read files, call tools, edit code, remember context, and decide the next action.

Claude CodeCodexHermes Agent

API + routing

Defines how a harness talks to a model host. Compatibility layers let one harness swap among local and cloud backends.

Anthropic MessagesOpenAI APILiteLLM

Inference host

Loads models, owns memory and caches, schedules requests, exposes an API, and often manages downloads.

OllamaoMLXLM StudiovLLM

Engine + model

Executes tensors on Metal/CPU and defines weight formats, quantization, attention, and the actual model intelligence.

MLXllama.cppGGUFSafetensors

Request beginsA terminal, chat surface, or scheduler starts the work.

Harness reasonsClaude Code, Codex, or Hermes constructs prompts and tools.

API carriesAn Anthropic- or OpenAI-compatible request travels locally.

Host schedulesOllama, oMLX, LM Studio, or vLLM loads and serves.

Engine computesMLX or llama.cpp runs the model on unified memory.

02 / Field guide

Seven tools, four different jobs

Filter by category. Notice that the strongest “versus” comparisons happen within a category—not across the whole list.

Host · manager · APIRecommended default

Ollama

A headless model manager and background service that exposes local APIs to agent clients.

Now supports MLX acceleration and GGUF/llama.cpp coverage
Snapshot caching is designed for branching agent conversations
Direct launch integrations for Claude Code and Codex
Simple model registry and reproducible configuration files

Best suited toA first persistent local inference service on Apple silicon.

Apple-native hostPerformance challenger

oMLX

An MLX-native server built specifically for long-running agent workloads on Apple silicon.

Continuous batching and paged SSD cache
OpenAI and Anthropic-compatible APIs
Native menu-bar app or Homebrew service
Focuses on long, shifting coding-agent prefixes

Best suited toA measured A/B test against Ollama after the baseline works.

GUI · lab · APIRecommended workbench

LM Studio

A visual model lab that also serves production-style APIs. Browse, estimate memory, download, tune, compare, and expose models.

Runs GGUF and MLX models
Anthropic endpoint explicitly supports Claude Code
GUI, CLI, and headless llmster daemon
Best place to learn model/context/quantization tradeoffs

Best suited toExploration, model comparisons, and debugging alongside a clear layered model.

Engine · reference serverKeep in the toolbox

llama.cpp

The low-level, portable inference foundation behind much of the ecosystem. Apple Silicon and Metal are first-class targets.

Broad GGUF model and quantization support
Lightweight CLI and multi-user HTTP server
OpenAI and Anthropic-compatible endpoints
Excellent benchmark and compatibility baseline

Best suited toMaximum control, format coverage, benchmarking, and diagnosing wrappers.

Throughput engine · APIAdvanced evaluation

vLLM-Metal

vLLM’s community-maintained Apple-silicon plugin now brings MLX-backed paged attention and vLLM serving semantics to macOS.

Built for throughput, batching, and server-style workloads
Uses MLX-optimized model weights on Apple Silicon
Newer and narrower model matrix than mature CUDA vLLM
More installation and operational surface

Best suited toMulti-agent concurrency experiments after a simpler baseline is established.

Framework · engineFoundation layer

MLX / MLX-LM

Apple’s open-source framework for Apple silicon; MLX-LM adds model loading, quantization, generation, tuning, and a basic server.

Purpose-built for unified memory and Metal
M5 Neural Accelerator support in current releases
Excellent direct experimentation and fine-tuning surface
Basic MLX-LM server is not recommended for production

Best suited toUnderstanding and tuning the engine layer; let a host package it day to day.

Agent harnessIsolated pilot

Hermes Agent

A provider-flexible agent system from Nous Research with tools, memory, skills, subagents, messaging, scheduling, and sandboxes.

Consumes local or hosted model APIs
Overlaps with Claude Code/Codex—not LM Studio
Also overlaps with control-plane skills and orchestration
Useful open-model-native harness experiment

Best suited toA separate named tmux lane with no control-plane authority during evaluation.

Correct comparison sets

Ollama vs oMLX vs LM Studio vs vLLM-Metal compares model hosts. MLX vs llama.cpp compares engines and model formats. Hermes vs Claude Code vs Codex compares agent loops.

03 / Comparison

Same ecosystem, different responsibilities

A product can span layers, but its center of gravity still matters.

Tool	Center of gravity	GUI	Downloads/models	API server	Primary format/backend	Own agent loop	Typical role
Ollama	Model host + manager	Light app	Yes	Native + OpenAI-compatible	MLX + GGUF/llama.cpp	No	Canonical headless service
oMLX	Apple-native host	Menu bar + admin UI	Yes	OpenAI + Anthropic	MLX models	No	Agent-cache performance challenger
LM Studio	Visual workbench + host	Full	Yes	Native + OpenAI + Anthropic	MLX + GGUF	No	Model lab and easiest first contact
llama.cpp	Portable engine + server	Basic web UI	HF helper	OpenAI + Anthropic	GGUF · Metal · CPU	No	Reference baseline and escape hatch
vLLM-Metal	Throughput engine + server	No	Hugging Face	vLLM APIs	MLX via Metal plugin	No	Concurrency and serving research
MLX-LM	Framework + model library	No	Hugging Face	Basic server	MLX safetensors	No	Direct engine experiments and tuning
Hermes Agent	Agent harness	CLI / desktop surfaces	Uses providers	Gateway, not inference	Provider-agnostic	Yes	Open-model harness pilot

04 / Recommendation

Evaluate by layer, not in one giant bake-off

A useful evaluation selects a primary tool at each layer, then keeps explicit challengers for learning and benchmarking.

05 / Architecture

How local inference fits alongside an agent fleet

A local inference lane can be introduced without replacing the fleet’s authority, quality, identity, or handoff systems.

Human surfacesTerminal · messagingPeople remain the authority and observation point.

→

Control planeorchestrator · tmux · Harness DBIdentity, claims, liveness, handoffs, and role ownership stay put.

→

Agent harnessClaude / Codex / Hermes pilotEach lane uses one explicit loop; harnesses are not accidentally nested.

→

Local inferenceOllama primary · oMLX challengerOne service owns the model and cache at a time.

Keep cloud-first

Orchestration + final QA

Architecture judgment, security decisions, orchestration, and merge gates initially remain on the strongest hosted models.

Move local first

Bounded implementation work

Repository scanning, test writing, documentation drafts, mechanical refactors, issue summarization, and evidence extraction are ideal pilots.

Isolate as experiments

Hermes + new servers

Each candidate receives a named tmux session, narrow repository permissions, no autonomous merge, and disk-backed witnesses before promotion.

Important boundary

Hermes carries its own memory, skills, subagents, scheduler, messaging gateway, and tool loop. Those capabilities overlap with an established orchestration stack, so Hermes belongs in a peer harness lane that reports into the same human-observable control plane—not inside another agent harness.

06 / Quick chooser

Choose an optimization target

The right starting point depends on whether the priority is simplicity, visibility, control, throughput, or harness experimentation.

Primary recommendation

Ollama

Use Ollama as the first persistent model service. It minimizes operational friction while now taking advantage of MLX on M5 hardware.

Simple lifecycle and model management
Agent-aware snapshot caching
Direct Claude Code and Codex launch paths

07 / Rollout

A four-day, low-drama trial

A useful trial measures repeated tool calls over a growing repository context—not a single chat response.

Build the lab

Install LM Studio, load one fast model and one 20–35B coding model, then test raw chat and tool calling.

Target: ≥25K context, Q4/NVFP4, green memory pressure

Establish Ollama

Run the same coding model class through Ollama and connect one fresh Claude Code session using a local-only launcher.

Keep normal cloud launchers untouched

Run a real task

Use a bounded repo task with reads, edits, tests, and a handoff. Capture TTFT, tokens/sec, memory, retries, tool errors, and correctness.

One model · one task · one context budget

Challenge the winner

Repeat through oMLX. Add vLLM-Metal only when concurrency is part of the hypothesis. Pilot Hermes separately against the winning host.

Promote from evidence, not dashboard appeal

One model resident at a time

Two hosts loading the same 20–35B model duplicate weights and crush unified-memory headroom.

Score task success before tokens/sec

A fast model that misses tools or corrupts edits is slower in the only metric that matters.

Keep 8–10GB of headroom

Yellow memory pressure and swap can erase the advantage of a larger model.

Expose through Tailscale, not the public LAN

A remote computer can consume the Mac’s server without exposing an unauthenticated model endpoint to the public internet.

Local doers, cloud judges

Start with low-risk implementation and extraction; retain strong cloud models for coordination and exact-head QA.

Keep launchers explicit

Names like claude-local and hermes-local-lab prevent silent provider drift in durable agents.

08 / Vocabulary

The six terms worth remembering

Model: The learned weights and architecture—the actual language intelligence. Examples are Qwen, Gemma, GPT-OSS, and Hermes-model checkpoints.
Quantization: Compressing weights to fewer bits. Q4/NVFP4 reduces memory and usually increases speed, with some quality cost.
Engine: The code that performs tensor operations on hardware. MLX and llama.cpp are engines/frameworks.
Inference server: A long-running process that loads weights, schedules requests, manages caches, and exposes an API.
Agent harness: The loop around a model: prompts, tools, permissions, file edits, memory, compaction, and next-action logic.
Control plane: The system above individual agents that manages identity, authority, ownership, handoffs, liveness, and observability.

09 / Sources

Primary references

Current as of July 10, 2026. Product capabilities move quickly; these links are the source of truth used for this guide.