Run a Model With Heartwood¶
Heartwood images and the generic Linux package include managed inference software but no model weights.
After a supported model is selected and downloaded or imported, the normal heartwood command verifies the files, plans context against available memory, starts the runtime, waits for readiness, and then opens the requested interface.
Downloading and verification are separate costs: a cached model avoids network transfer but Heartwood still verifies its immutable source and file hashes before use.
Download and Start¶
The guided path is:
Choose Run with Heartwood, inspect the plan, and confirm the download. The terminal displays transferred bytes and status; the browser shows a progress bar from the same gateway download state.
In the terminal, Heartwood continues automatically after the verified download and starts the model runtime.
In the browser setup flow, stop the process and repeat heartwood --interface web when the page reports that the model is ready.
The restarted browser process verifies the stored model, starts its local server, and keeps that server supervised for the session.
Hardware¶
| Runtime | Model Shape | Typical Environment |
|---|---|---|
| llama.cpp | One GGUF file | CPU workstation or portable container |
| vLLM | Standard safetensors snapshot | NVIDIA-enabled container, Terra GPU runtime, or Carina Slurm GPU allocation |
Heartwood reports model-specific minimum and recommended guidance before download. The estimates reserve space for runtime overhead and context, but no static estimate can account for every model architecture, driver, concurrent workload, or platform limit. For GPU configurations, Heartwood also checks the catalog GPU count, per-device memory, driver, precision, tool parser, tensor-parallel layout, and qualification status against the GPU compatibility matrix.
Context Window¶
Heartwood records the model's declared limit and chooses from conservative operating tiers when memory information permits. The smallest agent-capable runtime tier is 18,432 tokens: 16,384 tokens of OpenHands input capacity plus 2,048 tokens reserved for the model response. When model-size or memory information is unavailable, Heartwood uses that minimum tier rather than assuming that 32,768 tokens will fit. A 32,768-token plan provides 28,672 input tokens plus 4,096 response tokens when the model and available memory support it. Larger supported tiers range from 65,536 to the model and Heartwood maximum of 1,048,576 tokens.
The planner reserves memory for model weights, runtime overhead, and context, and retains headroom instead of always selecting the model maximum. The OpenHands SDK backend uses a rolling-history condenser before history exceeds the configured input budget, preserving recent events and a structured summary for long sessions.
Larger context is useful for broad repositories and long analyses but increases key/value-cache memory and latency. Prefer the largest window that fits with conservative headroom rather than the largest value printed on a model card.
Advanced Runtime Control¶
Normal users should start with heartwood.
Operators can inspect or control runtime allocation separately:
heartwood runtime start --dry-run
heartwood runtime start --task-profile powerful --partition dev --time 01:00:00
On Carina, Heartwood prints the complete Slurm request and asks before allocating a GPU.
On provisioned Terra compute, it uses the attached resources without submitting a scheduler request.
Model download and scheduler allocation require separate confirmation.
An advanced runtime start --dry-run may show a compatible recommendation when the project has no selected model, but it never persists that recommendation.
Complete the normal heartwood setup flow before starting or allocating the runtime.
Stop the Runtime¶
Exit the Heartwood process to stop the supervised model runtime.
Downloaded model files remain in .heartwood/models/ and are verified again on the next start.