public LocalLLMProvider(path as Path, device as Device = Device.auto, maxContext as Long = 2048l, verbose as Boolean = false)
Loads a local indexed Q8 model, validating its supported text architecture. Vision/MTP weights are not loaded. GPU failure never becomes silent CPU work. Tied output embeddings reuse the same packed device allocation. GGUF expresses this sharing by omitting output.weight; indexed checkpoints declare it in tie_word_embeddings. Neither path duplicates shared parameters.
| Parameter | Description |
|---|---|
path | Local Klyn Q8 directory or supported GGUF file. |
device | CPU/GPU selector; auto chooses a compatible GPU, otherwise CPU. |
maxContext | Bounded per-sequence context capacity, at most the model limit. |
verbose | Show a ProgressBar for weight-loading operations on stderr; false by default. Counts embeddings, output head, normalization and layers. |
provider = LocalLLMProvider(FolderPath("Qwen3.5-9B-Q8"), Device.gpu[0], maxContext=2048l)