Local LLM Inference

Local CPU And GPU Inference

klyn.llm.inference.LocalLLMProvider runs supported models locally and implements the same LLMProvider interface as OpenAIProvider. Use LLMChatSession for either provider: it owns the conversation history. Choose Device.cpu to use the CPU without a GPU driver, or Device.gpu[0] for NVIDIA GPU execution. Only the NVIDIA driver is required for GPU work; no CUDA Toolkit, Python or external inference engine is needed. Both execution paths are available on Linux and Windows.

import klyn.llm
import klyn.llm.inference
import klyn.math.nn

try provider = LocalLLMProvider.load(
    "bartowski/Qwen_Qwen3.5-9B-GGUF",
    quantization=ModelQuantization.Q8_0,
    device=Device.auto
):
    session = LLMChatSession(provider).withSystem("You are a helpful assistant.")
    print(session.ask("What is the capital of France?"))

The first invocation downloads only the requested checkpoint with SHA-256 verification. Console output is disabled by default; pass verbose=true to show klyn.gui.terminal.ProgressBar displays for download bytes and weight-loading stages on standard error. Later invocations reuse the pinned local snapshot; offline=true explicitly forbids network access. Close the provider with the resource block to release packed weights. Device.auto chooses a compatible NVIDIA GPU, otherwise the CPU; explicit GPU selections never silently fall back to CPU.

The model stays resident across calls to session.ask(). For low-level requests, provider.complete(LLMRequest(messages)) accepts typed LLMThreadMessage values and returns an LLMResponse, including prompt and completion token counts. Set provider.maxNewTokens to change the response bound (128 by default); prompt plus bound must fit maxContext. The completion count includes a generated EOS, but the response text does not.

The implementation supports single-file Qwen3.5 GGUF text chat in BF16, F16, Q8_0, Q6_K, Q6_K_L, Q5_K_M, Q5_K_S, Q4_K_M and Q4_K_S variants, with the embedded byte-level tokenizer, deterministic greedy decoding and thinking disabled. Vision, tools, split files, NPU inference and continuous batching are not implemented; unsupported requests raise an exception, never a silent conversion or fallback. availableQuantizations(repository) lists published variants, not backend capabilities. supportedQuantizations(repository, device) lists compatible published formats; it does not guarantee that the model and its context fit in memory. Try samples/llm/RunLLM.kn for the complete example.

Complete Sample

klyn samples/llm/RunLLM.kn

This interactive chat example uses the same LLMChatSession API with a local model. See Code Examples for the full list of samples.

Next Step

Continue with GPU 3D Scenes.