LocalLLMProvider

classin packageklyn.llm.inference

public class LocalLLMProvider implements LLMProvider, AutoClosable

LocalLLMProvider

All Implemented Interfaces: LLMProvider, AutoClosable

Local token-level Qwen3.5 text inference on CPU or NVIDIA GPU. Parameters stay packed on the selected device. Activations, KV caches and gated delta states accumulate in FP32. No Python, CUDA Toolkit or external model runtime is invoked. Implements LLMProvider for LLMChatSession; the session owns conversation history, while this provider owns model resources. Close the provider explicitly to release parameter storage.

Supports local Klyn Q8 checkpoints and GGUF F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K tensors, including mixed K-quantization variants. Text chat uses the embedded Qwen35 tokenizer and text-only template contract. Generation is deterministic greedy decoding with sequential prefill, not continuous batching or sampling. Unsupported variants are rejected. Concurrent use is unsupported.

Example

try provider = LocalLLMProvider.load("bartowski/Qwen_Qwen3.5-9B-GGUF"): session = LLMChatSession(provider).withSystem("You are a helpful assistant.") print(session.ask("Hello"))

Fields

Modifier and Type	Member	Description
public	maxNewTokens `maxNewTokens as Long = 128l`	Maximum generated tokens per complete() request, including a terminating EOS token.

Properties

Modifier and Type	Member	Description
public readonly property	device `device as Device`	Concrete device, fixed for this provider's lifetime.
public readonly property	maxContext `maxContext as Long`	Maximum prompt plus generated sequence length.
public readonly property	parameterBytes `parameterBytes as ULong`	Packed device parameter bytes, counting shared embeddings only once and excluding caches/activation buffers.
public readonly property	vocabularySize `vocabularySize as Long`	Vocabulary size, including reserved IDs.

Constructors

Modifier and Type	Member	Description
public	LocalLLMProvider `LocalLLMProvider(path as Path, device as Device = Device.auto, maxContext as Long = 2048l, verbose as Boolean = false)`	Loads a local indexed Q8 model, validating its supported text architecture.

Methods

Modifier and Type	Member	Description
public static	availableQuantizations `availableQuantizations(repository as String, revision as String = "main") as ArrayList<ModelQuantization>`	Lists published GGUF variants without downloading weights.
public	chat `chat<MESSAGE>(messages as IList<MESSAGE>, maxNewTokens as Long = 128l) as String`	Generates a textual assistant response from a text-only Qwen conversation.
public override	close `close() as Void`	Releases all packed parameter allocations, idempotently.
public override	complete `complete(request as LLMRequest) as LLMResponse throws Exception`	Completes a typed text conversation using resident model weights.
public	generateTokens `generateTokens(notnull prompt as IList<Long>, maxNewTokens as Long = 128l) as ArrayList<Long>`	Generates new token IDs by finite-logit argmax, without decoding text.
public static	load `load(repository as String, quantization as ModelQuantization = ModelQuantization.AUTO, device as Device = Device.auto, maxContext as Long = 2048l, revision as String = "main", offline as Boolean = false, verbose as Boolean = false, token as String = "") as LocalLLMProvider`	Downloads the requested Hub variant if necessary, verifies it and loads it on the selected CPU/GPU.
public static	supportedQuantizations `supportedQuantizations(repository as String, device as Device, revision as String = "main") as ArrayList<ModelQuantization>`	Lists published weight formats executable by this provider on the selected device.

Inherited Methods

methodInherited Methods from AutoClosable: close