LocalLLMProvider

classin packageklyn.genai.llm.inference

public class LocalLLMProvider implements LLMProvider, AutoClosable

LocalLLMProvider

All Implemented Interfaces: LLMProvider, AutoClosable

Local token-level Qwen3.5 text inference on CPU or NVIDIA GPU. Parameters stay packed on the selected device. Activations, KV caches and gated delta states accumulate in FP32. No Python, CUDA Toolkit or external model runtime is invoked. Implements LLMProvider for LLMChatSession; the session owns conversation history, while this provider owns model resources. Close the provider explicitly to release parameter storage.

Supports original F32/F16/BF16 Safetensors, local Klyn Q8 checkpoints and GGUF F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K tensors, including mixed K-quantization variants. Text chat uses the embedded Qwen35 tokenizer and pinned text/tool template. Native typed tool_calls are returned to the caller; this provider never executes tools. Generation is deterministic greedy decoding with sequential prefill, not continuous batching or sampling. Optional exact system/schema-prefix checkpoints avoid repeating that prefill; every request receives independent sequence storage. Unsupported variants are rejected. Concurrent use is unsupported.

Example

try provider = LocalLLMProvider.load("bartowski/Qwen_Qwen3.5-9B-GGUF"): session = LLMChatSession(provider).withSystem("You are a helpful assistant.") print(session.ask("Hello"))

Fields

Modifier and Type	Member	Description
public	contextCacheMinTokens `contextCacheMinTokens as Long = 128l`	Minimum input tokens beyond the static frame before retaining a cache.
public	contextCacheScope `contextCacheScope as String = ""`	Optional client/conversation identity for one additional message cache.
public	maxNewTokens `maxNewTokens as Long = 128l`	Maximum generated tokens per complete() request, including a terminating EOS token.
public	prefixCaching `prefixCaching as Boolean = false`	Opt-in reuse of the exact initial system/tool-schema frame only.

Properties

Modifier and Type	Member	Description
public readonly property	cachedContextTokens `cachedContextTokens as Long`	Retained complete message-frame tokens in the current scope.
public readonly property	cachedPrefixTokens `cachedPrefixTokens as Long`	Number of resident initial-frame tokens, not a generation speed estimate.
public readonly property	device `device as Device`	Concrete device, fixed for this provider's lifetime.
public readonly property	maxContext `maxContext as Long`	Maximum prompt plus generated sequence length.
public readonly property	parameterBytes `parameterBytes as ULong`	Packed device parameter bytes, counting shared embeddings only once and excluding caches/activation buffers.
public readonly property	vocabularySize `vocabularySize as Long`	Vocabulary size, including reserved IDs.

Constructors

Modifier and Type	Member	Description
public	LocalLLMProvider `LocalLLMProvider(path as Path, device as Device = Device.auto, maxContext as Long = 2048l, verbose as Boolean = false)`	Loads a local indexed Q8 model, validating its supported text architecture.

Methods

Modifier and Type	Member	Description
public static	availableQuantizations `availableQuantizations(repository as String, revision as String = "main") as ArrayList<ModelQuantization>`	Lists published GGUF variants without downloading weights.
public	chat `chat<MESSAGE>(messages as IList<MESSAGE>, maxNewTokens as Long = 128l) as String`	Generates a textual assistant response from a text-only Qwen conversation.
public	clearContextCache `clearContextCache() as Void`	Erases the conversation checkpoint while retaining model/system state.
public	clearPrefixCache `clearPrefixCache() as Void`	Releases the optional sequence checkpoint without unloading weights.
public override	close `close() as Void`	Releases all packed parameter allocations, idempotently.
public override	complete `complete(request as LLMRequest) as LLMResponse throws Exception`	Completes a typed text conversation using resident model weights.
public	completeStreaming `completeStreaming(request as LLMRequest, observer as LocalGenerationObserver) as LLMResponse throws Exception`	Generates incrementally on the calling thread, with cooperative stopping.
public	generateTokens `generateTokens(notnull prompt as IList<Long>, maxNewTokens as Long = 128l) as ArrayList<Long>`	Generates new token IDs by finite-logit argmax, without decoding text.
public static	load `load(repository as String, quantization as ModelQuantization = ModelQuantization.AUTO, device as Device = Device.auto, maxContext as Long = 2048l, revision as String = "main", offline as Boolean = false, verbose as Boolean = false, token as String = "", format as ModelFormat = ModelFormat.AUTO) as LocalLLMProvider`	Downloads the requested Hub variant if necessary, verifies it and loads it on the selected CPU/GPU.
public	primePrefix `primePrefix(request as LLMRequest, observer as LocalGenerationObserver) as Long`	Precomputes the initial system/schema frame without generating text.
public	promptTokenCount `promptTokenCount(request as LLMRequest) as Long`	Counts the exact rendered prompt, including template markers, without allocating inference states.
public static	supportedQuantizations `supportedQuantizations(repository as String, device as Device, revision as String = "main") as ArrayList<ModelQuantization>`	Lists published weight formats executable by this provider on the selected device.

Inherited Methods

methodInherited Methods from AutoClosable: close