DeviceQuantizedMatrix

classin packageklyn.math.nn.quantization

public class DeviceQuantizedMatrix implements AutoClosable

All Implemented Interfaces: AutoClosable

Resident CPU or NVIDIA weights, loaded in bounded blocks. GGUF F16, BF16 and K-quants retain their original packed storage; Q8 uses unchanged codes with FP32 scales. No full floating expansion or duplicate host weight copy is retained. Close explicitly after use; the GC also releases abandoned device allocations. Operations on a closed resource fail rather than selecting another execution device. Higher-rank checkpoint weights flatten all dimensions except the last.

Example

try matrix = DeviceQuantizedMatrix(Path("model-q8.safetensors"), "projection.weight"): print(matrix.multiply(NDArray<Float>.ones([1l, matrix.columns])))

Properties

Modifier and Type	Member	Description
public readonly property	columns `columns as Long`	Last tensor dimension.
public readonly property	device `device as Device`	Concrete CPU or NVIDIA device owning this allocation.
public readonly property	rows `rows as Long`	Flattened output rows.
public readonly property	shape `shape as IList<Long>`	Original checkpoint dimensions.
public readonly property	storageBytes `storageBytes as ULong`	Physical packed weight and scale bytes.

Constructors

Modifier and Type	Member	Description
public	DeviceQuantizedMatrix `DeviceQuantizedMatrix(notnull values as NDArray<Float>, device as Device = Device.auto)`	Copies a computed F32 matrix into owned resident storage in bounded blocks.
public	DeviceQuantizedMatrix `DeviceQuantizedMatrix(path as Path, name as String, device as Device = Device.auto)`	Loads one Q8 tensor without retaining the open file.
public	DeviceQuantizedMatrix `DeviceQuantizedMatrix(reader as GGUFReader, info as GGUFTensor, device as Device, buffer as Byte[], reorder as Int = 0, keyHeads as Long = 1l, valueHeads as Long = 1l, valueDim as Long = 1l)`	Loads exact GGUF F32, F16, BF16, Q8_0, Q4_K, Q5_K or Q6_K parameters.
public	DeviceQuantizedMatrix `DeviceQuantizedMatrix(reader as SafetensorsReader, info as TensorInfo, device as Device, buffer as Byte[])`	Loads a published F32/F16/BF16 Safetensors matrix in its original encoding, without silently quantizing it.
public	DeviceQuantizedMatrix `DeviceQuantizedMatrix(reader as SafetensorsReader, name as String, device as Device, buffer as Byte[] = null)`	Loads through a shared descriptor reader; the caller owns and closes it.

Methods

Modifier and Type	Member	Description
public	close `close() as Void`	Releases packed device storage immediately and idempotently.
public	convolveInto `convolveInto(output as NDArray<Float>, input as NDArray<Float>, state as NDArray<Float>) as Void`	One causal depthwise convolution followed by SiLU, updating its history.
public	embeddingInto `embeddingInto(output as NDArray<Float>, row as Long) as Void`	Decodes one embedding row into a reusable Float buffer, on its device.
public	multiply `multiply(inputs as NDArray<Float>) as NDArray<Float>`	Computes [batch, columns] times the packed weight transpose on its device.
public	multiplyInto `multiplyInto(output as NDArray<Float>, inputs as NDArray<Float>) as Void`	Uses a reusable result buffer inside the owning CPU/GPU execution scope.

Inherited Methods

methodInherited Methods from AutoClosable: close