GpuQuantizedMatrix

classin packageklyn.math.nn.quantization

public class GpuQuantizedMatrix extends DeviceQuantizedMatrix

DeviceQuantizedMatrix

└ GpuQuantizedMatrix

NVIDIA-only specialization of DeviceQuantizedMatrix. Resident weights are loaded in bounded blocks; GGUF F16, BF16 and K-quants retain their original packing, while Q8 retains integer codes with FP32 scales. No full floating or host weight copy is retained. Higher-rank tensors flatten all dimensions except the last. Close explicitly; the GC also releases abandoned allocations. Closed resources fail rather than selecting a different execution device. Use DeviceQuantizedMatrix when CPU execution or automatic CPU fallback is wanted.

Example

try matrix = GpuQuantizedMatrix(Path("model-q8.safetensors"), "projection.weight"): print(matrix.multiply(NDArray<Float>.ones([1l, matrix.columns])))

Inherited Properties

propertyInherited Properties from DeviceQuantizedMatrix: columns, device, rows, shape, storageBytes

Constructors

Modifier and Type	Member	Description
public	GpuQuantizedMatrix `GpuQuantizedMatrix(path as Path, name as String, device as Device = Device.gpu)`	Loads one Q8 tensor without retaining the open file.
public	GpuQuantizedMatrix `GpuQuantizedMatrix(reader as GGUFReader, info as GGUFTensor, device as Device, buffer as Byte[], reorder as Int = 0, keyHeads as Long = 1l, valueHeads as Long = 1l, valueDim as Long = 1l)`	Loads exact GGUF F32, F16, BF16, Q8_0, Q4_K, Q5_K or Q6_K parameters.
public	GpuQuantizedMatrix `GpuQuantizedMatrix(reader as SafetensorsReader, name as String, device as Device, buffer as Byte[] = null)`	Loads through a shared descriptor reader, owned and closed by the caller.

Inherited Methods

methodInherited Methods from DeviceQuantizedMatrix: close, convolveInto, embeddingInto, multiply, multiplyInto

methodInherited Methods from AutoClosable: close