Deep Learning

Build, Train and Evaluate

A Sequential<T> composes typed layers in order. This small classification example uses four rows of two features and integer class labels. The class is 1 when the first feature is 1, and 0 otherwise.

import klyn.math
import klyn.math.nn
import klyn.math.nn.layers

features = NDArray<Float>([
    [0f, 0f], [0f, 1f], [1f, 0f], [1f, 1f]
])
labels = NDArray<Int>([0, 0, 1, 1])

model = Sequential<Float>([
    Input<Float>((2,)),
    Dense<Float>(8, seed=17l, activation="relu"),
    Dense<Float>(2, seed=23l, activation="softmax")
])
model.compile(optimizer="adam")
model.to(Device.cpu)

history = model.fit(features, labels, epochs=200l, batchSize=4l, verbose=0)
result = model.evaluate(features, labels)
print(f"loss={result.loss:.4f}, accuracy={result.accuracy:.4f}")

predictedClasses = model.predict(features).argMax(1)
print(predictedClasses)

The first tensor dimension is the batch: each feature row is one observation. Input<Float>((2,)) declares the per-observation shape and builds the network immediately. The final two values are class probabilities. compile() defaults to sparse categorical cross-entropy, so labels contain class indices rather than one-hot vectors. argMax(1) selects one class index per row.

For multidimensional arrays and their arithmetic, see Math and Scientific Computing. Use separate training and validation data for a real experiment; fit(..., validationData=(xTest, yTest)) records validation metrics after each epoch.

Load Datasets as NDArrays

Dataset loaders live in klyn.math.datasets and return four independent, statically typed arrays in the order xTrain, yTrain, xTest, yTest. Missing data is downloaded once over HTTPS, verified, and cached outside your project. Downloads display progress on an interactive error console; cached loads are silent. Downloading uses the curl executable; decompression is performed by Klyn. Pass download=false to require already available local data.

import klyn.math.datasets

xTrain, yTrain, xTest, yTest = MNIST.load()
# Images: NDArray<Float>, shape [samples, 28, 28], normalized to [0, 1].
# Labels: NDArray<Int>, shape [samples], containing classes 0 through 9.

xTrain, yTrain, xTest, yTest = CaliforniaHousing.load()
# Features: NDArray<Float>, shape [samples, 8].
# Targets: NDArray<Float>, shape [samples], containing continuous values.

CaliforniaHousing contains 20,640 observations. The default seeded shuffle (seed=42l, testSplit=0.2) produces 16,512 training and 4,128 test observations. CaliforniaHousing.featureNames is an immutable list identifying the columns: median income, house age, average rooms, average bedrooms, population, average occupancy, latitude and longitude. Average counts are computed per household; the target is the median house value in units of 100,000 US dollars. Features are not automatically standardized: fit any preprocessing statistics on training data only, then apply them to both splits.

import klyn.io
import klyn.math.datasets

xTrain, yTrain, xTest, yTest = CaliforniaHousing.load<Double>(
    folder=Path("datasets/housing"),
    testSplit=0.25,
    seed=123l
)
print(CaliforniaHousing.featureNames)
print(xTrain.shape, yTrain.shape, xTest.shape, yTest.shape)

All four CaliforniaHousing tensors use the selected precision: Float by default, or Float16 / Double. FP16 storage is genuinely 16-bit; values overflowing that precision are rejected. shuffle=false preserves source row order. The local shuffle does not alter application-wide randomness and is reproducible within Klyn, not an identical permutation to that of other libraries. Optional trainLimit and testLimit select prefixes of the established partitions without changing their membership.

The default cache is ~/datasets/mnist for MNIST and ~/datasets/california_housing/cal_housing.data for CaliforniaHousing. A supplied local CaliforniaHousing subset may contain 2 to 20,640 records in the original, headerless nine-column format, one record per physical line. Loading uses bounded buffers, a row permutation, and the four output tensors; no additional full-sized feature tensor is retained.

Regression with CaliforniaHousing

import klyn.math.datasets
import klyn.math.nn
import klyn.math.nn.layers
import klyn.math.preprocessing

xTrain, yTrain, xTest, yTest = CaliforniaHousing.load()
scaler = StandardScaler<Float>()
scaler.fitTransform(xTrain, inPlace=true)
scaler.transform(xTest, inPlace=true)
model = Sequential<Float>([
    Input<Float>((Long(xTrain.shape[1]),)),
    Dense<Float>(64, activation="sigmoid"),
    Dense<Float>(32, activation="sigmoid"),
    Dense<Float>(16, activation="sigmoid"),
    Dense<Float>(1)
])
model.compile(optimizer="adam", loss="mean_squared_error")
model.summary()
history = model.fit(xTrain, yTrain, epochs=20,
                    validationData=(xTest, yTest))
predictions = model.predict(xTest)
print(history.validationLoss[-1])

"mse" and "mean_squared_error" are equivalent. The typed objective is MeanSquaredError<T> in klyn.math.nn.losses. It averages squared residuals over every observation and output coordinate and computes the corresponding mean-reduced derivative. CPU and GPU paths fuse these operations without temporary residual and squared-residual tensors. Float16 keeps its 16-bit tensor storage.

Regression targets use the model's precision and match its output shape. A [samples] vector is also accepted for a single linear output [samples, 1]. A final sigmoid or Softmax is not added implicitly. The default metrics list is empty for MSE; asking for metrics=["accuracy"] raises ValueException. Classification still defaults to accuracy.

Progress shows loss and, when supplied, val_loss. History provides cached immutable epochNumbers, trainingLoss and validationLoss series. Evaluation.hasAccuracy is false; reading regression accuracy or accuracy history raises ValueException rather than inventing a metric.

The complete samples/math/nn/CaliforniaDNN.kn example standardizes both feature splits with training-only statistics, trains for 20 epochs and compares observed and predicted house values using two coolwarm scatter charts. Marker area is population divided by 100. Both charts share fixed color limits from the observed targets and draw the highest-value housing last. Prediction outliers saturate the palette without rescaling the observed chart or changing the values used to calculate loss. A commented checkpoint line saves the model; retain the preprocessing statistics separately for future inputs.

Neural Network Activations

Sigmoid<T> maps each value into [0, 1], while Tanh<T> maps into [-1, 1]. Both preserve the batch shape, support Float16, Float and Double, and run on CPU or a supported NVIDIA GPU. Their derivatives reuse saved outputs during training, without materializing a Jacobian.

import klyn.math
import klyn.math.nn
import klyn.math.nn.layers
import klyn.math.nn.activations

inputs = NDArray<Float>([[-1f, 0f, 1f]])
probabilities = Sigmoid<Float>().forward(inputs)
centered = Tanh<Float>().forward(inputs)
assert probabilities[0, 1] == 0.5f
assert centered[0, 1] == 0f

model = Sequential<Float>([
    Input<Float>((3,)),
    Dense<Float>(16, activation="sigmoid"),
    Dense<Float>(8, activation="tanh"),
    Dense<Float>(2, activation="softmax")
])
model.compile()

These activations can also follow a convolution, or be configured directly with Conv2D(..., activation="sigmoid") or activation="tanh", including transposed convolutions. NaN propagates; infinities saturate to the activation's endpoints. The classification loss still requires a Softmax output, or linear logits when configured with fromLogits=true.

FP16 Storage and Training

Use Sequential<Float16> with Float16 layers and MNIST.load<Float16>() to keep images, parameters and activations in FP16. Matrix products accumulate in FP32 before rounding their output. Adam keeps separate FP32 master weights and moments so small updates are not lost; training uses dynamic gradient scaling. These optimizer buffers are additional memory, not FP16 parameter storage. summary() reports parameter bytes (weights and biases), excluding optimizer state, activations and object metadata. FP16 reduces tensor storage, but is not a promise of faster arithmetic on every CPU.

On compatible x86 CPUs, matrix products and Float16 Adam updates use AVX2 and hardware FP16 conversions. Other CPUs retain a portable implementation. Parameters remain stored in two bytes on either path; no full-size FP32 copy of the tensors is needed for matrix multiplication.

klyn samples/math/nn/MnistDNN-FP16.kn

CPU and GPU Execution

Neural-network models default to Device.auto. Use model.to(Device.cpu) to require CPU execution, model.to(Device.gpu) to select one GPU, or model.to(Device.gpu[0]) to select a specific GPU. Device.list() lists detected devices and model.device reports the resolved device. NVIDIA execution on Linux and Windows requires only a compatible GPU and its installed driver, not CUDA Toolkit. Weights and optimizer state survive device changes. Unsupported explicit selections raise an exception; only automatic selection may choose CPU instead. NPU selectors exist, but currently report that execution support is coming in a future release.

With verbose=1 or verbose=2, fit() prints the training device before the first epoch. Training metrics aggregate predictions made while parameters change between batches; validation and evaluation use the final, fixed parameters with dropout disabled. Their accuracy can therefore exceed the training accuracy, especially during the first epochs. Compare the two dataset splits with evaluate() when you need metrics for the same model state.

Complete Samples

The MNIST examples demonstrate larger dense and convolutional networks. Training may download the dataset on its first invocation; choose a workload and device that fit your machine.

klyn samples/math/nn/MnistCNN.kn

The dense versions are samples/math/nn/MnistDNN.kn and samples/math/nn/MnistDNN-FP16.kn. Find these files alongside the convolutional example.

Next Step

Continue with Model Checkpoints.