Deep Learning & GenAIAudio

Speech Synthesis And Recognition

Generate and recognize speech with typed audio, explicit playback and explicit microphone access.

Packages And Devices

klyn.genai.llm contains language-model providers; klyn.genai.tts contains speech synthesis, and klyn.genai.stt contains speech recognition. They share the klyn.genai.hub download cache. LLM/TTS GGUF selection uses klyn.genai.ModelQuantization. All three APIs use ModelFormat.AUTO, GGUF or SAFETENSORS to select the physical container. A format is not a model architecture: unsupported families still fail explicitly. Device remains in klyn.math.nn. There are no legacy package aliases.

On Linux and Windows, Device.auto selects a compatible NVIDIA GPU when available, otherwise the CPU. Use Device.gpu[0] to require a specific GPU, or Device.cpu for CPU-only execution. Packed projections and embeddings stay on the selected device; temporal audio convolutions, sampling and sliding attention currently run on CPU with synchronized tensors. An explicit GPU request never silently falls back to CPU. NPU requests raise NotSupportedException before downloading a model. The engine is implemented directly in Klyn/C++ using Klyn's existing NN and quantized-matrix backend. Importing GenAI does not load weights or initialize a device. No additional inference library, Python service or external speech executable is used.

The first exercised checkpoint is Qwen3-TTS CustomVoice 0.6B Q8_0. Model dimensions, code-group counts, sampling settings and temporal strides come from metadata. The weight reader accepts F32, F16, BF16, Q8_0, Q4_K, Q5_K and Q6_K: projections stay packed, while temporal convolution kernels are decoded once to Float32. This prepares other quantizations; it does not imply that every model architecture or variant has been tested.

klyn.audio is not an AI package. Its AudioClip stores finite, interleaved Float32 samples, a sample rate and a channel count. frames counts complete frames; duration is measured in seconds. The supplied sample array is retained, not copied.

Download, Synthesize And Play

import klyn.genai
import klyn.genai.tts
import klyn.io
import klyn.math.nn

try speech = SpeechSynthesizer.load(
    "sakasegawa/Qwen3-TTS-12Hz-0.6B-CustomVoice-GGUF",
    ModelQuantization.Q8_0,
    device=Device.cpu,
    verbose=true
):
    print(speech.device)
    print(speech.voices)
    clip = speech.synthesize("Hello, Klyn.", voice="ryan", language="en")
    clip.save(FilePath("speech.wav"))
    clip.play()

Weights remain resident until the synthesizer is closed. To reuse an existing local GGUF file, construct SpeechSynthesizer(FilePath("voice.gguf")). Model architecture, voice and language options are validated by the backend; a text-model GGUF is not a speech checkpoint.

The same Qwen3-TTS CustomVoice architecture can load original HF Safetensors, including its speech_tokenizer subdirectory: SpeechSynthesizer.load("Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice", format=ModelFormat.SAFETENSORS). Its published precision is retained, without requantization. A local snapshot uses SpeechSynthesizer(FolderPath("voice-model")). EMA codebooks, Snake scales and temporal filter layouts are prepared once during loading; no intermediate model file or Python process is used.

The first end-to-end validation targets Qwen3-TTS CustomVoice 0.6B in Q8. Matrix encodings F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K use the existing packed backend. Layer sizes, head geometry, code-group counts and temporal strides come from metadata. A separate code-predictor input projection is supported when the talker and predictor have different widths. These extension paths do not mean every checkpoint has been tested.

maxContext bounds prompt tokens plus generated codec frames (default 1024); maxFrames bounds generated frames (default 512). Reaching the latter before EOS raises an explicit error rather than silently presenting truncated speech as complete. seed controls repeatable per-utterance sampling without changing global random state.

The optional instructions argument to stream() and synthesize() is prepared for compatible larger CustomVoice models. The 0.6B checkpoint rejects nonempty instructions explicitly. Unknown model families and layout versions are also rejected, never replaced by another synthesizer.

synthesize() creates a complete clip and does not play it. play() sends audio to the operating system's default output, which can be speakers, headphones or another configured device, and blocks until completion. Linux uses PulseAudio; Windows uses its native audio service. An unavailable output raises an explicit exception. No microphone is opened.

WAV loading supports integer PCM8/16/24/32 and IEEE Float32. Saving uses IEEE Float32 without resampling, normalization or clipping. Other encodings are rejected explicitly.

Incremental PCM And Cancellation

try stream = speech.stream("Hello, Klyn.", voice="ryan", language="en"):
    while true:
        block = stream.read()
        if block is null:
            break
        print(block.frames, block.sampleRate, block.channels)

A stream has one consumer. Its producer queues at most eight blocks of 2048 mono Float32 samples (64 KiB), waiting when the queue is full. read() waits for a block, returns null at the end or after cancellation, and throws on failure. Returned clips remain valid after stream closure. The adapter does not call Klyn callbacks on its native worker.

stream.cancel() may be called from another thread and wakes a blocked producer. Unread blocks are discarded; already returned blocks remain valid. generatedSamples counts generated scalar samples, including queued blocks. Always close the stream, including on early exit; closure cancels and joins its worker. A synthesizer accepts one active request at a time. Use the same synthesizer for later utterances.

The PCM contract is reusable by KlynOS without coupling synthesis to its audio device or assistant transport. A runtime without a speech engine or output device reports that service as unavailable; it does not simulate it.

Transcribe Recorded Audio

import klyn.audio
import klyn.genai.stt
import klyn.io
import klyn.math.nn

try speech = SpeechRecognizer.load("openai/whisper-base", device=Device.auto):
    clip = AudioClip.load(FilePath("recording.wav"))
    result = speech.transcribe(clip, language="auto")
    print(result.language, result.text)
    # Equivalent WAV shortcut:
    print(speech.transcribe(FilePath("recording.wav"), language="fr").text)

The engine implements Whisper and Qwen3-ASR in Klyn/C++, using the existing NN backend; it does not launch Python, an external executable or an additional inference library. The multilingual openai/whisper-base checkpoint is downloaded once and reused. Original F32/F16/BF16 Safetensors are accepted. Qwen3-ASR additionally accepts paired GGUF decoder/audio-projector files. Other architectures are rejected explicitly. For a local Whisper snapshot, use SpeechRecognizer(FolderPath("models/whisper-base")). verbose=false is the default; offline=true requires a complete cached snapshot.

import klyn.genai
import klyn.genai.stt
import klyn.io

try speech = SpeechRecognizer.load(
    "unslothai/Qwen3-ASR-0.6B-GGUF", ModelQuantization.Q8_0
):
    result = speech.transcribe(FilePath("recording.wav"), language="fr")
    print(result.text)

# Same architecture, original HF precision:
try speech = SpeechRecognizer.load(
    "Qwen/Qwen3-ASR-0.6B", format=ModelFormat.SAFETENSORS
):
    print(speech.transcribe(FilePath("recording.wav")).text)

The GGUF download includes both the selected decoder and its matching mmproj- audio encoder, with pinned revision, SHA-256 verification and a separate completion manifest. A projector is never treated as a complete model. For local paired files use SpeechRecognizer(FilePath("model-Q8_0.gguf"), FilePath("mmproj-model-Q8_0.gguf")). Whisper and Qwen3-ASR remain separate architecture adapters. Legacy PyTorch Pickle .bin and Whisper GGML binaries are not supported.

Device.auto selects a compatible NVIDIA GPU, otherwise CPU. Encoder attention and projections use that device; Qwen3-ASR convolutions use the shared NN backend. Spectral processing, Whisper convolutions and cached decoder attention currently use CPU storage. Explicit GPU requests do not silently fall back; NPU is not implemented. Audio is downmixed and resampled to mono 16 kHz without modifying the input. Files longer than 30 seconds are processed in consecutive windows. Segment start/end values are window bounds, not word alignments.

Transcription exposes readonly text, language and an immutable segments catalogue. language="auto" detects the language; a multilingual model also accepts "fr" and "en". This API transcribes; it does not translate. Silence produces an empty result. An unsupported language, malformed tensor or decoder context exhaustion raises an error, rather than silently returning a truncated result.

Live Microphone Recognition

import klyn.audio
import klyn.genai.stt

try speech = SpeechRecognizer.load("openai/whisper-base"):
    try microphone = Microphone.open(bufferSeconds=30):
        try updates = speech.stream(microphone, language="auto"):
            for update in updates:
                if update.isFinal:
                    print(update.text)
                # Another thread/UI action calls microphone.stop() to finish.

Opening a microphone is an explicit recording action. Linux uses the system PulseAudio service (including compatible PipeWire/WSLg services); Windows uses native microphone capture and requires microphone access to be permitted by the OS. No capture starts when importing a package. A missing/denied input raises IOException. KlynOS microphone capture is not connected yet and raises NotSupportedException.

inputs = Microphone.listDevices()  # Discovery only: does not record.
for audioDevice in inputs:
    print(audioDevice.name, audioDevice.id)

if inputs.size > 0ul:
    try microphone = Microphone.open(bufferSeconds=30, device=inputs[0l]):
        block = microphone.read()

AudioInputDevice exposes readonly id and name. Keep the descriptor rather than its catalogue position: names can be identical and enumeration order can change. The ID is opaque and specific to the OS backend. Passing device=null (the default) follows the system default input at opening time; an explicit unavailable input raises an error instead of selecting another microphone. Discovery returns a fresh catalogue and may wait for the audio service; run it outside GUI callbacks. Linux may also expose monitor/loopback sources, and WSLg exposes the inputs provided by its audio service.

microphone.level reads the latest input RMS without consuming audio, maximumLevel retains the largest block RMS since opening, and capturedFrames counts accepted mono frames, including unread audio. These constant-time statistics remain available after stop(), until close(). They distinguish missing input, silence and model processing; no extra recording is started.

This API currently recomputes bounded Whisper/Qwen3-ASR audio windows; it does not claim native incremental decoding. The adapter uses energy-based voice activity detection, 200 ms pre-roll, 700 ms trailing silence and a maximum 25-second utterance. It recomputes partial hypotheses every four audio seconds. Latency and VAD accuracy depend on the device, microphone level and background noise; this is not a hard real-time guarantee.

minimumSpeechLevel configures the energy floor (default RMS 0.0003, with adaptive noise estimation). Zero disables this gate and lets the model inspect quiet audio. With partialResults=false, inference occurs only at utterance boundaries and stop, rather than repeatedly computing unused partial text. Keep partialResults=true when displaying revisable text during recording; use false only for a final-only consumer. Audio is still bounded to 25-second inference windows and cancellation remains cooperative.

An update exposes segmentId, text, isFinal, language, start and end. Replace earlier partial text for the same segment; append it only when isFinal is true. Normal microphone.stop() preserves queued PCM and finalizes its tail. updates.cancel() stops capture and discards pending hypotheses. Closing the stream joins its worker but leaves microphone ownership to the caller. Always close the stream before closing the microphone.

One recognizer serves one request at a time. The capture buffer defaults to ten seconds and is configurable from one to sixty seconds. Unlike an output stream it cannot apply backpressure to incoming real-world audio: overrun raises an explicit error, never silent sample loss. The result queue holds at most eight updates. Never perform blocking model loading, recognition or read() calls inside GUI callbacks.

Sample

Run klyn samples/genai/TTS.kn. LLM and speech examples share the flat samples/genai directory.

Run klyn samples/genai/STT.kn for a GUI with a Record/Stop button, an audio input selector, an Automatic/French/English language selector and a selectable, scrollable multiline transcript. Input discovery, model loading and transcription run on workers; it uses unslothai/Qwen3-ASR-0.6B-GGUF in Q8_0. The transcript updates while recording; Stop finalizes the captured tail. Partial hypotheses replace the active utterance instead of being appended repeatedly. The model is reused for subsequent recordings. An input-level/frame counter distinguishes silent or missing capture from transcription in progress. The GUI disables the energy gate for quiet inputs and enables intermediate hypotheses. Record is enabled after background model preparation, before any microphone is opened. Closing during the first download waits for that operation to finish before releasing its resources.