public GatedDeltaAttention(keyHeads as Long, valueHeads as Long, keyDim as Long, valueDim as Long, kernelWidth as Long, epsilon as Double = 1e-6)
Creates an initially zero recurrent state.
| Parameter | Description |
|---|---|
keyHeads | Key head count. |
valueHeads | Value count, a multiple of keyHeads. |
keyDim | Key dimension. |
valueDim | Value dimension. |
kernelWidth | Causal convolution width. |
epsilon | Positive output RMS stabilizer. |
state = GatedDeltaAttention(16l, 32l, 128l, 128l, 4l)