Virtual Booth

Project Abstract

Delta is a chat language model that replaces attention with a recurrent, attention-free stack, built in JAX and Equinox without a Hugging Face backbone. The project asks whether such an architecture can be trained into a chat model on a single GPU, avoiding the quadratic cost of attention at 512 to 2,048 token contexts. Each of its 64 layers (width 512) combines a recurrent time mix that carries context along the sequence, a per-token FFT frequency mix with a learned filter, and an MLP channel mix with a learnable xIELU activation, giving 134.6M parameters in the logged run. Training follows a two-stage pipeline: pretraining on about 1B unique tokens of FineWeb-Edu, split evenly between clean and noisy text, followed by SmolTalk chat fine-tuning using [TURN] symbols. Length buckets, automatic batch sizing, and Schedule-Free AdamW keep training efficient on one A100 80 GB. In the logged run, validation perplexity fell from 37.0 to 23.4 and accuracy rose from 33.2% to 39.2% over three epochs (228M tokens), with validation loss still declining when training was stopped. Measured throughput (~15.6k tokens/s) trails the ~44k benchmark, so future work will close this gap before full pretraining, chat fine-tuning, and evaluation.