Tiny Stories Diffusion Language Model

A diffusion language model built from scratch to tell tiny stories.

Formulated and trained a discrete, non-autoregressive diffusion language model from scratch that replaces left-to-right decoding with parallel, confidence-based iterative token unmasking.

  • Training pipeline with SwiGLU activations, TikToken (BPE) encoding, mask-token injection, and Rotary Positional Embeddings (RoPE) on the TinyStories dataset.
  • Reached a perplexity of 18.88, outperforming GPT-2’s baseline score of 19.93.

Tech Stack

  • Python, PyTorch, Hugging Face