reading

I'm not a blogger, but I do read a lot of papers. This is where I put down what I actually understood, what confused me, and sometimes a working implementation. Mostly for my own clarity — I would love to know if it helps you too.

Grokking: Generalisation Beyond Overfitting on Small Algorithmic Datasets May 2025

Power et al., 2022 · OpenAI

Reproduction of the grokking phenomenon on modular arithmetic. I traced which circuits become active before and after the generalisation phase transition, and found that weight norm regularisation alone substantially accelerates grokking — which has direct implications for my own work on subspace crystallisation.

mechanistic interp generalization implementation analysis notes → code → paper →
Progress Measures for Grokking via Mechanistic Interpretability — upcoming —

Nanda et al., 2023 · DeepMind

Nanda et al. identify progress measures — internal quantities that grow monotonically during training and predict generalisation. I want to verify these on my own grokking experiments and check whether the same measures hold for non-modular tasks.

mechanistic interp planned paper →

// more coming — I read faster than I write