A mathematical perspective on Transformers

 Physics has long supplied mathematics with some of its central equations. I will argue that Transformers, the engine of large language models, are becoming a source of the same kind. Starting from raw code, under various scaling assumptions, one can interpret Transformers as an interacting particle system in continuous time; tools from PDE and kinetic theory then explain empirically observed phenomena such as representation collapse and “lost in the middle”. Unlike in physics, however, the laws here are designed rather than given, so mathematical understanding can be written back into the architecture. I will close with some thoughts on what this two-way exchange means for applied mathematics.