Progress in machine learning has been driven largely by empirical and engineering advances, and at times it has looked more like alchemy than science1See Ali Rahimi and Ben Recht’s NeurIPS 2017 Test of Time talk. archives.argmin.net . Understanding has often lagged behind progress, and the gap has widened as models have become more capable. I believe in a scientific approach to machine learning, which has two complementary sides: empirical science, which studies AI systems as natural phenomena and seeks empirical laws, and mathematical theory, which seeks a deep and precise formal understanding.
Good theory explains observed phenomena through simple, general principles, predicts phenomena not yet observedSee Karl Popper, The Logic of Scientific Discovery (1959)., and guides the design of new architectures and algorithms. My theoretical work aims to give formal accounts of how architecture, learning algorithm, and problem structure determine what models can learn, and how efficiently.
Papers in this thread:
- CoT Information: a statistical learning theory of chain-of-thought supervision, centered on the CoT information, an information-theoretic measure of the statistical value of observing intermediate steps which governs sample complexity.
- Transformer theories: evaluates seven formal theories of transformer capabilities against experiments, mapping where each theory’s predictions hold and where they break down.
- Relation functions and attention: shows that inner products of neural networks, the primitive behind attention and relational architectures, can approximate broad classes of relation functions.
- Information structure in RL: formalizes the information structurethe causal dependencies in the dynamics of a sequential decision-making problem as a graph and bounds the sample complexity of reinforcement learning in terms of it, recovering known tractable classes and identifying new ones.
Broadly, I’m interested in developing theoretical frameworks that explain modern AI methods through statistical and computational principles. This includes questions such as what makes complex, multi-step tasks efficiently learnable, how inductive biases shape the solutions a model finds, and when those solutions generalize beyond training.