One useful way to think about machine learning is in terms of two interacting elements: (1) the architecturee.g., transformers, recurrent networks, state space models defines the model’s computational primitives and the space of solutions it can represent; and (2) the learning algorithmthe objective, supervision scheme, and optimizer that fit the model to data; e.g., next-token prediction or reinforcement learning from verifiable rewards, optimized by gradient descent defines the search procedure that selects a model using data. In addition to defining the search space, the architecture encodes inductive biases that interact with the learning algorithm to favor some solutions over others. The right biases steer learning toward solutions with desirable properties, enabling greater data efficiency and the formation of abstractions that generalize robustly beyond training.
This raises a natural question: which architectural mechanisms and inductive biases are the right ones? Are there fundamental primitives that a model needs in order to learn efficiently and generalize? I think answering this is one of the grand goals of machine learning research, just as the study of human and animal intelligence aims to explain intelligent behavior through a small set of simple principles.
Much of the challenge lies in the tension between structure and generality: while building in task-specific inductive biases can make learning more efficient in the short term, the history of AI shows that general methods that scale with data and computation eventually outperform methods that build in human knowledge1See Rich Sutton’s essay “The Bitter Lesson” (2019). incompleteideas.net . In my view, the resolution is to find fundamental primitives that are general enough to scale, forming flexible building blocks that the learning algorithm can compose, without prescribing the structure of the solution.
My own work on this question began with relational reasoning, which is believed to underlie human abilities for abstraction and systematic generalization. Its power lies in drawing inferences across instances that look very different on the surface but share an abstract structure, which gives humans a capacity for data-efficient, out-of-distribution generalization that machine learning has yet to match. A second mechanism that I believe is central to generalizing beyond experience is recurrence, which favors compact, recursive solutions and allows the model to adapt how much computation it uses to the difficulty of the problem.
Papers in this thread:
- Recursive latent space reasoning: explores recurrence, input-adaptive computation, and discrete latent states as inductive biases for depth-invariant computation, enabling transformers to learn algorithms that extend to problems far larger than those seen in training.
- Abstractors: a relational information bottleneck2 The Relational Bottleneck as an Inductive Bias for Efficient Abstraction arXiv inside the transformer, using relational cross-attention to separate the relations between objects from the objects’ own features, which makes relational learning more sample-efficient.
- Relational convolutions: an architecture for learning hierarchical relational feature representations by composing convolution-like relational operations.
- Dual Attention Transformer: adds relational attention as a primitive alongside standard attention and lets the model learn how to combine relational and sensory information, improving data and parameter efficiency, including in language modeling.