Awni Altabaa Blog https://awni.xyz/blog Academic website for Awni Altabaa, a PhD candidate studying the foundations of artificial intelligence. Putting Transformer Theories to the Test https://awni.xyz/blog/transformer-theories https://awni.xyz/blog/transformer-theories Fri, 23 Oct 2026 00:00:00 GMT A comparison and empirical evaluation of various formal theories of the capabilities & limitations of transformers. Uncovering Latent Reasoning Strategies in Language Models https://awni.xyz/blog/latent-strategies-in-lms https://awni.xyz/blog/latent-strategies-in-lms Mon, 20 Jul 2026 00:00:00 GMT Recovering a disentangled, strategy-aligned latent-variable factorization of a pretrained language model, and the model-directed reconstruction objective that makes it possible despite posterior collapse at initialization. Unlocking OOD Generalization via Latent Recurrence https://awni.xyz/blog/recursive-latent-reasoning https://awni.xyz/blog/recursive-latent-reasoning Wed, 15 Oct 2025 00:00:00 GMT Four architectural mechanisms that let Transformers learn algorithms that scale, on circuits several times larger than those seen in training. CoT Information: Improved Sample Complexity under Chain-of-Thought Supervision https://awni.xyz/blog/cot-info https://awni.xyz/blog/cot-info Wed, 21 May 2025 00:00:00 GMT A statistical learning theory for chain-of-thought supervision, centered on a CoT information measure that governs sample complexity. Disentangling and Integrating Relational and Sensory Information in Transformer Architectures https://awni.xyz/blog/dual-attention https://awni.xyz/blog/dual-attention Sun, 26 May 2024 00:00:00 GMT Dual Attention augments Transformers with separate mechanisms for routing sensory and relational information. On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games https://awni.xyz/blog/information-structure-rl https://awni.xyz/blog/information-structure-rl Fri, 01 Mar 2024 00:00:00 GMT We introduce a reinforcement learning model that explicitly represents information structure in sequential decision-making problems. Approximation of Relation Functions and Attention Mechanisms https://awni.xyz/blog/relation-function-approximation https://awni.xyz/blog/relation-function-approximation Tue, 13 Feb 2024 00:00:00 GMT Inner products of neural networks are universal approximators of relation functions; as an application, attention can approximate any preference-based selection. Learning Hierarchical Relational Representations through Relational Convolutions https://awni.xyz/blog/relational-convolutions https://awni.xyz/blog/relational-convolutions Thu, 05 Oct 2023 00:00:00 GMT Relational convolutional networks learn higher-order relational features by convolving graphlet filters against subsets of objects. The Relational Bottleneck as an Inductive Bias for Efficient Abstraction https://awni.xyz/blog/relational-bottleneck https://awni.xyz/blog/relational-bottleneck Tue, 12 Sep 2023 00:00:00 GMT We review relational bottleneck models as an inductive bias for data-efficient abstraction in minds and machines. Abstractors and Relational Cross-Attention: An Inductive Bias for Explicit Relational Reasoning in Transformers https://awni.xyz/blog/abstractor https://awni.xyz/blog/abstractor Sat, 01 Apr 2023 00:00:00 GMT Abstractors add relational cross-attention to Transformers as an inductive bias for explicit relational reasoning and abstraction. Decentralized Multi-Agent Reinforcement Learning for Continuous-Space Stochastic Games https://awni.xyz/blog/decentralized-marl-continuous-spaces https://awni.xyz/blog/decentralized-marl-continuous-spaces Thu, 16 Mar 2023 00:00:00 GMT We propose a decentralized multi-agent reinforcement learning algorithm for stochastic games with continuous state spaces.