Anson Ho

Tamay Besiroglu

Ege Erdil

David Owen

Robi Rahman

Zifan Carl Guo

David Atkinson

Neil Thompson

Jaime Sevilla

March 9, 2024

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using 200+ evaluations (2012–2023) on Wikitext and Penn Treebank, we find the compute required to reach a fixed performance halves roughly every 8 months (95% CI ≈ 5–14 months), faster than Moore's Law. Augmented scaling laws quantify algorithmic progress versus scaling, showing that increased compute still contributed even more to improvements over this period, despite rapid algorithmic advances.

Paper Link

Cite this work

March 9, 2024

Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, Jaime Sevilla (). Algorithmic progress in language models. Published in Taylor & Francis. Retrieved from https://arxiv.org/abs/2403.05812. Accessed .

Copied

@article{,
  title     = { Algorithmic progress in language models },
  author    = { Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, Jaime Sevilla },
  journal   = { Taylor & Francis },
  year      = {  },
  url       = { https://arxiv.org/abs/2403.05812 }
}

Copied

Related Publications

Introducing the Quantum Economic Advantage Online Calculator

August 2025

Frederick Mejia, Hans Gundlach, Jayson Lynch, Carl Dukatz, Andrew Lucas, Eleanor Crane, Prashant Shukla, Neil Thompson

We introduce an open-access web tool that estimates when quantum systems will outperform classical computers for specific algorithms on a cost‑equivalent basis. Built on the Choi–Moses–Thompson framework, it lets users vary assumptions about error correction, gate speeds, overheads, connectivity, hardware roadmaps, and classical baselines. We find that expected timing for advantage is robust for some algorithms (e.g., Shor's) but contingent for others (e.g., Grover's), depending on technical factors.

The Computational Limits of Deep Learning

July 2020

Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, Gabriel F. Manso

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

How Fast do Algorithms Improve?

September 2021

Yash Sherry, Neil Thompson

Algorithms are one of the fundamental building blocks of computing. But current evidence about how fast algorithms improve is anecdotal, using small numbers of case studies to extrapolate. In this work, we gather data from 57 textbooks and more than 1,137 research papers to present the first systematic view of algorithm progress ever assembled. There is enormous variation. Around half of all algorithm families experience little or no improvement. At the other extreme, 13% experience transformative improvements, radically changing how and where they can be used. Overall, we find that, for moderate-sized problems, 30% to 45% of algorithmic families had improvements comparable or greater than those that users experienced from Moore's Law and other hardware advances

On the Origin of Algorithmic Progress in AI

November 2025

Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson

Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 (ho2024algorithmic). Running small-scale ablation experiments on key innovations from this time period, we are able to account for less than 10× of these gains. Surveying the broader literature, we estimate that additional innovations not included in our ablations account for less than 10×, yielding a total under 100×. This leads us to conduct scaling experiments, which reveal that much of this efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that – contrary to standard assumptions – an algorithm's efficiency gains are tied to compute scale. Using experimental extrapolation and literature estimates, we account for 6,930× efficiency gains over the same time period, with the scale-dependent LSTM-to-Transformer transition accounting for the majority of gains. Our results indicate that algorithmic progress for small models has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.

Deep Learning's Diminishing Returns: The Cost of Improvement is Becoming Unsustainable

October 2021

Neil C. Thompson; Kristjan Greenewald; Keeheon Lee; Gabriel F. Manso

Collectively, our study provides clear empirical evidence of a new form of influence on judges' application of the law—easily accessible, user-generated online content. Because such content is not authoritative, our analysis reveals a policy gap: if easily accessible analysis of legal questions is already being relied on, it behooves the legal community to accelerate efforts to ensure that such analysis is both comprehensive and expert.

Neural Scaling Laws in Robotics

May 2024

Sebastian Sartor, Neil Thompson

Neural scaling laws have driven advances in ML, yet robotics remains underexplored. Via a meta-analysis of 327 papers, we measure how data, model size, and compute affect performance for robot foundation models (RFMs) and LLMs on robotic tasks. Performance improves with resources following a power-law, scaling faster than in language tasks; new robot capabilities emerge as scale increases, suggesting notable gains with more data and compute.