Large Language Model Routing with Benchmark Datasets

Tal Shnitzer

Anthony Ou

Mírian Silva

Kate Soule

Yuekai Sun

Justin Solomon

Neil Thompson

Mikhail Yurochkin

September 27, 2023
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.
Read More...

Related Publications

How Fast do Algorithms Improve?

September 2021
Yash Sherry, Neil Thompson

Algorithms are one of the fundamental building blocks of computing. But current evidence about how fast algorithms improve is anecdotal, using small numbers of case studies to extrapolate. In this work, we gather data from 57 textbooks and more than 1,137 research papers to present the first systematic view of algorithm progress ever assembled. There is enormous variation. Around half of all algorithm families experience little or no improvement. At the other extreme, 13% experience transformative improvements, radically changing how and where they can be used. Overall, we find that, for moderate-sized problems, 30% to 45% of algorithmic families had improvements comparable or greater than those that users experienced from Moore's Law and other hardware advances

What should be done about the growing influence of industry in AI research?

December 2023
Nur Ahmed and Neil C. Thompson

Artificial intelligence (AI) is becoming increasingly important for society and the economy. However, modern AI research requires expensive resources that are often only available to a small group of for-profit firms. Consequently, there is a growing concern that only a handful of companies will be able to perform cutting-edge research, leading to a worrisome concentration of power in AI development and its future.

The Grand Illusion: The Myth of Software Portability and Implications for ML Progress

September 2023
Fraser Mince, Dzung Dinh, Jonas Kgomo, Neil Thompson, Sara Hooker

Pushing the boundaries of machine learning often requires exploring different hardware and software combinations. However, the freedom to experiment across different tooling stacks can be at odds with the drive for efficiency, which has produced increasingly specialized AI hardware and incentivized consolidation around a narrow set of ML frameworks. Exploratory research can be restricted if software and hardware are co-evolving, making it even harder to stray away from mainstream ideas that work well with popular tooling stacks. While this friction increasingly impacts the rate of innovation in machine learning, to our knowledge the lack of portability in tooling has not been quantified. In this work, we ask: How portable are popular ML software frameworks? We conduct a large-scale study of the portability of mainstream ML frameworks across different hardware types. Our findings paint an uncomfortable picture -- frameworks can lose more than 40% of their key functions when ported to other hardware. Worse, even when functions are portable, the slowdown in their performance can be extreme and render performance untenable. Collectively, our results reveal how costly straying from a narrow set of hardware-software combinations can be - and suggest that specialization of hardware impedes innovation in machine learning research.

The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

January 2024
Tamay Besiroglu, Sage Andrus Bergerson, Amelia Michael, Lennart Heim, Xueyun Luo, Neil Thompson

Industrial and academic AI labs differ sharply in compute access. We document how this divide coincides with reduced representation of academic-only teams in compute‑intensive topics (e.g., foundation models) and argue academia may play a smaller role in advancing techniques, scrutiny, and diffusion. We recommend expanding academic access (e.g., national compute, open science), plus structured access and auditing to enable measured external evaluation of industry systems.

MIT FutureTech
‍MIT Computer Science and Artificial Intelligence Laboratory
32 Vassar Street, Office 386, Cambridge, MA 02139
617-258-5030

© 2026 FutureTech.
All rights reserved.

MIT Accessibility