Tal Shnitzer

Anthony Ou

Mírian Silva

Kate Soule

Yuekai Sun

Justin Solomon

Neil Thompson

Mikhail Yurochkin

September 27, 2023

There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.

Paper Link

Cite this work

Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, Mikhail Yurochkin (). Large Language Model Routing with Benchmark Datasets. Published in arXiv.org. Retrieved from https://arxiv.org/abs/2309.15789. Accessed .

@article{,
  title     = { Large Language Model Routing with Benchmark Datasets },
  author    = { Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, Mikhail Yurochkin },
  journal   = { arXiv.org },
  year      = {  },
  url       = { https://arxiv.org/abs/2309.15789 }
}

Large Language Model Routing with Benchmark Datasets

Tal Shnitzer

Anthony Ou

Mírian Silva

Kate Soule

Yuekai Sun

Justin Solomon

Neil Thompson

Mikhail Yurochkin

Cite this work

Related Publications

What should be done about the growing influence of industry in AI research?

The Grand Illusion: The Myth of Software Portability and Implications for ML Progress

The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

EvilGenie: A Reward Hacking Benchmark

What should be done about the growing influence of industry in AI research?

The Grand Illusion: The Myth of Software Portability and Implications for ML Progress