Скачать или смотреть FrontierMath: When will AI match the best human mathematicians?

What would it mean if AI is able to replace the best human mathematicians? In this interview, recorded in November 2024, Epoch’s Lead Mathematician Elliot Glazer explains why we developed FrontierMath, a very hard mathematical evaluation designed to test the limits of AI reasoning capabilities.

One of the big bottlenecks in AI development is the ability of AI models to come up with a coherent worldview and notion of truth, and engage in long and complex chains of logic. Advanced mathematics offers a very good environment for evaluating this type of complex reasoning, which makes FrontierMath a unique tool for reliably tracking AI progress.

The FrontierMath benchmark captures contemporary mathematics, spanning major fields—from computationally intensive number theory problems to abstract questions in algebraic geometry and category theory. Elliot discusses how FrontierMath has evolved to incorporate increasingly difficult problems expected to challenge AI systems significantly.

This conversation offers insights into how carefully designed benchmarks help us understand AI’s genuine capabilities, potentially tracking and guiding advancements in the systematic, innovative thinking required for scientific research.

Notes*
• Since this video was recorded (November 2024) to its release (April 2025), Elliot's estimates for when the first version of FrontierMath will be solved have shortened from 5 to 2 years, given recent models scoring higher than expected on the benchmark. Other researchers at Epoch have updated to even more optimistic timelines. We decided to leave the original predictions for transparency, and because these updates demonstrate that FrontierMath is doing its job: helping us make our beliefs about the future of AI more accurate.
• When this video was recorded, FrontierMath was composed of only three difficulty tiers. More recently, Epoch has been working on FrontierMath Tier 4, introducing even harder problems—requiring weeks to even months of expert effort to solve. Explore the FrontierMath methodology, review sample problems with solutions, and check AI performance here: https://epoch.ai/frontiermath/the-ben...


Timestamps
00:00 - Preview
00:13 - Why has Epoch AI put all this effort into a math benchmark specifically?
01:19 - What do AI models need to achieve before they can perform well on FrontierMath?
02:33 - Why can't LLM's beat FrontierMath just by training on existing mathematical data?
04:29 - How long do you expect FrontierMath to last?
05:02 - How representative are FrontierMath questions of research problems?
07:14 - What were the main difficulties encountered when making FrontierMath?
10:14 - Did you encounter significant challenges making hard problems?
10:52 - How do you measures the difficulty of problems on FrontierMath?
12:31 - How has the FrontierMath project evolved from its original goals?
13:57 - What excites you about this project as a mathematician?