Скачать или смотреть Ilya Sutskever's comments about the end of pre-training scaling and shift to test-time compute

If you combine Ilya Sutskever's comments from Friday’s 12.13 presentation - “Data is the fossil fuel of AI” - with those from Chetan Puttagunta and Modest Proposal on their 12.06 interview on “Invest Like The Best” with Patrick O'Shaughnessy, CFA, there are overlapping themes with heavy duty AI implications for 2025 and beyond...

Oversimplifying it of course…

1: End of pre-training scaling and shift to test-time compute / inference

The core technical insight - we’ve hit / are about to hit the ceiling on pre-training LLMs - because as Ilya said “data is the fossil fuel of AI,” a finite resource as “we have but one internet” translates to a pivot from the approach focused on ever-larger pre-trained models to test-time compute and reasoning.

2: Business implications of the shift and aligning expenditure with usage aka building a power plant vs metered use of electricity

The new focus on test-time compute aligns expenditure with usage, creating a more sustainable business model. Increasing alignment between revenue with expenditures.

It’s the difference between building a massive power plant upfront vs paying for electricity as you use it.

For example, using the electricity to build companies that create a refrigerator and / or ultimately the The Coca-Cola Company that goes inside…s / o Chamath Palihapitiya

3: Democratization of AI development

This shift may mark the beginning of AI’s most creative and productive phase yet.

The convergence of technical limitations and business reality may create new opportunities while forcing a rethinking of existing strategies, operating models etc. across the board.

Democratization of AI enables smaller teams to fine-tune existing models for specific use cases, dramatically reducing capital requirements and time to market.

As described multiple times by Chetan and Modest Proposal, 2-5 person teams are now achieving results that previously required hundreds of millions in investment…

“In the last, call it six weeks of time, the number of teams we've met here at Benchmark that are two to five people…at this plateauing time, we're starting to see these small teams catch up to the frontier…"

“We're seeing these small teams of quite literally two to five people jumping to the frontier with spend that is not one order, but multiple orders of magnitude less than what these large labs were spending to get there."

To be abundantly clear, this isn’t just a start-up thing.

This is all about what small, empowered, AI-first teams in businesses of ANY size can accomplish by building AI applications…

VERY exciting times.

Chamath's comments about electricity, ice, refrigeration, coca-cola etc:   / alecchh_imagine-a-world-before-refrigerati...  

Chetan Puttagunta + Modest Proposal interview on Invest Like The Best: https://www.linkedin.com/posts/alecch...