Скачать или смотреть Topic #4 of "24 AI Topics of 2024": Inference via Interview of Chetan Puttagunta + Modest Proposal

Let’s dig into Inference (#4 on my “24 AI topics of 2024”) via the phenomenal interview of Chetan Puttagunta + Modest Proposal on “Invest Like the Best” with Patrick O'Shaughnessy, CFA.

Oversimplifying it, but LLMs are pre-trained on vast amounts of data to develop general language understanding and knowledge through a process that typically happens once, in a resource-intensive way.

Inference is the practical application of this learned knowledge, where the model processes new inputs to produce outputs such as answering questions without further learning.

1: Inference costs have plummeted
“In the private markets one of the things that's happening is just the dramatic drop in prices of just compute whether it's inference or training or whatever, because it's just becoming way more available.”

“If you're sitting here today as an application developer versus two years ago, the cost of inference of these models is down 100x, 200x.”

“It's frankly outrageous. You've never seen cost curves that look this steep, that fast.”

2: Intelligent routing systems
“And today, most companies don't even think about inference costs because it's just like, ‘well, we've broken this task up and then we're using these small models for these tasks that are pretty basic, and then the stuff we're hitting with the most frontier models are these very few prompts.”

“And the rest of the stuff, we've just created this intelligent routing system.’”

“And so our cost of inference is essentially zero. And our gross margin for this task is 95%.”

“You're starting to look at and see the entire application stack for these new AI applications. And that starts with people that provide inference.”

“It starts with the tooling and the orchestration layer.”

“So we have a portfolio company that's extremely popular called LangChain...”

s / o Harrison Chase, Jacob Lee

3: Conversations with Enterprise CIOs...15-30 minutes to “let’s do this”
“We're now going to CIOs at Fortune 500 companies, showing these demos.”

“And two years ago there were really nice demos.”

“Today it's a really nice demo combined with five customer references of peers that are using it in production and experiencing great success.”

“And what becomes very clear in that conversation is that what we're presenting is not a 5% improvement over an existing SaaS solution.”

“It's about how we can eliminate significant amounts of software spend and human capital spend and move this to this AI solution.”

“And your 10x traditional ROI definition of software is easily justified and people get it within 30 minutes.”

“What used to be a very long sales cycle for SaaS in AI applications, it's 15 minutes to a yes, 30 minutes to a yes.”

“And then the procurement process for an enterprise, completely tenuous.”

“Now the CIO says something like, ‘Let's put this in as quickly as possible.’”

“We're going to run a 30-day pilot. The minute that's successful, we're signing a contract and we're deploying right away.”

Link to the "Invest Like The Best" interview: https://joincolossus.com/episode/chet...