Скачать или смотреть ai - attention is all you need - Transformer model: LLM

@Neuneworks
ai - attention is all you need - Transformer model

Attention:
Beyond Basics: Unpacking Attention in Large Language Models (LLMs)

What is Attention?

Attention is a DL mechanism, that was introduced in 2014, which is specialized in finding the most relevant part of the input sequence and generates the Output Sequence.

Lets see this with an Example

Let's take a Sentence

What is the capital of France? - if we feed this sentence to attention it'll find the relevant part of the sentence.

In this sentence the relevant parts are Capital and France.

From that it'll find the Output sequence. Capital of "France" is Paris

Let's dig little more on how it works

Let's take another example in which the

User prompts "What is Transformer?" - it finds Transformer as relevant Part

And It finds below sentences as reponse

1. Transformer is a DL technique - Tags #Transformer #AI #DL
2. Transformer is an Electrical Equipment - Tags #Transformer #Electrical
3. Transformer is an Animation Movie - Tags #Transformer #Movie #Animation
4. Transformer is a Game - Tags #Transformer #Toy #Game

And each sentence has tags associated with it.

Attention is composed of 3 parts. Query, Key and Value

In this example the The user Prompt "What is Transformer" is the Query. and the tags for each response is the key and actual response is Value.


When the prompt is fed to the Attention, it compares the keys of the responses With relevant parts of the Query.
In this examples it compares the "Transformer" from the query with the tags in each sentence.Like Transformer, DL, AI, Movie and so on.

, and finds the distance between the query and key. Distances are calculated by using similarity scores.
There are different techniques Like Euclidian distance, Cosine, Dot Product similarity, used to calculated the similarity scores.
These scores are then used as weights and the weighted sum is computed against the values.


from those the model randomly picks the values.
All the models have the parameter to control The randomness.