Day 22

What exactly is Self-Attention?

Here's what I've understood so far.

1. Every word depends on other words.

When we read a sentence, we don't understand each word in isolation.
We naturally use the surrounding words to understand its meaning.

2. Self-Attention helps AI do the same thing.

For every word, the model looks at the other words in the sentence and decides:
Which words are the most important for understanding this one?

3. Not every word gets the same attention.

For ex. - The puppy was playing with the ball because it was new.

What does "it" refer to?
The puppy? Or the ball?

We look at the surrounding words to understand that "it" most likely refers to the ball.

That's exactly what Self-Attention helps the model do.

Day 22 of 1% better.
Learning. Reflecting. Improving