01
Attention and the transformer
What problem was attention invented to solve?
Primary source★ LandmarkFree
Attention Is All You Need ↗Vaswani et al., 2017
The architecture behind essentially every modern language model.
AI · Module 14
Language Models & Modern AI
The most consequential technology of the decade, built from ideas in the previous three modules. Learn the mechanism, not the mythology.
By the end of this module
You can explain what a transformer does, token by token, and what it cannot do.
0%0 / 4 complete
02
Build a GPT from scratch
Can I actually implement one myself?
VideoFree
Let's build GPT: from scratch, in code, spelled out ↗Andrej Karpathy
Two hours, and you have written a working transformer.
03
Scaling laws, fine-tuning and alignment
Why does making it bigger keep working, and what breaks?
Primary source★ LandmarkFree
Training language models to follow instructions with human feedback ↗Ouyang et al., 2022
04
What language models cannot do
Where is the boundary between capability and appearance?
Primary sourceFree
On the Dangers of Stochastic Parrots ↗Bender, Gebru, McMillan-Major & Shmitchell, 2021
Read the strongest critique alongside the strongest results.
Quiz
Not written yet for this module. The lectures above are complete and the module still counts toward your progress.