Attention
A mechanism that lets one position decide how much other positions should influence it.
COMP10002 Foundations of Algorithms
Concepts
The core ideas you need for the assignment, plus optional demos, tooling, and background if you want extra help.
Needed for assignment
One-Hot Encoding
How a single active slot can represent a category or label.
Q, K, V Projections
How one vector is transformed into query, key, and value versions used by attention.
Dot Products and Scores
A way to measure how strongly two vectors line up, often used to build attention scores.
Softmax and Stable Softmax
A function that turns raw scores into non-negative weights that add up to 1.
Causal and Padding Masks
Rules that prevent certain positions from being used, such as future tokens or padding.
KV Cache
A store of previously computed keys and values so generation can reuse past work.
Practical resources
Annotated Scaffold
A guided reading of the provided `a1.c`, with direct links from the main spec and tooltip-style explanations for the names, arrays, and helper functions you keep seeing.
Sample input file
A small concrete example of the assignment input format, showing what a real input file looks like and how to read each line.
Checking and debugging your C code
How to compile with stronger warnings, run useful checking tools, and catch bugs in your own `a1.c` before submission.
Strings in C
The small amount of C string handling you may need for Stage 1: storing tokens, comparing them, sorting them, and copying them safely.
qsort and function pointers in C
A practical guide to using `qsort(...)` for Stage 1 and understanding the comparison function it calls.
Toy demo
A concrete numeric walkthrough that connects the assignment data back to token positions and attention weights.
Optional concepts and background
Tokens and Embeddings
How text is split into tokens, then converted into numeric vectors a model can work with.
Reading Transformer Paper Notation
How to read the compact notation used in the original Transformer paper and later ones.
Transformer explainer
A one-page view of the larger model architecture so you can see where this assignment fits.
Reading LLM architecture diagrams
How to read the boxes, arrows, repeated stacks, and omissions in the architecture figures that appear in papers.
Transformer history
Why attention mattered historically, including the 2017 paper and why the impact took time to become obvious.
Parameters and model scale
What terms like “7B parameters” mean, where those numbers live, and what they imply for memory.
How this assignment was built
Why the assignment was developed as a human-AI collaboration, what stayed human, and what AI helped with.
The main assignment page links the needed material inline at the point where it matters, so you can either read from here or jump in from the spec as needed.