1. The Process Address Space
When the kernel loads your program, it carves a virtual address space into named regions. Hover any band to see what lives there. This is what every running process looks like on x86_64 Linux.
2. Meet the CPU Registers
Before we follow a real call chain, let's see what those mysterious register names (RIP, RBP, RSP, RAX, RDI) actually mean. A register is a tiny piece of storage inside the CPU itself, not in RAM. There are only a handful of them and they're the fastest memory the CPU can touch. Five of them matter for understanding this example. Step through the tiny program one instruction at a time and watch each register update; hover any register name on the right to highlight what it points to.
int square(int n) {
return n * n;
}
int main(void) {
int result = square(5);
return result;
}
3. The Stack: Function Calls in Motion
Now that you know what RIP, RBP, RSP, and RAX are, let's follow a longer call chain. Every time a function is called, the CPU gives it a private workspace on the call stack, a little box holding its inputs, its local variables, and a note saying where to jump back to when it's done. When the function returns, the box vanishes. Below, we trace main calling average, which calls divide, and watch the boxes pile up and disappear.
The disassembly panel on the right shows the actual CPU instructions if you're curious; you can safely ignore it and just watch the boxes on the left.
int divide(int sum, int count) {
return sum / count;
}
int average(int a, int b) {
int sum = a + b;
return divide(sum, 2);
}
int main(void) {
return average(10, 20);
}
4. The Heap: malloc, free, and the Mess in Between
Unlike the stack, the heap is yours to manage. malloc hands out chunks with a hidden header; free returns them. Misuse it and you get fragmentation, use-after-free, or a double free, each rendered live below.
char *p1 = malloc(24);
char *p2 = malloc(48);
char *p3 = malloc(16);
free(p2);
char *p4 = malloc(16); // reuses p2's hole
*p2 = 'X'; // use-after-free!
free(p2); // double free!
5. Stack Smashing: How a Buffer Overflow Wins
What's a buffer overflow? Imagine a function reserves 16 mailboxes for some incoming text, but the code filling them doesn't bother to check the limit, it just keeps writing: mailbox 17, mailbox 18, mailbox 99, into whatever memory happens to sit next door. On the stack, those "next door" boxes are not random, they're the function's other local variables, plus the bookkeeping the CPU placed there: the saved RBP and, most importantly, the return address (the note from section 3 telling the CPU where to jump back to when this function ends).
This is the dangerous part. If an attacker can choose what spills past mailbox 16, they can rewrite that return address with one of their own choosing, sending the CPU wherever they want. That's "hijacking control flow", and it's the bug class behind some of the most famous attacks in computing history (the Morris Worm, Code Red, Slammer).
The function below has exactly this bug. It copies an input string into a 16-byte buffer using strcpy, which has no idea where the buffer ends, it just copies until it hits a zero byte. Feed it 17 characters and it writes 17. Feed it 100 and it writes 100. Step through to watch the bytes spill in real time, clobbering a nearby local, then the canary, then the saved RBP, then the return address itself. Then toggle each mitigation below to see how it slams the door.
void vuln(const char *src) {
long secret = 0xCAFEBABE;
char buf[16];
strcpy(buf, src); // no bounds check
}
Toggle a mitigation, then step through the overflow again to see how it changes the outcome. Each one is what a real Linux compiler/kernel turns on by default today.
__stack_chk_fail and aborts before ret ever runs. Cheap, catches almost every linear stack overflow.
buf, the CPU faults the moment it tries to fetch an instruction from a non-executable page. Killed the classic "shellcode in the buffer" trick; attackers now use ROP instead.