Linux Processes Under the Hood: fork, exec, Signals and Zombies

You type ls and press Enter. A list of files appears, and the prompt comes back. Behind that, the kernel has created a process, replaced its program, collected its exit status and cleaned up after it, using just a handful of system calls that every program on Linux relies on.

Knowing those calls explains a lot of things that otherwise seem mysterious: why cd can't be an external program, what exit code 137 means, why kill -9 should be your last resort, where zombie processes come from, and why a container sometimes ignores docker stop for exactly ten seconds. The examples use Python's os module, which maps almost one-to-one onto the underlying system calls, plus standard tools like ps. You'll get more out of it if you've written a few shell scripts; see Writing Bash Scripts That Don't Break.

fork and exec: how every program starts#

Linux creates new processes in two separate steps:

• fork() makes a copy of the calling process. Parent and child continue from the same line of code, with the same memory contents, open files and environment. The only difference is fork()'s return value: 0 in the child, and the child's PID in the parent.
• exec() replaces the program running in the current process with a new one. The PID stays the same; the code, memory and stack are thrown away and loaded fresh from the new executable.

A shell runs ls by forking itself, then making the child exec ls, while the parent waits. Here's the same thing in Python:

import os

print(f"parent: my PID is {os.getpid()}", flush=True)
pid = os.fork()

if pid == 0:
    # Child: a copy of the parent. Replace it with the `ls` program.
    print(f"child:  my PID is {os.getpid()}, my parent is {os.getppid()}", flush=True)
    os.execvp("ls", ["ls", "-d", "/tmp"])
    print("never printed: exec only returns if it fails")
else:
    # Parent: wait for the child and collect its exit status.
    _, status = os.waitpid(pid, 0)
    print(f"parent: child {pid} exited with code {os.waitstatus_to_exitcode(status)}")

$ python3 forkexec.py
parent: my PID is 726920
child:  my PID is 726922, my parent is 726920
/tmp
parent: child 726922 exited with code 0

The split into two calls is what makes Unix shells so flexible. Between fork and exec, the child is still running the shell's code, so it can rearrange its own environment first: redirect file descriptors for > file or |, change directory, set environment variables. Then it execs the program, which inherits all of it.

It also explains why cd must be a shell builtin. A child process can only change its own working directory, and that change disappears when it exits. To change the shell's directory, the shell has to do it itself.

Exit codes and wait#

When a process ends, it leaves behind an exit status for its parent: 0 for success, 1–255 for failure, by convention. The parent collects it with wait() or waitpid(), and the shell puts it in $?.

If the process was killed by a signal, the shell reports 128 + the signal number. These come up constantly in logs and CI output:

Signals: notifying a process#

A signal is a small asynchronous notification sent to a process, by the kernel or by another process with kill. Despite the name, kill sends any signal; SIGTERM is just the default. The ones you'll meet most:

(The numbers are for x86 and ARM Linux; run kill -l to see the full list on your system.)

A process can handle most signals with its own function, or ignore them. Two can never be caught or ignored: SIGKILL and SIGSTOP. The kernel acts on those directly.

Graceful shutdown#

That's why you should send SIGTERM first and SIGKILL only as a last resort. SIGTERM lets a program finish the request it's serving, flush data to disk, close database connections and remove lock files. SIGKILL stops it mid-instruction, with no chance to clean up.

Handling SIGTERM takes a few lines:

import signal, time

running = True

def stop(signum, frame):
    global running
    print(f"got {signal.Signals(signum).name}, finishing the current job...", flush=True)
    running = False

signal.signal(signal.SIGTERM, stop)

while running:
    time.sleep(1)        # pretend to work
print("clean exit", flush=True)

$ python3 worker.py &
$ kill %1
got SIGTERM, finishing the current job...
clean exit

The handler only sets a flag; the main loop checks it and exits at a safe point. Doing real work inside a signal handler is risky, because it can interrupt your program anywhere. Bash scripts get the same ability with trap 'cleanup' TERM.

Service managers follow the same pattern: systemctl stop sends SIGTERM, waits (90 seconds by default), then sends SIGKILL. docker stop does the same with a 10-second grace period.

Zombies#

When a child exits, the kernel can't throw it away completely yet: the parent might still want its exit status. So the process becomes a zombie. It has no memory or open files any more, just a process-table entry holding its PID and exit status, waiting for the parent to call wait(). Collecting the status is called reaping, and it removes the zombie.

A parent that never calls wait() leaves zombies behind. Let's make one:

import os, time

pid = os.fork()
if pid == 0:
    os._exit(0)          # the child exits immediately...
print(f"child PID: {pid}")
time.sleep(60)           # ...but the parent never calls wait()

$ python3 zombie.py &
child PID: 726905
$ ps -o pid,ppid,stat,cmd --ppid $!
    PID    PPID STAT CMD
 726905  726903 Z    [python3] <defunct>

Z in the STAT column and <defunct> mark a zombie. A few zombies are harmless, but each one holds a PID, and a buggy server that leaks zombies can eventually exhaust the PID limit, at which point nothing on the machine can start a new process.

You can't kill a zombie: it's already dead, so signals have no effect. Fix the parent so it calls wait(), or terminate the parent. When a parent dies, its children (living or zombie) are re-parented to PID 1 (or to the nearest process marked as a "subreaper"), which reaps them automatically. On a normal system, PID 1 is systemd, and it's very good at reaping.

PID 1 is special, and containers make it your problem#

Inside a container, there's no systemd. Whatever command the image starts becomes PID 1 in the container's PID namespace, and PID 1 has two extra responsibilities:

1. The kernel doesn't apply default signal actions to it. A signal that PID 1 hasn't installed a handler for is simply ignored. A normal program that relies on SIGTERM's default "terminate" action will ignore docker stop, sit there for the 10-second grace period, and get SIGKILLed.
2. It inherits every orphan, so it has to reap them, or zombies pile up.

You can see the first one without Docker. unshare starts a program as PID 1 in a new PID namespace:

$ sudo unshare --fork --pid --mount-proc python3 -c 'import time; time.sleep(30)' &
$ pid=$(pgrep -n -f 'time.sleep')    # the Python process, as seen from outside
$ sudo kill -TERM "$pid"             # nothing happens: PID 1 has no SIGTERM handler
$ sudo kill -KILL "$pid"             # this works: SIGKILL from the parent namespace

The fixes are simple:

• Run a tiny init process as PID 1: docker run --init adds tini, which forwards signals to your program and reaps zombies. Kubernetes users can add tini (or dumb-init) to the image as the entrypoint.
• Make your program handle SIGTERM explicitly, like worker.py above.
• Use the exec form in Dockerfiles, CMD ["python3", "app.py"], not CMD python3 app.py. The shell form starts /bin/sh -c "python3 app.py", so the shell becomes PID 1, and it doesn't pass SIGTERM on to your program.
• In entrypoint scripts, end with exec "$@". exec replaces the shell with your program (the same exec as above), so your program becomes PID 1 and receives signals directly.

Inspecting processes#

Every process has a directory under /proc/<PID>/, and most tools just read from it:

ps -eo pid,ppid,stat,etime,cmd --forest   # every process, as a tree
pstree -p                                 # a compact tree with PIDs
cat /proc/1234/status                     # state, parent, memory, threads, signal masks
tr '\0' ' ' < /proc/1234/cmdline          # the full command line
ls -l /proc/1234/fd                       # open files, sockets and pipes
cat /proc/1234/environ | tr '\0' '\n'     # its environment (needs permission)

The STAT column's first letter tells you what a process is doing:

A process stuck in D that can't be killed almost always points to a storage problem (a hung NFS mount, a failing disk), not to the program itself.

Recap#

• New processes are made with fork() (copy the current process) and exec() (replace its program). The gap between them is where shells set up redirection and pipes.
• The parent collects the exit status with wait(); exit codes above 128 mean "killed by signal (code − 128)": 130 Ctrl+C, 137 SIGKILL/OOM, 143 SIGTERM.
• Send SIGTERM and give programs time to shut down; SIGKILL can't be caught and skips all cleanup.
• Zombies are exited children nobody has reaped; fix or kill the parent, not the zombie.
• In containers, PID 1 ignores signals it doesn't handle and must reap orphans: use --init/tini, exec-form CMD, and exec "$@" in entrypoints.