Cryo Learn

Writing Bash Scripts That Don't Break

Thu, 29 Oct 2026 09:00:00 +0000

Bash scripts tend to start as a few commands pasted into a file. They work on your machine, with your files, on a good day. Then one evening a file name contains a space, a disk fills up, or a command fails halfway through, and the script keeps going as if nothing happened, sometimes deleting the wrong things on the way.

The fixes are few and mechanical. This article goes through the habits that make scripts fail loudly and safely, then combines them in a backup script you can actually use. It assumes you're comfortable with the commands in Essential Linux Commands Every Developer Should Know.

Start every script the same way

#!/usr/bin/env bash
set -euo pipefail

The first line (the shebang) says which program runs the file. /usr/bin/env bash finds bash wherever it's installed, which matters on macOS and BSD. Make the file executable once with chmod +x script.sh and run it as ./script.sh.

The second line turns on what's often called strict mode. Each letter fixes a different silent failure.

What strict mode catches: set -e stops after a failed cd, set -u turns a misspelled variable into an error, and pipefail makes a pipeline fail when any command in it fails

set -e: stop on the first error

By default Bash ignores failures and runs the next line anyway:

cd /var/www/myapp-typo        # fails: No such file or directory
rm -rf ./*                    # ...and now runs in whatever directory you were in

With set -e, the script exits as soon as a command fails (returns a non-zero exit code). Failures that are expected, such as a grep that finds nothing, can be handled explicitly:

if grep -q "ERROR" app.log; then
    echo "Errors found"
fi

grep -q "ERROR" app.log || echo "No errors"   # || also counts as handling it

Where set -e doesn't help

set -e is ignored in conditions (if, while, &&, ||), and it has two well-known blind spots:

local out=$(some_command) never fails, because the exit code you get is the one from local. Declare and assign on separate lines: local out then out=$(some_command).
((count++)) returns 1 when count was 0, which kills the script under set -e. Use count=$((count + 1)) instead.

set -u: no undefined variables

A typo in a variable name normally expands to an empty string, silently:

name="Ada"
echo "Hello, $nme"            # prints "Hello, " without set -u

That's annoying in an echo, and dangerous in rm -rf "$build_dir/"*. With set -u, using an unset variable is an error:

./greet.sh: line 3: nme: unbound variable

For variables that are allowed to be missing, give a default with ${var:-default}, or fail with your own message with ${var:?message}:

port=${PORT:-8080}                          # use $PORT if set, otherwise 8080
: "${API_KEY:?API_KEY must be set}"         # stop with this error if it's missing

set -o pipefail: failures inside pipelines

A pipeline's exit code is normally that of its last command, so failures earlier in the pipe vanish:

$ false | true; echo $?
0
$ set -o pipefail
$ false | true; echo $?
1

Without it, mysqldump mydb | gzip > backup.sql.gz "succeeds" even when the dump fails, because gzip happily compresses an empty input. With pipefail, the pipeline fails if any command in it fails.

Quote your variables

This is the single biggest source of shell bugs. An unquoted $var is split on spaces and then expanded as a wildcard pattern:

file="my report.txt"
rm $file          # rm: cannot remove 'my': No such file or directory
                  # rm: cannot remove 'report.txt': No such file or directory
rm "$file"        # removes "my report.txt"

And if a variable contains *, an unquoted expansion turns it into a list of files in the current directory. The rule is simple: always put double quotes around variables and command substitutions: "$file", "$(date +%F)", "${files[@]}". The rare exceptions are deliberate, and deserve a comment.

To pass all of a script's arguments on to another command, use "$@". It keeps each argument intact, even when it contains spaces; $* and unquoted $@ don't:

# called as: ./wrapper.sh "a b" c
for arg in "$@"; do echo "[$arg]"; done   # [a b] [c]
for arg in $*;   do echo "[$arg]"; done   # [a] [b] [c]

Use [[ ]] for tests

In Bash, prefer [[ -f "$file" ]] to [ -f "$file" ]. It doesn't split words, so a missing quote can't break it, and it supports &&, || and pattern matching like [[ $name == *.txt ]].

Read files line by line correctly

for line in $(cat file) loops over words, not lines, and expands wildcards on the way. The correct idiom is:

while IFS= read -r line; do
    echo "Processing: $line"
done < hosts.txt

IFS= keeps leading and trailing spaces, and -r stops backslashes from being treated as escapes.

Clean up with trap

Scripts often create temporary files. mktemp creates one with a unique, unguessable name (never hard-code /tmp/myscript.tmp), and trap ... EXIT guarantees it's removed however the script ends: normally, because of an error under set -e, or when you press Ctrl+C:

tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT

curl -fsSL https://example.com/data.csv -o "$tmpdir/data.csv"
# ...work with the file; no matter what happens, $tmpdir is deleted at the end

Note the single quotes: $tmpdir is expanded when the trap runs, not when it's defined. Only kill -9 (SIGKILL) skips the trap, because nothing can catch that signal.

Functions, arguments and errors

die() {
    echo "Error: $*" >&2      # error messages go to stderr
    exit 1
}

require_dir() {
    local dir=$1              # local: don't leak into the rest of the script
    [[ -d "$dir" ]] || die "'$dir' is not a directory"
}

Inside a function, $1, $2… and "$@" are the function's own arguments. Use local for every variable a function creates, print errors to standard error (>&2) so they don't mix with real output, and exit with a non-zero code so the caller (a person, cron or CI) knows something went wrong.

A complete example: a backup script

Here's a script that archives a directory, keeps only the newest few archives, and puts everything above into practice:

#!/usr/bin/env bash
# backup.sh: archive a directory and keep only the newest few archives.
set -euo pipefail

usage() {
    echo "Usage: $0 SOURCE_DIR DEST_DIR [KEEP]" >&2
    exit 2
}

[[ $# -ge 2 ]] || usage
src=$1
dest=$2
keep=${3:-7}

[[ -d "$src" ]] || { echo "Error: '$src' is not a directory" >&2; exit 1; }
mkdir -p "$dest"

# Write to a temporary file first, so a failed run never leaves a half-written archive behind
tmpfile=$(mktemp "$dest/.backup.XXXXXX")
trap 'rm -f "$tmpfile"' EXIT

name="$(basename "$src")-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -czf "$tmpfile" -C "$(dirname "$src")" "$(basename "$src")"
mv "$tmpfile" "$dest/$name"
echo "Created $dest/$name"

# The timestamp in the name sorts alphabetically, so the glob lists the oldest first
archives=("$dest/$(basename "$src")"-*.tar.gz)
excess=$(( ${#archives[@]} - keep ))
for (( i = 0; i < excess; i++ )); do
    rm -- "${archives[i]}"
    echo "Removed ${archives[i]}"
done

$ ./backup.sh "my site" backups 2
Created backups/my site-20261029-081312.tar.gz
$ ./backup.sh "my site" backups 2
Created backups/my site-20261029-081313.tar.gz
$ ./backup.sh "my site" backups 2
Created backups/my site-20261029-081314.tar.gz
Removed backups/my site-20261029-081312.tar.gz

A few details worth noticing:

It handles a source directory with a space in its name, because every variable is quoted.
The archive is written under a temporary name and renamed at the end. Renaming within one directory is atomic, so the backups folder never contains a half-written archive, even if tar fails or the disk fills up. (mktemp also makes the file readable only by you, which suits a backup.)
rm -- marks the end of options, so a file name that starts with - can't be mistaken for one.
Wrong usage exits with code 2 and real errors with code 1, so a cron job or CI pipeline can tell them apart.

Let ShellCheck review your scripts

ShellCheck is a linter for shell scripts. It catches missing quotes, the local trap, $* vs "$@" and dozens of other mistakes, each with an explanation. Install it with sudo apt install shellcheck (or brew install shellcheck), run shellcheck backup.sh, and add it to your editor or CI. It's the fastest way to absorb all of these habits.

Recap

Start with #!/usr/bin/env bash and set -euo pipefail, and know where set -e doesn't reach.
Quote every expansion: "$var", "$(cmd)", "$@".
Use ${var:-default} and ${var:?message} for optional and required values.
Create temporary files with mktemp and remove them with trap '...' EXIT.
Send errors to >&2, exit with meaningful codes, and run ShellCheck.

The JavaScript Event Loop: Tasks, Microtasks and Rendering

Thu, 29 Oct 2026 09:00:00 +0000

You already know that await pauses a function without freezing the page. But when exactly does that function resume? Why does a resolved promise always run before setTimeout(fn, 0)? And why can a "Loading…" message you set right before a heavy calculation never appear on screen?

All three questions have the same answer: the event loop. Once you can run it in your head, a whole class of timing bugs, flaky tests and janky interfaces stops being mysterious. This article assumes you're comfortable with promises and async/await; if not, start with Understanding async/await in JavaScript.

The pieces

JavaScript in the browser runs on one main thread, and that thread does more than run your code. It also handles input and paints the screen. Four things share it:

The call stack: the functions running right now. The event loop only steps in when it is empty.
The task queue: whole units of work waiting their turn, such as a setTimeout callback, a click handler, or a postMessage from a worker. (Specs call these tasks; you'll also see them called macrotasks.)
The microtask queue: small follow-up jobs, mainly promise reactions (.then() callbacks and code after an await), plus queueMicrotask() and MutationObserver callbacks.
The rendering steps: requestAnimationFrame callbacks, then style, layout and paint.

One turn of the loop

Stripped to its essentials, the browser repeats this forever:

while (true) {
  const task = taskQueue.takeOldest();
  if (task) run(task);                     // 1. exactly ONE task

  while (microtaskQueue.length > 0) {      // 2. EVERY microtask,
    run(microtaskQueue.shift());           //    including ones added meanwhile
  }

  if (isTimeToRender()) {                  // 3. maybe a frame
    runAnimationFrameCallbacks();
    updateStyleLayoutAndPaint();
  }
}

Each turn of the event loop runs one task, then empties the microtask queue, then renders a frame if one is due

The asymmetry is the whole story: the loop takes one task at a time, but drains the microtask queue completely, even if microtasks keep adding more microtasks. Rendering only happens between tasks, and only when the display is ready for a new frame (about every 16.7 ms on a 60 Hz screen).

Your script is a task too

The initial run of a <script> is itself a task. That's why promise callbacks created during it run as soon as the script finishes, before any timer.

Predict the output

Here's a classic puzzle. Work out the order before reading the answer:

console.log("script start");

setTimeout(() => console.log("timeout"), 0);

Promise.resolve()
  .then(() => console.log("then 1"))
  .then(() => console.log("then 2"));

queueMicrotask(() => console.log("microtask"));

(async () => {
  console.log("async start");
  await null;
  console.log("async after await");
})();

console.log("script end");

script start
async start
script end
then 1
microtask
async after await
then 2
timeout

Step by step:

The synchronous code runs first: script start, then async start (an async function runs synchronously up to its first await), then script end.
While that happened, three microtasks were queued in order: then 1, microtask, and the rest of the async function. The timer callback went to the task queue.
The script task ends, so the microtask queue is drained. Running then 1 resolves the next promise in the chain, which queues then 2 at the back of the queue. That's why then 2 comes after async after await.
Only when the microtask queue is empty does the loop pick the next task: the timer.

setTimeout(fn, 0) means "run fn in a future task", not "run it now". Every microtask already queued, and every microtask they queue, runs first.

Microtasks can starve the page

Because the loop won't leave step 2 until the microtask queue is empty, a microtask that keeps scheduling another one never lets the page render or respond:

function spinForever() {
  Promise.resolve().then(spinForever);   // freezes the tab: rendering never gets a turn
}

Swap the promise for a timer and the page stays responsive, because every timer callback is a separate task and the loop can render between them:

function spinPolitely() {
  setTimeout(spinPolitely, 0);           // one task per call: the browser can paint in between
}

setTimeout(fn, 0) isn't really 0

After timers have been nested five levels deep, browsers clamp the delay to at least 4 ms, and background tabs throttle timers much harder. Never use timers for precise timing.

Why your loading message never shows

This is the most common real-world event loop bug:

button.addEventListener("click", () => {
  status.textContent = "Crunching numbers…";
  crunchNumbers();                 // synchronous, takes ~2 seconds
  status.textContent = "Done!";
});

Users never see "Crunching numbers…". The DOM was updated, but painting only happens between tasks, and the click handler is one long task. By the time the browser gets to paint, the text already says "Done!". Meanwhile the page ignores every click and keystroke for two seconds.

The fix is to break long work into chunks and yield to the event loop between them. The yield must create a new task: await Promise.resolve() only queues a microtask, so it never lets the browser paint.

function yieldToMain() {
  if (globalThis.scheduler?.yield) {
    return scheduler.yield();                              // newer browsers
  }
  return new Promise((resolve) => setTimeout(resolve, 0)); // works everywhere
}

async function processAll(items) {
  let lastYield = performance.now();

  for (const item of items) {
    processItem(item);

    if (performance.now() - lastYield > 50) {   // a "long task" is anything over 50 ms
      await yieldToMain();                      // let the browser paint and handle input
      lastYield = performance.now();
    }
  }
}

Now the loading message appears, clicks are handled, and the work still finishes quickly because it only yields every 50 ms rather than after every item. scheduler.yield() is the better choice where it exists: the browser gives your continuation priority over other waiting tasks, so other code can't keep jumping in ahead of your work.

When chunking isn't enough

If the work itself takes seconds of pure computation (parsing a huge file, image processing), move it to a Web Worker. Workers have their own thread and their own event loop, so the main thread stays free.

requestAnimationFrame: work at the right moment

requestAnimationFrame callbacks run in step 3, right before the browser paints. That makes them the right place for visual updates: you run exactly once per frame, never twice between paints and never in the middle of one.

function animate(time) {
  box.style.transform = `translateX(${(time / 5) % 600}px)`;
  requestAnimationFrame(animate);       // schedule the next frame
}

requestAnimationFrame(animate);

Driving the same animation with setTimeout(animate, 16) drifts out of sync with the display, so some frames get two updates and others none, which is exactly the stutter users notice.

A subtle one: microtasks between event listeners

Microtasks run whenever the call stack becomes empty, not only at the end of a task. Look at two listeners on the same button:

button.addEventListener("click", () => {
  Promise.resolve().then(() => console.log("microtask 1"));
  console.log("listener 1");
});

button.addEventListener("click", () => {
  Promise.resolve().then(() => console.log("microtask 2"));
  console.log("listener 2");
});

Triggered by	Output
A real user click	`listener 1`, `microtask 1`, `listener 2`, `microtask 2`
`button.click()` in code	`listener 1`, `listener 2`, `microtask 1`, `microtask 2`

When a user clicks, the browser calls each listener with an empty stack, so microtasks run in between. When your code calls button.click(), your code is still on the stack, so every microtask waits until both listeners have finished. Keep this in mind when a test that simulates clicks behaves differently from the real page.

Node.js adds two extra queues

Node.js uses the same model (one task at a time, microtasks drained after each callback) but adds two scheduling tools of its own:

setTimeout(() => console.log("timeout"), 0);
setImmediate(() => console.log("immediate"));
Promise.resolve().then(() => console.log("promise"));
process.nextTick(() => console.log("nextTick"));

Run as a CommonJS script, this prints nextTick, then promise, then timeout and immediate in an order that can change from run to run.

process.nextTick() callbacks run before promise microtasks. Use it sparingly: recursive nextTick calls starve the loop just like recursive microtasks.
setImmediate() runs in a dedicated phase after I/O callbacks. Inside an I/O callback it always fires before a setTimeout(fn, 0); from the main script the order isn't guaranteed.

ES modules

In an ES module (.mjs), promise may print before nextTick, because the module body itself is evaluated from inside the microtask queue. Don't write code that depends on the exact order of nextTick and promises.

Recap

The event loop runs one task, then all microtasks, then renders if a frame is due, and repeats.
Promise callbacks, code after await and queueMicrotask() are microtasks: they always run before the next timer, event or paint.
A microtask that schedules microtasks forever freezes the page; timers don't, because each one is a separate task.
Long synchronous work blocks painting and input. Split it up and yield with setTimeout or scheduler.yield(). await Promise.resolve() does not yield.
Use requestAnimationFrame for visual updates and Web Workers for heavy computation.

Next time something runs "in the wrong order", ask yourself: is this a task or a microtask, and what else is in that queue?

The CSS Cascade and Specificity: Why Your Styles Don't Apply

Thu, 29 Oct 2026 09:00:00 +0000

You write a CSS rule. You reload. Nothing changes. You open DevTools and there it is, crossed out: another rule somewhere is winning. So you add a class, then an ID, then !important, and six months later nobody dares touch the stylesheet.

There's nothing random about which rule wins. When several declarations set the same property on the same element, the browser runs the cascade: a fixed sequence of tie-breakers. Learn the sequence and you can predict the winner every time, and write CSS that doesn't need !important at all. (If you're still getting comfortable with layout, CSS Flexbox vs Grid is a good first read.)

The cascade, in order

For each property of each element, the browser collects every declaration that applies, then compares them in this order. The first step that finds a difference decides; later steps only break ties.

The cascade compares declarations step by step: origin and importance, inline style, cascade layer, specificity, then order of appearance; the first step with a difference picks the winner

Origin and importance. Your stylesheets beat the browser's default styles. A declaration marked !important beats every normal one.
Inline styles. A style="…" attribute beats any selector in a stylesheet.
Cascade layers. If you use @layer, styles in later layers beat earlier ones, whatever their selectors say.
Specificity. The more specific selector wins.
Order of appearance. If everything else is equal, the declaration that comes last wins.

Most day-to-day surprises happen at steps 4 and 5, so let's start there.

Calculating specificity

Specificity is a score made of three numbers, usually written (A, B, C):

A: the number of ID selectors (#header)
B: the number of class selectors (.card), attribute selectors ([type="email"]) and pseudo-classes (:hover, :focus, :nth-child())
C: the number of type selectors (p, button) and pseudo-elements (::before)

The universal selector * and combinators (, >, +, ~) count for nothing.

Selector	Specificity
`p`	(0, 0, 1)
`.card p`	(0, 1, 1)
`.card .title`	(0, 2, 0)
`a:hover`	(0, 1, 1)
`input[type="email"]`	(0, 1, 1)
`ul li::before`	(0, 0, 3)
`#sidebar a`	(1, 0, 1)
`#nav .item a:hover`	(1, 2, 1)
`*`	(0, 0, 0)

To compare two selectors, compare A first. Only if A is equal do you look at B, and only if B is equal do you look at C. There's no carrying over between columns: one ID (1, 0, 0) beats a selector with twenty classes (0, 20, 0).

Here's a classic surprise:

#main .button { background: gray; }   /* (1, 1, 0) */
.button.primary { background: blue; } /* (0, 2, 0) */

<div id="main">
  <button class="button primary">Save</button>   <!-- gray! -->
</div>

The primary button is gray, even though .button.primary looks more specific to a human and comes later. The ID wins at column A, and step 5 is never reached.

Let DevTools do the maths

In Chromium-based browsers, hover over a selector in the Styles pane to see its specificity. The Computed tab shows the final value of each property and, when you expand it, every rule that tried to set it, winner first.

:is(), :not(), :has() and :where()

These pseudo-classes take a list of selectors, and they count differently:

:is(), :not() and :has() take the specificity of their most specific argument. :is(#nav, .menu) a is (1, 0, 1), even when it matches through .menu.
:where() always has zero specificity, whatever is inside it.

That makes :where() perfect for defaults that should be easy to override:

/* (0, 0, 1): any later class beats it */
:where(.prose) a { color: royalblue; }

/* (0, 1, 1): would beat most component styles, which is usually not what you want */
.prose a { color: royalblue; }

Inheritance is not the cascade

Some properties, mostly text-related ones like color, font-family and line-height, are inherited: if nothing sets them on an element, it takes its parent's value. Others, like margin, border and background, are not.

The trap: an inherited value always loses to any rule that targets the element directly, however weak that rule's selector is.

#article { color: darkred; }   /* (1, 0, 0), but applies to the div */
p { color: black; }            /* (0, 0, 1), applies to the p itself */

<div id="article">
  <p>This text is black, not dark red.</p>
</div>

Specificity only compares rules that match the same element. The #article rule matches the div; the paragraph only inherits from it, and inheritance only kicks in when no rule sets color on the paragraph itself.

Four keywords let you control this explicitly on any property:

Keyword	Meaning
`inherit`	Take the parent's value
`initial`	Use the property's default value from the CSS spec
`unset`	`inherit` for inherited properties, `initial` for the others
`revert`	Go back to what the browser's own stylesheet would set

!important and inline styles

!important jumps the queue at step 1, before specificity is even looked at. That's exactly why it causes trouble: the only way to override an !important declaration is with another !important declaration that wins on the later steps, and so the arms race begins.

Inline style attributes (step 2) beat any selector, but lose to !important in a stylesheet.

Keep !important for utilities

A reasonable use is a small utility class that must always win, like .hidden { display: none !important; }. Using it to fix a specificity problem only hides the problem until the next one.

Cascade layers: specificity under control

Cascade layers (@layer, supported in all major browsers since 2022) add a step above specificity. You declare the order of your layers once, and from then on a later layer beats an earlier one, whatever the selectors:

@layer reset, vendor, components, utilities;

@layer vendor {
  #app .btn { background: gray; }       /* (1, 1, 0) */
}

@layer components {
  .btn { background: royalblue; }       /* (0, 1, 0), and it wins */
}

components comes after vendor in the @layer statement, so its .btn wins despite much lower specificity. This is ideal for third-party CSS: wrap it in an early layer (@import url("library.css") layer(vendor);) and your own styles override it without specificity tricks.

Two details worth knowing: styles that aren't in any layer beat all layered styles, and for !important declarations the layer order is reversed, so an early layer's !important beats a later one's.

Habits that prevent specificity wars

Style with classes, not IDs. Keep IDs for anchors and JavaScript.
Keep selectors short and flat. .card-title instead of .sidebar .card > h2.title. Flat selectors all have similar specificity, so source order and layers decide, and those are easy to reason about.
Wrap defaults and resets in :where() so they're trivial to override.
Use @layer to put third-party and base styles below your components.
When a rule doesn't apply, check DevTools first: is it crossed out (lost the cascade), missing (selector doesn't match), or shown with a warning (invalid value)?

Recap

The cascade picks a winner in a fixed order: origin and !important, inline styles, layers, specificity, then order of appearance.
Specificity is (IDs, classes/attributes/pseudo-classes, elements/pseudo-elements), compared column by column, with no carrying over.
:is(), :not() and :has() take their strongest argument's specificity; :where() adds zero.
Inherited values lose to any rule that targets the element directly.
Prefer flat class selectors, :where() and @layer over IDs and !important.

SQL Window Functions: Running Totals, Rankings and Moving Averages

Thu, 29 Oct 2026 09:00:00 +0000

Sooner or later a report asks for something GROUP BY can't do. Show each sale next to its region's total. Rank sales reps within each region. Show this month's revenue next to last month's. With GROUP BY you get one row per group and lose the details; people often fall back to self-joins, subqueries or a loop in application code.

Window functions solve all of these in a single query. They compute values across a set of related rows (the window) without collapsing them: every input row is still in the output, with the calculated value added as a new column.

They work in PostgreSQL, MySQL 8+, SQL Server, Oracle and SQLite 3.25+. The examples below run as-is in SQLite. If JOINs still feel shaky, start with SQL JOINs Explained with Simple Examples.

Our example data

CREATE TABLE sales (rep TEXT, region TEXT, amount INTEGER);
INSERT INTO sales VALUES
  ('Ana', 'North', 900), ('Ben', 'North', 700), ('Cleo', 'North', 700), ('Dev', 'North', 400),
  ('Eve', 'South', 1200), ('Finn', 'South', 650), ('Gus', 'South', 300);

CREATE TABLE monthly (month TEXT, revenue INTEGER);
INSERT INTO monthly VALUES
  ('2026-01', 12000), ('2026-02', 15000), ('2026-03', 11000),
  ('2026-04', 18000), ('2026-05', 21000), ('2026-06', 19500);

GROUP BY vs a window

Here's the total per region with GROUP BY. Seven rows go in, two come out:

SELECT region, SUM(amount) AS total
FROM sales
GROUP BY region;

region	total
North	2700
South	2150

Now the same SUM, but followed by OVER, which turns it into a window function:

SELECT rep, region, amount,
       SUM(amount) OVER (PARTITION BY region) AS region_total,
       ROUND(100.0 * amount / SUM(amount) OVER (PARTITION BY region), 1) AS pct_of_region
FROM sales
ORDER BY region, amount DESC;

rep	region	amount	region_total	pct_of_region
Ana	North	900	2700	33.3
Ben	North	700	2700	25.9
Cleo	North	700	2700	25.9
Dev	North	400	2700	14.8
Eve	South	1200	2150	55.8
Finn	South	650	2150	30.2
Gus	South	300	2150	14.0

All seven rows survive. PARTITION BY region splits the rows into groups, like GROUP BY does, but instead of collapsing each group into one row, it gives every row access to its whole group. With an empty OVER (), the window is the entire result.

PARTITION BY splits the rows into a North window and a South window; every row keeps its own values and gains its window's total, and with ORDER BY the sum becomes a running total

The anatomy of OVER

function(...) OVER (
    PARTITION BY ...   -- which rows belong together (optional)
    ORDER BY ...       -- the order of rows inside each partition (optional)
    ROWS BETWEEN ...   -- which of those rows to use for this row (optional)
)

Every part is optional, and each one changes the result. We'll go through them one at a time.

Ranking: ROW_NUMBER, RANK and DENSE_RANK

Ranking functions need an ORDER BY inside the window. They differ only in how they treat ties, and our data has one: Ben and Cleo both sold 700.

SELECT rep, amount,
       ROW_NUMBER() OVER (ORDER BY amount DESC) AS row_num,
       RANK()       OVER (ORDER BY amount DESC) AS rnk,
       DENSE_RANK() OVER (ORDER BY amount DESC) AS dense_rnk
FROM sales
ORDER BY amount DESC, rep;

rep	amount	row_num	rnk	dense_rnk
Eve	1200	1	1	1
Ana	900	2	2	2
Ben	700	3	3	3
Cleo	700	4	3	3
Finn	650	5	5	4
Dev	400	6	6	5
Gus	300	7	7	6

ROW_NUMBER always gives 1, 2, 3, 4… with no ties. Which of Ben and Cleo gets 3 is up to the database, unless you add a tiebreaker such as ORDER BY amount DESC, rep.
RANK gives ties the same number, then skips: 3, 3, 5. Like a race where two runners share bronze.
DENSE_RANK gives ties the same number and doesn't skip: 3, 3, 4.

Top N per group

"The two best reps in each region" is one of the most common reporting questions, and window functions make it easy. But you can't filter on a window function directly:

SELECT rep FROM sales WHERE ROW_NUMBER() OVER (ORDER BY amount) <= 2;
-- Error: misuse of window function ROW_NUMBER()

Window functions are calculated after WHERE, GROUP BY and HAVING, just before the final ORDER BY and LIMIT. So you compute the rank in a subquery or CTE, then filter the result:

WITH ranked AS (
  SELECT rep, region, amount,
         ROW_NUMBER() OVER (PARTITION BY region ORDER BY amount DESC, rep) AS pos
  FROM sales
)
SELECT region, pos, rep, amount
FROM ranked
WHERE pos <= 2
ORDER BY region, pos;

region	pos	rep	amount
North	1	Ana	900
North	2	Ben	700
South	1	Eve	1200
South	2	Finn	650

Use RANK() instead of ROW_NUMBER() if tied reps should both make the list.

Running totals

Add ORDER BY to an aggregate's window and it becomes cumulative: each row sums everything from the start of the partition up to itself.

SELECT month, revenue,
       SUM(revenue) OVER (ORDER BY month) AS running_total
FROM monthly;

month	revenue	running_total
2026-01	12000	12000
2026-02	15000	27000
2026-03	11000	38000
2026-04	18000	56000
2026-05	21000	77000
2026-06	19500	96500

Add PARTITION BY to restart the running total for each group, for example a year-to-date total that resets every January.

Comparing with the previous row: LAG and LEAD

LAG(column) returns the value from the previous row in the window, and LEAD(column) from the next one. Month-over-month change becomes a simple subtraction:

SELECT month, revenue,
       LAG(revenue) OVER (ORDER BY month) AS prev_month,
       revenue - LAG(revenue) OVER (ORDER BY month) AS change
FROM monthly;

month	revenue	prev_month	change
2026-01	12000	NULL	NULL
2026-02	15000	12000	3000
2026-03	11000	15000	-4000
2026-04	18000	11000	7000
2026-05	21000	18000	3000
2026-06	19500	21000	-1500

The first month has no previous row, so LAG returns NULL. LAG(revenue, 1, 0) would return 0 instead, and LAG(revenue, 12) reaches back twelve rows, handy for year-over-year comparisons.

Window frames: moving averages

The frame picks which rows of the partition count for the current row. A three-month moving average uses the current row and the two before it:

SELECT month, revenue,
       ROUND(AVG(revenue) OVER (
         ORDER BY month
         ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
       ), 1) AS avg_3m
FROM monthly;

month	revenue	avg_3m
2026-01	12000	12000.0
2026-02	15000	13500.0
2026-03	11000	12666.7
2026-04	18000	14666.7
2026-05	21000	16666.7
2026-06	19500	19500.0

The first two months average over fewer rows because there's nothing before them. Other useful frames are ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING (a centred window) and ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING (the whole partition).

The default frame has a surprise

With ORDER BY and no frame, the default is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. RANGE treats rows with equal ORDER BY values as one step. Run a running total over amount DESC and Ben and Cleo, both at 700, both show 3500, because each one includes the other. Write ROWS UNBOUNDED PRECEDING when you want a strict row-by-row total.

The same default explains a classic puzzle: LAST_VALUE(revenue) OVER (ORDER BY month) returns the current row's value, not the last month's, because the default frame ends at the current row. Give it the whole partition with ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Windows over grouped results

Window functions run after GROUP BY, so they can work on the groups themselves. Here, each region's share of the grand total: the inner SUM(amount) is the region total, and the outer SUM(...) OVER () adds up those totals across all regions:

SELECT region,
       SUM(amount) AS total,
       ROUND(100.0 * SUM(amount) / SUM(SUM(amount)) OVER (), 1) AS pct_of_all
FROM sales
GROUP BY region;

region	total	pct_of_all
North	2700	55.7
South	2150	44.3

Name a window you use more than once

If several columns share the same window, define it once with a WINDOW clause after WHERE/GROUP BY/HAVING: SELECT ..., FIRST_VALUE(revenue) OVER w, LAG(revenue) OVER w FROM monthly WINDOW w AS (ORDER BY month);

Cheat sheet

You want	Use
A group total next to every row	`SUM(x) OVER (PARTITION BY g)`
A unique row number per group	`ROW_NUMBER() OVER (PARTITION BY g ORDER BY x)`
A ranking with ties	`RANK()` or `DENSE_RANK()`
A running total	`SUM(x) OVER (ORDER BY t ROWS UNBOUNDED PRECEDING)`
The previous / next row's value	`LAG(x)` / `LEAD(x)`
A moving average	`AVG(x) OVER (ORDER BY t ROWS BETWEEN n PRECEDING AND CURRENT ROW)`
Top N per group	`ROW_NUMBER()` in a CTE, then `WHERE pos <= N`

Recap

A window function is a normal function followed by OVER (...). It calculates across related rows but keeps every row.
PARTITION BY splits rows into groups, ORDER BY orders them inside each group, and the frame (ROWS BETWEEN ...) picks the rows used for each calculation.
ROW_NUMBER, RANK and DENSE_RANK differ only in how they handle ties.
Window functions run after WHERE and GROUP BY, so filter on them in a CTE or subquery.
Watch the default frame: use ROWS for strict running totals, and give LAST_VALUE the whole partition.

Once your queries summarise millions of rows, speed starts to matter; SQL Indexes and Query Plans is the natural next step.

SQL Indexes and Query Plans: Making Slow Queries Fast

Thu, 29 Oct 2026 09:00:00 +0000

A query that returns instantly on your laptop can take seconds in production. Usually nothing is wrong with the query itself: the table just grew from a thousand rows to a few million, and the database is reading every single one of them to find the handful you asked for.

Indexes fix that, but only the right index, and only if your query lets the database use it. In this article you'll build a million-row table, watch a query go from 100 ms to under a millisecond, and learn to read the query plan that tells you why.

We'll use SQLite because it's already installed almost everywhere and needs no server. The ideas carry over directly to PostgreSQL and MySQL; there's a translation table at the end. If JOINs are still new to you, start with SQL JOINs Explained with Simple Examples.

Set up a million rows

Open a new database with sqlite3 shop.db and paste this in. It creates an orders table and fills it with one million random orders from 50,000 customers:

CREATE TABLE orders (
  id          INTEGER PRIMARY KEY,
  customer_id INTEGER NOT NULL,
  status      TEXT    NOT NULL,
  total       REAL    NOT NULL,
  created_at  TEXT    NOT NULL
);

WITH RECURSIVE n(i) AS (
  SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 1000000
)
INSERT INTO orders (customer_id, status, total, created_at)
SELECT abs(random()) % 50000 + 1,
       CASE abs(random()) % 10 WHEN 0 THEN 'pending' WHEN 1 THEN 'cancelled' ELSE 'shipped' END,
       (abs(random()) % 50000) / 100.0,
       datetime('2024-01-01', '+' || (abs(random()) % 86400000) || ' seconds')
FROM n;

Then turn on timing with .timer on. Every query will now report how long it took.

Reading a query plan

Prefix any query with EXPLAIN QUERY PLAN and SQLite shows its strategy instead of running it:

EXPLAIN QUERY PLAN
SELECT * FROM orders WHERE customer_id = 4242;

QUERY PLAN
`--SCAN orders

SCAN means a full table scan: SQLite reads all one million rows and checks each one. Run the query itself and, in our test, it took about 100 ms to return a handful of rows. That doesn't sound like much until a page runs it twenty times, or a thousand users load that page at once.

How a B-tree index works

An index is a separate, sorted copy of one or more columns, where each entry points back to its row. Databases store it as a B-tree: a shallow tree of pages, each holding hundreds of sorted keys. To find 4242, the database reads the root page, follows the pointer for the range containing 4242, and repeats until it reaches a leaf page. Because each page fans out to hundreds of children, a million rows need only three or four levels.

A B-tree lookup for customer_id 4242 reads one root page, one branch page and one leaf page, then fetches only the matching rows

Create one and look at the plan again:

CREATE INDEX idx_orders_customer ON orders (customer_id);

EXPLAIN QUERY PLAN
SELECT * FROM orders WHERE customer_id = 4242;

QUERY PLAN
`--SEARCH orders USING INDEX idx_orders_customer (customer_id=?)

SEARCH ... USING INDEX means SQLite walks the index to the matching entries, then fetches just those rows from the table. The same query now runs in under a millisecond, roughly a hundred times faster, and the gap keeps growing with the table: a scan doubles in cost when the table doubles, while an index lookup barely notices.

Composite indexes: column order is everything

Real queries filter on more than one column. Here's a typical one: a customer's recent orders, newest last.

SELECT * FROM orders
WHERE customer_id = 4242 AND created_at >= '2025-01-01'
ORDER BY created_at;

With only idx_orders_customer, the plan has a second line:

QUERY PLAN
|--SEARCH orders USING INDEX idx_orders_customer (customer_id=?)
`--USE TEMP B-TREE FOR ORDER BY

USE TEMP B-TREE FOR ORDER BY means the rows come back unsorted and SQLite has to sort them itself. A composite index on both columns solves the filter and the sort at once:

CREATE INDEX idx_orders_customer_created ON orders (customer_id, created_at);

QUERY PLAN
`--SEARCH orders USING INDEX idx_orders_customer_created (customer_id=? AND created_at>?)

Think of a composite index like a phone book sorted by last name, then first name. Inside the "4242" section, entries are already sorted by created_at, so the database jumps to the first date it needs and reads forward. No sort step.

Now try the columns the other way round. Drop the two customer indexes first, so SQLite can't pick them instead:

DROP INDEX idx_orders_customer;
DROP INDEX idx_orders_customer_created;
CREATE INDEX idx_orders_created_customer ON orders (created_at, customer_id);

QUERY PLAN
`--SEARCH orders USING INDEX idx_orders_created_customer (created_at>?)

It still says SEARCH, so it looks fine. But it took 95 ms in our test, almost as slow as no index at all. Look inside the parentheses: only created_at>? uses the index. The range covers about two thirds of all orders, and SQLite has to check every one of those entries for customer_id = 4242. Drop this index (DROP INDEX idx_orders_created_customer;) before moving on.

The rule for column order

Put columns compared with = first, then one column used for a range (>, <, BETWEEN) or for ORDER BY last. An index on (a, b, c) can also serve queries on a alone or on a and b, but not queries on b or c alone. This is called the leftmost prefix rule.

SEARCH doesn't mean fast

Always read which columns appear inside the parentheses. A SEARCH that only uses a broad range can be nearly as slow as a SCAN.

Covering indexes: skip the table entirely

Even with a perfect index, each match costs an extra jump from the index into the table to fetch the rest of the row. If the index already contains every column the query needs, the database can skip the table altogether:

CREATE INDEX idx_orders_customer_created_total
  ON orders (customer_id, created_at, total);

EXPLAIN QUERY PLAN
SELECT created_at, total FROM orders WHERE customer_id = 4242;

QUERY PLAN
`--SEARCH orders USING COVERING INDEX idx_orders_customer_created_total (customer_id=?)

COVERING INDEX is the best plan you can get for a lookup. It matters most for queries that return many rows, such as reports and dashboards. It's another good reason to avoid SELECT *: asking for fewer columns makes a covering index possible.

Mistakes that silently disable an index

An index only helps if the database can compare your condition directly against the sorted values. Wrap the column in a function or expression, and it can't. Say we add a plain index on created_at and look for one day's orders:

CREATE INDEX idx_orders_created ON orders (created_at);

-- ✗ a full SCAN: date() has to be computed for every one of the million entries (≈600 ms)
SELECT count(*) FROM orders WHERE date(created_at) = '2025-03-14';

-- ✓ a SEARCH on idx_orders_created (created_at>? AND created_at<?) (<1 ms)
SELECT count(*) FROM orders
WHERE created_at >= '2025-03-14' AND created_at < '2025-03-15';

Both return the same count. The second form is called sargable (from Search ARGument ABLE): the column stands alone on one side of the comparison. Other common index killers:

Instead of…	Write…
`WHERE date(created_at) = '2025-03-14'`	`WHERE created_at >= '2025-03-14' AND created_at < '2025-03-15'`
`WHERE price * 1.2 > 100`	`WHERE price > 100 / 1.2`
`WHERE lower(email) = 'ada@example.com'`	store emails lowercased, or index the expression (below)
`WHERE name LIKE '%son'`	a leading wildcard can't use a B-tree; consider full-text search

When you genuinely need to filter on an expression, index the expression itself. SQLite, PostgreSQL and MySQL 8 all support this:

CREATE INDEX idx_orders_day ON orders (date(created_at));
-- now WHERE date(created_at) = '2025-03-14' uses: SEARCH orders USING INDEX idx_orders_day (<expr>=?)

Partial indexes: index only the rows you query

Suppose a background job keeps asking for pending orders, which are only 10% of the table. A partial index includes just those rows:

CREATE INDEX idx_orders_pending ON orders (created_at) WHERE status = 'pending';

EXPLAIN QUERY PLAN
SELECT id, created_at FROM orders
WHERE status = 'pending' AND created_at < '2024-02-01';

QUERY PLAN
`--SEARCH orders USING INDEX idx_orders_pending (created_at<?)

In our database this index takes 2.7 MB, against 27 MB for a full index on created_at. Smaller indexes are faster to search, fit in memory more easily, and cost less to update. PostgreSQL supports partial indexes too; MySQL doesn't.

Indexes aren't free

Every index is a second data structure the database must keep sorted. Each INSERT, UPDATE of an indexed column, and DELETE has to update every index on the table. In our test, inserting 200,000 more rows into the million-row table took about half a second with no secondary indexes and over seven seconds with three of them.

So don't index every column "just in case":

Index columns that appear in frequent WHERE, JOIN and ORDER BY clauses.
Prefer one well-designed composite index over several single-column ones.
Look for redundant indexes: (customer_id) is redundant once (customer_id, created_at) exists, because of the leftmost prefix rule.
Columns with only a few distinct values (like status) rarely make good indexes on their own. Use them in a partial index or as a later column in a composite one.
After big data changes, run ANALYZE so the planner has up-to-date statistics to choose between indexes.

Reading plans in PostgreSQL and MySQL

The vocabulary differs, but the questions are the same: is it scanning or searching, which index is used, and is there an extra sort?

What happens	SQLite	PostgreSQL `EXPLAIN`	MySQL `EXPLAIN`
Reads the whole table	`SCAN orders`	`Seq Scan on orders`	`type: ALL`
Uses an index, then fetches rows	`SEARCH ... USING INDEX`	`Index Scan` / `Bitmap Heap Scan`	`type: ref` or `range`
Answers from the index alone	`USING COVERING INDEX`	`Index Only Scan`	`Extra: Using index`
Extra sorting step	`USE TEMP B-TREE FOR ORDER BY`	`Sort`	`Extra: Using filesort`

In PostgreSQL, use EXPLAIN ANALYZE to actually run the query and see real row counts and timings next to the planner's estimates. When an estimate is wildly off from the real count, outdated statistics are the usual suspect.

Recap

SCAN reads every row; SEARCH ... USING INDEX jumps straight to the matches through a sorted B-tree.
In composite indexes, put equality columns first, then one range or sort column. Indexes only serve their leftmost prefix.
A covering index answers the query without touching the table.
Functions and arithmetic on an indexed column disable the index. Rewrite the condition, or index the expression.
Partial indexes keep indexes small when you only query a subset of rows.
Every index slows down writes. Add indexes for real queries, and check the plan before and after.

The habit that matters most: before you add an index, run EXPLAIN. After you add it, run EXPLAIN again.

Python Descriptors: The Protocol Behind Properties and Methods

Thu, 29 Oct 2026 09:00:00 +0000

Have you ever wondered how @property turns a method into something that looks like a plain attribute? Or where self actually comes from when you call obj.method()? Or how Django model fields and SQLAlchemy columns know which object they belong to?

They all rely on the same small mechanism: the descriptor protocol. It's one of the few genuinely "magic" parts of Python, and once you understand it you can read the language's internals and write cleaner, reusable attribute logic of your own.

If decorators are still new to you, read Python Decorators Explained Step by Step first. We'll use them below.

What is a descriptor?

A descriptor is any object whose class defines at least one of these methods:

__get__(self, obj, objtype=None)   # called when the attribute is read
__set__(self, obj, value)          # called when it is assigned
__delete__(self, obj)              # called on `del obj.attr`

The catch: it only works when the descriptor is stored as a class attribute. When Python looks up obj.attr and finds a descriptor on the class, it calls the descriptor's method instead of returning the descriptor itself.

Your first descriptor: a validated attribute

Say every Product needs a positive price and quantity. With properties you'd write the same getter and setter twice. With a descriptor you write the rule once and reuse it:

class Positive:
    """A number that must stay above zero."""

    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = "_" + name

    def __get__(self, obj, objtype=None):
        if obj is None:          # accessed on the class: Product.price
            return self
        return getattr(obj, self.private_name)

    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError(f"{self.public_name} must be positive, got {value!r}")
        setattr(obj, self.private_name, value)


class Product:
    price = Positive()
    quantity = Positive()

    def __init__(self, name, price, quantity):
        self.name = name
        self.price = price          # goes through Positive.__set__
        self.quantity = quantity

>>> p = Product("Keyboard", 49.0, 3)
>>> p.price
49.0
>>> vars(p)
{'name': 'Keyboard', '_price': 49.0, '_quantity': 3}
>>> p.quantity = -1
ValueError: quantity must be positive, got -1

Three details are worth noticing:

__set_name__ (Python 3.6+) is called once, when the class is created, with the attribute name the descriptor was assigned to. That's how one Positive class knows whether it's guarding price or quantity.
obj is None means the attribute was read from the class (Product.price) rather than an instance. Returning the descriptor itself is the convention, and it keeps tools like help() working.
The real values live on the instance, under _price and _quantity. The descriptor only holds configuration.

The attribute lookup order

To use descriptors confidently you need the exact rules object.__getattribute__ follows for obj.attr. Descriptors come in two kinds:

A data descriptor defines __set__ or __delete__ (Positive and property are data descriptors).
A non-data descriptor defines only __get__ (plain functions are non-data descriptors).

Python checks a data descriptor on the class first, then the instance dictionary, then non-data descriptors and class attributes, and finally __getattr__

The difference matters because the instance dictionary sits between them. You can prove it:

class DataDesc:
    def __get__(self, obj, objtype=None):
        return "data descriptor"

    def __set__(self, obj, value):
        raise AttributeError("read-only")


class NonDataDesc:
    def __get__(self, obj, objtype=None):
        return "non-data descriptor"


class Demo:
    a = DataDesc()
    b = NonDataDesc()


d = Demo()
d.__dict__["a"] = "instance dict"     # bypass __set__ and write directly
d.__dict__["b"] = "instance dict"

print(d.a)   # data descriptor   ← beats the instance dict
print(d.b)   # instance dict     ← beats the non-data descriptor

That one rule explains a surprising amount of Python's behaviour, as the next sections show.

Methods are descriptors

Functions have a __get__ method. When you read a function through an instance, __get__ returns a bound method, a small object that remembers both the function and the instance. That's where self comes from:

>>> class Greeter:
...     def hello(self):
...         return "hi"
...
>>> g = Greeter()
>>> f = Greeter.__dict__["hello"]    # the raw function, no descriptor magic
>>> f
<function Greeter.hello at 0x7fc890f70160>
>>> f.__get__(g, Greeter)            # exactly what g.hello does
<bound method Greeter.hello of <__main__.Greeter object at 0x7fc890f63760>>
>>> g.hello == f.__get__(g, Greeter)
True

Because functions are non-data descriptors, an instance attribute with the same name wins. That's why you can override a method on a single object (handy for mocks in tests):

>>> g.hello = lambda: "patched"
>>> g.hello(), Greeter().hello()
('patched', 'hi')

classmethod and staticmethod are non-data descriptors too: their __get__ binds the class instead of the instance, or binds nothing at all.

Rebuilding @property

property is a data descriptor written in C, but a working pure-Python version fits on one screen:

class MyProperty:
    def __init__(self, fget, fset=None):
        self.fget = fget
        self.fset = fset

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError("read-only attribute")
        self.fset(obj, value)

    def setter(self, fset):
        return type(self)(self.fget, fset)    # a new descriptor with both functions


class Temperature:
    def __init__(self, celsius):
        self.celsius = celsius

    @MyProperty
    def fahrenheit(self):
        return self.celsius * 9 / 5 + 32

    @fahrenheit.setter
    def fahrenheit(self, value):
        self.celsius = (value - 32) * 5 / 9

>>> t = Temperature(100)
>>> t.fahrenheit
212.0
>>> t.fahrenheit = 32
>>> t.celsius
0.0

@MyProperty is just fahrenheit = MyProperty(fahrenheit). Then @fahrenheit.setter builds a second descriptor holding both functions and rebinds the name to it. Because MyProperty always defines __set__, it's a data descriptor even without a setter, which is why a read-only property can't be shadowed by assigning to the instance.

cached_property: using the lookup order on purpose

functools.cached_property is a clever use of the rule from earlier. It's a non-data descriptor: the first access runs your method and stores the result in the instance's __dict__ under the same name. From then on the instance dictionary wins the lookup, so __get__ is never called again:

>>> from functools import cached_property
>>> class Report:
...     def __init__(self, rows):
...         self.rows = rows
...
...     @cached_property
...     def total(self):
...         print("computing…")
...         return sum(self.rows)
...
>>> r = Report([3, 4, 5])
>>> r.total
computing…
12
>>> r.total          # served straight from r.__dict__
12
>>> vars(r)
{'rows': [3, 4, 5], 'total': 12}
>>> del r.total      # clear the cache; the next access recomputes

cached_property needs an instance dict

Since the cache lives in obj.__dict__, cached_property doesn't work on classes that define __slots__ without including "__dict__".

The classic bug: storing state on the descriptor

There's only one descriptor object per class attribute, shared by every instance. Store per-object data on self inside a descriptor and all instances end up sharing it:

class Broken:
    def __get__(self, obj, objtype=None):
        return self.value

    def __set__(self, obj, value):
        self.value = value     # stored on the descriptor, which every instance shares


class Account:
    balance = Broken()


a, b = Account(), Account()
a.balance = 100
b.balance = 5
print(a.balance)   # 5  ← a's balance was overwritten by b

Store values on obj instead, as Positive does with setattr(obj, self.private_name, value).

Avoid infinite recursion

Inside __set__, never write setattr(obj, self.public_name, value). Assigning to the public name calls __set__ again, forever. Always store under a different name, or write to obj.__dict__ directly.

Where else you'll meet descriptors

Once you know the pattern you'll spot it everywhere:

Feature	Descriptor kind	What `__get__` does
Plain methods	non-data	returns a bound method
`classmethod` / `staticmethod`	non-data	binds the class / binds nothing
`property`	data	calls the getter function
`functools.cached_property`	non-data	computes once, caches in `obj.__dict__`
`__slots__` entries	data	reads a fixed slot in the object's memory
Django fields, SQLAlchemy columns	depends on the field	load the value from the database, convert or track it

You can even see the __slots__ ones directly:

>>> class Point:
...     __slots__ = ("x", "y")
...
>>> Point.__dict__["x"]
<member 'x' of 'Point' objects>

Recap

A descriptor is a class attribute whose class defines __get__, __set__ or __delete__.
Data descriptors (with __set__ or __delete__) beat the instance dictionary; non-data descriptors lose to it.
Methods, property, classmethod, staticmethod, cached_property and __slots__ are all built on descriptors.
Use __set_name__ to learn the attribute's name, and store per-instance values on the instance, never on the descriptor.
Reach for a custom descriptor when the same attribute logic (validation, conversion, lazy loading) repeats across several attributes or classes.

The best way to make this stick: open a REPL, write a descriptor that prints in each method, and watch exactly when Python calls it.

Linux Processes Under the Hood: fork, exec, Signals and Zombies

Thu, 29 Oct 2026 09:00:00 +0000

You type ls and press Enter. A list of files appears, and the prompt comes back. Behind that, the kernel has created a process, replaced its program, collected its exit status and cleaned up after it, using just a handful of system calls that every program on Linux relies on.

Knowing those calls explains a lot of things that otherwise seem mysterious: why cd can't be an external program, what exit code 137 means, why kill -9 should be your last resort, where zombie processes come from, and why a container sometimes ignores docker stop for exactly ten seconds. The examples use Python's os module, which maps almost one-to-one onto the underlying system calls, plus standard tools like ps. You'll get more out of it if you've written a few shell scripts; see Writing Bash Scripts That Don't Break.

fork and exec: how every program starts

Linux creates new processes in two separate steps:

fork() makes a copy of the calling process. Parent and child continue from the same line of code, with the same memory contents, open files and environment. The only difference is fork()'s return value: 0 in the child, and the child's PID in the parent.
exec() replaces the program running in the current process with a new one. The PID stays the same; the code, memory and stack are thrown away and loaded fresh from the new executable.

A shell runs ls by forking itself, then making the child exec ls, while the parent waits. Here's the same thing in Python:

import os

print(f"parent: my PID is {os.getpid()}", flush=True)
pid = os.fork()

if pid == 0:
    # Child: a copy of the parent. Replace it with the `ls` program.
    print(f"child:  my PID is {os.getpid()}, my parent is {os.getppid()}", flush=True)
    os.execvp("ls", ["ls", "-d", "/tmp"])
    print("never printed: exec only returns if it fails")
else:
    # Parent: wait for the child and collect its exit status.
    _, status = os.waitpid(pid, 0)
    print(f"parent: child {pid} exited with code {os.waitstatus_to_exitcode(status)}")

$ python3 forkexec.py
parent: my PID is 726920
child:  my PID is 726922, my parent is 726920
/tmp
parent: child 726922 exited with code 0

Why flush=True matters

exec replaces the whole program, including Python's output buffer. If the output goes to a pipe or a file, it's buffered, and anything not yet flushed is silently lost. The same applies to C's printf. Flush before you fork or exec.

The split into two calls is what makes Unix shells so flexible. Between fork and exec, the child is still running the shell's code, so it can rearrange its own environment first: redirect file descriptors for > file or |, change directory, set environment variables. Then it execs the program, which inherits all of it.

It also explains why cd must be a shell builtin. A child process can only change its own working directory, and that change disappears when it exits. To change the shell's directory, the shell has to do it itself.

The life of a process: the shell (PID 4100) forks a child (PID 4101), the child execs ls, ls exits with code 0, the child stays a zombie until the shell calls wait, and then it disappears

Exit codes and wait

When a process ends, it leaves behind an exit status for its parent: 0 for success, 1–255 for failure, by convention. The parent collects it with wait() or waitpid(), and the shell puts it in $?.

If the process was killed by a signal, the shell reports 128 + the signal number. These come up constantly in logs and CI output:

Exit code	Meaning
`0`	Success
`1`, `2`, …	The program reported an error (meaning varies by program)
`126` / `127`	Command found but not executable / command not found
`130`	Killed by SIGINT (2): someone pressed `Ctrl`+`C`
`137`	Killed by SIGKILL (9): often the out-of-memory killer, or a container exceeding its memory limit
`143`	Killed by SIGTERM (15): a normal shutdown request, e.g. `docker stop` or a deploy

Signals: notifying a process

A signal is a small asynchronous notification sent to a process, by the kernel or by another process with kill. Despite the name, kill sends any signal; SIGTERM is just the default. The ones you'll meet most:

Signal	Number	Default action	Typical source
SIGHUP	1	terminate	Terminal closed; many daemons reload their config on it
SIGINT	2	terminate	`Ctrl`+`C`
SIGQUIT	3	terminate + core dump	`Ctrl`+`\`
SIGKILL	9	terminate, can't be caught	`kill -9`, the OOM killer
SIGTERM	15	terminate	`kill`, `systemctl stop`, `docker stop`
SIGCHLD	17	ignore	A child process exited or stopped
SIGSTOP / SIGTSTP	19 / 20	pause	`kill -STOP` / `Ctrl`+`Z`

(The numbers are for x86 and ARM Linux; run kill -l to see the full list on your system.)

A process can handle most signals with its own function, or ignore them. Two can never be caught or ignored: SIGKILL and SIGSTOP. The kernel acts on those directly.

Graceful shutdown

That's why you should send SIGTERM first and SIGKILL only as a last resort. SIGTERM lets a program finish the request it's serving, flush data to disk, close database connections and remove lock files. SIGKILL stops it mid-instruction, with no chance to clean up.

Handling SIGTERM takes a few lines:

import signal, time

running = True

def stop(signum, frame):
    global running
    print(f"got {signal.Signals(signum).name}, finishing the current job...", flush=True)
    running = False

signal.signal(signal.SIGTERM, stop)

while running:
    time.sleep(1)        # pretend to work
print("clean exit", flush=True)

$ python3 worker.py &
$ kill %1
got SIGTERM, finishing the current job...
clean exit

The handler only sets a flag; the main loop checks it and exits at a safe point. Doing real work inside a signal handler is risky, because it can interrupt your program anywhere. Bash scripts get the same ability with trap 'cleanup' TERM.

Service managers follow the same pattern: systemctl stop sends SIGTERM, waits (90 seconds by default), then sends SIGKILL. docker stop does the same with a 10-second grace period.

Zombies

When a child exits, the kernel can't throw it away completely yet: the parent might still want its exit status. So the process becomes a zombie. It has no memory or open files any more, just a process-table entry holding its PID and exit status, waiting for the parent to call wait(). Collecting the status is called reaping, and it removes the zombie.

A parent that never calls wait() leaves zombies behind. Let's make one:

import os, time

pid = os.fork()
if pid == 0:
    os._exit(0)          # the child exits immediately...
print(f"child PID: {pid}")
time.sleep(60)           # ...but the parent never calls wait()

$ python3 zombie.py &
child PID: 726905
$ ps -o pid,ppid,stat,cmd --ppid $!
    PID    PPID STAT CMD
 726905  726903 Z    [python3] <defunct>

Z in the STAT column and <defunct> mark a zombie. A few zombies are harmless, but each one holds a PID, and a buggy server that leaks zombies can eventually exhaust the PID limit, at which point nothing on the machine can start a new process.

You can't kill a zombie: it's already dead, so signals have no effect. Fix the parent so it calls wait(), or terminate the parent. When a parent dies, its children (living or zombie) are re-parented to PID 1 (or to the nearest process marked as a "subreaper"), which reaps them automatically. On a normal system, PID 1 is systemd, and it's very good at reaping.

PID 1 is special, and containers make it your problem

Inside a container, there's no systemd. Whatever command the image starts becomes PID 1 in the container's PID namespace, and PID 1 has two extra responsibilities:

The kernel doesn't apply default signal actions to it. A signal that PID 1 hasn't installed a handler for is simply ignored. A normal program that relies on SIGTERM's default "terminate" action will ignore docker stop, sit there for the 10-second grace period, and get SIGKILLed.
It inherits every orphan, so it has to reap them, or zombies pile up.

You can see the first one without Docker. unshare starts a program as PID 1 in a new PID namespace:

$ sudo unshare --fork --pid --mount-proc python3 -c 'import time; time.sleep(30)' &
$ pid=$(pgrep -n -f 'time.sleep')    # the Python process, as seen from outside
$ sudo kill -TERM "$pid"             # nothing happens: PID 1 has no SIGTERM handler
$ sudo kill -KILL "$pid"             # this works: SIGKILL from the parent namespace

The fixes are simple:

Run a tiny init process as PID 1: docker run --init adds tini, which forwards signals to your program and reaps zombies. Kubernetes users can add tini (or dumb-init) to the image as the entrypoint.
Make your program handle SIGTERM explicitly, like worker.py above.
Use the exec form in Dockerfiles, CMD ["python3", "app.py"], not CMD python3 app.py. The shell form starts /bin/sh -c "python3 app.py", so the shell becomes PID 1, and it doesn't pass SIGTERM on to your program.
In entrypoint scripts, end with exec "$@". exec replaces the shell with your program (the same exec as above), so your program becomes PID 1 and receives signals directly.

Inspecting processes

Every process has a directory under /proc/<PID>/, and most tools just read from it:

ps -eo pid,ppid,stat,etime,cmd --forest   # every process, as a tree
pstree -p                                 # a compact tree with PIDs
cat /proc/1234/status                     # state, parent, memory, threads, signal masks
tr '\0' ' ' < /proc/1234/cmdline          # the full command line
ls -l /proc/1234/fd                       # open files, sockets and pipes
cat /proc/1234/environ | tr '\0' '\n'     # its environment (needs permission)

The STAT column's first letter tells you what a process is doing:

State	Meaning
`R`	Running, or ready to run
`S`	Sleeping, waiting for something (most processes, most of the time)
`D`	Uninterruptible sleep, usually waiting on disk or network storage; even SIGKILL waits until it returns
`T`	Stopped (SIGSTOP, `Ctrl`+`Z`, or a debugger)
`Z`	Zombie: exited, waiting to be reaped

A process stuck in D that can't be killed almost always points to a storage problem (a hung NFS mount, a failing disk), not to the program itself.

Recap

New processes are made with fork() (copy the current process) and exec() (replace its program). The gap between them is where shells set up redirection and pipes.
The parent collects the exit status with wait(); exit codes above 128 mean "killed by signal (code − 128)": 130 Ctrl+C, 137 SIGKILL/OOM, 143 SIGTERM.
Send SIGTERM and give programs time to shut down; SIGKILL can't be caught and skips all cleanup.
Zombies are exited children nobody has reaped; fix or kill the parent, not the zombie.
In containers, PID 1 ignores signals it doesn't handle and must reap orphans: use --init/tini, exec-form CMD, and exec "$@" in entrypoints.

Interactive Rebase: Clean Up Your Commits Before You Share Them

Thu, 29 Oct 2026 09:00:00 +0000

Real work is messy. You commit a half-finished form, then "WIP", then "fix typo", then a debug print you forgot to remove. That's fine while you're working, but it's a poor history to hand to a reviewer, and an even worse one to find six months later in git log or git blame.

Interactive rebase lets you rewrite your own branch before you share it: combine commits, rewrite their messages, change their order, split them up or delete them. It's one of the most useful Git skills to learn once you're comfortable with branches. If you aren't yet, read A Practical Git Branching Workflow for Beginners first.

The messy branch

Here's a feature branch with five commits on top of main:

$ git log --oneline main..
746aec7 debug print
451c410 fix typo
63d1135 Add password validation
0ce6ff7 WIP
4430f61 Add login form

main.. means "commits on this branch that aren't on main". What we'd like to share is just two commits: Add login form (including the WIP and the typo fix, which both touched the form) and Add password validation, without the debug print.

Starting an interactive rebase

git rebase -i main

Git opens your editor with a todo list, one line per commit:

pick 4430f61 Add login form
pick 0ce6ff7 WIP
pick 63d1135 Add password validation
pick 451c410 fix typo
pick 746aec7 debug print

# Rebase 1c893f9..746aec7 onto 1c893f9 (5 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
#                    commit's log message, unless -C is used, ...
# d, drop <commit> = remove commit
# ...

The todo list is upside down compared to git log

git log shows the newest commit first. The todo list shows the oldest first, because that's the order Git will replay them in. Read it from top to bottom like a script.

Git will start from main and run each line in turn. Change the word at the start of a line to change what happens to that commit:

Command	What it does
`pick`	Keep the commit as it is
`reword`	Keep the changes, but stop to edit the message
`edit`	Apply the commit, then pause so you can change it (amend, split…)
`squash`	Merge into the commit above, and combine both messages
`fixup`	Merge into the commit above, and throw this commit's message away
`drop`	Delete the commit (removing the line does the same)
`exec`	Run a shell command at this point, and stop if it fails

Squash, reorder and drop

Edit the todo list like this, then save and close the editor:

pick 4430f61 Add login form
fixup 0ce6ff7 WIP
fixup 451c410 fix typo
pick 63d1135 Add password validation
drop 746aec7 debug print

We changed three things: the two small commits become fixups of Add login form, fix typo moved up so that it sits right under the commit it fixes, and the debug print is dropped.

$ git rebase -i main
Successfully rebased and updated refs/heads/login-form.

$ git log --oneline main..
486c6b2 Add password validation
24c6dfe Add login form

The todo list read from top to bottom: two fixups melt into Add login form, Add password validation is kept, and the debug print is dropped, leaving two clean commits

Notice the hashes changed, even for Add password validation, whose content we didn't touch. Rebase never edits commits: it creates new ones with the same changes. That detail matters when we push, below.

squash or fixup?

Use squash when the second message has something worth keeping: Git opens the editor with both messages so you can merge them. Use fixup for "oops" commits whose message you'd delete anyway.

Fixup commits: clean up as you go

Reordering lines by hand gets tedious. If you already know which commit a change belongs to, tell Git when you commit:

# Spotted a bug in "Add password validation"? Fix it, then:
git commit -a --fixup=486c6b2

Git creates a commit called fixup! Add password validation. Later, --autosquash moves every such commit under its target and marks it fixup for you:

$ git rebase -i --autosquash main

pick 24c6dfe Add login form
pick 486c6b2 Add password validation
fixup 4eb2adb fixup! Add password validation

Just save and close. To make this the default, run git config --global rebase.autoSquash true.

Splitting a commit with edit

Sometimes one commit does two unrelated things. Mark it edit, and Git pauses right after applying it:

edit 9a1c2d3 Add login form and fix footer links
pick 3f7e8b1 Add password validation

git reset HEAD^                 # undo the commit, keep its changes as unstaged edits
git add login.html
git commit -m "Add login form"
git add footer.html
git commit -m "Fix footer links"
git rebase --continue           # carry on with the rest of the todo list

git add -p is handy here when both changes are in the same file: it asks about each block of changes one at a time.

When there's a conflict

Reordering or dropping commits can produce conflicts, exactly like a merge. Git stops and tells you which files conflict. You have three options:

# fix the files, then mark them as resolved and continue
git add login.html
git rebase --continue

git rebase --skip      # drop the commit that conflicts and move on
git rebase --abort     # give up: put the branch back exactly as it was

--abort is your safety net. As long as the rebase hasn't finished, nothing has changed for real.

Run your tests on every commit

git rebase -i main --exec "npm test" (or -x) adds an exec line after every commit. If the tests fail anywhere, the rebase stops on the commit that broke them. It's a great final check that each commit works on its own, which is what makes git bisect useful later.

Rebase without moving your branch

git rebase -i main also moves your branch on top of the latest main. Usually that's what you want, but if main has moved a lot, you'll get conflicts that have nothing to do with your cleanup. To tidy commits in place, keeping the same starting point:

git rebase -i --keep-base main

You can also name the starting point directly: git rebase -i HEAD~3 lets you edit the last three commits.

Pushing a rewritten branch

Because the commits are new, a branch you pushed before the rebase no longer matches the remote, and a normal git push is rejected. Use:

git push --force-with-lease

Unlike --force, --force-with-lease refuses to overwrite the remote branch if someone else pushed to it since you last fetched, so you can't wipe out a teammate's work by accident.

The golden rule of rebasing

Only rewrite commits that nobody else has built on. Cleaning up your own feature branch before (or during) a pull request is fine. Rebasing main, or a branch your teammates have pulled, leaves them with copies of the old commits and a painful merge.

Undoing a rebase

Finished a rebase and regret it? Git keeps the old position of your branch in ORIG_HEAD:

git reset --hard ORIG_HEAD

This only works right after the rebase, because other commands also update ORIG_HEAD. Further back, the reflog remembers every place your branch has been (git reflog). How Git Really Works explains why the old commits are still there.

Recap

git rebase -i main opens a todo list of your branch's commits, oldest first.
Change pick to reword, edit, squash, fixup or drop, and move lines to reorder commits.
git commit --fixup=<commit> plus --autosquash makes cleanup almost automatic.
--abort stops a rebase in progress; ORIG_HEAD and the reflog undo a finished one.
Rebasing creates new commits, so push with --force-with-lease, and never rebase commits other people have built on.

How Git Really Works: Blobs, Trees, Commits and Refs

Thu, 29 Oct 2026 09:00:00 +0000

Most people learn Git as a list of commands to memorise. That works until something goes wrong: a rebase rewrites history you didn't expect, a reset --hard "deletes" a day's work, or you find yourself in a detached HEAD. Then the commands stop making sense.

Underneath, Git is surprisingly simple. It's a key-value store of immutable objects, plus a handful of movable pointers called refs. Once you've seen both with your own eyes, every Git command becomes "add some objects, then move a pointer".

You'll get the most from this if you're comfortable with everyday branching. If not, read A Practical Git Branching Workflow for Beginners first.

Content-addressed storage

Git stores everything (file contents, directories, commits) as objects, and each object's name is the SHA-1 hash of its content. Ask Git to hash some text:

$ echo 'hello' | git hash-object --stdin
ce013625030ba8dba906f756967f9e9ca394464a

Run that on any machine, in any repository, and you get the same hash. Git hashes a small header (the word blob, a space, the size in bytes and a null byte) followed by the content, and you can reproduce it without Git at all:

$ printf 'blob 6\0hello\n' | sha1sum
ce013625030ba8dba906f756967f9e9ca394464a  -

This design has big consequences. Identical content is stored only once, however many files or commits contain it. And since the name depends on the content, an object can never change: change one byte and you get a different object with a different name.

The four object types

Type	Stores	Points to
blob	the contents of one file (no name, no permissions)	nothing
tree	one directory: names, modes and the hashes of its entries	blobs and other trees
commit	a snapshot: author, committer, date and message	one tree and its parent commit(s)
tag	an annotated tag: tagger, message, optional signature	usually a commit

Let's make a tiny repository and look at real objects:

git init -b main demo && cd demo
mkdir src
echo "# Demo" > README.md
echo 'print("hello")' > src/app.py
git add . && git commit -m "First commit"

That one commit created five objects, stored as compressed files under .git/objects/, named by the first two characters of the hash, then the rest:

$ find .git/objects -type f
.git/objects/08/05455a24b6c68fbc38d0fa5d121f735984285d
.git/objects/11/b15b1a4584b08fa423a57964bdbf018b0da0d5
.git/objects/16/ae6a2afcc370fa1a06bc31edcacc3a7f2c41cc
.git/objects/60/546c2588e9d75d822392d6f296e1fd99f04c96
.git/objects/ef/6ad6746727fa1d0efcf4a020ec3caeda5ccf64

Your hashes will mostly match

Blobs and trees depend only on content, so if you used the same files, those four hashes will be identical on your machine. The commit hash won't be: it includes your name, email and the exact time.

Reading objects with cat-file

git cat-file -p pretty-prints any object. Start from the commit that HEAD points to:

$ git cat-file -p HEAD
tree 60546c2588e9d75d822392d6f296e1fd99f04c96
author Ada <ada@example.com> 1793256281 +0000
committer Ada <ada@example.com> 1793256281 +0000

First commit

A commit is just a few lines of text. It names exactly one tree, the snapshot of the whole project. Follow it:

$ git cat-file -p 60546c2
100644 blob 0805455a24b6c68fbc38d0fa5d121f735984285d    README.md
040000 tree ef6ad6746727fa1d0efcf4a020ec3caeda5ccf64    src

$ git cat-file -p ef6ad67
100644 blob 11b15b1a4584b08fa423a57964bdbf018b0da0d5    app.py

$ git cat-file -p 11b15b1
print("hello")

Notice where the file names live: in the trees, not in the blobs. Renaming a file creates a new tree but reuses the same blob.

Now change one file and commit again:

$ echo 'print("hello, world")' > src/app.py
$ git commit -am "Greet the world"
$ git cat-file -p HEAD
tree 17b6cd9de955edb28d8a9b0e4668f588ba3c664a
parent 16ae6a2afcc370fa1a06bc31edcacc3a7f2c41cc
...
$ git cat-file -p 'HEAD^{tree}'
100644 blob 0805455a24b6c68fbc38d0fa5d121f735984285d    README.md
040000 tree b6dcf64039da33b42fb025d2cc49c6d070c65686    src

The new commit has a parent line pointing to the first commit. Its root tree has a new src tree (because app.py changed), but README.md still points to the very same blob, 0805455. Unchanged files cost nothing.

Two commits: the second points to its parent and to a new tree; the unchanged README blob is shared by both trees, and main and HEAD point to the newest commit

Snapshots, not diffs

Each commit stores a full snapshot (a tree), not a list of changes. git diff and git show compute changes on the fly by comparing two trees. To save space on disk, git gc later packs objects into packfiles and delta-compresses similar ones, but that's a storage detail, invisible to everything else.

Refs: branches are 41-byte files

If objects never change, how does a branch "move"? Branches aren't objects. They're refs: tiny files that contain a commit hash.

$ cat .git/HEAD
ref: refs/heads/main

$ cat .git/refs/heads/main
ffb68609991f7ac36c595767d2d91545868364cb

HEAD usually holds the name of the current branch, and the branch file holds a commit hash. When you commit, Git writes the new objects and then rewrites that one branch file with the new hash. That's all a commit is, from the branch's point of view.

This explains several things that otherwise feel like magic:

Creating a branch is instant, whatever the size of the repository: git branch experiment just writes a new 41-byte file. (Git may later move refs into a single .git/packed-refs file, which works the same way.)
Detached HEAD means HEAD contains a commit hash directly instead of ref: refs/heads/.... New commits still work, but no branch moves to remember them.
Tags are refs under refs/tags/ that, by convention, never move.

Build a commit by hand

To prove there's nothing more to it, let's commit a file without git add or git commit, using only Git's low-level "plumbing" commands:

# 1. Store the content as a blob (-w writes it into .git/objects)
blob=$(echo "Made without git add" | git hash-object -w --stdin)

# 2. Put it into the staging area (the index) under a file name
git update-index --add --cacheinfo 100644 "$blob" NOTES.txt

# 3. Turn the index into a tree object
tree=$(git write-tree)

# 4. Create a commit object pointing to that tree, with HEAD as its parent
commit=$(git commit-tree "$tree" -p HEAD -m "Commit made by hand")

# 5. Move the branch pointer
git update-ref refs/heads/main "$commit"

$ git log --oneline
8e0d264 Commit made by hand
ffb6860 Greet the world
16ae6a2 First commit

A completely normal commit. Run git status now and you'll see NOTES.txt listed as deleted: it's in the commit and the index, but we never wrote it to the working directory. git restore NOTES.txt fixes that. Those are Git's three places for a file: the object database, the index, and your working directory.

What this explains

Rebase creates new commits. A commit's hash covers its parent's hash. When you rebase a branch, every commit gets a new parent, so Git has to create brand-new commits with new hashes, even if the code changes are identical. The old commits still exist; the branch simply points somewhere else now. That's why rebasing commits that others have already pulled causes trouble: they still have the old ones.

reset just moves a pointer. git reset --hard HEAD~2 points the current branch two commits back (and updates your files to match). The commits themselves aren't deleted.

The reflog remembers where pointers were. Every time HEAD or a branch moves, Git logs it. So a "lost" commit is usually one command away:

$ git reset --hard HEAD~2        # oops
$ git reflog
16ae6a2 HEAD@{0}: reset: moving to HEAD~2
8e0d264 HEAD@{1}: ...
$ git reset --hard 8e0d264       # or: git branch rescue 8e0d264

The safety net has an expiry date

Commits that no branch, tag or reflog entry can reach are eventually deleted by git gc. By default, reflog entries for unreachable commits expire after 30 days (90 days for reachable ones). Recover lost work soon, and never assume an unpushed commit is safe forever.

Integrity is built in. Because every commit's hash covers its tree, and every tree's hash covers its blobs, one commit hash vouches for the entire history behind it. Change a single byte anywhere in the past and every hash after it changes. (Git is gradually moving from SHA-1 to SHA-256 for exactly this reason; you can already try it with git init --object-format=sha256.)

Plumbing commands worth knowing

Command	What it does
`git cat-file -t <obj>`	Show an object's type
`git cat-file -p <obj>`	Pretty-print an object
`git ls-tree -r HEAD`	List every file in a commit's tree, recursively
`git rev-parse main`	Turn a name (branch, tag, `HEAD~2`…) into a hash
`git hash-object -w <file>`	Store a file as a blob
`git update-ref <ref> <hash>`	Point a ref at a commit
`git count-objects -vH`	Show how many objects there are and how much space they use

Recap

Git is a content-addressed object store: every object's name is the hash of its content, so objects never change.
Blobs hold file contents, trees hold directories and names, commits point to one tree and their parents.
Commits are snapshots; unchanged files are shared between them for free.
Branches, tags and HEAD are refs: small files containing a hash or another ref's name.
Rebase, reset and amend create new objects and move pointers. They don't edit history in place, which is why the reflog can usually bring things back.

Next time a Git command surprises you, ask two questions: which objects did it create, and which pointer did it move?

Dijkstra's Algorithm: Shortest Paths with a Priority Queue

Thu, 29 Oct 2026 09:00:00 +0000

Your maps app finding a route, a network router choosing where to forward a packet, a game character walking around obstacles: they're all versions of the same question. Given a graph with weighted edges, what's the cheapest path from here to everywhere else?

For unweighted graphs, breadth-first search answers it: explore one step away, then two, then three. Weights break that. A path with three cheap edges can beat one with a single expensive edge, so "fewest steps" is no longer "shortest". Dijkstra's algorithm, published by Edsger Dijkstra in 1959, fixes this by always expanding the closest unfinished node next.

This article builds it in Python, traces it on a real graph, explains why it works, and covers the details that separate a textbook version from one you can trust: stale heap entries, ties, negative weights and complexity. You should be comfortable with Big-O (Big-O Notation Explained with Real Code) and with Python dictionaries.

The example graph

Six places joined by roads, each road labelled with its travel time. Roads go both ways.

A graph of six nodes. Shortest distances from A: A 0, C 2, B 3, D 8, E 10, F 12. The highlighted shortest-path tree uses the edges A–C, C–B, B–D, D–E and E–F

We'll store it as an adjacency map: for each node, a dictionary of neighbours and edge weights.

from collections import defaultdict

edges = [("A", "B", 4), ("A", "C", 2), ("B", "C", 1), ("B", "D", 5), ("C", "D", 8),
         ("C", "E", 10), ("D", "E", 2), ("D", "F", 6), ("E", "F", 2)]

graph = defaultdict(dict)
for u, v, w in edges:
    graph[u][v] = w
    graph[v][u] = w      # undirected: add both directions

The idea

Keep a tentative distance for every node: 0 for the source, infinity for everything else. Then repeat:

Pick the unfinished node with the smallest tentative distance. Its distance is now final.
Relax each of its edges: if going through this node gives a neighbour a shorter distance than the one it has, update the neighbour.

The only hard part is step 1: finding the smallest distance quickly. Scanning every node each time costs O(V) per step. A priority queue (a binary heap) gets it down to O(log V).

The implementation

Python's heapq module turns a plain list into a min-heap. It has no "decrease key" operation, so when a node's distance improves we simply push a new entry and leave the old one in the heap. When the old one eventually comes out, we recognise it as stale and skip it. This is called lazy deletion, and it's how almost every Python implementation works.

import heapq

def dijkstra(graph, source):
    dist = {source: 0}
    prev = {}                             # prev[node] = the node we reached it from
    heap = [(0, source)]
    while heap:
        d, node = heapq.heappop(heap)
        if d > dist[node]:
            continue                      # stale entry: we already found a shorter way here
        for neighbor, weight in graph[node].items():
            new_dist = d + weight
            if new_dist < dist.get(neighbor, float("inf")):
                dist[neighbor] = new_dist
                prev[neighbor] = node
                heapq.heappush(heap, (new_dist, neighbor))
    return dist, prev

>>> dist, prev = dijkstra(graph, "A")
>>> dist
{'A': 0, 'B': 3, 'C': 2, 'D': 8, 'E': 10, 'F': 12}

Nodes that can't be reached from the source simply don't appear in dist.

Rebuilding the path

dist gives the length of each shortest path; prev lets us rebuild the path itself by walking backwards from the target:

def path_to(prev, source, target):
    path = [target]
    while path[-1] != source:
        path.append(prev[path[-1]])       # KeyError means target is unreachable
    return path[::-1]

>>> path_to(prev, "A", "F")
['A', 'C', 'B', 'D', 'E', 'F']

The prev pointers form a shortest-path tree rooted at the source: the highlighted edges in the diagram. Notice that the best route to B isn't the direct road (4) but the detour through C (2 + 1 = 3).

A step-by-step trace

Here's every heap operation for our graph. Watch for the stale entries: B, D, E and F each get pushed twice because a shorter route turned up after the first push.

Pop	Action	Heap afterwards
(0, A)	settle A; B = 4, C = 2	(2, C) (4, B)
(2, C)	settle C; B = 3, D = 10, E = 12	(3, B) (4, B) (10, D) (12, E)
(3, B)	settle B; D = 8	(4, B) (8, D) (10, D) (12, E)
(4, B)	stale, skip	(8, D) (10, D) (12, E)
(8, D)	settle D; E = 10, F = 14	(10, D) (10, E) (12, E) (14, F)
(10, D)	stale, skip	(10, E) (12, E) (14, F)
(10, E)	settle E; F = 12	(12, E) (12, F) (14, F)
(12, E)	stale, skip	(12, F) (14, F)
(12, F)	settle F	(14, F)
(14, F)	stale, skip	empty

Why it's correct

The key claim is: when a node is popped with its current distance d, no shorter path to it exists.

Suppose a shorter path did exist. It starts at the source (already settled) and ends at our node, so somewhere along it there's a first node that isn't settled yet; call it u. The part of the path before u runs through settled nodes, so u was relaxed with that distance and sits in the heap with a key no larger than the length of the path up to u. Since every edge weight is non-negative, the remaining edges can only add to that, so u's key is at most the shorter path's length, which is less than d. But then the heap would have given us u before our node. Contradiction.

That argument leans on exactly one assumption: non-negative weights. Take it away and the whole thing falls apart.

Negative edges break it

graph = {"A": {"B": 2, "C": 3}, "B": {"D": 2}, "C": {"B": -2}, "D": {}}

The true distance to B is 1 (A → C → B), and to D it's 3. But Dijkstra pops B (distance 2) before C (distance 3), so it considers B finished before the cheaper route through C has been seen. Textbook versions that mark nodes as "done" return B = 2 and D = 4, which is wrong.

Our lazy version happens to fix itself here, because it re-pushes B when the shorter distance shows up and then processes B again. Don't rely on that: with negative edges a node can be re-processed many times, and on adversarial graphs the running time becomes exponential. If your graph can have negative weights, use the Bellman–Ford algorithm (O(V·E)), which also detects negative cycles, where no shortest path exists at all.

Testing against Bellman–Ford

Bellman–Ford is short and slow, which makes it a great reference implementation: relax every edge V − 1 times. Comparing dijkstra() with it on a few thousand small random graphs with non-negative weights is a quick way to catch bugs in your implementation.

Complexity

Every node is processed only once (later entries for it are skipped as stale), so each edge is relaxed at most once in each direction and causes at most one push. The heap therefore holds O(E) entries, and every push and pop costs O(log E). Since E is at most V², log E is at most 2 log V, which gives:

Time: O((V + E) log V) with a binary heap.
Space: O(V + E) for the graph, the distances and the heap.

For dense graphs, where E is close to V², a simpler O(V²) version without a heap is actually faster: keep an array of distances and scan it for the minimum each time. Theoretical structures like Fibonacci heaps reach O(E + V log V), but their overhead rarely pays off in practice.

Practical details

Stop early. If you only need the path to one target, return as soon as you pop it. Its distance is final at that moment, and on a big graph you can skip most of the work:

        d, node = heapq.heappop(heap)
        if node == target:
            return d

Ties in the heap. Heap entries are tuples, so when two distances are equal Python compares the nodes themselves. That's fine for strings and numbers, but node objects that don't support < raise TypeError. Add a counter as a tiebreaker:

from itertools import count

tiebreak = count()
heapq.heappush(heap, (new_dist, next(tiebreak), neighbor))

Floating-point weights. With float weights, two routes of "equal" length can differ by rounding errors. Compare with a tolerance, or scale to integers (centimetres, milliseconds), when the exact path matters.

Where to go from here

A* is Dijkstra plus a heuristic: it orders the heap by distance + estimate_to_target. With a good estimate (like straight-line distance on a map) it explores far fewer nodes while still finding the shortest path, as long as the estimate never overestimates.
Network routing uses Dijkstra directly: OSPF routers compute shortest paths over the network map with it.
Road networks at continental scale use preprocessing techniques such as contraction hierarchies, with Dijkstra-style searches at their core.

Recap

Dijkstra finds shortest paths from one source in graphs with non-negative edge weights.
Always expand the closest unfinished node; a min-heap (heapq) finds it in O(log V).
Without decrease-key, push duplicates and skip stale entries (if d > dist[node]: continue).
Keep prev pointers to rebuild paths; they form a shortest-path tree.
Runs in O((V + E) log V); use Bellman–Ford for negative edges and an O(V²) array scan for very dense graphs.

CSS Stacking Contexts: Why z-index: 9999 Doesn't Work

Thu, 29 Oct 2026 09:00:00 +0000

It's one of the most common CSS bugs there is. The modal has z-index: 9999. The site header has z-index: 10. And the header is drawn on top of the modal. You try 99999, then !important; nothing changes.

The browser isn't broken. z-index is not a global ranking of every element on the page. It only orders elements within the same stacking context, and your modal is trapped inside a context that, as a whole, sits below the header. This article explains how the browser decides what's on top, what creates those contexts (often by accident), and how to find and fix the one causing your bug. It assumes you're comfortable with the cascade (The CSS Cascade and Specificity) and with positioning.

The bug

<header class="site-header">…</header>
<main class="page">
  <div class="modal">…</div>
</main>

.site-header { position: sticky; top: 0; z-index: 10; }
.page        { position: relative; z-index: 1; }
.modal       { position: fixed; inset: 0; z-index: 9999; }

The modal is drawn underneath the header. To see why, we need two ideas: the painting order inside one stacking context, and how contexts nest.

Painting order inside a stacking context

A stacking context is a group of elements that the browser paints together, as a unit. Inside one, it paints in this order, from bottom to top:

The background and borders of the element that forms the context
Positioned descendants with a negative z-index (most negative first)
Normal block boxes, in document order
Floats
Inline content: text, images, inline-blocks
Positioned descendants with z-index: auto or 0, in document order
Positioned descendants with a positive z-index, lowest first

Two consequences surprise people. First, positioned elements paint above non-positioned ones even without any z-index: adding position: relative alone can bring an element to the front. Second, z-index only has an effect on positioned elements (relative, absolute, fixed, sticky) and on flex and grid items. On a plain static block it's ignored.

Contexts nest like version numbers

Here's the crucial rule: a stacking context is painted as a single layer inside its parent context, at the position given by its own z-index. Everything inside it is drawn within that layer, so a child can never escape its parent's position, however high its own z-index is.

A good mental model is version numbers. In our example, the root context contains two layers, the header (10) and .page (1). The modal's 9999 only matters inside .page, so its effective position is something like 1.9999:

The root stacking context contains the header at z-index 10 and .page at z-index 1. The modal's z-index 9999 only applies inside .page, so its effective order is 1.9999, which is below 10

The browser first compares 10 with 1 at the root level, and the header wins. The 9999 is never compared with the header at all. That's the whole bug.

What creates a stacking context

This is the list that matters, because most stacking contexts are created by accident, as a side effect of a property added for a different reason:

Property	Creates a context when…
`position: relative` / `absolute`	`z-index` is anything other than `auto`
`position: fixed` / `sticky`	always
flex or grid item	`z-index` is anything other than `auto`
`opacity`	less than `1`
`transform`, `scale`, `rotate`, `translate`	anything other than `none`
`filter`, `backdrop-filter`	anything other than `none`
`clip-path`, `mask`, `perspective`	anything other than `none`
`mix-blend-mode`	anything other than `normal`
`isolation`	`isolate`
`will-change`	names one of the properties above
`contain`	`layout`, `paint`, `strict` or `content`
`container-type`	`size` or `inline-size`

Plus the root element (<html>), which is always one.

So a fade-in animation (opacity: 0.99 during the transition), a transform: translateZ(0) "performance hack", or a will-change: transform on a wrapper can each trap every z-index inside it. Classic real-world causes:

An animated page wrapper with transform, which traps every modal and dropdown inside it.
A sticky header in one part of the page and a dropdown inside a different sticky section.
A card with opacity: 0.95 whose tooltip slides under the next card.

transform also breaks position: fixed

An ancestor with transform, filter, perspective or will-change: transform doesn't just create a stacking context: it also becomes the containing block for position: fixed descendants. Your "fixed" modal then positions itself relative to that ancestor instead of the viewport, and scrolls with it. If a fixed element is in the wrong place and in the wrong layer, look for a transform.

Finding the context that traps you

DevTools don't show stacking contexts directly, but you can find them with a few lines of JavaScript. Select the misbehaving element in the Elements panel, then paste this into the console ($0 is the selected element):

function stackingContexts(el) {
  const found = [];
  for (let node = el.parentElement; node; node = node.parentElement) {
    const s = getComputedStyle(node);
    const parent = node.parentElement && getComputedStyle(node.parentElement);
    const reasons = [];
    const flexOrGridItem = /flex|grid/.test(parent?.display ?? "");
    if (s.zIndex !== "auto" && (s.position !== "static" || flexOrGridItem))
      reasons.push(`z-index: ${s.zIndex}`);
    if (s.position === "fixed" || s.position === "sticky") reasons.push(`position: ${s.position}`);
    if (s.opacity !== "1") reasons.push(`opacity: ${s.opacity}`);
    for (const prop of ["transform", "scale", "rotate", "translate", "filter", "backdropFilter", "perspective", "clipPath"])
      if (s[prop] && s[prop] !== "none") reasons.push(prop);
    if (s.mixBlendMode !== "normal") reasons.push(`mix-blend-mode: ${s.mixBlendMode}`);
    if (s.isolation === "isolate") reasons.push("isolation: isolate");
    if (/transform|opacity|filter/.test(s.willChange)) reasons.push(`will-change: ${s.willChange}`);
    if (reasons.length) found.push({ element: node, reasons: reasons.join(", ") });
  }
  return found;
}

console.table(stackingContexts($0));

It walks up from the element and lists every ancestor that forms a stacking context, with the reason. It covers the common causes rather than every rule in the table. The first row is the context your z-index is trapped in; compare the z-index of the element you're fighting with against that ancestor's, not against your own.

Fixing it

Once you know which context traps the element, pick the fix that matches the cause:

1. Remove the accidental context. If .page doesn't actually need z-index: 1 (or that transform, or will-change), delete it. This is the best fix when the property was added "just in case".

2. Move the element out. Overlays like modals, toasts and dropdown menus are best rendered as direct children of <body>, outside every component's context. Most UI frameworks have a tool for it: React's createPortal, Vue's <Teleport>, Svelte's portal libraries.

3. Use the top layer. Modern browsers have a top layer that sits above the entire document, outside all stacking contexts. Elements get there through the platform's own APIs, so no z-index is needed at all:

<dialog id="confirm">
  <p>Delete this file?</p>
  <form method="dialog"><button>Cancel</button> <button value="ok">Delete</button></form>
</dialog>

<script>
  document.getElementById("confirm").showModal();   // rendered in the top layer
</script>

<dialog> opened with showModal(), elements with the popover attribute, and fullscreen elements all render in the top layer, wherever they are in the DOM. showModal() also adds a ::backdrop, traps focus and closes on Esc, which are things a hand-made modal usually gets wrong.

4. Raise the whole context. If neither of the above works, change the z-index of the context root (.page), not the element inside it. Remember this moves everything inside .page along with it.

isolation: isolate, the useful kind of context

Sometimes you want a stacking context. A common example is a decorative layer behind a card's content:

.card { position: relative; background: white; }
.card::before {
  content: "";
  position: absolute;
  inset: -4px;
  z-index: -1;                 /* behind the card's content... */
  background: linear-gradient(135deg, violet, orange);
  border-radius: inherit;
}

Without a stacking context on .card, the ::before with z-index: -1 is painted in whatever context the card lives in, at step 2 of the painting order: below the card's own white background. The glow is hidden behind the card it was meant to frame, and could even slide behind the backgrounds of the card's ancestors.

Give the card its own context and the problem disappears:

.card { position: relative; background: white; isolation: isolate; }

Now the ::before is painted inside the card's context: above the card's background, below its content. isolation: isolate exists for exactly this. It creates a context and does nothing else: no positioning, no z-index value to manage, no side effects.

A z-index scale for the rest

For elements that really do share a context, avoid an arms race of ever-bigger numbers. Define a small scale once, for example --z-dropdown: 10; --z-sticky: 20; --z-overlay: 30; --z-toast: 40;, and only use those values. If something needs to go between two levels, that's a design decision, not a number to make up.

Recap

z-index only orders elements inside the same stacking context; contexts nest, so think in version numbers like 1.9999 < 10.
Within a context the painting order is: background, negative z-index, blocks, floats, inline content, z-index: auto/0, positive z-index.
z-index only works on positioned elements and flex/grid items.
Many properties create contexts as a side effect: opacity < 1, transform, filter, will-change, fixed/sticky positioning and more.
Fix overlap bugs by removing accidental contexts, moving overlays to <body>, or using the top layer (<dialog>, popover). Use isolation: isolate when you want a context on purpose.

Binary Search Without Off-by-One Bugs

Thu, 29 Oct 2026 09:00:00 +0000

Binary search fits in one sentence: look at the middle item, then throw away the half that can't contain what you're looking for. Each step halves the remaining range, so even a billion sorted items need only about 30 steps. That's O(log n); if Big-O is new to you, start with Big-O Notation Explained with Real Code.

Writing it correctly is another story. In Programming Pearls, Jon Bentley reports that when he asked professional programmers to write binary search, about 90% of them had bugs. And the version in Java's standard library had an overflow bug that went unnoticed for about nine years. The usual culprits are off-by-one errors: < or <=? mid or mid + 1? When does the loop stop?

This article gives you one template that answers those questions for you, and shows how far it stretches.

The question to ask

"Is x in the list?" is the weakest form of binary search. A much more useful question is:

Where is the first item that is greater than or equal to x?

This is usually called lower bound. It answers several questions at once: whether x is present, where it starts if it appears several times, and where you'd insert it to keep the list sorted.

One template: a half-open range

def lower_bound(a, x):
    """Index of the first item >= x, or len(a) if every item is smaller."""
    lo, hi = 0, len(a)            # invariant: the answer is between lo and hi (inclusive)
    while lo < hi:
        mid = (lo + hi) // 2
        if a[mid] < x:
            lo = mid + 1          # a[mid] is too small, so the answer is to its right
        else:
            hi = mid              # a[mid] could be the answer: keep it in range
    return lo

Every line follows from the invariant in the comment:

hi starts at len(a), not len(a) - 1, because "no item is big enough" is a valid answer.
If a[mid] < x, then mid and everything before it are too small, so the answer is at least mid + 1.
Otherwise a[mid] >= x, so mid might be the answer. We can't skip it, so hi = mid, not mid - 1.
The range shrinks on every pass (mid is always less than hi), so the loop ends. When lo == hi, there's only one candidate left, and that's the answer.

Let's trace a search for 23:

lower_bound searching for 23: the range starts as the whole list, then shrinks to indices 0–5, 3–5, 3–4, and finally index 4, the first 23

a = [3, 8, 12, 15, 23, 23, 31, 42, 56, 70]
lower_bound(a, 23)   # 4  (the first 23)
lower_bound(a, 20)   # 4  (20 would be inserted before the first 23)
lower_bound(a, 99)   # 10 (bigger than everything: len(a))
lower_bound(a, 1)    # 0

Checking membership is now a one-liner, with no special cases inside the loop:

def contains(a, x):
    i = lower_bound(a, x)
    return i < len(a) and a[i] == x

Upper bound and counting

Change < to <= and you get upper bound: the first item strictly greater than x.

def upper_bound(a, x):
    lo, hi = 0, len(a)
    while lo < hi:
        mid = (lo + hi) // 2
        if a[mid] <= x:
            lo = mid + 1
        else:
            hi = mid
    return lo

Together they give you the whole block of equal items, so counting duplicates in a sorted list takes O(log n), not O(n):

upper_bound(a, 23) - lower_bound(a, 23)   # 2
a[lower_bound(a, 23):upper_bound(a, 23)]  # [23, 23]

Python already has these

The bisect module is exactly this: bisect.bisect_left(a, x) is lower_bound and bisect.bisect_right(a, x) is upper_bound. bisect.insort(a, x) inserts while keeping the list sorted. Since Python 3.10 they accept key=, e.g. bisect_left(users, 35, key=lambda u: u.age). Use them in real code; write your own when you need the general version below.

The general version: first True

Look at the two functions again. All that changes is a test that is False for a while and then True from some point on:

index:      0      1      2      3      4     5     6     ...
a[i] >= 23: False  False  False  False  True  True  True  ...
                                        ↑ the answer

So binary search really finds the boundary of a yes/no question, as long as the answers never flip back from True to False. That's worth its own function:

def first_true(lo, hi, pred):
    """Smallest i in [lo, hi) where pred(i) is True, or hi if there is none.
    pred must be False, False, ..., True, True over that range."""
    while lo < hi:
        mid = (lo + hi) // 2
        if pred(mid):
            hi = mid
        else:
            lo = mid + 1
    return lo

lower_bound(a, x) is just first_true(0, len(a), lambda i: a[i] >= x). But the predicate doesn't need an array at all.

Binary search on the answer

The integer square root of n is the largest r with r * r <= n. In other words, it's one less than the first r where r * r > n:

def isqrt(n):
    return first_true(0, n + 1, lambda r: r * r > n) - 1

isqrt(50)          # 7
isqrt(10**12)      # 1000000, in about 40 steps

This trick solves a whole family of "find the minimum value that works" problems. Say a truck must deliver packages in order, the truck can carry capacity kilograms per day, and everything has to be delivered within days days. What's the smallest capacity that works?

def days_needed(weights, capacity):
    days, load = 1, 0
    for w in weights:
        if load + w > capacity:      # doesn't fit: start a new day
            days, load = days + 1, 0
        load += w
    return days

def min_capacity(weights, days):
    lo, hi = max(weights), sum(weights)
    return first_true(lo, hi, lambda c: days_needed(weights, c) <= days)

min_capacity([3, 2, 2, 4, 1, 4], days=3)   # 6: [3, 2] [2, 4] [1, 4]

Finding the capacity directly is hard, but checking one capacity is easy, and the answer is monotonic: if capacity 6 works, 7 works too. Binary search turns an easy check into a fast search: O(n log S) instead of trying every capacity up to S = sum(weights).

Some boundary details, all following from the template: the capacity can't be below the heaviest package, so lo = max(weights). We pass hi = sum(weights) even though first_true never tests hi itself: if no smaller capacity works, it returns hi, which is correct because carrying everything in one day always works.

You already know one: git bisect

git bisect is this exact algorithm over commits: the question is "is the bug present in this commit?", and the answers go from no to yes at exactly one point. A thousand commits need only about 10 tests.

The bugs to watch for

Bug	What goes wrong	Fix
`hi = len(a) - 1` with `while lo < hi`	`len(a)` can never be returned, and the last item is never checked	Pick one convention (we use half-open `[lo, hi)`) and stick to it
`lo = mid` instead of `mid + 1`	When `hi = lo + 1`, `mid == lo` forever: an infinite loop	Each branch must shrink the range
`hi = mid - 1`	Can skip over the answer when `a[mid]` is the answer	Only exclude `mid` when it definitely isn't the answer
`(lo + hi) / 2` in Java, C, Go…	`lo + hi` can overflow a 32-bit int on huge arrays	Use `lo + (hi - lo) / 2` (Python's integers can't overflow)
Searching unsorted data	Returns a plausible but wrong answer, silently	Sort first, or check the predicate really is monotonic

Test it against the slow version

Binary search is the perfect candidate for a brute-force comparison test: a linear scan is obviously correct, so compare against it on thousands of random inputs, including empty lists and duplicates:

import random

for _ in range(10_000):
    a = sorted(random.randint(0, 20) for _ in range(random.randint(0, 15)))
    x = random.randint(-2, 22)
    expected = next((i for i, v in enumerate(a) if v >= x), len(a))
    assert lower_bound(a, x) == expected, (a, x)

Small values (0–20) force lots of duplicates and edge cases. This kind of test catches every bug in the table above within a fraction of a second.

Recap

Ask "where is the first item that is ≥ x?" (lower bound) rather than "is x here?".
Use a half-open range [lo, hi) with while lo < hi, lo = mid + 1 and hi = mid. Every step shrinks the range, and lo is the answer at the end.
Binary search finds the boundary of any monotonic yes/no question, so you can search over answers (capacities, sizes, times, commits), not just arrays.
In Python, reach for bisect first.
Test against a brute-force version with small random inputs.

Python List Comprehensions: From Loops to Elegant One-Liners

Mon, 26 Oct 2026 09:00:00 +0000

If you've read other people's Python code, you've almost certainly seen something like this:

squares = [n * n for n in range(10)]

That's a list comprehension: a compact way to build a new list from an existing iterable. Once it clicks, you'll use it constantly. In this guide we'll build up from the plain for loop you already know, cover filtering and transformation, look at dictionary and set comprehensions, and finish with the cases where you shouldn't use one.

Start with the loop you already know

Here's the classic way to build a list of squares:

squares = []
for n in range(10):
    squares.append(n * n)

print(squares)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Three lines, three jobs: create an empty list, loop, append. A comprehension expresses the same idea in one line:

squares = [n * n for n in range(10)]

Read it left to right as "give me n * n for each n in range(10)." The expression you want to collect comes first, followed by the loop that produces the values.

The anatomy of a comprehension

Every list comprehension follows this shape:

[expression for item in iterable if condition]

The same three parts in a real comprehension that squares the even numbers below 10

Part	Meaning	Required?
`expression`	What to put in the new list	Yes
`for item in iterable`	Where the values come from	Yes
`if condition`	Keep only items that pass the test	No

Filtering with `if`

Add a condition at the end to keep only some items. Here we keep the even numbers:

evens = [n for n in range(20) if n % 2 == 0]
# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

You can transform and filter at the same time. This collects the lengths of words longer than three characters:

words = ["git", "python", "sql", "linux", "css", "docker"]
long_word_lengths = [len(w) for w in words if len(w) > 3]
# [6, 5, 6]

Choosing a value with `if ... else`

What if you want to keep every item but change some of them? Put an if ... else before the for. This is Python's conditional expression, not a filter:

scores = [92, 47, 78, 30, 85]
results = ["pass" if s >= 50 else "fail" for s in scores]
# ['pass', 'fail', 'pass', 'fail', 'pass']

Remember the position rule

An if after the for filters items out. An if ... else before the for chooses what each item becomes. The first can shrink your list; the second never does.

Real-world examples

Comprehensions shine when you're cleaning or reshaping data.

Normalising user input:

raw_emails = ["  Ada@Example.com", "grace@example.com  ", "", "LINUS@example.com"]
emails = [e.strip().lower() for e in raw_emails if e.strip()]
# ['ada@example.com', 'grace@example.com', 'linus@example.com']

Extracting a field from a list of dictionaries:

users = [
    {"name": "Ada", "active": True},
    {"name": "Grace", "active": False},
    {"name": "Linus", "active": True},
]
active_names = [u["name"] for u in users if u["active"]]
# ['Ada', 'Linus']

Flattening a nested list. The loops appear in the same order as they would in nested for statements:

matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [value for row in matrix for value in row]
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

Dictionary and set comprehensions

The same syntax works with curly braces. Use key: value to build a dictionary:

words = ["git", "python", "sql"]
lengths = {w: len(w) for w in words}
# {'git': 3, 'python': 6, 'sql': 3}

Drop the colon and you get a set, which removes duplicates automatically:

unique_lengths = {len(w) for w in words}
# {3, 6}

Generator expressions: comprehensions without the list

Swap the square brackets for parentheses and you get a generator expression. It produces values one at a time instead of building the whole list in memory, which is perfect when you only need to consume the values once:

total = sum(n * n for n in range(1_000_000))

When a generator expression is the only argument to a function, you can drop the extra parentheses, as above. Functions like sum(), any(), all(), max() and min() pair beautifully with them:

has_admin = any(u.get("role") == "admin" for u in users)

When not to use a comprehension

Comprehensions exist to make code more readable. If one stops being readable, use a loop. Watch for these warning signs:

It doesn't fit comfortably on one or two lines. Nested comprehensions with several conditions are hard to scan.
You need side effects. Don't write [print(x) for x in items]. It builds a throwaway list of None values. Use a for loop.
You need try/except or several statements per item. A comprehension holds a single expression; a loop or a helper function is clearer.

# Hard to read: avoid
result = [x * y for x in range(10) if x % 2 for y in range(x) if y > 2 and x + y < 12]

# Clearer: give the logic a name
def odd_pair_products(limit):
    for x in range(1, limit, 2):
        for y in range(3, x):
            if x + y < 12:
                yield x * y

result = list(odd_pair_products(10))

Recap

[expr for x in items] transforms every item.
[expr for x in items if cond] filters as well.
[a if cond else b for x in items] chooses a value for each item.
Use {k: v ...} for dicts, {x ...} for sets and (x ...) for lazy generators.
If it's getting clever, a loop is usually kinder to the next reader, who might be you.

Practice: open a script you wrote recently and look for an "empty list, loop, append" pattern. Rewrite it as a comprehension, then decide honestly which version reads better.

Understanding async/await in JavaScript

Wed, 21 Oct 2026 09:00:00 +0000

JavaScript runs your code on a single thread, yet it can download files, wait for timers and talk to databases without freezing. The secret is asynchronous code, and async/await is the most readable way to write it.

In this article you'll learn what a promise actually is, how async/await builds on it, how to handle errors properly, and how to avoid the most common performance mistake developers make.

The problem: waiting without blocking

Imagine fetching a user from an API. The network might take 300 ms. If JavaScript simply stopped and waited, your whole page would freeze: no clicks, no scrolling, no animations.

Instead, asynchronous functions return immediately with a promise, an object that stands for a value that will be available later.

Promises in 60 seconds

A promise is always in one of three states:

pending: the work is still in progress
fulfilled: it finished successfully with a value
rejected: it failed with an error

You react to a promise with .then() and .catch():

fetch("https://api.github.com/users/octocat")
  .then((response) => response.json())
  .then((user) => console.log(user.name))
  .catch((error) => console.error("Request failed:", error));

This works, but chains get hard to follow as the logic grows, especially when one step depends on the result of another.

Enter async/await

async/await is syntax built on top of promises. It lets you write asynchronous code that reads like synchronous code:

async function showUser(username) {
  const response = await fetch(`https://api.github.com/users/${username}`);
  const user = await response.json();
  console.log(user.name);
}

showUser("octocat");

Two rules make this work:

async before a function makes it always return a promise.
await pauses that function until the promise settles, then hands you its value. Other code keeps running in the meantime. Only this function waits.

await doesn't block the page

While showUser is paused at an await, the browser is free to handle clicks, run other functions and repaint the screen. The function resumes when the promise settles.

Handling errors with try/catch

With .then() chains you use .catch(). With async/await you use the familiar try/catch:

async function getUser(username) {
  try {
    const response = await fetch(`https://api.github.com/users/${username}`);

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    return await response.json();
  } catch (error) {
    console.error(`Could not load ${username}:`, error.message);
    return null;
  }
}

fetch only rejects on network failure

A 404 or 500 response does not reject the promise returned by fetch(). Always check response.ok (or response.status) yourself, as shown above.

The #1 mistake: accidentally sequential code

Look at this function. Can you spot the problem?

async function loadDashboard() {
  const user = await getUser("octocat");     // ~300 ms
  const repos = await getRepos("octocat");   // ~300 ms
  const events = await getEvents("octocat"); // ~300 ms
  return { user, repos, events };            // total: ~900 ms
}

Each request waits for the previous one to finish, even though none of them depend on each other! Start them all at once and wait for them together with Promise.all():

async function loadDashboard() {
  const [user, repos, events] = await Promise.all([
    getUser("octocat"),
    getRepos("octocat"),
    getEvents("octocat"),
  ]);
  return { user, repos, events };            // total: ~300 ms
}

Same result, roughly three times faster.

Awaiting independent requests one by one adds their times together; Promise.all runs them at the same time

Choosing the right Promise combinator

Method	Fulfils when…	Rejects when…
`Promise.all()`	all promises fulfil	any promise rejects
`Promise.allSettled()`	all promises settle	never
`Promise.race()`	the first promise to settle fulfils	the first promise to settle rejects
`Promise.any()`	the first promise fulfils	all promises reject

Use Promise.allSettled() when you want every result even if some fail, for example when loading several optional widgets:

const results = await Promise.allSettled([getRepos("octocat"), getEvents("octocat")]);

for (const result of results) {
  if (result.status === "fulfilled") {
    render(result.value);
  } else {
    console.warn("Widget failed:", result.reason);
  }
}

Adding a timeout

fetch() has no timeout by default. In modern browsers and Node.js 18+, AbortSignal.timeout() gives you one in a single line:

async function fetchWithTimeout(url, ms = 5000) {
  const response = await fetch(url, { signal: AbortSignal.timeout(ms) });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  return response.json();
}

If the request takes longer than ms milliseconds, the promise rejects with a TimeoutError, which you can handle in a try/catch like any other error.

Loops and await

Using await inside a for...of loop processes items one at a time. That's exactly what you want when order matters or when you must avoid hammering an API:

for (const id of orderIds) {
  await processOrder(id); // waits for each one before starting the next
}

When the tasks are independent, map them to promises and wait for them together instead:

await Promise.all(orderIds.map((id) => processOrder(id)));

forEach doesn't wait

orderIds.forEach(async (id) => await processOrder(id)) does not wait for anything, because forEach ignores the promises your callback returns. Use for...of or Promise.all() instead.

Recap

A promise represents a future value: pending, fulfilled or rejected.
async functions always return a promise; await pauses only that function.
Use try/catch for errors, and remember to check response.ok with fetch().
Run independent tasks in parallel with Promise.all() instead of awaiting them one by one.
Never rely on forEach to wait for async callbacks.

Next time you write an await, ask yourself: does this line really need to wait for the one above it? That one question will make your apps noticeably faster.

A Practical Git Branching Workflow for Beginners

Wed, 14 Oct 2026 09:00:00 +0000

When you first learn Git, it's natural to commit everything straight to main. That works until you need to fix a bug halfway through a feature, or a teammate pushes changes that collide with yours.

Branches solve this. In this guide you'll learn a simple workflow used by teams of every size: create a branch for each piece of work, commit freely, then merge it back when it's ready.

What a branch really is

A branch in Git is just a movable pointer to a commit. Creating one is instant and costs almost nothing, because Git doesn't copy your files. When you make a new commit, the branch you're on moves forward to point at it.

A feature branch splits from main, gets its own commits, then merges back

Here, feature/login branches off main after commit C and gets two commits of its own, D and E. Work on one branch doesn't touch the other until you merge, which creates the merge commit M.

The workflow at a glance

Update your local main.
Create a feature branch.
Commit your work in small steps.
Push the branch and open a pull request.
Keep the branch up to date with main.
Merge, then delete the branch.

Let's walk through each step.

Step 1: Start from an up-to-date main

git switch main
git pull origin main

git switch (Git 2.23+) is the modern, clearer alternative to git checkout for changing branches.

Step 2: Create a feature branch

git switch -c feature/login-form

The -c flag creates the branch and switches to it in one go. Use descriptive, prefixed names so everyone can tell what a branch is for:

Prefix	Use it for	Example
`feature/`	New functionality	`feature/login-form`
`fix/`	Bug fixes	`fix/password-reset-email`
`docs/`	Documentation	`docs/api-examples`
`refactor/`	Code improvements with no behaviour change	`refactor/user-service`

Step 3: Commit in small, meaningful steps

Make a change, review it, then commit:

git status              # what changed?
git diff                # review the exact changes
git add src/login.js    # stage specific files
git commit -m "Add login form markup and validation"

Write commit messages for your future self

Use the imperative mood ("Add validation", not "Added validation") and describe what changed and why, not how. A good test: your message should complete the sentence "If applied, this commit will…"

Step 4: Push and open a pull request

The first time you push a new branch, tell Git which remote branch it should track:

git push -u origin feature/login-form

-u sets the upstream, so afterwards a plain git push is enough. On GitHub, GitLab or Bitbucket you'll now see a prompt to open a pull request (called a merge request on GitLab). This is where teammates review your code, leave comments and run automated tests before anything reaches main.

Step 5: Keep your branch up to date

If main moves on while you work, bring those changes into your branch so you resolve conflicts early and in small pieces:

git switch main
git pull origin main
git switch feature/login-form
git merge main

Some teams prefer git rebase main instead of git merge main. It replays your commits on top of the latest main for a straight-line history. Both are valid, so follow your team's convention.

Never rebase shared history

Rebasing rewrites commits. Only rebase branches that nobody else has based work on. Otherwise you'll create painful conflicts for your teammates.

Resolving a merge conflict

A conflict happens when two branches change the same lines. Git pauses and marks the file:

<<<<<<< HEAD
const MAX_ATTEMPTS = 5;
=======
const MAX_ATTEMPTS = 3;
>>>>>>> main

The HEAD section is your branch's version and the bottom section comes from main. Edit the file to keep the correct version, delete the markers, then finish the merge:

git add src/config.js
git commit

Step 6: Merge and clean up

Once the pull request is approved, merge it through your hosting platform. Then tidy up locally:

git switch main
git pull origin main
git branch -d feature/login-form

git branch -d refuses to delete a branch that hasn't been merged, so it's a safe default. If your team uses squash merges, Git may not recognise the branch as merged. Once you've confirmed the pull request was merged, git branch -D forces the deletion.

Handy commands to remember

git branch                        # list local branches
git branch -a                     # include remote branches
git log --oneline --graph --all   # visualise history
git switch -                      # jump back to the previous branch
git stash                         # shelve uncommitted changes temporarily
git stash pop                     # bring them back

Recap

A branch is a lightweight pointer, so create them freely.
One branch per feature or fix keeps main always deployable.
Commit in small steps, write clear messages and open a pull request for review.
Merge main into your branch regularly to avoid giant conflicts.
Delete branches after merging to keep your repository tidy.

Adopt this workflow even on solo projects. It builds habits that make joining a team effortless.

Big-O Notation Explained with Real Code

Wed, 07 Oct 2026 09:00:00 +0000

Your code works on 100 items. Will it still work on 1,000,000? Big-O notation answers that question. It describes how the running time (or memory use) of an algorithm grows as its input gets bigger.

You don't need advanced maths to use it. In this article we'll look at real functions, count the work they do, and build an intuition you can apply to your own code.

The core idea: growth, not speed

Big-O doesn't tell you that a function takes 3 milliseconds. It tells you how the time scales:

Double the input of an O(n) function and the time roughly doubles.
Double the input of an O(n²) function and the time roughly quadruples.

We also drop constants and smaller terms. A function that does 3n + 10 steps is simply O(n), because as n gets large the 3 and the 10 stop mattering compared to the growth itself.

O(1): constant time

The work doesn't depend on the input size at all:

def get_first(items):
    return items[0]

def is_even(n):
    return n % 2 == 0

Indexing a list, looking up a key in a dictionary and appending to a list (on average) are all O(1).

O(n): linear time

The work grows in direct proportion to the input. Looping over every item once is the classic example:

def find_max(numbers):
    largest = numbers[0]
    for n in numbers:          # runs n times
        if n > largest:
            largest = n
    return largest

Hidden O(n) operations are worth knowing too: x in some_list, some_list.index(x), sum() and max() all scan the whole list.

O(n²): quadratic time

A loop inside a loop, where both run over the input, gives you roughly n × n steps:

def has_duplicates(items):
    for i in range(len(items)):
        for j in range(i + 1, len(items)):   # nested loop
            if items[i] == items[j]:
                return True
    return False

For 1,000 items that's about half a million comparisons. For 1,000,000 items it's about 500 billion. That's the difference between instant and "come back tomorrow".

Fixing it: trade memory for speed

We can make has_duplicates O(n) by remembering what we've already seen in a set, whose membership checks are O(1) on average:

def has_duplicates(items):
    seen = set()
    for item in items:          # runs n times
        if item in seen:        # O(1) on average
            return True
        seen.add(item)          # O(1) on average
    return False

Or, more concisely:

def has_duplicates(items):
    return len(set(items)) != len(items)

We spend a little extra memory (O(n) space) to save an enormous amount of time. This trade-off is one of the most common optimisations in real-world code.

O(log n): logarithmic time

Logarithmic algorithms cut the problem in half at every step. Binary search on a sorted list is the textbook example:

def binary_search(sorted_items, target):
    low, high = 0, len(sorted_items) - 1
    while low <= high:
        mid = (low + high) // 2
        if sorted_items[mid] == target:
            return mid
        if sorted_items[mid] < target:
            low = mid + 1       # discard the left half
        else:
            high = mid - 1      # discard the right half
    return -1

Searching 1,000,000 sorted items takes at most about 20 steps, because 2²⁰ ≈ 1,000,000. Double the data and you add just one more step.

O(n log n): the sweet spot for sorting

Efficient general-purpose sorting algorithms run in O(n log n). That includes merge sort and Timsort, which powers Python's sorted() and list.sort(). It's only slightly worse than linear, and it's the best possible for sorting by comparing elements.

names = sorted(names)   # O(n log n)

Comparing growth rates

Approximate number of steps for different input sizes:

Complexity	n = 10	n = 1,000	n = 1,000,000
O(1)	1	1	1
O(log n)	3	10	20
O(n)	10	1,000	1,000,000
O(n log n)	33	10,000	20,000,000
O(n²)	100	1,000,000	1,000,000,000,000

The same growth rates as curves: the gap between them widens dramatically as n grows

A quick rule of thumb

A modern CPU running compiled code manages very roughly 10⁸ simple operations per second, and Python manages fewer. If your algorithm needs 10¹² steps, no hardware upgrade will save you. A better algorithm will.

How to analyse your own code

Find the loops. A loop over the input is usually O(n). Nested loops over the input multiply: O(n²).
Look inside function calls. item in my_list inside a loop is a hidden nested loop.
Keep the biggest term. O(n² + n) simplifies to O(n²).
Think about the worst case. Unless stated otherwise, Big-O usually describes the worst case.

Here's a sneaky example. What's its complexity?

def common_items(a, b):
    return [x for x in a if x in b]

If b is a list, x in b scans it every time, so this is O(n × m). Convert b to a set first and it drops to O(n + m):

def common_items(a, b):
    b_set = set(b)                          # O(m)
    return [x for x in a if x in b_set]     # O(n)

Recap

Big-O describes how work grows with input size, ignoring constants.
Common classes, fastest to slowest: O(1), O(log n), O(n), O(n log n), O(n²).
Nested loops and hidden scans (in on a list) are the usual culprits behind slow code.
Sets and dictionaries turn O(n) lookups into O(1), which is often the easiest big win.

Next time your code slows down as your data grows, don't reach for a faster server first. Reach for Big-O.

CSS Flexbox vs Grid: When to Use Which

Wed, 30 Sep 2026 09:00:00 +0000

Flexbox and CSS Grid are the two modern layout systems in CSS, and beginners often ask which one is "better". The honest answer is neither. They solve different problems, and professional layouts use both, often on the same page.

The one rule to remember

Flexbox lays things out in one dimension: a row or a column.

Grid lays things out in two dimensions: rows and columns at the same time.

Another way to put it: Flexbox is content-first. Items decide how much space they need, and the container distributes what's left. Grid is layout-first. You define the tracks, then place items into them.

Flexbox distributes items along a single axis; Grid aligns them on rows and columns at once

Flexbox basics

Turn any element into a flex container with display: flex. Its direct children become flex items arranged in a row:

.toolbar {
  display: flex;
  align-items: center;            /* alignment on the cross axis */
  justify-content: space-between; /* distribution on the main axis */
  gap: 12px;
}

The properties you'll use 90% of the time:

Property	What it does
`flex-direction`	`row` (default) or `column`
`justify-content`	Distributes space along the main axis
`align-items`	Aligns items on the cross axis
`gap`	Adds space between items
`flex-wrap`	Lets items wrap onto new lines
`flex: 1` (on an item)	Makes the item grow to fill available space

<nav class="navbar">
  <a class="logo" href="/">Brand</a>
  <ul class="links">
    <li><a href="/docs">Docs</a></li>
    <li><a href="/blog">Blog</a></li>
  </ul>
  <a class="button" href="/signup">Sign up</a>
</nav>

.navbar {
  display: flex;
  align-items: center;
  gap: 24px;
}

.links {
  display: flex;
  gap: 16px;
  margin: 0 auto 0 0;   /* the auto right margin pushes the button to the far right */
  padding: 0;
  list-style: none;
}

The auto-margin trick

In a flex container, margin-left: auto or margin-right: auto on an item absorbs all the free space on that side. It's the cleanest way to push one item away from the others.

Example: perfect centring

The famous "how do I centre a div?" question has a short answer with Flexbox:

.center {
  display: flex;
  justify-content: center;
  align-items: center;
}

With Grid, display: grid; place-items: center; does the same thing.

Grid basics

Grid shines when you need items aligned in both directions. Define columns (and optionally rows) on the container:

.gallery {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  gap: 20px;
}

The fr unit means "one fraction of the free space", so repeat(3, 1fr) creates three equal columns.

Example: a responsive card grid with no media queries

This one line is worth memorising:

.cards {
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
  gap: 24px;
}

It says "fit as many columns as you can, each at least 250px wide, and stretch them to fill the row." On a phone you get one column; on a wide monitor you might get four, all without a single media query.

Example: a full page layout

Grid's grid-template-areas lets you draw your layout as text:

.page {
  display: grid;
  grid-template-columns: 240px 1fr;
  grid-template-rows: auto 1fr auto;
  grid-template-areas:
    "header  header"
    "sidebar main"
    "footer  footer";
  min-height: 100vh;
}

.page > header { grid-area: header; }
.page > aside  { grid-area: sidebar; }
.page > main   { grid-area: main; }
.page > footer { grid-area: footer; }

@media (max-width: 700px) {
  .page {
    grid-template-columns: 1fr;
    grid-template-rows: auto 1fr auto auto;
    grid-template-areas:
      "header"
      "main"
      "sidebar"
      "footer";
  }
}

Notice how the mobile version simply rearranges the areas. The HTML doesn't change at all.

Side-by-side comparison

	Flexbox	Grid
Dimensions	One (row or column)	Two (rows and columns)
Approach	Content-first	Layout-first
Best for	Navbars, toolbars, button groups, centring	Page layouts, galleries, dashboards, forms
Sizing	Items size themselves, then flex	Tracks define sizes, items fill them
Overlapping items	Awkward	Easy: place items in the same cell

Using them together

Real interfaces nest the two. A typical pattern is Grid for the overall structure, Flexbox inside each component:

/* Grid arranges the cards */
.cards {
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
  gap: 24px;
}

/* Flexbox arranges the content inside each card */
.card {
  display: flex;
  flex-direction: column;
}

.card .footer {
  margin-top: auto;   /* pins the footer to the bottom of every card */
}

Grid stretches every card in a row to the same height, and the margin-top: auto trick keeps the footers aligned even when the cards contain different amounts of text. It's a small detail that makes designs look polished.

Quick decision guide

Ask yourself:

Am I arranging items in a single line? Use Flexbox.
Do I need items to line up in rows and columns? Use Grid.
Should the content decide the size? Use Flexbox.
Should the layout decide the size? Use Grid.

When in doubt, start with Flexbox for small components and Grid for anything that looks like a table, a gallery or a page skeleton.

Recap

Flexbox is one-dimensional and content-first, which makes it perfect for components.
Grid is two-dimensional and layout-first, which makes it perfect for page structure and galleries.
repeat(auto-fill, minmax(…, 1fr)) gives you responsive grids without media queries.
Combine them: Grid for the structure, Flexbox inside the pieces.

Open your browser's DevTools on any modern website and inspect the layout. You'll find both systems working together everywhere.

SQL JOINs Explained with Simple Examples

Wed, 23 Sep 2026 09:00:00 +0000

Relational databases split data into separate tables to avoid repetition: customers in one table, orders in another. JOINs are how you put that data back together when you query it.

The best way to understand joins is to watch them work on real rows. We'll use two tiny tables throughout this guide.

Our example tables

customers

id	name
1	Ada
2	Grace
3	Linus

orders

id	customer_id	total
101	1	49.00
102	1	15.50
103	2	99.99
104	4	25.00

Notice two things: Linus has no orders, and order 104 belongs to customer 4, who doesn't exist (perhaps the account was deleted). These edge cases are exactly where the different joins behave differently.

Want to follow along? Here's the setup SQL, which works in SQLite, PostgreSQL and MySQL:

CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT);
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, total DECIMAL(10, 2));

INSERT INTO customers VALUES (1, 'Ada'), (2, 'Grace'), (3, 'Linus');
INSERT INTO orders VALUES (101, 1, 49.00), (102, 1, 15.50), (103, 2, 99.99), (104, 4, 25.00);

Databases don't guarantee row order without ORDER BY, so the results below are sorted for readability.

INNER JOIN: only matching rows

An INNER JOIN returns rows where the join condition matches in both tables:

SELECT c.name, o.id AS order_id, o.total
FROM customers AS c
INNER JOIN orders AS o ON o.customer_id = c.id;

name	order_id	total
Ada	101	49.00
Ada	102	15.50
Grace	103	99.99

Linus disappears because he has no orders, and so does order 104 because it has no customer. Writing JOIN on its own means INNER JOIN.

LEFT JOIN: everything from the left table

A LEFT JOIN keeps every row from the left table (the one after FROM) and fills in NULL where there's no match on the right:

SELECT c.name, o.id AS order_id, o.total
FROM customers AS c
LEFT JOIN orders AS o ON o.customer_id = c.id;

name	order_id	total
Ada	101	49.00
Ada	102	15.50
Grace	103	99.99
Linus	NULL	NULL

Linus is back, with NULLs in the order columns. After INNER JOIN, this is the join you'll use most often.

Finding rows with no match

A classic trick is to combine LEFT JOIN with WHERE ... IS NULL to find "orphans". Here we find customers who have never ordered:

SELECT c.name
FROM customers AS c
LEFT JOIN orders AS o ON o.customer_id = c.id
WHERE o.id IS NULL;

Result: Linus.

RIGHT JOIN: everything from the right table

A RIGHT JOIN is the mirror image: it keeps every row from the right table.

SELECT c.name, o.id AS order_id, o.total
FROM customers AS c
RIGHT JOIN orders AS o ON o.customer_id = c.id;

name	order_id	total
Ada	101	49.00
Ada	102	15.50
Grace	103	99.99
NULL	104	25.00

Now the orphaned order 104 appears with no customer name. In practice, most developers rewrite right joins as left joins by swapping the table order. A query is easier to read when the "keep everything" table always comes first.

FULL OUTER JOIN: everything from both

A FULL OUTER JOIN keeps all rows from both tables, matching them where it can:

SELECT c.name, o.id AS order_id, o.total
FROM customers AS c
FULL OUTER JOIN orders AS o ON o.customer_id = c.id;

name	order_id	total
Ada	101	49.00
Ada	102	15.50
Grace	103	99.99
Linus	NULL	NULL
NULL	104	25.00

It's handy for reconciling data: comparing two lists and spotting what's missing on either side.

Database support

MySQL doesn't support FULL OUTER JOIN; you can emulate it by combining a LEFT JOIN and a RIGHT JOIN with UNION. SQLite supports RIGHT and FULL joins from version 3.39.0 onwards.

Summary table

Which rows each join keeps, shown as Venn diagrams

Join	Keeps unmatched rows from…	Typical use
`INNER JOIN`	neither table	Data that must exist on both sides
`LEFT JOIN`	the left table	"All customers, plus their orders if any"
`RIGHT JOIN`	the right table	Rarely used; usually rewritten as a LEFT JOIN
`FULL OUTER JOIN`	both tables	Reconciling two datasets

Aggregating after a join

Joins become really powerful with GROUP BY. Here's each customer's order count and total spend, including customers with zero orders:

SELECT
  c.name,
  COUNT(o.id)               AS order_count,
  COALESCE(SUM(o.total), 0) AS total_spent
FROM customers AS c
LEFT JOIN orders AS o ON o.customer_id = c.id
GROUP BY c.id, c.name
ORDER BY total_spent DESC;

name	order_count	total_spent
Grace	1	99.99
Ada	2	64.50
Linus	0	0.00

COUNT(column) vs COUNT(*)

COUNT(o.id) counts only non-NULL values, so Linus correctly gets 0. COUNT(*) counts rows, and would give Linus 1 because the LEFT JOIN still produces one row for him.

Common mistakes

Filtering the right table in WHERE after a LEFT JOIN. A condition like WHERE o.total > 20 silently removes the NULL rows and turns your LEFT JOIN into an INNER JOIN. Move the condition into the ON clause instead: LEFT JOIN orders AS o ON o.customer_id = c.id AND o.total > 20.
Using the wrong join condition. Joining on the wrong columns, or with no real condition at all, can produce a Cartesian product: every row paired with every other row.
Joining on columns with different types. This can stop the database from using its indexes, which makes queries slow.

Recap

INNER JOIN returns only rows that match on both sides.
LEFT JOIN returns all rows from the left table, with NULLs where there's no match.
RIGHT JOIN is the mirror of LEFT and is usually rewritten as a LEFT JOIN.
FULL OUTER JOIN returns everything from both tables.
LEFT JOIN ... WHERE right.id IS NULL finds rows without a match.

Set up the two tables above in SQLite (sqlite3 shop.db) and run each query yourself. Five minutes of experimenting beats an hour of reading.

Essential Linux Commands Every Developer Should Know

Wed, 16 Sep 2026 09:00:00 +0000

Whether you're deploying to a cloud server, working inside Docker, or just using a terminal on your Mac, sooner or later you'll be typing Linux commands. The good news is that a few dozen commands cover the vast majority of daily work.

Each section below shows the commands, the options you'll actually use, and real-world examples.

Navigating the file system

pwd                 # where am I?
ls -lah             # list everything: long format, hidden files, human-readable sizes
cd /var/log         # go to an absolute path
cd ..               # up one level
cd -                # jump back to the previous directory
cd                  # go home (same as cd ~)

Tab completion saves hours

Type the first few letters of a file or directory name and press Tab. The shell completes it for you, or shows the options if there's more than one match.

Creating, copying, moving and deleting

mkdir -p projects/blog/src     # create nested directories in one go
touch notes.txt                # create an empty file (or update its timestamp)
cp config.yml config.yml.bak   # copy a file
cp -r src/ backup/             # copy a directory recursively
mv draft.md posts/final.md     # move and/or rename
rm old.log                     # delete a file
rm -r build/                   # delete a directory and everything in it

There is no recycle bin

rm deletes immediately and permanently. Double-check paths before running rm -r, especially with wildcards or sudo. Adding -i makes rm ask before each deletion.

Reading files

cat package.json        # print a whole (short) file
less /var/log/syslog    # scroll through a long file (q to quit, / to search)
head -n 20 app.log      # first 20 lines
tail -n 50 app.log      # last 50 lines
tail -f app.log         # follow new lines as they're written, great for live logs

Searching inside files with grep

grep finds lines that match a pattern. It's one of the most useful tools you'll ever learn:

grep "ERROR" app.log            # lines containing ERROR
grep -i "timeout" app.log       # case-insensitive
grep -rn "TODO" src/            # search recursively, show line numbers
grep -v "DEBUG" app.log         # lines that do NOT match
grep -c "404" access.log        # count matching lines

Finding files with find

find . -name "*.py"                       # all Python files below the current directory
find /var/log -name "*.log" -mtime -1     # .log files modified in the last 24 hours
find . -type f -size +100M                # files bigger than 100 MB
find . -type d -name node_modules         # directories named node_modules

Pipes and redirection: combining commands

The real power of the command line comes from chaining small tools together. The pipe | sends one command's output into the next command's input:

# Top 5 IP addresses hitting your web server
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -5

Read it left to right: take the first field (the IP address) from each line, sort the addresses, count the duplicates, sort by count with the highest first, and show the top five.

Each command transforms the data and passes it to the next one through the pipe

Redirection sends output to files:

ls -la > files.txt          # write output to a file (overwrites it)
echo "done" >> build.log    # append to a file
npm test > test.log 2>&1    # capture both normal output and errors
wc -l < data.csv            # use a file as input: counts its lines

Permissions with chmod and chown

Run ls -l and you'll see lines like this:

-rwxr-xr-- 1 deploy www-data 2048 Oct 12 09:30 deploy.sh

The permission string breaks down into three groups of three: the owner (rwx), the group (r-x) and everyone else (r--). Here r means read, w means write and x means execute.

chmod +x deploy.sh                              # make a script executable
chmod 644 index.html                            # owner: read/write, everyone else: read-only
chmod 755 scripts/                              # owner: full access, everyone else: read and enter
sudo chown -R www-data:www-data /var/www/site   # change owner and group recursively

Numeric modes add up: read = 4, write = 2, execute = 1. So 7 = 4+2+1 (rwx), 5 = 4+1 (r-x) and 4 means read-only.

Processes: what's running?

ps aux | grep node          # find running node processes
top                         # live view of CPU and memory use (q to quit)
kill 12345                  # politely ask process 12345 to stop
kill -9 12345               # force it to stop (last resort)

Many systems also have htop, a friendlier, colourful alternative to top.

Disk and memory

df -h               # free disk space per filesystem
du -sh *            # size of each item in the current directory
du -sh * | sort -h  # ...sorted from smallest to largest
free -h             # memory usage

Networking basics

curl https://example.com          # fetch a page
curl -I https://example.com       # headers only: check status codes and caching
curl -X POST -H "Content-Type: application/json" \
     -d '{"name": "Ada"}' https://httpbin.org/post
sudo ss -tlnp                     # which programs are listening on which ports
ping -c 4 example.com             # is the host reachable?

Working on remote servers

ssh user@203.0.113.10                                 # log in to a server
scp report.pdf user@203.0.113.10:~/                   # copy a file to it
rsync -avz ./site/ user@203.0.113.10:/var/www/site/   # sync a whole directory efficiently

Getting help

You don't need to memorise every option, because the system documents itself:

man grep                 # the full manual (q to quit)
grep --help              # a quick summary of options
history | grep docker    # find that command you ran last week

And here's the best shortcut of all: press Ctrl+R and start typing to search your command history interactively.

Recap

Task	Commands
Navigate	`pwd`, `ls`, `cd`
Manage files	`mkdir`, `touch`, `cp`, `mv`, `rm`
Read files	`cat`, `less`, `head`, `tail -f`
Search	`grep`, `find`
Combine	pipes, `>`, `>>`, `2>&1`
Permissions	`chmod`, `chown`
System	`ps`, `top`, `kill`, `df`, `du`, `free`
Network	`curl`, `ss`, `ssh`, `rsync`

Open a terminal and try each command as you read. Muscle memory is the real goal, and within a week you'll wonder how you ever worked without them.

Python Decorators Explained Step by Step

Wed, 09 Sep 2026 09:00:00 +0000

You've probably seen code like this in Flask, FastAPI or Django:

@app.route("/")
def home():
    return "Hello!"

That @ line is a decorator. Decorators let you add behaviour to a function (logging, timing, caching, access control) without changing the function's own code. They can look like magic, but they're built from three simple ideas. Let's take them one at a time.

Idea 1: functions are objects

In Python, functions are values like any other. You can assign them to variables and pass them to other functions:

def greet(name):
    return f"Hello, {name}!"

say_hello = greet          # no parentheses: we're not calling it
print(say_hello("Ada"))    # Hello, Ada!

Idea 2: functions can be defined inside functions

def outer():
    def inner():
        return "I'm inside!"
    return inner           # return the function itself

fn = outer()
print(fn())                # I'm inside!

Idea 3: inner functions remember their surroundings

An inner function can use variables from the function that created it, even after that outer function has finished. This is called a closure:

def make_multiplier(factor):
    def multiply(x):
        return x * factor  # "remembers" factor
    return multiply

double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5), triple(5))  # 10 15

Putting it together: your first decorator

A decorator is a function that takes a function and returns a new function, usually one that wraps the original with extra behaviour:

def shout(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result.upper()
    return wrapper

def greet(name):
    return f"Hello, {name}!"

greet = shout(greet)       # wrap it manually
print(greet("Ada"))        # HELLO, ADA!

The @ syntax is simply a shortcut for that greet = shout(greet) line:

@shout
def greet(name):
    return f"Hello, {name}!"

print(greet("Ada"))        # HELLO, ADA!

What the @ line actually does: the decorator receives your function and returns a replacement

*args and **kwargs let the wrapper accept any arguments and pass them straight through, so the decorator works on functions with any signature.

Don't forget functools.wraps

Our decorator has a subtle problem: the wrapped function has lost its identity.

print(greet.__name__)      # wrapper

That breaks debugging, documentation tools and some frameworks. Fix it with functools.wraps, which copies the original function's name, docstring and other metadata onto the wrapper:

from functools import wraps

def shout(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs).upper()
    return wrapper

Make it a habit

Add @wraps(func) to every decorator you write. It costs one line and prevents confusing bugs later.

A practical decorator: timing

Let's build something useful: a decorator that reports how long a function takes.

import time
from functools import wraps

def timed(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            elapsed = time.perf_counter() - start
            print(f"{func.__name__} took {elapsed:.3f}s")
    return wrapper

@timed
def build_report(rows):
    return sum(i * i for i in range(rows))

build_report(5_000_000)    # build_report took 0.312s

The try/finally makes sure the time is printed even if the function raises an exception.

Decorators that take arguments

What if you want to write @retry(times=3)? Then you need one more layer: a function that takes the arguments and returns a decorator.

import time
from functools import wraps

def retry(times=3, delay=1.0, exceptions=(Exception,)):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as error:
                    if attempt == times:
                        raise
                    print(f"Attempt {attempt} failed: {error}. Retrying in {delay}s…")
                    time.sleep(delay)
        return wrapper
    return decorator

@retry(times=5, delay=0.5, exceptions=(ConnectionError,))
def fetch_prices():
    ...

@retry(times=5) first calls retry(...), which returns decorator, which is then applied to fetch_prices. Three layers sounds complicated, but each one has a single job:

Layer	Receives	Returns
`retry`	The options (`times`, `delay`…)	`decorator`
`decorator`	The function to wrap	`wrapper`
`wrapper`	The arguments of each call	The function's result

Built-in decorators you should know

Python ships with several excellent decorators. For example, you don't need to write your own caching, because functools.cache (Python 3.9+) does it for you:

from functools import cache

@cache
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

print(fibonacci(100))   # 354224848179261915075, instantly

Without @cache, this naive recursion would make more than 10²⁰ function calls. With it, each value is computed exactly once. Use functools.lru_cache(maxsize=…) if you need to cap memory use.

Other built-in decorators worth knowing:

@property exposes a method as a read-only attribute.
@staticmethod and @classmethod define methods that don't need an instance.
@dataclass generates __init__, __repr__ and __eq__ for you.
@contextlib.contextmanager turns a generator into a with-statement context manager.

Recap

Functions are objects: they can be passed around, returned and defined inside other functions.
A decorator takes a function and returns a wrapped version; @deco is shorthand for func = deco(func).
Use *args, **kwargs to support any signature, and always use @functools.wraps.
Decorators with arguments need one extra layer: a function that returns the decorator.
Reach for built-ins like @cache, @property and @dataclass before writing your own.

Challenge: write a @log_calls decorator that prints the function's name and arguments every time it's called. Then stack it with @timed and observe the order in which they run.