In this post, you will learn what webhooks are and how to use one to send notifications from your scripts straight to your phone. Webhooks are one of the most common building blocks in software development, and once you understand them, a big part of modern automation stops being mysterious.

You have probably created an automation with Python before. Perhaps a script that sends an email, or something that scrapes data from a weather website. Now imagine the script crashed for some reason. You would not know immediately. You might come back days later and see that no data has been scraped for a week.

So what most people do is add a loop somewhere that keeps checking:

import time

while True:
    if check_if_done():
        break
    time.sleep(10)

This approach is called polling. Your script keeps asking “is it done yet?” over and over. It works, but think about what is actually happening here: your script sends thousands of requests a day just to hear “no” almost every time. It is like calling the pizza place every two minutes to ask if your order is ready. They will not enjoy that, and neither will the server you are polling.

The pizza place solves this differently, of course. You give them your number, and they call you when the order is ready. In software, that pattern is called a webhook.

A webhook is not a technology

So what is a webhook exactly? It is an HTTP request that someone else’s computer sends to a URL of your choice when an event happens. That is really the whole definition. It is not a library you install or a protocol you have to study. It is just an agreement between two computers: when X happens, send a POST request to this address.

Until now, you have always been the caller. Every requests.get() you have written is your script phoning a server. A webhook is the same thing in the opposite direction: the server phones you.

The script that texts you

Let me show you the fastest way to experience this. We will make a script send a notification to your phone.

The tool we will use is ntfy.sh. There is no signup, no API key, and no credit card involved. I have seen enough free services die under my projects to be careful with recommendations, but ntfy has been reliable for me.

First, install the ntfy app on your phone (free on iOS and Android) and subscribe to a topic. A topic is just a name you invent. Keep in mind that anyone who knows the name can send messages to it, so pick something hard to guess, like da-topic-4821, not test.

Then run this script:

import requests

requests.post(
    "https://ntfy.sh/da-topic-4821",
    data="Scraper finished. 340 rows."
)

About one second later, your phone buzzes.

That is four lines of Python, and your script can now reach you anywhere.

What actually traveled over the wire

Let’s look at what your script actually sent to ntfy’s server.

It is a verb (POST), an address, a couple of headers, and a body with the message. Every webhook in the world has this same shape. When Stripe informs an online shop that a payment succeeded, it sends this exact structure, just with a bigger JSON body. When GitHub tells a build server that someone pushed code, same thing.

The other side of the phone call

So far you have been the sender. Receiving a webhook is the reverse: you have a URL, and other computers send POST requests to it when something happens.

You might think you need Flask or FastAPI for this, but you don’t. Python comes with everything built in:

from http.server import HTTPServer, BaseHTTPRequestHandler

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers["Content-Length"])
        body = self.rfile.read(length).decode()
        print("Webhook received:", body)
        self.send_response(200)
        self.end_headers()

HTTPServer(("", 8000), Hook).serve_forever()

Run this and you have a webhook receiver at http://localhost:8000. If you send it a POST request from a second terminal (reuse the four-line script from above and just change the URL), you will see the message printed. You just ran both sides of a webhook on one machine, with the standard library only.

By the way, if you want to watch a real webhook arrive before writing any code, open webhook.site in your browser. It gives you a temporary URL and shows every request that hits it, live. Five minutes there and webhooks will never feel mysterious again.

The catch: localhost is a private address

There is one thing that catches almost everyone at this point. Your receiver works when you send to it. However, if you paste http://localhost:8000 into Stripe’s webhook settings, nothing will ever arrive.

That is because localhost means “this machine”. It is a name only your own computer can hear. Your laptop has no public address on the internet. It sits behind your router, and the router has no idea it should forward anything to you.

There are two solutions. For testing, you can use a tunnel like ngrok, which gives you a temporary public URL that forwards to your laptop. For real applications, you deploy the receiver to a server, which is simply a computer with a public address that stays on. That trip from laptop to server is a topic we will keep coming back to in this publication.

The pattern to steal today

Here is what I recommend you do with all this. Wrap the sender in a small function and keep it in every project:

import requests

def notify(message):
    requests.post("https://ntfy.sh/da-topic-4821", data=message)

notify("Backup complete: 2.3 GB")
notify(f"Script crashed: {error}")
notify("Price dropped below 40 EUR")

My personal rule is that any script running longer than a coffee gets a notify() call at the end and one in the error handler. It changes how you work, because your scripts stop being things you watch and start being things that report to you.

And once you know the shape, you will notice webhooks everywhere: GitHub triggering tests, Stripe unlocking accounts, Telegram bots replying to messages.

In a future post, we will take the next step: giving your own function a public URL, so that anything on the internet can call your code.

-Ardit

Today we build that.

Projects in this week’s series:

Day 1 (Yesterday): CLI recorder + local Whisper transcription

When the user runs the Day 1 script, they get prompted to press Enter, speak, and press Enter again. The transcript appears in the terminal and the audio + text are saved to ~/VoiceNotes/:

Everything runs locally — the transcript appears within a second or two of hitting Enter to stop. It works, but you have to be in the terminal.

👉 Day 2 (Today): “Whisper Flow” mode — press Right Option anywhere, dictate anything

When the user presses and holds Right Option (or Right Alt on Windows) inside any app on their computer, the voice notes daemon starts recording. On release, the transcript is typed directly into whichever app has focus. Here’s it landing in a Slack message input:

No window. No terminal to switch to. Just dictate, release, done. Same experience as the commercial dictation apps, running fully offline on your own hardware.

View All Projects This Week

Today’s Project

The Day 1 script had two things wrong for real daily use:

1. You had to be in the terminal. No app you actually use fires up in the terminal — email, Slack, docs, code editors all live in their own windows.

2. The transcript went to a file. You still had to open that file, copy the text, tab back to your target app, paste. More clicks than just typing.

The fix is a background daemon that listens for a global hotkey. When you press it, the app records — no matter which window you’re in. When you release, the app transcribes and types the text into whichever window has keyboard focus. Nothing visible, no context switching, no copy-paste.

The daemon reuses Day 1’s record_until_stopped() and transcribe() functions unchanged. What changes is the trigger (a hotkey instead of ENTER) and the destination (a keystroke stream instead of a file). Same underlying pipeline; radically different UX.

Project Task

Build a background daemon that:

• Loads a faster-whisper model on startup and prints a “ready” banner

• Uses pynput to listen for a global hotkey (default: Right Option / Right Alt)

• On key press: starts recording from the microphone in a background thread

• On key release: stops recording, transcribes, and types the result into whichever window has focus

• Runs forever until Ctrl+C

• Filters out accidental quick taps (recordings under 0.3s)

• Archives each note to ~/VoiceNotes/ for later reference

This project teaches you pynput (global hotkeys + typing), how threading events coordinate press-and-release-driven state, and — critically — the macOS Accessibility permission dance that trips up everyone the first time.

Setup Instructions

This one’s more involved than Day 1 because we’re doing operating-system-level things — listening to keyboard events across all apps, injecting keystrokes into whichever app has focus. macOS in particular has security requirements that need explicit setup. Read this section carefully. Half the errors you’ll hit are permissions, not code.

Python dependencies (all platforms):

pip install sounddevice soundfile numpy faster-whisper pynput

If Day 1 already installed the first four, only pynput is new.

macOS setup — the Accessibility permission

You need to grant Accessibility permission to whichever app runs the script. This is macOS’s security model for anything that observes or injects keyboard events — Python needs it, Chrome needs it, Zoom needs it. Without it, pynput still runs, but macOS feeds it garbage events. The daemon will appear to record and stop on its own, in an infinite loop, without you touching anything. If that happens to you, permission is the problem, not the code.

The app that needs permission is the one running python — usually Terminal, iTerm2, Warp, or your IDE. Not Python itself.

To grant it:

1. Open System Settings → Privacy & Security → Accessibility

2. Click the + button

3. Add Terminal (path: /System/Applications/Utilities/Terminal.app), or whichever terminal you actually use

4. Toggle the switch next to it on

5. Fully quit Terminal with Cmd+Q (not just close the window) and reopen it

That last step is non-negotiable. macOS only checks the permission list at process startup. A running Terminal keeps behaving as unpermitted even after you flip the toggle, until you fully quit and relaunch.

macOS setup — the SSL cert gotcha

If you get an SSL certificate error the first time faster-whisper tries to download the model — something like FileNotFoundError: [Errno 2] No such file or directory deep in the traceback, pointing at a .pem file — you’ve been bitten by a stale SSL_CERT_FILE environment variable. Fix it with:

unset SSL_CERT_FILE
python3 voice_notes_daemon.py

If that works, find where the variable is being set and remove it permanently:

grep -H SSL_CERT_FILE ~/.zshrc ~/.zprofile ~/.bash_profile 2>/dev/null

Delete the offending line from whichever file grep finds it in. This isn’t a problem with your code or with faster-whisper — it’s a common misconfiguration on Macs that had older Python setups or Anaconda installed at some point.

Windows setup

pynput works out of the box on Windows. No permissions, no config. Just install and run.

One note: some antivirus software flags apps that install global keyboard hooks, which is what pynput does under the hood. This is a legitimate flag in general — apps that watch keystrokes globally could be keyloggers — but our code only reads press/release events for the hotkey and never captures other keys’ values. Whitelist the script or the Python interpreter if your antivirus balks.

Linux setup

pynput needs an X server. If you’re on desktop Linux, you already have one. Wayland users get “failed to acquire X connection” errors — either switch to an X11 session or install xwayland. pynput‘s Wayland support is limited as of writing.

Choosing your hotkey

The default hotkey is Right Option on Mac, called Right Alt on Windows keyboards. In pynput it’s keyboard.Key.alt_r — one code, both platforms.

Why not CAPS LOCK, which is what commercial apps seem to use?

Because CAPS LOCK on macOS is a broken abstraction for userspace keyboard libraries. Unlike every other key, CAPS LOCK doesn’t send normal press-and-release events at the OS level — it’s a hardware toggle, and macOS emits weird 0-millisecond press-plus-release event pairs that pynput can’t reliably interpret. You end up with the daemon firing “recording started / recording stopped” on its own, in a loop, without touching the keyboard. I spent longer than I’d like to admit figuring this out.

Commercial dictation apps that use CAPS LOCK reach for lower-level APIs than Python has access to from userspace. The nearest equivalent you can build in plain Python is Right Option, which sits on the same row as CAPS LOCK, is rarely used for anything else, and behaves like a normal key.

If you really want CAPS LOCK on Mac: install Karabiner Elements — free, well-maintained, the standard tool for Mac keyboard remapping — and remap Caps Lock → F13. Then set the daemon’s hotkey to keyboard.Key.f13. Karabiner handles the CAPS LOCK weirdness at the hardware event level, and pynput never sees anything unusual. But honestly? Right Option works great and requires zero extra software. Don’t add complexity for a wart.

Windows users don’t have a “Right Option” key — the same physical key is labeled “Right Alt” or “AltGr” on international keyboards. Same pynput code (Key.alt_r), different physical label. No remapping needed.

Running it

python voice_notes_daemon.py

You should see this in the terminal:

🎙️  Voice Notes daemon
    Model: base (int8)
    Hotkey: Key.alt_r
    Archive: /Users/you/VoiceNotes

Loading Whisper model...
✓ Model ready.

🟢 Ready. Hold Key.alt_r anywhere on your system to dictate.

Then the daemon goes quiet. Silence is correct. If “Recording...” messages appear immediately without you pressing anything, that means macOS Accessibility permission isn’t granted, or the terminal wasn’t fully quit after granting it. Go back to the Accessibility section above.

Once it’s idle at the “🟢 Ready” line: open any other app, put your cursor in a text field, hold Right Option, speak a sentence, release. The daemon prints a transcription line, and the text appears in the app you had focused.

Understanding Global Hotkeys with pynput

Yesterday’s script used input() — a built-in that reads from stdin. It only fires when the terminal window has focus. That’s the opposite of what we want.

We want to listen for key presses anywhere on the system, regardless of which app has focus. That’s called a global hotkey, and it requires operating-system-level integration:

• On macOS: the Quartz event system

• On Windows: the SetWindowsHookEx API

• On Linux: the X server or evdev

pynput wraps all three behind one Python API. You never touch the platform-specific code:

from pynput import keyboard

def on_press(key):
    print(f"pressed: {key}")

def on_release(key):
    print(f"released: {key}")

with keyboard.Listener(on_press=on_press, on_release=on_release) as listener:
    listener.join()

The Listener runs in its own thread, calling your callbacks whenever a key event happens anywhere on the system. listener.join() blocks the main thread until the listener stops.

This is the entire core of what commercial dictation apps do. Everything else is polish.

Understanding the Press-Release State Machine

Our daemon has three states:

• Idle — no key held, no recording

• Recording — hotkey pressed, audio flowing

• Transcribing — hotkey released, waiting for Whisper

The state transitions are triggered by press/release events, not by ENTER. Two subtleties:

The listener callback thread must not block. If we started the actual recording synchronously in on_press, the listener would freeze — meaning it would miss the release event. Instead, we hand recording off to a new thread and use a threading.Event from Day 1 to signal stop:

recording = False
stop_event = threading.Event()

def on_press(key):
    global recording
    if key == HOTKEY and not recording:
        recording = True
        stop_event.clear()
        threading.Thread(target=record_and_transcribe, daemon=True).start()

def on_release(key):
    global recording
    if key == HOTKEY and recording:
        stop_event.set()
        recording = False

The recording flag prevents double-starts when a key auto-repeats. On some keyboards, holding a key down makes the OS fire “press” events repeatedly. Without the flag, each auto-repeat would spawn a fresh recording thread — chaos. The flag ensures one press = one recording session, no matter how many repeat events arrive during the hold.

Three threads coordinating: the main thread runs the listener, the listener thread calls our callbacks, and each recording session spawns a worker thread. Sounds like a lot, but it maps cleanly: one thread per concern, no thread ever blocking another.

Understanding Typing Into the Focused Window

Once we have transcribed text, we inject it into whichever app has keyboard focus. pynput handles this with a Controller:

from pynput.keyboard import Controller

kb = Controller()
kb.type("hello world")

That’s it. type() iterates through each character and synthesizes press+release events for it. From the OS’s perspective, it’s indistinguishable from a human typing very fast. The text lands in whichever field currently has focus — which is exactly what we want, because the user’s cursor was there before they held the hotkey.

There’s one gotcha worth flagging: type() can be slow for long text because it’s really simulating individual keystrokes. For a 200-word paragraph you might see a subtle typing animation. That’s authentic UX (matches what commercial apps do) and rarely a problem, but if you want instant paste-style insertion, put the text on the clipboard with pyperclip and simulate Cmd+V / Ctrl+V instead. Left as an exercise.

Understanding the Accessibility Permission Under the Hood

macOS-specific, but worth explaining what’s actually going on because the failure mode is confusing.

macOS enforces TCC (Transparency, Consent, and Control) — a database of which apps can do which sensitive things. Reading global keyboard events is one of the sensitive things. Every app that wants to observe keyboard events across other apps must be individually approved in System Settings → Accessibility.

The reason pynput doesn’t just crash without permission is that macOS deliberately lets the process create the event listener — it just feeds it garbage. This is a security design: apps that don’t have permission are silently sandboxed from the real event stream, so a malicious keylogger can’t tell whether the user knows about it. From the app’s point of view, everything looks fine; it’s just receiving fake events.

The practical implication for us: when the terminal doesn’t have Accessibility permission, pynput sees phantom events — 0ms-duration press-plus-release pairs, ghost keystrokes — that trigger the daemon’s callbacks. That’s why the symptom of missing permission is “recording started / recording stopped fires by itself” instead of a clean error message. It’s not the code being buggy. It’s macOS feeding fake data as a security measure.

After granting permission and fully quitting the terminal, the event stream becomes real. Press a key, pynput sees a real press. Release, it sees a real release. Silence when idle.

Understanding Why This Composes with Any App

The reason the daemon works in Slack, VS Code, Google Docs, terminal, email, and every other app is that we’re not integrating with any of them. We’re operating at the OS level, one layer below the applications. macOS, Windows, and Linux each own their own keyboard focus model. Whichever app currently has focus receives the keystrokes we generate — the app has no idea whether they came from a human, a Bluetooth keyboard, or a Python daemon.

This is the architectural insight behind commercial dictation apps. They don’t have Slack integrations. They don’t have VS Code integrations. They just type into the OS’s keystroke stream, and every app that receives keystrokes normally receives their keystrokes too. Universal by construction.

The trade-off: we can’t do app-specific behavior. We can’t detect that we’re in Slack and format the text as a reply-to-thread. We can’t notice we’re in VS Code and put the text in a comment vs. code. But 95% of the daily value is just “make the text appear where my cursor is” — and that’s what we do, in every app, forever, with no integrations to maintain.

Practical Use Cases

Meeting notes on the fly: Cursor in your notes app, dictate a bullet, release. Cursor in your Slack channel, dictate a message, release. No app switching, no re-focusing.

Writing code comments: Cursor in a code file. Hold, describe what the function does, release. The comment types itself. This is how you get comments on functions you’d normally skip because typing them is friction.

Dictating emails: The hardest part of email is starting. Voice makes the first draft trivial. Cursor in Gmail’s compose box, hold hotkey, ramble, release, edit. Ships faster.

Language learning: Whisper is multilingual. Dictate in Spanish, get Spanish text. Dictate in French, get French text. Detects the language automatically per recording.

Accessibility: For anyone whose typing is limited by RSI, tremor, or vision. Fully offline means no cloud dependency, no privacy trade-off, works on airplanes.

What You’ve Accomplished This Week

Two days ago, you had no dictation tool. Now you have one that’s competitive with $12/month commercial apps, running fully on your machine, forever.

You’ve learned:

• Real-time audio recording with sounddevice and thread-safe queues

• Local AI transcription with faster-whisper — no API keys, no data leaves your machine

• Global keyboard event capture with pynput

• Programmatic keystroke injection into the OS event stream

• How macOS Accessibility permission actually works under the hood

• Why CAPS LOCK is a landmine on Mac for userspace apps, and how to route around it

The tool is useful daily, and you own every line of the code. Modify the hotkey. Add auto-save to a specific folder per app. Trigger different behavior when you double-tap the hotkey. Change the model size for accuracy vs. speed. This isn’t a toy tutorial — it’s a real personal productivity tool that will still work five years from now, because it depends on nothing except your local machine.

Next steps you might explore:

• Add sound feedback (a subtle beep on recording start/stop) using simpleaudio or playsound

• Add a menu bar icon (macOS) or system tray icon (Windows) using rumps or pystray so you can see the daemon status at a glance

• Swap type() for clipboard-paste (pyperclip + Cmd+V) for instant insertion instead of simulated typing

• Add language selection — press the hotkey once for English, twice quickly for Spanish, etc.

• Swap the base model for small or medium if you want more accuracy and can spare the extra CPU

That’s the finish line for this week. See you next Tuesday.

Solution

The complete daemon code — hotkey listener, threading choreography, and the type-into-focused-window integration — is below. Save it as voice_notes_daemon.py, install the dependencies, grant Accessibility permission (macOS), and run.

Read more

This week we build it in Python. Running fully on your machine. No cloud. No API keys. No subscription — ever.

Projects in this week’s series:

👉 Day 1 (Today): CLI recorder + local Whisper transcription

When the user runs the program, they get asked to press Enter to record their voice:

After pressing Enter again, the program will transcribe their voice, print out the transcribed text in the terminal and also indicate where the saved audio and transcribed file were saved.

Day 2 (Tomorrow): “Whisper Flow” mode — press Right Option anywhere, dictate anything

When the user presses and holds the Right Option / Right Alt key on their keyboard inside any app on their computer, the voice notes daemon starts recording on the background. On release of the button, the transcript is typed directly into whichever app has focus — here shown landing in a Slack message input as an example:

No terminal to switch to. No copy-paste. The exact same experience as Whisper Flow and Superwhisper, running fully offline on the user’s machine.

View All Projects This Week

By Wednesday, you’ll have this:

You’re in Slack. Cursor in the message box. Press Right Key/ Right Option. Say “hey team, running 5 minutes late for standup.” Release. The words appear in the message box, typed by the app. No browser tab open, no cloud upload, no monthly fee.

Same in VS Code — you’re writing a code comment. Right Key / Right Option, dictate, release. Text lands in the file. Same in Google Docs, Notes, terminal, email — anywhere your cursor is. Nothing on your screen changes except the words appearing.

That’s Wednesday. Today we build the foundation that makes it possible: recording and transcribing audio locally, without touching the cloud.

Today’s Project

The CLI voice recorder.

Run the script. Press ENTER. Speak. Press ENTER again. Watch the transcript appear. The audio and text are auto-saved to ~/VoiceNotes/ for later reference.

Simple, useful, works entirely offline after the first run. And it teaches the full pipeline that Wednesday’s magic depends on:

1. Stream audio from the microphone into a numpy array

2. Save the array as a WAV file

3. Feed it to Whisper for transcription

4. Print + save the result

Wednesday just changes how those steps are triggered — from an ENTER prompt to a system-wide hotkey — and where the transcript goes: not to a file, but typed directly into whichever app has focus.

About Whisper (and Why “Faster”)

Whisper is an open-source speech recognition model from OpenAI, released in 2022. Trained on 680,000 hours of multilingual audio, it’s genuinely excellent — accurate, robust to accents, handles background noise, and works in 99 languages.

The original whisper library from OpenAI uses PyTorch and is somewhat slow on CPU. faster-whisper is a re-implementation using CTranslate2 — the same model, but 4x faster on CPU and using less memory. Same accuracy, better performance. It’s what nearly every real-world Whisper app uses today.

Model sizes:

• tiny — 75 MB, fastest, less accurate

• base — 150 MB, great balance for voice notes ← our default

• small — 500 MB, more accurate

• medium — 1.5 GB, close to human parity

• large-v3 — 3 GB, state of the art

For personal voice notes on a laptop, base is the sweet spot — fast enough to transcribe a 30-second note in ~1 second on a modern CPU, accurate enough for casual dictation.

Project Task

Build a CLI script that:

• Loads a faster-whisper model on startup

• Loops: prompts the user, records audio, transcribes, saves, repeats

• Uses sounddevice to stream from the microphone into a numpy array

• Streams audio through a thread-safe queue so recording keeps working while the main thread waits for input

• Uses a threading.Event to signal “stop recording” from the main thread

• Saves each note as a timestamped pair of .wav + .txt files in ~/VoiceNotes/

• Handles Ctrl+C cleanly

• Filters out accidental too-short recordings (under 0.3s)

This project gives you hands-on practice with sounddevice, thread-safe programming, threading.Event, queue.Queue, faster-whisper, and soundfile.

Setup Instructions

System dependencies (macOS + Linux):

macOS:

brew install portaudio

Linux:

sudo apt install portaudio19-dev

Windows: no system dependencies needed — sounddevice bundles PortAudio.

Python dependencies:

pip install sounddevice soundfile numpy faster-whisper

Run it:

python voice_notes.py

First run takes ~30 seconds to download the Whisper model. After that, model load is 2-3 seconds and everything works offline. You’ll see a startup banner, then a “Press ENTER to start recording” prompt. Speak, press ENTER again, and the transcript appears in the terminal — matching the screenshot at the top of this post.

A note on microphone permissions: on macOS, the first time you run this, macOS will pop up a dialog asking to grant microphone permission to whichever terminal you’re using. Grant it, then restart the script. This is a one-time setup.

Understanding sounddevice and Streaming Audio

Recording audio in real time is fundamentally different from reading a file. A file has a fixed size; a live audio stream has no end until you tell it to stop. So instead of “read all the audio,” we say “start a stream, and call this function every time a chunk of audio is available.”

sounddevice gives us exactly that — the InputStream with a callback:

import sounddevice as sd

def my_callback(indata, frames_count, time_info, status):
    # Called every ~10ms with a fresh chunk of audio.
    # indata is a numpy array of shape (frames, channels)
    print(f"Got {frames_count} frames")

with sd.InputStream(samplerate=16000, channels=1,
                    dtype="float32", callback=my_callback):
    time.sleep(3.0)  # let it run for 3 seconds

Three things worth internalizing:

• The callback runs on a separate thread, managed by PortAudio internally. It fires at whatever rate the audio hardware ticks (usually every 10-30ms).

• indata is a view into a stream-owned buffer that will be overwritten as soon as the callback returns. If you want to keep the data, you MUST .copy() it. Failing to do this is the #1 rookie sounddevice bug.

• The with block is what starts and stops the stream. As soon as you enter it, audio starts flowing; as soon as you exit, it stops. Clean lifetime management.

Understanding queue.Queue for Cross-Thread Data

Now we have a problem: the audio callback is running on the audio thread. We want to process the audio on the main thread. How do we get data safely from one thread to the other?

queue.Queue is the answer. It’s a thread-safe FIFO buffer. One thread puts items in, another thread takes them out. All the locking is handled internally.

import queue

audio_queue = queue.Queue()

def audio_callback(indata, frames_count, time_info, status):
    audio_queue.put(indata.copy())    # audio thread: producer

# Main thread: consumer
while not stop_event.is_set():
    try:
        block = audio_queue.get(timeout=0.1)
        frames.append(block)
    except queue.Empty:
        continue

Two habits worth forming:

• .copy() before putting into the queue. Otherwise the audio thread will overwrite what you queued.

• .get(timeout=0.1) instead of .get() — the timeout lets the main thread wake up periodically and check stop_event, instead of blocking forever on an empty queue. 100ms is a good compromise between responsiveness and CPU wake-ups.

The producer-consumer pattern with a queue is foundational for any streaming-data work. Once you’ve done it with audio, you know how to do it with sensor data, live network feeds, real-time file monitoring, and everything in between.

Understanding threading.Event for Stop Signals

We need a clean way for the main thread to tell the recording thread “stop now, wrap it up.” A boolean flag would work but has subtle threading issues. threading.Event is the idiomatic answer:

import threading

stop_event = threading.Event()   # starts unset

# In recording thread:
while not stop_event.is_set():
    # ... record ...

# In main thread:
stop_event.set()                 # signal stop
recording_thread.join()          # wait for it to actually stop

An Event is essentially a thread-safe boolean with three operations:

• .set() — turn it on

• .is_set() — check its state

• .wait(timeout) — block until it turns on

This is the pattern for graceful shutdown in any threaded Python program: web servers, background workers, real-time processors. Learn it once, use it forever.

Understanding the Main Loop Choreography

Now let’s zoom out and see how ENTER, the event, and the thread all coordinate:

while True:
    input("Press ENTER to start recording...")   # blocks main thread

    stop_event = threading.Event()               # fresh event per note
    result = {}                                  # place for thread's output

    def worker():
        result["audio"] = record_until_stopped(stop_event)

    recording_thread = threading.Thread(target=worker, daemon=True)
    recording_thread.start()                     # off it goes

    print("Recording... Press ENTER to stop.")
    input()                                       # blocks main thread again
    stop_event.set()                              # signal the worker
    recording_thread.join()                       # wait for actual shutdown

    audio = result["audio"]                       # retrieve results
    # ... transcribe, save, loop ...

Key ideas:

• The result = {} dict pattern — since Python functions can’t easily return values from a threading.Thread(target=...), we use a dict as a shared “result box.” The thread writes into it; the main thread reads after .join().

• daemon=True — daemon threads die automatically when the main program exits. If the user hits Ctrl+C during recording, the recording thread won’t linger.

• The two input() calls block the main thread — but only the main thread. The recording thread keeps happily filling the queue during both waits. This is why threading is essential here: without it, ENTER would never work while recording was happening.

Read this section twice if it’s new. Understanding this dance is the difference between “I can use threading” and “I can design threaded programs.”

Understanding faster-whisper

faster-whisper is dead simple to use:

from faster_whisper import WhisperModel

# Load once. This takes 2-3 seconds and downloads the model on first run.
model = WhisperModel("base", device="cpu", compute_type="int8")

# Transcribe. audio is a 1-D float32 numpy array at 16 kHz.
segments, info = model.transcribe(audio, beam_size=5, vad_filter=True)

# segments is a generator. Consume it to get the text.
text = " ".join(seg.text.strip() for seg in segments)

Three parameters worth knowing:

• compute_type="int8" — the fastest CPU inference mode. Slight accuracy loss vs. float32, but ~4x faster and half the RAM. Perfect for laptops.

• beam_size=5 — how many candidate sequences the model considers. Bigger = more accurate, slower. 5 is a great default; you can drop to 1 for speed or bump to 10 for the last few percent of accuracy.

• vad_filter=True — Voice Activity Detection. Whisper’s biggest annoyance is that it “hallucinates” transcriptions in silent audio (”thank you for watching” is a common one, from YouTube training data). VAD drops silent segments before transcription, dramatically reducing this problem.

The info object tells you what language Whisper detected and how confident it was — a handy little UX detail.

Understanding Saving with soundfile

soundfile is our WAV writer. One function does everything:

import soundfile as sf

sf.write(wav_path, audio, SAMPLE_RATE, subtype="PCM_16")

• subtype="PCM_16" — 16-bit signed integer WAV format. Widely compatible, ~half the size of float32 WAV. soundfile handles the float→int conversion automatically.

• The path can be a string or pathlib.Path — both work.

If you ever need to read an audio file, it’s the mirror:

audio, sample_rate = sf.read(wav_path)

Both soundfile and sounddevice use numpy arrays as their common language — no format juggling, no encoding conversions.

Understanding the Auto-Save Pattern

We save every note automatically with a timestamped filename:

from datetime import datetime

stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
wav_path = NOTES_DIR / f"note_{stamp}.wav"
txt_path = NOTES_DIR / f"note_{stamp}.txt"

Two design choices worth explaining:

• Timestamp format YYYYMMDD_HHMMSS — sorts lexically the same as chronologically. note_20260721_143000.wav sorts before note_20260721_143500.wav. Both ls and file explorers show them in order automatically.

• Paired WAV + TXT with the same stem — makes it trivial to find “the audio for this transcript” or “the transcript for this audio.”

Never save with just datetime.now().isoformat() — that produces strings like 2026-07-21T14:30:00.123456+00:00, which contain colons and special characters that break on Windows filenames. strftime with only safe characters is the right call.

Practical Use Cases

Personal dictation: Record thoughts on a walk, transcribe them later, paste into your journal.

Meeting notes: Record short “action item” reminders after a call. The transcripts feed right into your task list.

Language learning: Whisper is multilingual — record yourself speaking Spanish, get back the transcript to verify pronunciation.

Accessibility: Transcription for anyone who prefers speaking over typing. Fully local means it works on airplanes.

Foundation for Wednesday: The record + transcribe + save pipeline stays. Wednesday just changes what triggers it (a system-wide hotkey) and where the transcript goes (into whatever app you’re using).

Coming Wednesday

Wednesday: no ENTER prompt. No terminal you have to switch to. No copy-paste.

The app runs invisibly in the background. You press Right Key / Right Option anywhere on your system — inside Slack, inside VS Code, inside a Google Doc, inside your email. You speak. You release Right Key / Right Option. The transcription lands directly in whichever app you’re focused on, character by character, like you typed it yourself.

It’s the same experience as Whisper Flow and Superwhisper. It costs nothing. It runs on your laptop offline. And you’ll own every line of the code.

Wednesday’s the paid post — upgrade here if you’re not already a subscriber.

Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build a Climate Dashboard — a Streamlit web app that fetches historical weather data from a free public API and turns it into beautiful climatology charts.

• Day 1: Single city, monthly averages

• Day 2: Daily climatology with filled temperature bands

• Day 3: Multi-city comparison (WeatherSpark style) (Today)

View All Projects This Week

Today’s Project

Welcome to the finale — the recreated WeatherSpark chart.

Days 1 and 2 built the foundation: fetch, aggregate, smooth, chart. But a climatology chart of a single city can only answer questions about that city in isolation. The really interesting questions are comparative:

• Is Vienna colder than Tirana in January?

• When during the year is the gap the biggest?

• Which city has a more extreme annual swing?

• Are there any months where they’re at the same temperature?

Today we build the app that answers all of that. Two (or three) cities’ curves on the same chart. Translucent filled bands that show where they overlap. Floating labels on each curve. Prose insights beneath the chart that read the data for you.

Same UI shell as Days 1 and 2. Dramatically more powerful chart.

Project Task

Build the multi-city comparison view:

• Two text inputs (plus a third, optional) for city names

• Load and process each city using the pipeline from Days 1 & 2

• Overlay all cities’ max/min curves on a single Plotly chart

• Each city gets its own color palette and translucent filled band

• Add floating city-name labels near each curve’s peak

• Add a Today marker (like Day 2)

• Below the chart: a prose insights section — “Right now, Tirana is 6°C warmer”, “Biggest gap in October”, “Curves never cross”

• Below that: a monthly comparison table showing exact numbers

• And at the bottom: city info cards styled in each city’s color

Expected Output

Running the app:

streamlit run climatology_compare.py

Type Vienna and Tirana. Here’s what you get:

Two smooth curves per city, showing the lowest and highest daily average temperature. Vienna’s high/low in purple, Tirana’s in green. This is a great way to have a feeling of the weather of a city at a certain month by comparing it with your own city.

We also show some more insights below the chart area:

Setup Instructions

Same dependencies as Days 1 & 2:

pip install streamlit plotly requests
streamlit run climatology_compare.py

Understanding Reusing the Day 2 Pipeline

The whole point of the “same UI, growing intelligence” pattern is that we can call our old code with different arguments instead of rewriting anything. Day 3’s Load-A-City helper is literally the Day 1/2 pipeline in a single function:

def load_city_climatology(city_name, years):
    """Fetch + aggregate + smooth for one city."""
    city = geocode_city(city_name.strip())
    if city is None:
        return None

    today = date.today()
    end_year = today.year - 1
    start_year = end_year - years + 1
    start_date = f"{start_year}-01-01"
    end_date = f"{end_year}-12-31"

    daily = fetch_daily_history(
        city["latitude"], city["longitude"],
        start_date, end_date,
        city.get("timezone", "auto"),
    )

    raw_max, raw_min = daily_climatology(daily)
    avg_max = rolling_mean(raw_max, SMOOTHING_WINDOW)
    avg_min = rolling_mean(raw_min, SMOOTHING_WINDOW)

    return {
        "city": city,
        "avg_max": avg_max,
        "avg_min": avg_min,
        "years_range": (start_year, end_year),
    }

Every function called here — geocode_city, fetch_daily_history, daily_climatology, rolling_mean — is unchanged from Day 2. We just wrapped them in a function that returns a self-contained “city bundle” dict. Then we call it once per city:

bundles = []
for name in [city_a, city_b, city_c]:
    bundles.append(load_city_climatology(name, years))

This is the “bundle” pattern in action. Each bundle contains everything downstream code needs (the chart, insights, table) to work with that city. Downstream code doesn’t care how many cities there are — it just iterates the bundle list.

Understanding Overlapping Traces with Individual Fills

Each city needs its own max/min pair with its own filled band. Since we have 2-3 cities, that means 4-6 traces in total. The pattern:

for bundle, palette in zip(bundles, CITY_PALETTE):
    # Max curve for this city
    fig.add_trace(go.Scatter(
        x=x_labels, y=bundle["avg_max"],
        line=dict(color=palette["high"], width=2.5),
        legendgroup=bundle["city"]["name"],
    ))

    # Min curve WITH fill='tonexty' - fills to the previous trace (the max above)
    fig.add_trace(go.Scatter(
        x=x_labels, y=bundle["avg_min"],
        line=dict(color=palette["low"], width=2.5),
        fill="tonexty",
        fillcolor=palette["fill"],
        legendgroup=bundle["city"]["name"],
    ))

Two important tricks:

• Trace order matters (same as Day 2). Max first, then min with fill="tonexty". The fill fills to the previous trace, which is the max curve above.

• legendgroup groups both traces of one city together in the legend. Click “Vienna” in the legend and both the max and min curves toggle at the same time. Nice UX detail.

The color palette provides three hues per city — a bold high color, a slightly lighter low color, and a translucent fill color:

CITY_PALETTE = [
    {"high": "#7c3aed", "low": "#a78bfa", "fill": "rgba(124, 58, 237, 0.15)"},  # purple
    {"high": "#059669", "low": "#34d399", "fill": "rgba(5, 150, 105, 0.15)"},    # green
    {"high": "#ea580c", "low": "#fb923c", "fill": "rgba(234, 88, 12, 0.15)"},    # orange
]

The fill alpha (0.15) is crucial. Too dark and the cities’ bands hide each other where they overlap. At 0.15, they blend transparently — you can see both cities’ territory even in the shared middle.

Understanding Floating City Labels

Legends are helpful but require the eye to jump back and forth. Real chart craftsmanship puts labels directly on the data. WeatherSpark does this — so do we, using add_annotation:

for bundle, palette in zip(bundles, CITY_PALETTE):
    avg_max = bundle["avg_max"]
    # Find the day-of-year where this city's max is highest
    peak_i = max(range(365), key=lambda i: avg_max[i] if avg_max[i] is not None else -999)

    # Offset each city's label by a few days so multi-city labels don't overlap
    offset = bundles.index(bundle) * 8
    label_i = min(364, peak_i + offset)

    fig.add_annotation(
        x=x_labels[label_i],
        y=avg_max[label_i],
        text=f"<b>{bundle['city']['name']}</b>",
        showarrow=False,
        font=dict(color=palette["high"], size=13),
        xanchor="left", yanchor="bottom",
        xshift=6,
    )

Two concepts worth internalizing:

• showarrow=False — a bare annotation, no arrow pointing anywhere. Just text at a coordinate.

• xshift=6 — nudge the label 6 pixels right so it doesn’t sit on the data point. Small visual polish that makes a huge difference.

The offset = bundles.index(bundle) * 8 line prevents label collision when multiple cities peak in the same month. Without it, “Vienna” and “Tirana” (both peaking mid-July) would sit on top of each other.

Understanding the Smart Insights

The chart shows what the data says. The insights section explains it in prose. This is where Day 3 stops being “just a chart” and becomes a report.

Each insight is generated by comparing two cities’ arrays and finding the answer to a specific question:

def build_insights(bundles):
    insights = []
    a, b = bundles[0], bundles[1]

    # Right now: today's typical difference
    today_i = today_doy() - 1
    diff = a["avg_max"][today_i] - b["avg_max"][today_i]
    warmer = a["city"]["name"] if diff > 0 else b["city"]["name"]
    insights.append(
        f"📍 Right now, **{warmer}** is typically warmer by "
        f"**{abs(diff):.1f}°C**."
    )

    # July vs January comparison
    a_monthly = monthly_averages_from_daily(a["avg_max"])
    b_monthly = monthly_averages_from_daily(b["avg_max"])
    jul_diff = a_monthly[6] - b_monthly[6]
    # ...

Each insight is a few lines of pure data-summary logic. The chart shows what — the insights answer specific questions the user probably had in mind.

Understanding the Monthly Comparison Table

The insights tell you the story; the table gives you the numbers. st.dataframe accepts a list of dicts and renders them as an interactive table:

def build_comparison_table(bundles):
    rows = []
    monthly_pairs = [(b, monthly_averages_from_daily(b["avg_max"]))
                     for b in bundles]

    for m_i, month_name in enumerate(MONTH_NAMES):
        row = {"Month": month_name}
        for bundle, monthly in monthly_pairs:
            row[bundle["city"]["name"]] = round(monthly[m_i], 1)
        rows.append(row)

    return rows

st.dataframe(build_comparison_table(bundles),
             use_container_width=True, hide_index=True)

The dataframe is interactive — click a column header to sort, drag corners to resize. Every real dashboard should have a table below its chart. The chart shows the shape; the table gives you the exact values.

A design principle worth adopting: always pair a visualization with a data table. The chart is for pattern recognition; the table is for verification. Users use both, at different moments.

Understanding What You’ve Built

Let’s zoom out. Compare Day 1 with Day 3:

Day 1: One city, 12 monthly points. Simple, but limited.

Day 3: Two or three cities compared side-by-side, with 365-day smooth curves, filled bands, floating labels, generated prose insights, monthly comparison table, color-coded city cards. The pipeline was extended twice — once for daily granularity, once for multi-city. Nothing was thrown away.

That’s the shape of every real data app you’ll ever build:

• Fetch layer: cache-once functions per API

• Transform layer: small pure functions that turn raw data into usable structures

• Bundle layer: self-contained data envelopes with everything a view needs

• View layer: widgets that render bundles into UI

Change the fetch API tomorrow and only the fetch layer needs updating. Add a fourth visualization next week and only the view layer changes. This separation is what makes Streamlit apps maintainable.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete climate comparison dashboard:

• Day 1: Two-API pipeline, monthly aggregation, Plotly basics

• Day 2: 365-day climatology, rolling smoothing with wraparound, filled bands

• Day 3: Multi-city comparison, prose insights, comparison tables, color-coded cards

You now have:

• Two-API chaining — geocode + main data, the shape of every location app

• Time-series aggregation — the universal groupby-and-average pattern

• Rolling means for noise reduction — with wraparound for cyclical data

• Plotly for interactive charts — line traces, filled areas, annotations, custom ticks

• Streamlit at intermediate level — text inputs, sliders, spinner, metrics, dataframes, styled markdown

• The “bundle” pattern — one data structure per entity, downstream code stays clean

• Prose insights generation — reading data programmatically to write sentences

Real-world applications:

• Any climatology / seasonality analysis — retail sales by day-of-year, energy consumption by hour-of-day, website traffic by day-of-week

• Comparative dashboards — two products’ revenue over time, two competitors’ stock prices, two cities’ any-metric

• Any location-aware data app — real estate prices, air quality, restaurant density

• Time-series smoothing — the rolling-mean trick applies to any noisy signal

• Portfolio piece — deploy to Streamlit Cloud in 5 minutes, share the URL

Next steps:

• Add precipitation as a second chart tab

• Try other data types from Open-Meteo: humidity, wind, sunshine hours

• Build a “best time to visit” view combining temperature, rain, and daylight

• Add a city autocomplete using the geocoding API’s count=5 param

• Deploy to Streamlit Community Cloud — free, one-click GitHub deploy

You’ve built a fully-functional, deployable climate comparison tool. That’s a real portfolio piece. 🚀

View Code Evolution

Compare Day 3’s comparison view with Day 2’s single-city climatology and Day 1’s monthly averages — same fetch, same aggregate, same smooth. The chart and insights got smarter; the pipeline never changed. That’s the design lesson of the week.

Read more

This week, we build a Climate Dashboard — a Streamlit web app that fetches historical weather data from a free public API and turns it into beautiful climatology charts, culminating in a WeatherSpark-style multi-city comparison by Friday.

• Day 1: Single city, monthly averages

• Day 2: Daily climatology with filled temperature bands (Today)

• Day 3: Multi-city comparison (WeatherSpark style)

View All Projects This Week

Today’s Project

Yesterday we drew 12 points per line — one per month. Useful, but chunky. Real climate visualization sites like WeatherSpark draw smooth curves — one point for every single day of the year, forming a beautiful sinusoidal wave that shows the year’s temperature cycle in glorious detail.

Today we go from 12 points to 365. The trick: for each calendar day (Jan 1, Jan 2, ... Dec 31), we average that day’s temperature across many years. So the “Jan 15” value becomes the typical Jan 15 temperature for this city — averaged over a decade of data.

Then we add three visual upgrades:

• A filled band between the max and min curves (the classic WeatherSpark look)

• A vertical “Today” marker so you can see where the current date sits in the annual cycle

• A 7-day rolling smooth so the curves are silky-smooth instead of noisy

Same UI shell as Day 1. Dramatically better chart.

What’s Changing

The pipeline stays the same:

1. Geocode the city

2. Fetch daily temperatures from the archive

3. Aggregate

4. Chart

The aggregation step is what levels up. Instead of grouping by month (12 buckets), we group by day-of-year (365 buckets). Instead of a chunky monthly line, we get a smooth daily curve.

Same data, ~30× more detail.

Project Task

Build the daily climatology view on top of Day 1’s foundation:

• Reuse Day 1’s geocoding + fetch functions unchanged

• Group daily observations by day-of-year (1..365) instead of by month

• Collapse Feb 29 into Feb 28 (leap-year handling)

• Compute a raw 365-day average from 10 years of history

• Smooth the raw curve with a 7-day rolling mean (with year-boundary wraparound)

• Chart it as two curves with fill='tonexty' for the shaded band between them

• Add a vertical “Today” line using Plotly’s add_vline

• Update the metric cards to show today’s typical high/low

• Reduce the x-axis clutter (365 tick labels would be unreadable — show ~12)

Expected Output

Running the app:

streamlit run climatology_daily.py

Type a city (default Vienna, 10 years of history) and you will get these metrics:

• Warmest day of the year

• Coldest day of the year

• Typical hight temperature today

• Elevation

The “Typical high today” card is new — it tells you exactly what a normal mid-July day looks like in Vienna based on 10 years of data. Answers questions like “am I hotter than usual today?” at a glance.

For Vienna, you see the classic seasonal cycle: a smooth sine-like wave peaking in mid-July around 26°C and bottoming in mid-January around -4°C. For Cairo, you see a flatter, higher curve peaking in August around 35°C. For Sydney, the curve is inverted (peaks in January, valleys in July) because it’s the southern hemisphere.

Setup Instructions

Same dependencies as Day 1:

pip install streamlit plotly requests
streamlit run climatology_daily.py

Understanding Grouping by Day-of-Year

Yesterday we grouped by month (12 buckets). Today we group by day-of-year — the integer position of each date within its calendar year, from 1 (Jan 1) to 365 (Dec 31).

The idea is beautifully simple: every Jan 15 across every year goes into the same bucket. Same for Feb 3, and July 14, and December 25. After 10 years of data, each bucket has ~10 observations. We average within each bucket → get the “typical” temperature for that calendar day.

buckets_max = defaultdict(list)
buckets_min = defaultdict(list)

for date_str, tmax, tmin in zip(daily["dates"], daily["max"], daily["min"]):
    doy = day_of_year_no_leap(date_str)      # "2020-07-14" -> 195
    if tmax is not None:
        buckets_max[doy].append(tmax)
    if tmin is not None:
        buckets_min[doy].append(tmin)

avg_max = [_avg(buckets_max[d]) for d in range(1, 366)]
avg_min = [_avg(buckets_min[d]) for d in range(1, 366)]

Same defaultdict(list) pattern as yesterday — just with 365 buckets instead of 12. The universality of the “bucket-and-average” pattern is worth appreciating: it’s the same shape whether you’re aggregating temperatures by day, sales by category, or website visits by hour of day.

Understanding the Feb 29 Problem

Leap years have 366 days. Non-leap years have 365. So Feb 29 exists in some years but not others.

If we naively used the “true” day-of-year:

• Mar 1 would be day 60 in non-leap years and day 61 in leap years

• The bucket for “day 61” would mix Mar 1 (leap) with Mar 2 (non-leap) — polluted data

• Feb 29 would have ~2 data points (from leap years only) while other days have ~10 — massively unbalanced

The cleanest fix is to use a 365-day calendar and collapse Feb 29 into Feb 28:

def day_of_year_no_leap(date_str):
    y, m, d = int(date_str[0:4]), int(date_str[5:7]), int(date_str[8:10])

    # Handle the leap day: treat Feb 29 the same as Feb 28.
    if m == 2 and d == 29:
        d = 28

    days_before_month = [0, 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334]
    return days_before_month[m] + d

Now every calendar day gets ~the same number of data points, Mar 1 is always day 60, and Feb 29’s 2 or 3 observations quietly merge into Feb 28’s bucket. Data integrity preserved.

This is what “handling edge cases correctly” looks like in practice. Not fancy — just knowing your data well enough to spot the trap before it bites you.

Understanding the Rolling Mean

Even with 10 years of data per bucket, our daily curve is still noisy. A random cold snap in July 2019 will make July 14 look colder than July 13 in the raw average. Look at any real climate data closely and you’ll see this daily jitter — it’s not signal, it’s noise.

The fix is a rolling mean — replace each day’s value with the average of it and its neighbors. Our version uses a 7-day window:

def rolling_mean(values, window):
    n = len(values)
    half = window // 2
    smoothed = []
    for i in range(n):
        window_vals = []
        for offset in range(-half, half + 1):
            j = (i + offset) % n         # wrap Dec 31 <-> Jan 1
            v = values[j]
            if v is not None:
                window_vals.append(v)
        if window_vals:
            smoothed.append(sum(window_vals) / len(window_vals))
        else:
            smoothed.append(None)
    return smoothed

Three ideas worth understanding:

• The window size matters. 3 days = still noisy; 30 days = over-smoothed (real seasonal transitions get flattened). 7 days is the sweet spot for daily climate data — one week worth of neighbors on each side.

• (i + offset) % n — the wraparound trick. December 30’s smoothed value legitimately includes Jan 1’s data because climate is cyclical. Without the modulo, Dec 30 would have half a window (only left neighbors) and look artificially bumpy. This little detail is what makes the curve seamlessly continuous.

• Skip None values in the window — even if some day has no data, the rolling mean fills in gracefully using its neighbors.

In our tests: the raw curve had a day-to-day wobble of 1.05°C. After smoothing, it dropped to 0.20°C — an 81% reduction in noise, with the seasonal shape completely preserved.

Understanding the “Today” Marker

Adding a vertical line at today’s date is one Plotly call:

today_i = today_doy() - 1
today_label = doy_to_display_date(today_i + 1)

fig.add_vline(
    x=today_label,
    line_width=2,
    line_dash="dash",
    line_color="#f97316",
    annotation_text="Today",
    annotation_position="top right",
    annotation_font_color="#f97316",
)

add_vline draws a full-height vertical line at the given x-position, with an optional annotation. line_dash="dash" makes it dashed — visually distinct from the temperature curves. The annotation floats near the top of the chart.

This is the contextualization trick that makes the chart feel alive — without the “Today” marker, you’d have to eyeball where you are in the year. With it, the current moment is obvious at a glance.

Understanding X-Axis Tick Reduction

If we told Plotly to show every one of our 365 x-axis labels, they’d overlap into an illegible blur. We manually pick ~12 evenly-spaced positions and only label those:

tick_positions = [(m - 1) * 30 for m in range(1, 13)]      # ~monthly
tick_text = [doy_to_display_date(p + 1) for p in tick_positions]

fig.update_layout(
    xaxis=dict(
        tickmode="array",
        tickvals=[x_labels[i] for i in tick_positions],
        ticktext=tick_text,
    ),
)

tickmode="array" tells Plotly “use exactly these positions and labels, ignore automatic ticking.” Perfect for cases where the data density on the x-axis is way higher than the labels can accommodate.

The lesson: in charts with many x-values, labels are for orientation, not for identification. 12 monthly labels give the reader “roughly where in the year” without cluttering the picture. The unified-hover tooltip gives them the exact day when they want it.

Understanding the Combined Effect

Look at the full transformation from Day 1 to Day 2:

Day 1 chart:

• 12 discrete monthly points connected by straight-ish segments

• Chunky, looks like a bar chart’s spiritual cousin

• Answers “roughly, what does this month look like?”

Day 2 chart:

• 365 smooth points forming a continuous curve

• Shaded band showing the daily temperature range

• Today marker for temporal context

• Answers “what’s the typical Vienna weather on any given day of the year?”

All of Day 2’s added intelligence lives in the aggregation step. The API fetching is unchanged. The Streamlit layout is nearly identical. Only the middle — how we bucket, how we smooth, how we plot — grew smarter.

This is the same “stable UI, growing intelligence underneath” pattern that ran through Week 25’s predictor. It’s a design principle worth internalizing: your users shouldn’t have to relearn the app because you upgraded the math.

Coming Tomorrow

Tomorrow is the finale — multi-city comparison. Type Vienna, then type Tirana, and watch both cities’ climate curves appear on the same chart with different colors. Which city is hotter in July? By how much? At what point in the year do their temperatures cross? The comparison chart answers all of these visually. This is what WeatherSpark does — and we’ll build it from scratch.

View Code Evolution

Compare Day 2’s daily climatology with Day 1’s monthly averages — same fetch pipeline, same UI shell, but the aggregation step levels up dramatically. That’s how real data apps evolve: not by rewriting from scratch, but by refining one piece at a time.

Read more

This week, we build a Climate Dashboard — a Streamlit web app that fetches historical weather data from a free public API and turns it into beautiful climatology charts, culminating in a WeatherSpark-style multi-city comparison by Friday.

Why build this? Because climate data is one of the most rewarding kinds of data to visualize — the seasonal cycle is right there in the numbers, but you need the right chart to see it. And this week teaches the exact “same UI, growing sophistication” pattern that shows up in real-world data apps every day.

What you’ll learn: This series teaches you how to consume two APIs together (geocoding + weather), aggregate time-series data by different time buckets, and build progressively better charts with Plotly. By Friday, you’ll be able to compare the climate of any two cities side-by-side in a single interactive chart.

Why this matters: Climate visualization is the exact shape of dozens of real data problems — sales seasonality, website traffic patterns, energy consumption cycles, stock market seasonality. Master this pattern and you master a huge class of dashboards.

• Day 1: Single city, monthly averages (Today)

• Day 2: Daily climatology with filled temperature bands

• Day 3: Multi-city comparison (WeatherSpark style)

View All Projects This Week

Today’s Project

We start with the foundation: the annual climate profile of a single city, shown as a 12-month line chart of average high and low temperatures.

Type “Vienna” — see Vienna’s climate. Type “Tokyo” — see Tokyo’s. Type “Cairo” — see Cairo’s. The app fetches five years of daily data, averages it by month, and plots the seasonal cycle.

By the end of today you’ll have real climate data on screen for anywhere in the world, working from two free public APIs with no API key required.

About the Data

Two free, public APIs from Open-Meteo — both need no signup, no API key, no credit card:

Historical Weather API: archive-api.open-meteo.com

• ERA5 reanalysis data from 1940 to today

• Global coverage (works for any lat/lng on Earth)

• Daily aggregates: max temp, min temp, precipitation, wind, humidity, and more

• CC BY 4.0 license

• Free tier: ~10,000 requests/day (way more than we’ll need)

Geocoding API: geocoding-api.open-meteo.com

• Turns “Vienna” into latitude: 48.21, longitude: 16.37

• Returns country, timezone, elevation, population as bonuses

• Also free, also no key

Together they let us build a location-agnostic climate app in under 300 lines.

Project Task

Build a Streamlit app that:

• Takes a city name as text input

• Geocodes the city into lat/lng using Open-Meteo’s geocoding API

• Fetches N years of daily historical temperatures for that location

• Aggregates the daily data into 12 monthly average high and low temperatures

• Displays the result as a Plotly line chart with two traces (high and low)

• Shows a summary strip with 4 metric cards: hottest month, coldest month, annual range, elevation

• Includes a small map showing the city location

• Caches API results so repeat interactions are instant

This project gives you hands-on practice with requests + query parameters, JSON parsing with parallel arrays, dict-based bucketing, defaultdict, Plotly graph_objects, Streamlit’s metric / columns / plotly_chart widgets, and @st.cache_data for performance.

Expected Output

Running the app:

streamlit run climatology.py

Type a city name at the top (e.g., Viena). You get:

Setup Instructions

Install dependencies:

pip install streamlit plotly requests

Run:

streamlit run climatology.py

Your browser opens automatically at http://localhost:8501.

Understanding Two APIs Working Together

A common real-world pattern is that the API you actually want to hit needs data the user doesn’t have. The user knows the city name “Vienna” — but the historical weather API needs precise coordinates. That means two API calls, chained.

User types "Vienna"
       ↓
Geocoding API → { latitude: 48.21, longitude: 16.37, ... }
       ↓
Historical API → { daily: { time: [...], temperature_2m_max: [...], ... } }
       ↓
Chart

Both calls happen in the backend of your Streamlit app. The user just sees a text box and a chart — the two-hop journey is invisible.

This is the pattern for any location-aware data app. Weather, air quality, timezone lookups, restaurant search, real-estate prices — all of them start with “turn this address/city into coordinates first.”

Understanding requests with Query Parameters

Both APIs use standard HTTP GET with query parameters. Our helper wraps requests.get cleanly:

@st.cache_data(ttl=86400)
def geocode_city(name):
    response = requests.get(
        "https://geocoding-api.open-meteo.com/v1/search",
        params={"name": name, "count": 1, "language": "en", "format": "json"},
        timeout=10,
    )
    response.raise_for_status()
    data = response.json()
    if not data.get("results"):
        return None
    return data["results"][0]

The params={...} argument tells requests to add these as query parameters — it will URL-encode them properly, so we don’t have to think about escaping. Never manually build a URL by concatenating strings. ?name=New%20York%20City becomes just params={"name": "New York City"} — Python handles the encoding.

response.raise_for_status() throws immediately on 4xx/5xx errors — better than silently getting a broken response and confusing yourself later.

data.get("results") safely returns None if the key is missing, instead of crashing. Different APIs are inconsistent about their “no results” behavior — some return an empty array, some omit the key entirely. .get() handles both.

Understanding the Archive API Response Shape

The Historical Weather API returns something like:

{
  "latitude": 48.2,
  "longitude": 16.37,
  "elevation": 190.0,
  "timezone": "Europe/Vienna",
  "daily": {
    "time":              ["2020-01-01", "2020-01-02", "2020-01-03", ...],
    "temperature_2m_max": [5.4,         6.1,         3.8,          ...],
    "temperature_2m_min": [-1.2,        0.8,         -2.4,         ...]
  }
}

Notice something specific: daily contains three parallel arrays, not a list of daily record objects. Element i in each array belongs to the same day.

This is a common shape for time-series APIs — it’s more compact than an array of objects (saving bandwidth on repeated key names), but requires you to iterate them together with zip():

for date_str, tmax, tmin in zip(daily["time"],
                                daily["temperature_2m_max"],
                                daily["temperature_2m_min"]):
    # date_str = "2020-01-01", tmax = 5.4, tmin = -1.2
    ...

zip is the tool for parallel arrays. Any time you see this shape, zip is what you want.

Understanding the Aggregation Idea

The core teaching point of today: grouping daily data into monthly buckets.

We fetch ~1,800 daily observations (5 years × ~365 days). We need 12 monthly averages. The pattern:

from collections import defaultdict

buckets_max = defaultdict(list)
buckets_min = defaultdict(list)

for date_str, tmax, tmin in zip(daily["dates"], daily["max"], daily["min"]):
    month = int(date_str[5:7])          # "2020-07-14" -> 7
    if tmax is not None:
        buckets_max[month].append(tmax)
    if tmin is not None:
        buckets_min[month].append(tmin)

avg_max = [sum(buckets_max[m]) / len(buckets_max[m]) for m in range(1, 13)]
avg_min = [sum(buckets_min[m]) / len(buckets_min[m]) for m in range(1, 13)]

Three concepts worth internalizing:

• defaultdict(list) — a dict where any missing key auto-initializes to an empty list. No if month not in buckets: buckets[month] = [] clutter. It just works.

• date_str[5:7] — string slicing to grab the month digits directly. Since Open-Meteo returns dates in YYYY-MM-DD format, positions 5 and 6 are always the month. Fast and no datetime.strptime overhead needed.

• The pattern — “group by X, aggregate within each group” is the most common data pattern in the world. This is pandas.groupby under the hood; this is SQL’s GROUP BY; this is what MapReduce is doing at scale.

Once you see this pattern once, you see it everywhere.

Understanding Safe Aggregation with None Values

Open-Meteo occasionally returns null for a day where the reanalysis is missing (this is very rare, but real). If we don’t guard for it, a None value slipping into a bucket crashes the average calculation.

Two guards handle this:

# Skip Nones when appending
if tmax is not None:
    buckets_max[month].append(tmax)

# Guard against empty buckets (e.g. very short date ranges)
def _avg(bucket):
    return sum(bucket) / len(bucket) if bucket else None

avg_max = [_avg(buckets_max[m]) for m in range(1, 13)]

If a whole month happens to be empty (say, the user picks a very narrow date range), we return None for that slot. Plotly happily renders a gap in the line instead of crashing.

This is the difference between fragile code and robust code. Fragile code assumes the data is perfect. Robust code assumes gaps exist and handles them gracefully. Real-world APIs always have gaps.

Understanding Plotly Chart Basics

We use plotly.graph_objects.Figure for maximum control. The pattern:

import plotly.graph_objects as go

fig = go.Figure()

fig.add_trace(go.Scatter(
    x=MONTHS,
    y=avg_max,
    mode="lines+markers",
    name="Average High",
    line=dict(color="#e11d48", width=3),
    marker=dict(size=8),
    hovertemplate="%{x}: <b>%{y:.1f}°C</b><extra></extra>",
))

fig.add_trace(go.Scatter(
    x=MONTHS,
    y=avg_min,
    mode="lines+markers",
    name="Average Low",
    line=dict(color="#2563eb", width=3),
    marker=dict(size=8),
    hovertemplate="%{x}: <b>%{y:.1f}°C</b><extra></extra>",
))

fig.update_layout(
    title=f"Monthly Average Temperature - {city_label}",
    xaxis_title="Month",
    yaxis_title="Temperature (°C)",
    hovermode="x unified",
    height=500,
)

A few Plotly ideas to know:

• add_trace(go.Scatter(...)) — each trace is one line/series. Add multiple to overlay.

• mode="lines+markers" — shows both the connected line and a dot at each data point.

• hovertemplate — customize what shows up on hover. %{x} and %{y} are the data values; <extra></extra> hides the trace name from the tooltip.

• hovermode="x unified" — the whole reason interactive charts feel great. Hover anywhere on the chart and both traces show their values for that x-position in a single tooltip.

Understanding Streamlit Caching for API Calls

API calls are expensive. If we re-fetched every time the user tweaked the years slider, we’d hit the API 5+ times per interaction. Streamlit’s @st.cache_data fixes this in one line:

@st.cache_data(ttl=86400)      # 86,400 seconds = 1 day
def fetch_daily_history(latitude, longitude, start_date, end_date, timezone):
    ...

Streamlit hashes the input arguments and caches the return value. Next time the function is called with the same arguments, the cached result is returned instantly — no HTTP call.

The ttl=86400 means “invalidate after 1 day” — perfect for historical data that only changes slightly. For live/current data you’d use a much smaller TTL.

Cache the API layer, not the UI. Historical weather doesn’t change; the same lat/lng + date range should return the same result forever. This is exactly the right layer to cache.

Understanding End-Date Choice

There’s a subtle bug you could easily make: fetch data through today’s date. Do that and December 2026 gets averaged in when the user picks a range including 2026, but only using days that happened so far. Then your December average is skewed — it only has data through July 14, missing the coldest half of the month.

The fix: end at the last completed year:

today = date.today()
end_year = today.year - 1
start_year = end_year - years + 1

If today is July 14, 2026, we ask for data through Dec 31, 2025. All months are complete. All averages are fair.

Little details like this are what separate good data apps from misleading ones. Whenever you’re averaging over a time bucket, ask yourself: “do all the buckets have equal coverage?” If not, fix it before the users trust bad numbers.

Understanding st.metric for Summary Cards

Streamlit’s st.metric is one of its best components — a big-number card with an optional caption underneath:

c1, c2, c3, c4 = st.columns(4)
c1.metric("Hottest month",
          stats["hottest_month"],
          f"{stats['hottest_high']:.1f}°C avg high")
c2.metric("Coldest month",
          stats["coldest_month"],
          f"{stats['coldest_low']:.1f}°C avg low")

Three args: label (small text above), value (big text), delta (small caption below). The delta is normally used for “changed by X” arrows, but works nicely as a supplementary caption too.

Four st.metric cards in a st.columns(4) row is the dashboard-header pattern. Almost every data app should start with a row of these.

Practical Use Cases

1. The two-API pattern:

User input → geocoding → main API → chart. Works for weather, air quality, restaurants, real estate.

2. Time-series bucketing:

Daily → monthly, hourly → daily, minute → hourly. Same defaultdict pattern for any resampling.

3. Streamlit-plus-Plotly:

Nearly the smallest way to ship a real interactive data app. Plotly gives you interactivity for free.

4. Handling messy real-world data:

None checks, empty-bucket guards, incomplete-year handling. Real data is always messy.

5. Foundation for the rest of the week:

Tomorrow's daily climatology reuses fetch + geocoding, changes only the aggregation. Friday's multi-city just adds another fetch.

Coming Tomorrow

Tomorrow we go daily. Instead of 12 monthly points, we compute the full 365-day climatology — average across years for every single day of the year. Then we plot the two curves with a filled area between them (the classic WeatherSpark look) and add a vertical “Today” marker. Same UI, dramatically smoother chart.

Skeleton and Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build a World Cup 2026 Match Predictor — a Streamlit web app that predicts upcoming matches using real football-analytics techniques.

• Day 1: Elo-based Predictor

• Day 2: Poisson Goal Model

• Day 3: Ensemble + Interactive Analysis (Today)

View All Projects This Week

Today’s Project

Welcome to the finale — the models learn to talk to each other.

Yesterday, our Poisson model surprised us: France vs Morocco came out as Morocco favored 51%, driven by Morocco’s elite defensive record over the last 3 years. That reveal was uncomfortable — most football fans would pick France without hesitation.

Here’s what we didn’t show yesterday: Elo, our Day 1 model, disagrees with Poisson on that match. Elo (trained on 150 years of history) picks France at 56%. Poisson picks Morocco at 51%. Which one is right?

Both. And neither. They’re capturing different signals — long-term class vs. recent form — and the truth almost always lies between them.

Today we combine them into an ensemble — a weighted average of both models’ probabilities. We also add:

• Confidence indicator — 🟢 High when models agree, 🔴 Low when they don’t

• Click-to-expand analysis — see both models side-by-side, recent form badges (W-W-D-L-W), head-to-head record

Same UI shell as Days 1 and 2. Smarter, more honest predictions underneath.

The Big Idea Behind Ensembles

Ensembling is one of the most important ideas in modern statistics and machine learning: combining multiple models produces predictions that are more accurate than any single model.

Why does this work? Because different models make different kinds of mistakes. If model A tends to overrate historically strong teams and model B tends to overrate recent form, then averaging them cancels out both biases. The classic academic result is that ensembles reduce variance without adding bias — as long as the individual models are reasonably good and independent in their errors, the average almost always beats each one.

Random forests do this by averaging hundreds of decision trees. Gradient boosting stacks weak models. Even the Netflix Prize was won by an ensemble of dozens of models. The idea is universal.

For our app: Elo and Poisson are wonderfully independent. Elo bakes in every result since 1872 and moves slowly; Poisson only sees the last 3 years. When they agree on a match, we should be very confident. When they disagree, we should express uncertainty. That’s what today’s app does.

Project Task

Extend yesterday’s app so it:

• Combines Elo and Poisson predictions into a weighted-average ensemble

• Adds a third outcome (draw probability) to the Elo model so the two are directly comparable

• Computes an agreement score between the two models (total-variation distance)

• Shows a confidence indicator (🟢/🟡/🔴) with a plain-English description

• Adds a click-to-expand section on each match card showing:

  • Both models’ individual predictions in a table

  • Recent form as W-D-L colored badges for each team’s last 5 matches

  • Head-to-head history between the two teams

• Ranks the sidebar by a combined ensemble rank (average of Elo rank + attack rank)

Expected Output

Running the app:

streamlit run predictor_ensemble.py

Now the user has the full picture. France has won 5 straight, but Morocco has drawn Brazil and the Netherlands — both models’ predictions are defensible; the ensemble captures the genuine uncertainty; the confidence indicator communicates that honestly.

Compare with a different match — Norway vs England:

🟢 High confidence — The two models strongly agree.

Both models pick England at ~51%. Agreement 96%. The app tells you when to trust it and when to hedge.

Setup Instructions

No new dependencies:

pip install streamlit requests
streamlit run predictor_ensemble.py

Understanding Adding Draws to Elo

Yesterday’s Elo model gave us just two numbers: P(team1 wins), P(team2 wins). No draws. That’s a real limitation — international football sees roughly 25-30% draws, so pretending they don’t exist inflates the win probabilities.

Adding draws to Elo turns out to be surprisingly simple with a trick: draw probability grows as the teams get closer in rating. Two teams of equal Elo? Around 28% chance of a draw. Two teams with a huge gap? Almost no chance of a draw (the stronger team just wins).

def elo_predict(ratings, team1, team2, neutral=True):
    r1 = ratings.get(team1, INITIAL_ELO) + (0 if neutral else HOME_ADVANTAGE)
    r2 = ratings.get(team2, INITIAL_ELO)

    p1_raw = expected_score(r1, r2)          # standard Elo win probability
    p2_raw = 1.0 - p1_raw

    # How close are the teams? 1.0 = identical, 0.0 = 100% vs 0%
    closeness = 1.0 - abs(p1_raw - 0.5) * 2
    p_draw = 0.28 * closeness

    # Scale down win probabilities to make room for the draw
    p_win1 = p1_raw * (1.0 - p_draw)
    p_win2 = p2_raw * (1.0 - p_draw)

    return p_win1, p_draw, p_win2

Three moves worth understanding:

• closeness = 1.0 - abs(p1_raw - 0.5) * 2 — a clean formula that returns 1.0 when the teams are perfectly balanced (p_raw = 0.5) and 0.0 when it’s a total mismatch (p_raw = 0 or 1). Perfect for scaling draw probability.

• p_draw = 0.28 * closeness — the 0.28 is the “peak draw rate” for perfectly balanced teams. It’s an empirical constant from international football.

• Scaling p_win1 *= (1 - p_draw) — we’re not adding a draw; we’re reallocating probability from wins to draws. That way our three numbers still sum to exactly 1.0.

The three probabilities sum to p1_raw * (1-p_draw) + p_draw + p2_raw * (1-p_draw) = (1-p_draw)(p1_raw + p2_raw) + p_draw = (1-p_draw) + p_draw = 1.0. ✓

Understanding the Ensemble Combination

The ensemble is the simplest possible thing: a weighted average of the two models’ probabilities:

def ensemble_predict(elo_probs, poisson_probs, w_elo=0.5, w_poi=0.5):
    p1 = w_elo * elo_probs[0] + w_poi * poisson_probs[0]
    pd = w_elo * elo_probs[1] + w_poi * poisson_probs[1]
    p2 = w_elo * elo_probs[2] + w_poi * poisson_probs[2]
    return p1, pd, p2

Because both inputs sum to 1.0, and the weights sum to 1.0, the ensemble automatically sums to 1.0 too. No normalization needed. Beautiful.

Choosing the weights matters. 50/50 is a good default — treat both models as equally valid. But you could bias toward one:

• Weight Elo higher if you trust long-term class more (0.7 / 0.3 → Elo-dominant)

• Weight Poisson higher during a hot tournament run when recent form matters more (0.3 / 0.7)

• Learn the weights by testing on past tournaments and picking whatever gave the highest accuracy — the classic ML approach

For teaching purposes, 50/50 is honest: “we don’t know which model is right; we’ll trust them equally.”

Understanding Model Agreement

To measure how much the two models agree, we use total variation distance — a standard measure between two probability distributions:

def model_agreement(elo_probs, poisson_probs):
    tv = 0.5 * sum(abs(e - p) for e, p in zip(elo_probs, poisson_probs))
    return 1.0 - tv

Total variation distance (TV) is half the sum of absolute differences. It ranges from 0 (identical) to 1 (totally opposed). We flip it (1 - tv) so higher = more agreement — reads intuitively.

In practice:

• Both models pick France at 55% → TV ≈ 0.05 → agreement ≈ 95% → 🟢 High confidence

• Elo picks France, Poisson picks Morocco → TV ≈ 0.35 → agreement ≈ 65% → 🔴 Low confidence

The threshold values (0.90, 0.75) that separate green/yellow/red are chosen by feel — the goal is honest communication of uncertainty, not statistical precision.

Understanding Recent Form Badges

To show a team’s last 5 matches, we compute a W/D/L character per match:

def recent_form(team, form_index, n=5):
    matches = form_index.get(team, [])[:n]
    results = []
    for m in matches:
        is_home = m["home_team"] == team
        my_score = int(m["home_score"] if is_home else m["away_score"])
        opp_score = int(m["away_score"] if is_home else m["home_score"])
        opp = m["away_team"] if is_home else m["home_team"]

        if my_score > opp_score: r = "W"
        elif my_score < opp_score: r = "L"
        else: r = "D"

        results.append((r, opp, f"{my_score}-{opp_score}", m["date"]))
    return results

The trick is that each match has a home team and away team, but the team we’re asking about could be either. The is_home = m["home_team"] == team check normalizes: we always want our team’s score and their opponent’s score, regardless of who was nominally at home.

Then we render each result as a colored badge (green W, yellow D, red L) using inline HTML:

colors = {"W": "#22c55e", "D": "#eab308", "L": "#ef4444"}

badges = []
for r, opp, score, dt in form_results:
    badges.append(
        f"<span style='background:{colors[r]}; color:white; "
        f"padding:2px 8px; border-radius:4px; font-weight:bold; "
        f"margin-right:4px;' title='vs {opp} ({score}) on {dt}'>{r}</span>"
    )

The title="..." attribute means users can hover any badge to see the match details. Small UX detail, big polish.

Understanding Building the Recent Form Index

Looking up a team’s recent matches quickly requires an index — a precomputed dict of team → list of matches, sorted newest first:

@st.cache_data(ttl=3600)
def build_recent_form_index(matches):
    idx = defaultdict(list)
    for m in matches:
        if m["date"] < FORM_WINDOW_START:
            continue
        idx[m["home_team"]].append(m)
        idx[m["away_team"]].append(m)
    for team in idx:
        idx[team].sort(key=lambda m: m["date"], reverse=True)
    return dict(idx)

We iterate once through history, adding each match to both teams’ lists. Then sort each list by date descending. Lookup is O(1) — no more scanning through 49,000 matches for every card. Same “walk-once, index, then instant-lookup” pattern that shows up in every real data application.

Understanding the Ensemble Ranking Sidebar

The sidebar now ranks by a combined score — average of Elo rank and attack strength rank:

by_elo = sorted(wc_teams_list, key=lambda t: -ratings[t])
by_atk = sorted(wc_teams_list, key=lambda t: -strength[t]["attack"])

elo_rank = {t: i for i, t in enumerate(by_elo)}
atk_rank = {t: i for i, t in enumerate(by_atk)}

combined = sorted(wc_teams_list,
                  key=lambda t: elo_rank[t] + atk_rank[t])

Ranking by rank (rather than by raw value) is a subtle but important choice. Elo values range from ~1400 to ~2200; attack strengths range from ~0.5 to ~2.0. Averaging them directly would let Elo dominate purely due to scale.

Averaging ranks avoids this entirely. A team ranked 3rd by Elo and 7th by attack has a combined score of 10. A team ranked 1st by Elo and 20th by attack scores 21. Rank-based combinations are dimensionless — no scale bias, no magic normalization constants. Same trick works for any leaderboard that combines fundamentally different metrics.

Understanding Why This Is the Right Finale

Look at the arc of the week:

• Day 1 built the Elo engine — the classic rating system.

• Day 2 added a Poisson goal model — a fundamentally different lens.

• Day 3 combines them into an ensemble with honest uncertainty.

Each layer added information without discarding the previous one. The Day 1 code still runs. The Day 2 code still runs. Day 3 is just a coordinator that talks to both.

This is the shape of every real ML system. You don’t build one model; you build several, and combine them. You don’t discard old models; you keep them running for comparison. You don’t hide model disagreement from users; you show it, because that’s what honest communication of uncertainty looks like.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete World Cup Match Predictor:

• Day 1: Elo rating system trained on 49,000+ historical matches

• Day 2: Poisson goal model with expected goals and W/D/L splits

• Day 3: Ensemble prediction with confidence indicators and click-to-expand analysis

You’ve built a real, working ensemble ML system. That’s not a beginner project. 🚀

View Code Evolution

Compare today’s ensemble with Day 1’s Elo predictor and Day 2’s Poisson model — and see how each layer added a new perspective without discarding the previous one. Same UI, three genuinely different models, one honest prediction.

Read more

This week, we build a World Cup 2026 Match Predictor — a Streamlit web app that predicts upcoming matches using real football-analytics techniques. Each day the same interface stays, but the model behind it gets smarter.

• Day 1: Elo-based Predictor

• Day 2: Poisson Goal Model (Today)

• Day 3: Ensemble + Interactive Analysis

View All Projects This Week

Today’s Project

Yesterday’s Elo model was good at one thing: telling us who is likely to win. But football fans care about more than that — they want to know how it’ll go. Will it be a rout or a close game? What’s the most likely scoreline? What’s the chance of a draw?

Today we add all of that. The Poisson goal model treats each team’s goals as random events drawn from a probability distribution, with rates estimated from recent form. From that we get:

• Expected goals (λ) for each team — the rate parameter

• A full grid of scoreline probabilities

• W / D / L split — draws now have their own line

• The most likely exact scoreline

Same Streamlit UI, better answers.

The Big Idea Behind Poisson

Football goals aren’t uniformly distributed. Teams don’t score exactly 1.5 goals per match — they score 0, 1, 2, sometimes 5, occasionally none. But the average rate over many matches is measurable, and the distribution of individual match totals around that average follows a very specific pattern.

That pattern is the Poisson distribution — the standard model for “count of independent events over a fixed period.” Radioactive decay, phone calls per hour to a call center, and football goals per team per match all follow the same shape.

The Poisson distribution says: if a team’s average scoring rate is λ (lambda), the probability they score exactly k goals in one match is:

$$P(k) = \frac{e^{-\lambda} \lambda^k}{k!}$$

For a team averaging 1.5 goals per match:

• Probability of 0 goals: 22%

• Probability of 1 goal: 33%

• Probability of 2 goals: 25%

• Probability of 3 goals: 13%

• Probability of 4 goals: 5%

Sums to essentially 100% (the tail is tiny). That’s it — one number (λ) fully describes a team’s scoring profile.

Project Task

Extend yesterday’s app so it:

• Estimates each team’s attack strength (goals scored vs league average) and defense strength (goals conceded vs league average) from recent matches

• Computes expected goals (λ) for each team in a matchup

• Builds a scoreline probability matrix using two independent Poisson distributions

• Sums the matrix into W / D / L probabilities

• Finds the most likely exact scoreline

• Displays all of the above in the same card-based Streamlit UI as Day 1

• Shows a sidebar of top-20 WC 2026 teams by attack strength

Expected Output

Running the app:

streamlit run predictor_poisson.py

Why is Morocco predicted to win here?

The screenshot above shows France vs Morocco in the Quarter-final. The app predicts Morocco to win at 50.5%, with a most likely scoreline of 0–1. If you’re a football fan you’re probably surprised — France is clearly the “bigger” team, right? So what’s going on?

The Poisson model looks at each team’s recent scoring form since 2023, not their historical prestige. And when you look at the actual numbers:

• France has an attack strength of 1.63 (63% above the international average of 1.4 goals/match) and a defense strength of 0.59.

• Morocco has an attack strength of 1.53 — very close to France — but a defense strength of 0.29. They’ve conceded just 23 goals in 56 matches since 2023.

That defense number is the whole story. Morocco has been exceptionally stingy defensively — including drawing Brazil 1-1 and Netherlands 1-1 in the group stage, and beating Canada 3-0 in the Round of 16. When we multiply France’s attack × Morocco’s tiny defense number, France’s expected goals collapses to 0.67. Meanwhile Morocco’s attack × France’s more permissive defense gives Morocco 1.26 expected goals.

The prediction reflects a specific truth: over the last 3 years, Morocco’s defensive record is elite, and France’s attack has been less prolific than reputation suggests. Whether that translates to a real Morocco win is up to Wednesday — the model captures a real signal but ignores things like tournament pressure and historical class.

This tension between “recent form” and “long-term class” is exactly why we’re building an ensemble on Day 3. Yesterday’s Elo model, which uses every match since 1872, still favors France at 68%. When the two models disagree like this, the truth usually lies in the middle.

Setup Instructions

Nothing new to install — same dependencies as Day 1.

pip install streamlit requests
streamlit run predictor_poisson.py

Understanding Team Attack and Defense Strength

Before the Poisson math can happen, we need to estimate each team’s scoring rate. But teams don’t play in a vacuum — a 3-0 win is impressive against a strong opponent, meaningless against a minnow. The trick is to normalize by opponent strength using multiplicative factors:

def compute_team_strength(matches):
    recent = [m for m in matches if m["date"] >= FORM_WINDOW_START]

    scored = defaultdict(list)
    conceded = defaultdict(list)

    for m in recent:
        hg, ag = int(m["home_score"]), int(m["away_score"])
        home, away = m["home_team"], m["away_team"]
        scored[home].append(hg)
        scored[away].append(ag)
        conceded[home].append(ag)
        conceded[away].append(hg)

    strength = {}
    for team in scored:
        avg_scored = sum(scored[team]) / len(scored[team])
        avg_conceded = sum(conceded[team]) / len(conceded[team])
        strength[team] = {
            "attack":  avg_scored / LEAGUE_AVG_GOALS,
            "defense": avg_conceded / LEAGUE_AVG_GOALS,
        }

    return strength

The output is two multipliers per team:

• attack = 1.0 — you score at league average.

• attack = 1.5 — you score 50% more than league average.

• defense = 0.7 — you concede 30% less than league average (defense good; the lower the number, the tighter the defense).

Real numbers from live data: Japan attack 2.02, defense 0.53 (dominant scoring, stingy defense). Spain attack 1.82, defense 0.53 (elite). These are relative to the international average of 1.4 goals per team per match.

Understanding the Sample Size Guard

There’s a classic gotcha here that will bite you the first time you build a ranking like this: teams with very few matches will dominate the top of the list. If you rank the whole world of national teams by attack strength, the top 20 gets flooded with tiny CONIFA teams — Isle of Man, Tamil Eelam, Jersey, Sápmi — all of whom play maybe 6-10 matches every few years against similarly small teams and end up with wildly inflated attack numbers.

The naive fix is a minimum-matches filter (s["matches"] >= 30). But we can do something cleaner and more relevant to our specific app: only show teams that are actually at the World Cup 2026.

def wc_teams(fixtures):
    """The set of real (non-placeholder) team names in the WC 2026 fixtures."""
    teams = set()
    for m in fixtures["matches"]:
        for name in (m["team1"], m["team2"]):
            if not is_placeholder(name):
                teams.add(name)
    return teams

# In the sidebar:
qualifying = {t: s for t, s in strength.items() if t in wc_team_set}
top = sorted(qualifying.items(), key=lambda x: -x[1]["attack"])[:20]

Two lines. But this is a much better solution than a raw sample-size filter for two reasons:

• It’s semantically correct. This is a World Cup app; only WC teams belong in the sidebar. Isle of Man’s attack strength is irrelevant to anyone using this tool.

• It solves the noise problem automatically. Every WC team has played ~40+ international matches since 2023 (they had to qualify, they play friendlies, they’re in continental competitions). The 48 WC teams are automatically the sample-size-clean subset.

The lesson: the right filter is often a domain filter, not a technical one. Instead of “teams with 30+ matches” (arbitrary threshold), we filter by “teams that matter for this specific view” (natural, defensible boundary). Any time you’re about to reach for a magic number to filter noise, ask first if there’s a domain fact that does the same job cleanly.

We still compute strengths for every team (Morocco’s defense number needs their whole history, not just their WC record) — we’re just choosing not to display the ones that aren’t relevant.

Understanding the Recent-Form Window

We only look at matches since a cutoff date:

FORM_WINDOW_START = "2023-01-01"

Why? Because a team’s 2010 attack strength doesn’t tell us much about their 2026 form. Coaches change, players retire, tactics evolve. Three years is the sweet spot for international football — enough matches to be meaningful (~40-50 per top team), recent enough to reflect current reality.

Compare this with Day 1’s Elo, which used every match ever. That’s the right choice for Elo — the rating evolves gradually with each result, so the current rating already captures recent form. For Poisson, we need a fresh per-team average, so a rolling window works better.

Different models want different data. That’s the meta-lesson of doing two models.

Understanding Expected Goals (Lambda)

Given both teams’ attack and defense multipliers, the expected goals for a matchup is a three-way product:

def expected_goals(team1, team2, strength):
    s1 = strength[team1]
    s2 = strength[team2]
    lambda1 = s1["attack"] * s2["defense"] * LEAGUE_AVG_GOALS
    lambda2 = s2["attack"] * s1["defense"] * LEAGUE_AVG_GOALS
    return lambda1, lambda2

The formula in plain English:

Team1’s expected goals = (how well they score) × (how easy team2 is to score against) × (the average goals in a match).

For France vs Morocco: France’s expected goals = France’s attack (1.63) × Morocco’s defense (0.29) × 1.4 = 0.67 (very low, because Morocco is so hard to score against). Morocco’s expected goals = Morocco’s attack (1.53) × France’s defense (0.59) × 1.4 = 1.26. Morocco is the Poisson favorite despite being lower-Elo.

This is why the models disagree. Elo bakes in the whole history; Poisson listens to what’s happened recently. Both perspectives are legitimate.

Understanding the Poisson PMF

The probability mass function for a Poisson distribution:

from math import exp, factorial

def poisson_pmf(k, lam):
    """Probability of exactly k events when the rate is lam."""
    return exp(-lam) * (lam ** k) / factorial(k)

That’s the whole math. Three built-in Python operations: exp, **, factorial. No NumPy, no SciPy, no dependencies beyond stdlib.

For λ = 1.5:

• poisson_pmf(0, 1.5) = 0.223 → 22% chance of 0 goals

• poisson_pmf(1, 1.5) = 0.335 → 33% chance of 1 goal

• poisson_pmf(2, 1.5) = 0.251 → 25% chance of 2 goals

• poisson_pmf(3, 1.5) = 0.126 → 13% chance of 3 goals

Sum: 93% by k=3, essentially 100% by k=5. Football scores are dominated by low numbers, and Poisson captures that shape naturally.

Understanding the Scoreline Matrix

To predict a match outcome, we build a joint probability grid — every possible scoreline gets a probability:

def score_matrix(lambda1, lambda2, max_goals=8):
    matrix = {}
    for g1 in range(max_goals + 1):
        p1 = poisson_pmf(g1, lambda1)
        for g2 in range(max_goals + 1):
            p2 = poisson_pmf(g2, lambda2)
            matrix[(g1, g2)] = p1 * p2
    return matrix

The critical assumption: team1’s goals and team2’s goals are independent. So P(1-0) = P(team1 scores 1) × P(team2 scores 0).

For most matches this assumption holds well enough. Real matches have some correlation (a team that’s winning big might ease off; a team that’s losing might risk more) but the independence assumption is a reasonable first-order model. Dixon and Coles’ famous 1997 improvement adjusted for exactly this correlation in 0-0 and 1-1 outcomes; you can bolt it on later.

max_goals=8 is our practical upper bound. The probability of any score beyond 8-8 is astronomically tiny; ignoring it costs us less than 0.01% of the total probability.

Understanding the Most-Likely Scoreline

Finding the single most-probable scoreline is a one-liner using max with a key:

def most_likely_score(matrix):
    return max(matrix, key=matrix.get)

matrix.get is the accessor that returns the probability for a given key; max(..., key=fn) returns the key whose function-value is largest. Python at its most elegant.

For Argentina vs Egypt (Argentina expected goals 1.30, Egypt 0.50), the top 5 scorelines are:

Score Probability 1-0 21.4% 0-0 16.5% 2-0 13.9% 1-1 10.8% 0-1 8.3%

The most likely single score is 1-0, but even the most likely score only happens 21% of the time — football is inherently high-variance. That’s why we care about probability distributions and not just point predictions.

Understanding the Model’s Limitations

Every model has weaknesses. Poisson’s are worth naming — and are exactly why the France vs Morocco prediction feels off:

1. Independence assumption: teams’ goals aren’t perfectly independent. Real matches have correlation — winning teams ease off, losing teams take risks. Dixon-Coles (1997) fixes this by adjusting the corners of the matrix.

2. Opponent quality isn’t weighted: we compute attack as raw goals scored / league average, without weighting by who was defended against. So a team that mostly plays weaker opponents will look artificially strong. Morocco’s 5-0 friendly rout of Burundi counts the same as their 1-1 vs Brazil.

3. Match importance isn’t weighted: friendlies count the same as World Cup matches. A team that runs up scores in exhibitions inflates its attack rating.

4. No context: injuries, tournament pressure, weather — none of it’s in the model.

Every one of these is fixable with more work. But even this vanilla Poisson model produces ~62% match-outcome accuracy in the football-analytics literature. Good baseline. The interesting question is what happens when we combine it with Elo — which has different blind spots.

Understanding Comparing the Models

Here’s what makes tomorrow interesting. Look at these live matchups from right now:

Match Elo says Poisson says France vs Morocco France 68% Morocco 51% Norway vs England England 64% England 51% Spain vs Belgium Spain 72% Spain 45% / Draw 25% / Belgium 30% Argentina vs Switzerland Argentina 78% Argentina 63%

Elo is confident. Poisson is more uncertain. And sometimes they disagree entirely on the favorite.

Which one is right? Both, and neither. Elo sees the long game (decades of results); Poisson listens to recent form (this year’s goals). When they agree, we can be confident. When they disagree, we should probably say so.

That’s the whole motivation for tomorrow’s ensemble — combining the two models into one prediction, and showing the user when the models are aligned vs when they’re arguing.

Coming Tomorrow

Tomorrow the two models meet. We build an ensemble — a weighted average of Elo and Poisson probabilities — that outperforms either alone. And we add interactivity: click any match card and expand it to see both model’s answers side-by-side, the recent form of each team (W-W-D-L-W last 5), and a confidence indicator that tells you when the models agree vs when they’re arguing. The finale is where it all clicks.

View Code Evolution

Compare today’s Poisson predictor with Day 1’s Elo predictor and see how a fundamentally different mathematical model plugs into the same UI shell — because we designed the UI to be model-agnostic from day one.

Read more

This week, we build a World Cup 2026 Match Predictor — a Streamlit web app that predicts upcoming matches using real football-analytics techniques. Each day the same interface stays, but the model behind it gets smarter. Day 1 is Elo, Day 2 adds a Poisson goal model, Day 3 combines them into an ensemble with interactive analysis.

Why build this? Because the World Cup 2026 is happening right now — Round of 16 starts July 4. This is the perfect moment to build a real prediction app that answers questions your friends actually care about. Plus the techniques (Elo, Poisson, ensembles) show up in far more places than football: chess ratings, video-game matchmaking, insurance risk, sports betting, any pairwise-competition domain.

What you’ll learn: This series teaches you Streamlit web apps, the Elo rating system, Poisson distributions applied to goal modeling, ensemble modeling, working with two data sources at once, and turning statistical models into a user-facing interface.

Why this matters: By Friday, you’ll have a real prediction tool that pulls live data and produces defensible predictions — with a UI polished enough to send to friends.

• Day 1: Elo-based Predictor (Today)

• Day 2: Poisson Goal Model

• Day 3: Ensemble + Interactive Analysis

View All Projects This Week

Today’s Project

We start with Elo ratings — the same system Arpad Elo invented in the 1950s for chess, now widely used in football analytics. Elo is beautiful because it’s simple: every team has a number, higher = better, and after each match the winner takes a bit of the loser’s rating.

Our app fetches 49,000+ international matches since 1872, computes an Elo rating for every national team, then displays upcoming World Cup 2026 matches with a “who wins %” prediction based on the rating difference.

About the Data

Two free, public data sources — both on GitHub, no API keys.

Historical results: martj42/international_results

• A CSV with 49,000+ international matches from 1872 to today

• Columns: date, home_team, away_team, home_score, away_score, tournament, city, country, neutral

• Includes friendlies, qualifiers, Copa América, Euros, World Cups — everything

Upcoming fixtures: openfootball/worldcup.json (same source as Week 24)

• Live World Cup 2026 fixtures and results

• Placeholder codes resolve to real teams as the bracket fills in

Both are public domain, updated daily by hand, and used by dozens of real projects.

Project Task

Build a Streamlit app that:

• Fetches historical international match results

• Fetches the live World Cup 2026 fixture list

• Computes an Elo rating for every national team from scratch

• Filters the fixture list to just upcoming matches with resolved teams

• Displays each upcoming match as a card with team names, Elo ratings, and prediction bars

• Shows the top 20 teams by Elo rating in a sidebar

• Uses Streamlit’s caching so the 49k-match walk only happens once per session

This project gives you hands-on practice with the Elo rating algorithm, Streamlit’s card/column/progress components, @st.cache_data, and the classic “walk historical data to build a state” pattern that shows up everywhere in analytics.

Expected Output

Running the app:

streamlit run predictor_elo.py

The app opens in your browser and you will see all the upcoming matches:

Setup Instructions

Install dependencies:

pip install streamlit requests

Run it:

streamlit run predictor_elo.py

Your browser opens automatically at http://localhost:8501.

First run downloads ~3.6MB of historical data. Subsequent runs (or interactions) use Streamlit’s cache — instant.

Understanding the Elo Rating System

Elo is the simplest useful rating system you’ll ever meet. Every team has a rating (starting at 1500 by convention). After each match, the winner’s rating goes up, the loser’s goes down. The magnitude depends on:

• The rating gap. Beating a much stronger team = large gain. Beating a much weaker team = tiny gain.

• The margin. A 4-1 win moves your rating more than a 1-0 win.

• Match importance. World Cups matter more than friendlies.

The core formula gives you the expected result of a matchup:

def expected_score(rating_a, rating_b):
    return 1.0 / (1.0 + 10.0 ** ((rating_b - rating_a) / 400.0))

That’s it. If A and B are equal, the formula returns 0.5. If A is 400 points ahead, A’s expected score is ~91%. Every 400 Elo points = 10x more likely to win. This is the formula behind chess ratings, video-game matchmaking, and half the football-analytics world.

Understanding the Update Rule

After the match, we compare what actually happened to what Elo expected:

actual_home = 1.0 if home_won else 0.0 if away_won else 0.5
expected_home = expected_score(home_rating, away_rating)

change = K_FACTOR * (actual_home - expected_home)
ratings[home] += change
ratings[away] -= change

Three key ideas:

• actual - expected is the surprise. If a strong team beats a weak team as predicted, surprise is small, change is small. If a weak team wins as an underdog, surprise is huge, change is big.

• The K-factor controls speed. K=30 is standard for football. Smaller = ratings move slowly (more stable), larger = ratings react faster (more responsive).

• What one team gains, the other loses — Elo is a zero-sum system. Total rating across all teams stays constant.
  

Understanding Walking the Historical Data

The whole ratings dict is built by replaying history — walking every match in chronological order, updating ratings match-by-match:

def compute_ratings(matches):
    ratings = defaultdict(lambda: INITIAL_ELO)

    for m in matches:
        home, away = m["home_team"], m["away_team"]
        hg, ag = int(m["home_score"]), int(m["away_score"])
        neutral = m["neutral"] == "TRUE"

        home_adj = ratings[home] + (0 if neutral else HOME_ADVANTAGE)
        actual_home = 1.0 if hg > ag else 0.0 if hg < ag else 0.5
        expected_home = expected_score(home_adj, ratings[away])
        g = goal_diff_multiplier(hg - ag)

        change = K_FACTOR * g * (actual_home - expected_home)
        ratings[home] += change
        ratings[away] -= change

    return dict(ratings)

Two clean patterns:

• defaultdict(lambda: INITIAL_ELO) — any team we haven’t seen before automatically gets a starting rating of 1500. No pre-initialization, no if team in ratings checks.

• CSV order = chronological. The martj42 CSV is already sorted by date. Just walk it top to bottom and history unfolds naturally. This “replay events to build state” pattern is at the heart of every event-sourced system in the software world — banking ledgers, git commit histories, database write-ahead logs. All the same shape.

Understanding Making the Prediction

Once we have the ratings dict, predicting a match is one call to expected_score:

def predict_match(ratings, team1, team2, neutral=True):
    r1 = ratings.get(team1, INITIAL_ELO)
    r2 = ratings.get(team2, INITIAL_ELO)
    if not neutral:
        r1 += HOME_ADVANTAGE
    p1 = expected_score(r1, r2)
    return p1, 1.0 - p1

Note we use .get(team, INITIAL_ELO) — if a team somehow isn’t in our history (very rare), it gets a neutral 1500. Never crash, always return a defensible number.

Understanding Streamlit Cards, Columns, Progress Bars

The whole prediction card is a few widget calls:

with st.container(border=True):
    st.caption(f"**{match['round']}**  •  {match['date']}  •  {match['ground']}")

    col1, col2, col3 = st.columns([3, 1, 3])
    with col1:
        st.markdown(f"### {team1}")
        st.caption(f"Elo: **{r1:.0f}**")
    with col2:
        st.markdown("<div style='text-align:center;padding-top:20px;'><b>vs</b></div>",
                    unsafe_allow_html=True)
    with col3:
        st.markdown(f"### {team2}")
        st.caption(f"Elo: **{r2:.0f}**")

    st.markdown(f"**{team1}**: {p1*100:.1f}%")
    st.progress(p1)
    st.markdown(f"**{team2}**: {p2*100:.1f}%")
    st.progress(p2)

Three concepts:

• st.container(border=True) — draws an outlined card. Everything inside is a group.

• st.columns([3, 1, 3]) — a row split into three columns with proportions 3:1:3.

• st.progress(value) — a filled progress bar, value is 0.0–1.0.

Streamlit’s superpower: this whole UI is written in Python, no HTML/CSS needed. And it looks polished by default. That’s what makes it perfect for prototyping data apps.

Understanding Filtering Upcoming Matches

We only want to predict matches that (1) haven’t been played and (2) have real team names (not "W73" bracket placeholders):

def upcoming_matches(all_matches):
    today_str = date.today().isoformat()
    upcoming = []
    for m in all_matches:
        if "score" in m:                                 # already played
            continue
        if m.get("date", "9999") < today_str:            # past-and-unplayed data lag
            continue
        if is_placeholder(m["team1"]) or is_placeholder(m["team2"]):
            continue
        upcoming.append(m)

    upcoming.sort(key=lambda m: (m.get("date", ""), m.get("time", "")))
    return upcoming

Three filters, each catching a real edge case:

• Skip played matches — no point predicting what we already know.

• Skip past-dated unplayed matches — sometimes the openfootball data lags for a day.

• Skip bracket placeholders — no point predicting "Winner of match 74" vs "Winner of match 75". Wait for those to resolve.

As Round of 16 finishes, placeholder slots in Quarter-finals fill in, and new match cards appear in the app. The prediction set grows organically as the tournament progresses. No manual updates.

Understanding the Data Flow

The whole app is a small pipeline:

1. Fetch — cached HTTP calls to two GitHub URLs

2. Compute — walk 49,000 matches, build the ratings dict

3. Filter — narrow WC 2026 fixtures to upcoming-with-real-teams

4. Render — one card per fixture, one bar per team

Everything downstream of the ratings dict is just presentation. The intelligence is in step 2. Day 2 and Day 3 will change only that step — same UI, same filtering, but a smarter model behind the scenes.

Coming Tomorrow

Tomorrow the model gets smarter. Elo tells us who is likely to win but not by how much — a 4-1 rout has the same win probability as a 1-0 nail-biter. Poisson distributions fix this by modeling goals as random events with team-specific rates. By Wednesday, our app predicts the most likely scoreline, expected goals for each team, and a full W / D / L breakdown (not just win-or-lose).

Skeleton and Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build a World Cup 2026 Tracker — a three-day project that turns a free public football API into your own personal tournament dashboard.

• Day 1: Fetch and Show the Knockout Phase

• Day 2: Bracket Printer (Terminal Art)

• Day 3: Visual Bracket Web App (Today)

View All Projects This Week

Today’s Project

Welcome to the finale — the tournament goes to the browser.

The terminal versions from Days 1 and 2 were useful, but nobody can share a terminal screenshot. Today we make a real web app: a Flask backend that serves the live match data, and a single-page HTML front-end that renders the whole 32-team knockout bracket visually, with color-coded winners, live match highlighting, and auto-refresh every 60 seconds.

Open it in your browser. Send it to a friend. Deploy it. This is the portfolio piece.

Project Task

Build a two-piece web app:

Flask backend (worldcup_web.py):

• Serves an HTML page at /

• Exposes the live match data at /api/matches as JSON

• Caches the upstream openfootball fetch (so multiple visitors don’t hammer their server)

• Handles fetch failures gracefully

Single-page frontend (templates/index.html):

• Fetches from /api/matches

• Renders the 32 knockout matches as a bracket-style grid using CSS Grid

• Highlights winners in green

• Highlights today’s matches with a bright yellow “TODAY” badge

• Dims placeholder team names (like “Winner of #74”)

• Shows scores including extra time and penalty shootouts

• Auto-refreshes every 60 seconds

This project gives you hands-on practice with Flask (routes, templates, JSON responses), CSS Grid, fetch() + async/await in JavaScript, and porting Python logic to JavaScript.

Expected Output

Running the server:

python worldcup_web.py

🏆 World Cup Bracket running at http://127.0.0.1:5000
 * Serving Flask app 'worldcup_web'
 * Running on http://127.0.0.1:5000

Open http://127.0.0.1:5000 in your browser:

The user can see up-to-date matches and their results. The match cards are clickable. Clicking them will show more information about the match in a modal:

Setup Instructions

Install one new dependency:

pip install flask

Create the project structure:

your-project/
├── worldcup_web.py          # the Flask server
└── templates/
    └── index.html           # the single-page frontend

Note we have an html file which is located inside a templates directory.

Run:

python worldcup_web.py

Then open http://127.0.0.1:5000 in your browser.

Understanding Two-Piece Web Apps

Every modern web app splits into two pieces:

1. The backend — a server that returns data (usually as JSON)

2. The frontend — HTML/CSS/JS that fetches the data and renders it

The pieces talk to each other over HTTP. Our backend has two routes:

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/matches")
def api_matches():
    data = get_matches()
    return jsonify({"ok": True, "matches": data["matches"], ...})

The first serves the HTML page (the frontend). The second serves the data (JSON). The frontend loads first, then it fetches from the second route to fill itself in.

This split scales endlessly: a single backend can serve a website, a mobile app, a CLI, and third-party integrations. Today’s tiny two-route Flask app is the exact shape of billion-dollar SaaS backends.

Understanding Flask Basics

Flask is Python’s most-loved web framework. The core is three concepts:

from flask import Flask, render_template, jsonify

app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/matches")
def api_matches():
    return jsonify({"ok": True, "matches": [...]})

if __name__ == "__main__":
    app.run(debug=True, port=5000)

• Flask(__name__) — creates the app. The __name__ tells Flask where your project’s root is (for finding the templates/ folder etc).

• @app.route("/path") — decorator that registers the function below as the handler for that URL.

• render_template("index.html") — Flask looks in templates/ and returns the file’s contents. (It can also do template variables, but we don’t need that today.)

• jsonify({...}) — serializes the dict to JSON and sets the Content-Type header correctly. Never do return json.dumps(...) — always use jsonify.

Six lines is a real, working web server.

Understanding Server-Side Caching

Our backend hits openfootball’s server every time someone loads the page. If 100 people had the page open, and it polls every 60 seconds, that’s 100 requests per minute to a static JSON file. Rude.

The fix: cache the upstream response for 60 seconds in the server’s memory. Regardless of how many browsers hit us, we hit openfootball at most once a minute:

CACHE_SECONDS = 60
_cache = {"data": None, "fetched_at": 0}

def get_matches():
    now = time.time()
    if _cache["data"] is None or now - _cache["fetched_at"] > CACHE_SECONDS:
        response = requests.get(DATA_URL, timeout=10)
        response.raise_for_status()
        _cache["data"] = response.json()
        _cache["fetched_at"] = now
    return _cache["data"]

Two-key dict, one check, one write. That’s the whole cache.

The layered cache picture: each browser polls our Flask backend every 60 seconds. Our backend polls openfootball at most once per 60 seconds. Result: openfootball sees one request per minute, regardless of how many users we have. This is the exact shape of every real API service — layered caching all the way up.

Understanding CSS Grid for the Bracket

The bracket needs six columns of matches side by side. CSS Grid is born for this:

.bracket {
  display: grid;
  grid-template-columns: repeat(6, minmax(220px, 1fr));
  gap: 24px;
  overflow-x: auto;
}

Three properties:

• display: grid — this container is a grid.

• grid-template-columns: repeat(6, minmax(220px, 1fr)) — six columns, each at least 220px wide but growing to share available space equally.

• overflow-x: auto — if the viewport is too narrow, allow horizontal scroll instead of squishing.

Then each column is a flexbox stacking match cards vertically:

.round-matches {
  display: flex;
  flex-direction: column;
  justify-content: space-around;
  gap: 10px;
}

justify-content: space-around gives each match card equal breathing room — which visually approximates how bracket lines would space matches to feed into the next round. No SVG lines needed.

Understanding Fetch and Async/Await

To get data from our own backend, the frontend uses JavaScript’s fetch():

async function fetchAndRender() {
  try {
    const response = await fetch("/api/matches");
    const data = await response.json();

    if (!data.ok) {
      showError(data.error);
      return;
    }

    renderBracket(data.matches);
  } catch (err) {
    showError(err.message);
  }
}

fetch returns a Promise. await waits for it to resolve. .json() also returns a Promise (parsing the body is async), so we await that too.

Three modern-JS habits worth internalizing:

• async on the function enables await inside it.

• await in front of every Promise-returning call — the code reads sequentially even though it’s asynchronous.

• try/catch — network errors are exceptions in async code, catch them the same way as sync errors.

This is the pattern for every frontend API call. Once you have it, you can talk to any HTTP API from a browser.

Understanding Auto-Refresh with setInterval

The page should update itself when new match results come in. setInterval schedules a repeating function call:

const REFRESH_MS = 60 * 1000;   // 60 seconds

fetchAndRender();                // initial render
setInterval(fetchAndRender, REFRESH_MS);

That’s the whole auto-refresh mechanism. One line does what a whole “reload the page” workflow would.

Because our backend caches upstream fetches for 60 seconds and the frontend polls every 60 seconds, the maximum freshness delay is 120 seconds. That’s plenty for a tournament where matches take 90 minutes.

Understanding Porting Python to JavaScript

Days 1 and 2 built is_placeholder, pretty_team, winner_of, and score-formatting functions in Python. The frontend needs them all — because it’s the one drawing the cards.

Rewriting them in JavaScript is a great exercise in seeing how the languages relate:

// Python:
//   if name.startswith("W") and name[1:].isdigit():
//       return "Winner of #" + name[1:]

// JavaScript:
if (/^W\d+$/.test(name)) return "Winner of #" + name.slice(1);

re.match in Python, RegExp.test in JavaScript. name[1:] in Python, name.slice(1) in JavaScript. Same ideas, different syntax — like learning to write left-handed after a career of writing with your right.

Same for winner_of:

function winnerOf(match) {
  if (!match.score) return 0;
  const s = match.score;
  if (s.p) return s.p[0] > s.p[1] ? 1 : (s.p[1] > s.p[0] ? 2 : 0);
  if (s.et && s.et[0] !== s.et[1]) return s.et[0] > s.et[1] ? 1 : 2;
  if (s.ft[0] !== s.ft[1]) return s.ft[0] > s.ft[1] ? 1 : 2;
  return 0;
}

Same logic, JavaScript syntax. Every language has some form of if, array indexing, comparison. Once you know one language deeply, the second one is 90% translation.

Understanding Rendering a Match Card

The heart of the frontend is renderMatch(match, todayStr) — takes a match dict, returns a chunk of HTML. Built with a template literal (multi-line string with ${expression} interpolation):

function renderMatch(match, todayStr) {
  const played = !!match.score;
  const isToday = match.date === todayStr;
  const w = winnerOf(match);

  // ...build class names, score values, etc.

  return `
    <div class="${cardClasses.join(' ')}">
      <div class="match-meta">
        <span class="match-num">${num}</span>
        <span>${dateHtml}</span>
      </div>
      <div class="team">
        <span class="${t1Cls.join(' ')}">${prettyTeam(match.team1)}</span>
        <span class="${s1Cls}">${s1}</span>
      </div>
      <div class="team">
        <span class="${t2Cls.join(' ')}">${prettyTeam(match.team2)}</span>
        <span class="${s2Cls}">${s2}</span>
      </div>
    </div>
  `;
}

The ${...} slots inject dynamic values. The resulting HTML string gets stitched together with matches.map(renderMatch).join('') and injected into the DOM with element.innerHTML = ....

This is “template literal” rendering — the simplest form of client-side templating. React, Vue, Svelte add sophistication (reactive updates, components, virtual DOM) but at the core they’re doing the same thing: turn state into HTML.

Understanding the Full Data Flow

Let’s trace one match result appearing on your screen, from data source to green pixels:

1. A referee blows the final whistle in Boston. Paraguay beats Germany 4-3 on penalties.

2. A few hours later, someone updates the openfootball JSON on GitHub.

3. Within 60 seconds, our Flask cache times out; on the next visitor request it re-fetches the JSON.

4. Within another 60 seconds, every open browser tab polls /api/matches and receives the fresh data.

5. renderBracket() runs; winnerOf() returns 2 for Paraguay; renderMatch() builds a card with class="team-name winner" on Paraguay’s name.

6. element.innerHTML = ... swaps in the new HTML; CSS applies --win (green) to Paraguay’s name.

Nowhere in that pipeline is anyone manually editing anything. From “match ends” to “your bracket updates” is fully automatic. That’s the shape of every modern data-driven web app.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete, deployable World Cup Bracket Tracker:

• Day 1: Fetch from a live public API, parse JSON, print the knockout stage

• Day 2: Style the terminal output with rich — panels, colors, winner highlighting

• Day 3: Wrap it in a Flask web app with a visual bracket UI and auto-refresh

Next steps:

• Deploy it: Render or Railway hosts Flask apps for free

• Add a matchup detail modal: click a card, see scorers

• Add team logos: openfootball has a companion repo with country flags

• Add group standings: reuse Day 1 logic from the older version

• Build a mobile-first view: the CSS Grid becomes a vertical stack on small screens

• Swap Flask for FastAPI to try async

You’ve built the foundation for a real full-stack data web app. 🚀

View Code Evolution

Get the code of this project down below and compare today’s web app with Day 1’s plain-text output and Day 2’s colored terminal panels:

Read more

This week, we build a World Cup 2026 Tracker — a three-day project that turns a free public football API into your own personal tournament dashboard.

• Day 1: Fetch and Show the Knockout Phase

• Day 2: Bracket Printer (Terminal Art) (Today)

• Day 3: Visual Bracket Web App

View All Projects This Week

Today’s Project

This World Cup has been quite unexpected so far. Just yesterday, two strong teams were eliminated. We continue our world cup Python project and today we make it look like a tournament.

We will use the Python rich library which turns any Python script’s terminal output into something that could pass for a designed UI. We’ll wrap each round in a colored panel, highlight winners in green, dim upcoming placeholder matches, mark today’s matches in yellow, and add a running “played / total” counter for each round.

Same data, dramatically better presentation. The only new dependency is rich.

Project Task

Extend yesterday’s script so it:

• Wraps each knockout round in a colored rich Panel

• Uses a different color for each round (blue → cyan → magenta → yellow → gold)

• Highlights the winning team’s name in bold green

• Dims placeholder team names (like “Winner of #74”) in italic gray

• Marks today’s matches with a bright yellow “TODAY” tag instead of “vs”

• Shows a “played / total” counter as each panel’s subtitle

• Includes a header banner and a legend

This project gives you hands-on practice with the rich library — Panels, Tables, Text styling — plus a small helper for detecting winners from the score object.

Expected Output

Running the script:

python worldcup_bracket.py

Output (rendered in your terminal with actual colors):

Same data as Day 1. Vastly better presentation.

Setup Instructions

Install the new dependency:

pip install rich

rich is the only new package. It works on Windows, Mac, and Linux terminals with no configuration.

Run it:

python worldcup_bracket.py

Understanding rich

rich is a Python library for making terminal output beautiful. It handles:

• Colors and text styles

• Panels (bordered boxes)

• Tables (aligned columns)

• Progress bars, spinners, live updates

• Markdown, syntax-highlighted code, tracebacks

We’re using a small subset today: Console (for printing), Panel (the bordered boxes), Table.grid (for aligned columns), and Text (styled strings). The whole library is thousands of features deep; you can go as far as you want with it.

The pattern is:

from rich.console import Console
from rich.panel import Panel
from rich.text import Text

console = Console()
console.print(Panel(Text("Hello!", style="bold green"), title="Greeting"))

Every console.print() renders to the terminal with full styling. Regular print() still works alongside it.

Understanding rich.Text vs f-strings

Yesterday we used f-strings for alignment. rich prefers you to build Text objects instead — they carry style information separately from the raw text:

from rich.text import Text

t = Text("Argentina")
t.stylize("bold bright_green")     # winner: bright green

The style syntax is one or more space-separated tokens:

• Colors: red, green, blue, cyan, magenta, yellow, white, bright_black, gold1, and dozens more

• Attributes: bold, italic, dim, underline, strike, reverse

• Background: on black, on grey11

• Combined: "bold bright_green" = bold + green; "dim italic" = dim + italic

Style is separate from content — much cleaner than mixing ANSI escape codes into your strings.

Understanding Panels

A Panel is a bordered box with optional title and subtitle:

from rich.panel import Panel

console.print(Panel(
    "Contents go here",
    title="[bold blue]Round of 32[/]",
    subtitle="[blue]7 / 16 played[/]",
    border_style="blue",
    padding=(1, 2),
))

Two things to notice:

• The title/subtitle use inline BBCode-style markup — [bold blue] opens the style, [/] closes it. Concise for one-off text bits.

• border_style sets the color of the actual border characters.

• padding=(vertical, horizontal) controls internal whitespace: (1, 2) means one blank line top/bottom and two spaces left/right.

Panels are our workhorse today — one per round, each in a different color.

Understanding Table.grid for Alignment

Regular rich.Table renders with visible borders between rows and columns. Table.grid gives you the same alignment behavior with no borders — perfect for laying out rows inside a Panel:

from rich.table import Table

table = Table.grid(padding=(0, 1))
table.add_column(justify="left")     # match number
table.add_column(justify="left")     # date
table.add_column(justify="right")    # team1
table.add_column(justify="center")   # score / vs
table.add_column(justify="left")     # team2

table.add_row("#73", "2026-06-28", "South Africa", "0–1", "Canada")
table.add_row("#74", "2026-06-29", "Germany", "1–1 (3-4 pens)", "Paraguay")

The five columns line up automatically no matter what row content you add. Right-align for team1 + center for the score + left-align for team2 gives you that classic “team score team” look:

     South Africa  0–1     Canada
          Germany  1–1 (3-4 pens)  Paraguay

This is dramatically easier than trying to do it with f-string width specifiers, especially once rich styling gets mixed in.

Understanding Detecting the Winner

To highlight a winner, we need to know who won. Football has three ways a knockout match can end, and each hides the winner in a different score field:

def winner_of(match):
    """Return 1 if team1 won, 2 if team2 won, or 0 for a draw / unplayed."""
    if "score" not in match:
        return 0
    score = match["score"]

    # Penalty shootout: the penalty score decides it.
    if "p" in score:
        p1, p2 = score["p"]
        return 1 if p1 > p2 else 2 if p2 > p1 else 0

    # Extra time: if there's an et score AND it's not tied, that decides it.
    if "et" in score:
        g1, g2 = score["et"]
        if g1 != g2:
            return 1 if g1 > g2 else 2

    # Regulation.
    g1, g2 = score["ft"]
    if g1 != g2:
        return 1 if g1 > g2 else 2

    return 0

The order matters again: check p first, then et, then ft. A match with a penalty shootout has all three fields; if we checked ft first we’d get “tied” and never look at the shootout.

This function is small but foundational — Day 3’s web app uses the exact same logic to pick which team’s name to bold in the browser.

Understanding Detecting Placeholders

To style unresolved team names differently (dim italic), we need a way to tell “Canada” apart from “Winner of #74”. The check is the mirror image of yesterday’s pretty_team():

def is_placeholder(name):
    if name.startswith("W") and name[1:].isdigit():
        return True
    if name.startswith("L") and name[1:].isdigit():
        return True
    if len(name) == 2 and name[0].isdigit() and name[1].isalpha():
        return True
    if "/" in name:
        return True
    return False

Same patterns as yesterday, but instead of transforming the string, we’re classifying it. Same regex-free approach — just startswith, isdigit, and a couple of shape checks.

Understanding Styling One Row

The core styling function ties it all together — for each match, pick the right style for the team names, the score, and the date:

def build_match_row(match, today_str):
    num = f"#{match.get('num', '?')}"
    date_str = match.get("date", "")
    played = "score" in match
    is_today = date_str == today_str
    w = winner_of(match)

    # Match number: always bold
    num_text = Text(num, style="bold")

    # Date: yellow if today AND unplayed, plain otherwise
    date_text = Text(date_str)
    if is_today and not played:
        date_text.stylize("bold yellow")

    # Teams: green if winner, dim italic if placeholder, plain white otherwise
    t1_text = styled_team(match["team1"], is_winner=(w == 1),
                          is_placeholder_name=is_placeholder(match["team1"]))
    t2_text = styled_team(match["team2"], is_winner=(w == 2),
                          is_placeholder_name=is_placeholder(match["team2"]))

    # Score column: real result, "TODAY", or "vs"
    if played:
        score_text = Text(format_score(match["score"]), style="bold white")
    elif is_today:
        score_text = Text("TODAY", style="bold yellow")
    else:
        score_text = Text("vs", style="dim")

    return num_text, date_text, t1_text, score_text, t2_text

The function returns a tuple of Text objects — one per column. Table.grid.add_row(*result) unpacks the tuple into the table’s columns. Small, composable, easy to test.

Understanding Color-Coding by Round

Each round gets its own color, and we keep them in a list of (name, color) tuples so the display order and color mapping stay in one place:

ROUNDS = [
    ("Round of 32",            "bright_blue"),
    ("Round of 16",            "cyan"),
    ("Quarter-final",          "magenta"),
    ("Semi-final",             "yellow"),
    ("Match for third place",  "bright_black"),
    ("Final",                  "gold1"),
]

for round_name, color in ROUNDS:
    round_matches = [m for m in matches if m["round"] == round_name]
    ...
    console.print(build_round_panel(round_name, round_matches, color, today_str))

The palette goes cool-to-warm, culminating in gold for the Final. It’s a visual metaphor — the tournament heats up as it narrows. Small aesthetic choice, big effect.

Understanding Adding a Legend

At the bottom, a two-row Table.grid explains what the styles mean:

legend = Table.grid(padding=(0, 2))
legend.add_column()
legend.add_column()
legend.add_column()

legend.add_row(
    Text("Legend:", style="bold"),
    Text("winner", style="bold bright_green"),
    Text("upcoming placeholder", style="dim italic"),
)
legend.add_row(
    Text(""),
    Text("today's match", style="bold yellow"),
    Text("score / result", style="bold white"),
)
console.print(legend)

Nothing dramatic — just a small ceremonial ending that helps users decode the colors. Every good chart has a legend; every good terminal UI can too.

Understanding Everything Together

Look at what the whole script does:

1. Fetch the JSON (same as Day 1)

2. Loop through rounds in the desired order

3. For each round: filter matches, sort by match number, build a Panel containing a Table.grid

4. For each match in the round: pick styles based on who won, whether it’s today, whether teams are placeholders

5. Print each panel

6. Print a legend

That’s the entire day. rich handles all the styling and alignment work — we just describe what we want. The data logic barely changed from Day 1. All the new complexity is presentation. That’s the point: the same fetch-and-parse pipeline can power a plain print, a terminal UI, or (as we’ll see tomorrow) a web app.

Practical Use Cases

1. Terminal dashboards:

Server monitoring, deployment logs, CI/CD summaries — rich turns any script into a designed UI.

2. Interactive CLIs:

Combine rich with prompt_toolkit for full-featured terminal apps.

3. Progress and status:

rich.progress and rich.live are worth exploring — real-time-updating panels are magical.

4. Foundation for Day 3:

The same match-styling logic (winner detection, placeholder detection) drives the web app tomorrow.

Coming Tomorrow

Tomorrow we ship it. The Visual Bracket Web App wraps this whole thing in a Flask server that serves an HTML page — with a real, actual visual bracket diagram in HTML/CSS. Auto-refreshing every minute. Deployable. Shareable. This is the finale.

View Code Evolution

Compare today’s colorful bracket with yesterday’s plain-text version — and see how a few rich components (Panel, Table.grid, Text) turn a functional script into something that looks designed.

Read more

This week, we build a World Cup 2026 Tracker — a three-day project that turns a free public football API into your own personal tournament dashboard. By Friday, you’ll have a live web app showing the entire knockout bracket, updating itself as the matches play out.

Why build this? Because we’re in the World Cup right now. The Round of 32 just kicked off, and through July 19, the bracket fills in match by match. This is the perfect moment to build something that pulls live tournament data and turns it into something useful. The skills — consuming an API, parsing JSON, building UIs around external data — apply to every kind of real-world data work, from finance to weather to fitness apps.

What you’ll learn: This series teaches you HTTP API consumption with requests, JSON parsing, terminal art with rich, Flask web servers, and bridging Python data to HTML/CSS/JS frontends.

Why this matters: By Day 3, you’ll have a deployable web app showing a live, visual knockout bracket — a portfolio piece you can share with anyone who likes football.

• Day 1: Fetch and Show the Knockout Phase (Today)

• Day 2: Bracket Printer (Terminal Art)

• Day 3: Visual Bracket Web App

View All Projects This Week

Today’s Project

We start small but real: fetch live World Cup 2026 data from a free public JSON API, parse it, and print every knockout-stage match in the terminal. By the end of today you’ll be making a real HTTP request, working with real-world JSON, and seeing real match results from a tournament happening right now.

The display is intentionally simple — we’ll polish it on Day 2 with terminal art, and Day 3 turns it into a proper visual web app. Today is about the data. Once you can fetch and parse it, the rest of the week is presentation.

The Data Source

We’re using the wonderful openfootball/worldcup.json project — a free, public-domain, no-API-key-required JSON dataset of every World Cup match. It’s maintained by hand by an Austrian developer named Gerald Bauer, updated daily, and used by dozens of real projects.

The endpoint is a single URL that returns all 104 matches:

https://raw.githubusercontent.com/openfootball/worldcup.json/master/2026/worldcup.json

That’s it. No authentication, no signup, no rate limits to worry about. Hit the URL, get JSON. Perfect for learning.

A note on “live”: the data is updated by hand (usually daily), not in real time. For our purposes — building a tracker, learning APIs — that’s fine. Day 3 will auto-refresh every few minutes. For real-time scores you’d need a paid API; that’s a different project.

Project Task

Build a Python script that:

• Fetches the openfootball JSON endpoint via HTTP

• Parses the response as JSON

• Filters the match list to just the knockout-stage matches

• Sorts each round in bracket order (by match number)

• Prints them grouped by round: Round of 32 → Round of 16 → QF → SF → 3rd place → Final

• Shows the score correctly whether the match was decided in regulation, extra time, or penalties

• Renders placeholder team codes (W73, 2A) as readable text (Winner of match 73, Group A runner-up)

This project gives you hands-on practice with requests, JSON parsing, list comprehensions, sort keys, dict access patterns, and the small details that make the difference between “the data is there” and “the output is readable.”

Expected Output

Running the script:

python worldcup.py

Output:

Played matches show their full score including penalty shootouts. Future matches show placeholders that automatically resolve into real team names as the bracket fills in.

Setup Instructions

Install one dependency:

pip install requests

That’s it. Everything else is standard library.

Run it:

python worldcup.py

Understanding the Data Source

Before writing any code, look at the data. Open the URL in a browser and scroll through it. The structure is:

{
  "name": "World Cup 2026",
  "matches": [
    {
      "round": "Matchday 1",
      "date": "2026-06-11",
      "time": "13:00 UTC-6",
      "team1": "Mexico",
      "team2": "South Africa",
      "score": { "ft": [2, 0], "ht": [1, 0] },
      "goals1": [...],
      "ground": "Mexico City"
    },
    {
      "round": "Round of 32",
      "num": 73,
      "date": "2026-06-28",
      "team1": "2A",
      "team2": "2B",
      "ground": "Los Angeles (Inglewood)"
    },
    ...
  ]
}

Three things to notice for today:

• Every match has round, team1, team2. The knockout matches also have num — their match number (73-104).

• score only appears after the match is played. Future matches have no score key at all.

• Knockout matches have placeholder teams until the dependency matches resolve. "2A" means “Group A runner-up.” "W73" means “Winner of match 73.” As real matches finish, openfootball replaces those placeholders with actual team names — automatically, for free, on our side.

Understanding requests for an API

requests is Python’s go-to library for HTTP. Hitting our endpoint is two lines:

import requests

response = requests.get(DATA_URL, timeout=10)
response.raise_for_status()
data = response.json()

Three habits worth forming on day one of API work:

• Always set a timeout. Without one, a slow server can hang your script forever. 10 seconds is sensible.

• Always call raise_for_status(). It throws on 4xx/5xx responses, so you catch network failures immediately instead of crashing while parsing an HTML error page as JSON.

• response.json() is the magic call. It does json.loads(response.text) for you, returning a Python dict (or list).

That’s the entire API interaction. The hard part of API work isn’t the HTTP call — it’s everything you do with the data afterward.

Understanding Filtering by Round

The JSON has all 104 matches — group stage plus knockouts. We want just the knockout ones. A list comprehension and a fixed list of round names handles it:

KNOCKOUT_ROUNDS = [
    "Round of 32",
    "Round of 16",
    "Quarter-final",
    "Semi-final",
    "Match for third place",
    "Final",
]

for round_name in KNOCKOUT_ROUNDS:
    round_matches = [m for m in matches if m["round"] == round_name]
    # ... print them

The order of KNOCKOUT_ROUNDS defines the display order — Round of 32 first, Final last. Just by looping that list in order, we get the bracket in the right sequence. No sorting algorithm needed for the rounds themselves.

Understanding Match Numbers and Sort Order

Within a round, matches have a num field — 73, 74, 75, etc. The order of matches in the JSON isn’t guaranteed, so we sort each round’s matches by num:

round_matches.sort(key=lambda m: m.get("num", 0))

A lambda is just a small inline function — lambda m: m.get("num", 0) takes a match m and returns its num field (or 0 if missing).

Why m.get("num", 0) and not m["num"]? Because group-stage matches don’t have a num field, and using m["num"] would crash if any non-knockout match slipped through. dict.get(key, default) is the safe accessor. Use it whenever a key might not exist — it returns the default value instead of raising KeyError.

Understanding the Score Structure

This is the most interesting JSON detail of the day. Football matches can end in three ways, and openfootball encodes each one slightly differently:

# 1. Regulation: just full-time
"score": {"ft": [2, 0], "ht": [1, 0]}

# 2. Extra time: full-time was a draw, extra time decided it
"score": {"ft": [1, 1], "et": [2, 1], "ht": [0, 1]}

# 3. Penalty shootout: full-time was a draw, extra time too, decided on pens
"score": {"ft": [1, 1], "et": [1, 1], "p": [4, 3], "ht": [0, 1]}

So three keys to know:

• ft — full-time score, always present

• et — extra-time aggregate (only if the match went to extra time)

• p — penalty shootout result (only if there was a shootout)

For the knockout stage, all three matter. A 1-1 draw in regulation that ended 4-3 on penalties looks very different on the bracket than just “1-1.” So our score formatter handles all three:

def format_score(score):
    g1, g2 = score["ft"]

    if "p" in score:                       # penalty shootout
        p1, p2 = score["p"]
        if "et" in score:
            g1, g2 = score["et"]           # show the ET tie, not the FT one
        return f"{g1} — {g2} ({p1}-{p2} pens)"

    if "et" in score:                       # decided in extra time
        g1, g2 = score["et"]
        return f"{g1} — {g2} (a.e.t.)"

    return f"{g1} — {g2}"                   # regulation

The order matters: check p first, then et, then default to regulation. A penalty shootout always also has et — if we checked et first, we’d report shootouts as “extra time” results and miss the decisive penalty score.

Practical Use Cases

1. The API consumption pattern:

fetch → parse → filter → display works for weather, crypto, stocks, GitHub, anything.

2. The placeholder-resolution pattern:

APIs often return raw codes that need translation. Build a small helper, keep callers clean.

3. Tournament data in general:

The same JSON format covers Premier League, La Liga, Bundesliga, every past World Cup.

4. Foundation for the rest of the week:

Day 2 makes the display beautiful. Day 3 puts it in the browser. Same fetch + parse pipeline.

Coming Tomorrow

Tomorrow we make this look like a bracket. We’ll use the rich library to draw the actual knockout bracket diagram in your terminal — with colors, lines connecting matches across rounds, winners highlighted, and the whole thing as one cohesive visual. Same data source, way more wow.

Skeleton and Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build an Email Productivity Suite — a complete set of tools for one of the most universal time-sinks in modern life: writing and sending email.

• Day 1: Email Sender Desktop App

• Day 2: Mail Merge from CSV

• Day 3: AI Email Assistant (Gemini) (Today)

View All Projects This Week

Today’s Project

Welcome to the finale — and the most fun one.

You have an email to write. Maybe to your professor about a missed deadline. Maybe to a client whose invoice you need to chase. Maybe to your landlord about a leaky tap. You know what you want to say in two seconds. Writing it politely — picking the right tone, the right opening, the right level of firmness — takes ten minutes you don’t have.

Today we will have AI write our emails in the GUI. You just type a rough note in plain language:

“tell my professor i can’t make tomorrow’s office hours, doctor’s appointment, ask if we can meet thursday”

Click the tone — polite, firm, concise, apologetic — and Gemini rewrites it as a polished email. One more click and it sends through the same Gmail engine you built on Day 1.

This is the kind of AI tool you’ll actually use — not a demo, a real piece of your workflow.

Project Task

Build an AI Email Assistant with Tkinter and Gemini that:

• Takes a rough draft as input — any tone, any messiness

• Lets the user pick a target tone: Polite / Firm / Concise / Apologetic / Friendly

• Sends the rough draft to Gemini with a tone-specific instruction

• Displays the polished version in a separate editable text box

• Lets the user tweak the AI output before sending

• Suggests a subject line automatically based on the content

• Sends the finished email through the same send_email() function from Day 1

• Handles missing API keys, network errors, and Gmail errors gracefully

This project gives you hands-on practice with the langchain-google-genai library, prompt engineering for tone control, integrating LLM output into a real workflow, two-pane editing UIs, and capping off a multi-day project series cleanly.

Expected Output

Running the app:

python ai_email_assistant.py

That will display the app. In the app, you can write the instructions to the AI about the email and pick a tone:

Then, further down an email with subject will be generated:

Pressing “SEND EMAIL” will send out the email to the sender’s email address. Here is what I received to my the sender’s address:

Cool, isn’t it?

Setup Instructions

Step 1 — Install the AI dependency:

pip install langchain-google-genai

That’s the only new thing this week.

Step 2 — Get a Gemini API key (free, 1 minute):

1. Go to aistudio.google.com/apikey.

2. Sign in with any Google account.

3. Click Create API key.

4. Copy the key it shows you.

The free tier gives you generous daily quota — plenty for personal email rewriting.

Step 3 — Paste your API key into the script:

Open ai_email_assistant.py and find this line near the top:

GOOGLE_API_KEY = "PASTE_YOUR_KEY_HERE"

Replace PASTE_YOUR_KEY_HERE with your actual key. Save.

Security note: keeping the key directly in the code is the simplest approach for learning. For real apps, store it in an environment variable or a .env file, then read it via os.environ["GOOGLE_API_KEY"]. We use the inline approach so the project runs without extra setup.

Step 4 — Use the same Gmail App Password from Day 1.

If you already set up your App Password earlier this week, you’re done. Otherwise see Day 1 — 5-minute setup.

Run it:

python ai_email_assistant.py

Understanding LangChain + Gemini

langchain-google-genai is the glue between Python and Google’s Gemini models. The basic shape is three steps:

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(
    model="gemini-2.5-flash",
    google_api_key=GOOGLE_API_KEY,
    temperature=0.4,        # how creative; lower = more focused
)

response = llm.invoke("Rewrite this email politely: tell my prof I'll be late")
print(response.content)

Three things to know:

• gemini-2.5-flash — Google’s fast, cheap model. Perfect for short text rewriting. The smarter gemini-3.5-flash exists too; you can swap it in any time, you just pay a small latency cost.

• temperature=0.4 — moderate creativity. 0 is fully deterministic (same input → same output every time); 1 is wildly creative. For tone rewriting, 0.3–0.5 hits the right spot: consistent voice with a bit of natural variation.

• response.content — the actual text. The full object has tokens, metadata, etc.; for our use we just want the string.

Understanding Prompt Engineering for Tone

The whole AI part of the app boils down to one well-crafted prompt template. We need Gemini to:

1. Read the user’s rough note.

2. Rewrite it as a professional email.

3. Apply the chosen tone.

4. Return only the email — no preamble, no explanation.

Here’s a template that does exactly that:

PROMPT_TEMPLATE = """You are an expert email writer. The user has a rough \
draft of an email they want to send. Rewrite it as a polished, well-structured \
email in a {tone} tone.

Rules:
- Output ONLY the email body. No preamble, no commentary, no markdown.
- Do not include a subject line - the user handles that separately.
- Keep it appropriately brief; do not pad with filler.
- Preserve every concrete detail from the rough draft.
- Sign off appropriately for the tone.

Tone: {tone}

Rough draft:
{rough_draft}

Polished email:"""

Three prompt-writing habits in this one example:

• Set the role — “You are an expert email writer” primes the model for the task.

• List explicit rules — bullet rules are dramatically more effective than vague prose at controlling output. “Output ONLY the email body” prevents the model’s tendency to add “Sure! Here’s your email:” preambles.

• Show the structure — ending with “Polished email:” tells the model what comes next is the answer. Sounds silly, works brilliantly.

The {tone} and {rough_draft} are placeholders we’ll fill at runtime — same str.format() trick from Day 2:

prompt = PROMPT_TEMPLATE.format(tone="polite", rough_draft=user_text)

Understanding Tone-Specific Variations

“Polite” and “Firm” and “Concise” each have different sweet spots. We can make the rewrite much better by giving each tone a bit of guidance:

TONES = {
    "Polite": "warm, respectful, hedged where appropriate",
    "Firm": "direct and assertive, but not aggressive; clear next steps",
    "Concise": "as short as possible while staying complete; no filler words",
    "Apologetic": "genuinely apologetic, takes ownership, offers to make it right",
    "Friendly": "warm and conversational; appropriate for someone you know well",
}

In the prompt:

prompt = PROMPT_TEMPLATE.format(
    tone=f"{tone_name} ({TONES[tone_name]})",
    rough_draft=user_text,
)

So the model sees Tone: Polite (warm, respectful, hedged where appropriate). The parenthesized hint is the real instruction — the label is just for the user. This is prompt scaffolding: giving the model enough context to do the job well, without making the user write a paragraph every time.

Understanding Subject Line Suggestions

Most rough drafts don’t include a subject — but the user needs one. We can have Gemini suggest one in the same call by asking it for a tiny JSON-shaped response:

SUBJECT_PROMPT = """Suggest a concise email subject line for this email body. \
Reply with ONLY the subject line - no quotes, no preamble.

Email body:
{body}

Subject line:"""

def suggest_subject(llm, body):
    response = llm.invoke(SUBJECT_PROMPT.format(body=body))
    return response.content.strip()

A separate small call keeps things simple — one call per output. Both are short (a few hundred tokens) so latency adds up to maybe two seconds total. Much cleaner than trying to coerce structured output from the model.

Aside: for production, you’d probably want with_structured_output (like we did in Week 19’s travel planner) and get both subject and body in one structured call. For a personal-use tool, two simple invoke calls are easier to reason about.

Understanding the Two-Pane Layout

The whole app is a top-down flow: rough on top, polished on bottom. Tkinter’s vertical stacking is perfect:

# Top pane: rough draft
tk.Label(root, text="YOUR ROUGH DRAFT").pack(anchor="w")
self.rough_text = tk.Text(root, height=6, wrap="word")
self.rough_text.pack(fill="x", padx=10)

# Tone buttons
tone_bar = tk.Frame(root)
tone_bar.pack(pady=8)
for tone in TONES:
    tk.Button(tone_bar, text=tone,
              command=lambda t=tone: self.rewrite(t)).pack(side="left", padx=4)

# Big "Rewrite" button
tk.Button(root, text="✨ REWRITE WITH AI", ...).pack(pady=4)

# Bottom pane: polished output
tk.Label(root, text="POLISHED EMAIL (edit before sending)").pack(anchor="w")
self.subject_entry = tk.Entry(root, textvariable=self.subject_var)
self.subject_entry.pack(fill="x", padx=10)
self.polished_text = tk.Text(root, height=10, wrap="word")
self.polished_text.pack(fill="both", expand=True, padx=10)

The polished pane is a regular Text widget — the user can edit it. That’s the key UX choice: AI suggests, human approves. The output isn’t locked; you’re free to fix any phrase before sending.

Understanding the Threading Question

LLM calls take 1–3 seconds. If we call llm.invoke() directly from the button handler, Tkinter freezes for that whole time — the window won’t repaint, won’t respond to clicks, looks crashed.

For a personal-use tool with quick calls, the simplest fix is root.update_idletasks() after a “Thinking...” status update, then let the call complete. The window freezes briefly but the user knows why:

self.set_status("✨ Thinking...")
self.root.update_idletasks()
result = llm.invoke(prompt)

For longer calls or a heavier app, you’d use threading.Thread and root.after() to keep the GUI responsive. For our case — a few seconds, a clear status message — the simple approach is the right tradeoff. Less code to teach, less code to break.

Understanding Reusing send_email Again

The whole point of the architecture lesson this week:

# Top of ai_email_assistant.py
from email_sender import send_email

Day 1’s send_email() ships emails from Day 1’s app, from Day 2’s mail-merge loop, and from Day 3’s AI assistant. Three completely different products, one engine. That’s what separating pure functions from UI gets you.

When the user clicks Send, we feed Gemini’s polished output into the same function:

def on_send(self):
    sender = self.from_var.get().strip()
    password = self.password_var.get()
    recipient = self.to_var.get().strip()
    subject = self.subject_var.get().strip()
    body = self.polished_text.get("1.0", "end-1c").strip()

    try:
        send_email(sender, password, recipient, subject, body)
        self.set_status(f"✓ Email sent to {recipient}")
    except smtplib.SMTPAuthenticationError:
        self.set_status("✗ Gmail authentication failed. Check your App Password.")
    except Exception as e:
        self.set_status(f"✗ Send failed: {e}")

Same shape as Day 1. Same SMTP errors. Same status updates.

Understanding the Bigger Picture

Three days of email tools. One pure send function. One templating helper. One AI rewriter.

• Day 1 built the engine.

• Day 2 wrapped it in a loop.

• Day 3 put intelligence in front of it.

Each layer adds value; none of them rewrite the layer below. That’s not a Python pattern — that’s the discipline behind every well-built software product. Today you wrote one piece of an “email-productivity-AI-assistant”; the same shape works for any AI-powered desktop app you’ll build from here.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete Email Productivity Suite:

• Day 1: Send a single email through Gmail SMTP via a desktop app

• Day 2: Send personalized emails to a CSV of recipients

• Day 3: AI-rewrite rough drafts in any tone before sending

You now have:

✅ Python email skills — smtplib, EmailMessage, Gmail SMTP, App Passwords ✅ CSV-driven personalization — DictReader, str.format(), batch processing ✅ LangChain + Gemini integration — building a real LLM-powered tool ✅ Prompt engineering — tone control, output constraints, structured prompts ✅ A reusable architecture — pure functions composed across three tools

Real-world applications:

• 📨 Personal email speed — quick drafts become professional emails in seconds

• 🎓 Student communications — emailing professors, advisors, departments

• 💼 Freelancer workflows — invoicing, follow-ups, client check-ins

• 🛠️ Automation triggers — long script finishes → script emails you the result

• 🤖 Custom AI tools — you now know how to wrap any LLM into a desktop app

Next steps:

• Add an inbox reader (IMAP) — let Gemini summarize and respond to incoming emails

• Save templates — keep your favorite rewritten emails for one-click reuse

• Multi-recipient AI batch — combine Day 2 mail merge with Day 3 AI rewriting

• Voice input — speak your rough draft, transcribe with Whisper, rewrite with Gemini

• Use keyring or python-dotenv to manage your API key and Gmail credentials safely

You’ve built a real, daily-use AI tool in three days. 🚀

View Code Evolution

Compare today’s AI assistant with Day 1’s single sender and Day 2’s mail merge — and see how the same pure send_email() function composes with templating and an LLM to produce three very different end-user tools.

Read more

This week, we build an Email Productivity Suite — a complete set of tools for one of the most universal time-sinks in modern life: writing and sending email.

• Day 1: Email Sender Desktop App

• Day 2: Mail Merge from CSV (Today)

• Day 3: AI Email Assistant (Gemini)

View All Projects This Week

Today’s Project

Yesterday we sent one email. Today we send a hundred — each personalized to its recipient.

Mail merge is the technique behind every personalized email you’ve ever received: the company has a list of names and a template (”Hi {first_name}, your order #{order_id} has shipped”), and a script fills the blanks for each row and sends. Today you build that script. Load a CSV of recipients, write your email once with {placeholder} variables, hit Send All, and watch personalized emails go out one by one.

Same send_email() function from Day 1 — wrapped in a loop with smarter UI around it.

Project Task

Build a CSV-driven mail merge tool with Tkinter that:

• Loads a CSV of recipients with at minimum an email column

• Auto-detects the other columns as available template variables

• Lets you write subject and body with {column_name} placeholders

• Shows a live preview of how the email will look for the first recipient

• Sends one personalized email per row when Send All is clicked

• Adds a polite delay between sends to avoid Gmail rate-limiting

• Reports progress per recipient (sent / failed / row count)

• Continues to the next row if one fails — never crashes mid-batch

• Logs every send with status to a scrollable log box

This project gives you hands-on practice with csv.DictReader, Python’s str.format() for templating, batch processing with error recovery, real-time progress reporting in Tkinter, and turning a single-use function into a reusable engine.

Expected Output

Running the tool:

python mail_merge.py

Here is what you will see when you run the program.

You can write an email in the GUI and use variables such as {first_name} and {course} in the email:

When you press SEND ALL, the app will pull the actual value from the respective column (i.e., first_name and course) in the CSV file for each row:

Each person in the CSV will get a personalized email. Here is the email Carol received:

Each member receives an email with their own data fetched from the CSV.

Setup Instructions

Install (nothing new):

Same as Day 1 — pure standard library (smtplib, csv, tkinter). No pip install.

Prepare a recipients CSV:

A sample students.csv is provided with this project. The format is simple:

email,first_name,course,amount
alice@example.com,Alice,Python Bootcamp,250
bob@example.com,Bob,Data Science 101,180
carol@example.com,Carol,Web Dev Foundations,200
david@example.com,David,Python Bootcamp,250

You can copy the above, paste it in an empty text file and save as students.csv.

Run the program:

python mail_merge.py

Understanding csv.DictReader

The CSV-handling workhorse for today is csv.DictReader. It reads the header row, then yields each subsequent row as a dictionary keyed by column name:

import csv

with open("students.csv", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    fieldnames = reader.fieldnames           # ['email', 'first_name', ...]
    recipients = list(reader)                # [{'email': 'alice@...', ...}, ...]

Two things to know:

• reader.fieldnames — the column headers. We’ll use these to tell the user which {variables} they can put in their template.

• list(reader) — materializes all rows into memory. Fine for hundreds of rows; for hundreds of thousands you’d iterate the reader directly.

DictReader gives us a clean shape: a list of dicts, each one perfectly ready to feed into .format(**row).

Understanding Python’s str.format with **kwargs

Templating in Python is built into the language — no Jinja, no Mustache, just str.format():

template = "Hi {first_name}, your {course} starts soon."
row = {"first_name": "Alice", "course": "Python Bootcamp"}

personalized = template.format(**row)
# "Hi Alice, your Python Bootcamp starts soon."

The **row syntax unpacks the dict into keyword arguments. So template.format(**row) is equivalent to template.format(first_name="Alice", course="Python Bootcamp", ...).

It works for any number of variables. The template grabs whichever placeholders it needs and ignores the rest. The dict can have 10 columns — your template might only use 2 of them; that’s fine.

Understanding Safe Personalization

What happens if your template has {first_name} but a CSV row is missing that column? format() raises KeyError. That’s exactly what you want for a typo — but you want it to fail cleanly for the user, not crash the script:

def personalize(template, row):
    """Fill {placeholders} in template with values from row dict."""
    try:
        return template.format(**row)
    except KeyError as e:
        raise ValueError(f"Template references {e} but the CSV has no such column.")

Wrapping in a function with a clearer error means the GUI can show “Template references ‘first_name’ but the CSV has no such column” instead of a stack trace. Errors as messages, not crashes.

Understanding the Live Preview

The live preview shows how the first recipient’s email will look — it runs the same personalize() function the batch will use:

def refresh_preview(self):
    if not self.recipients:
        self.preview_text.delete("1.0", "end")
        return

    first = self.recipients[0]
    try:
        subject = personalize(self.subject_var.get(), first)
        body = personalize(self.body_text.get("1.0", "end-1c"), first)
    except ValueError as e:
        self.preview_text.delete("1.0", "end")
        self.preview_text.insert("1.0", f"⚠️ {e}")
        return

    preview = f"To: {first['email']}\nSubject: {subject}\n\n{body}"
    self.preview_text.delete("1.0", "end")
    self.preview_text.insert("1.0", preview)

Bind it to keystrokes on the subject and body widgets:

subject_entry.bind("<KeyRelease>", lambda _e: self.refresh_preview())
self.body_text.bind("<KeyRelease>", lambda _e: self.refresh_preview())

The user types {first_name} — the preview instantly shows “Alice.” Type {first_nam} (typo) — the preview shows a clear warning before they hit Send. This is the small UX detail that separates I built a script from I built a tool.

Understanding the Send-All Loop

The batch loop wraps yesterday’s send_email() in a for loop with three additions: a delay between sends, per-row error handling, and progress logging:

import time

def send_all(self):
    sender = self.from_var.get().strip()
    password = self.password_var.get()
    subject_template = self.subject_var.get()
    body_template = self.body_text.get("1.0", "end-1c")

    successes = 0
    failures = 0
    total = len(self.recipients)

    self.log(f"Sending to {total} recipients...")

    for i, row in enumerate(self.recipients, 1):
        recipient = row.get("email", "").strip()

        try:
            subject = personalize(subject_template, row)
            body = personalize(body_template, row)
            send_email(sender, password, recipient, subject, body)
            self.log(f"  [{i}/{total}]  {recipient:<30}  →  ✓ sent")
            successes += 1
        except Exception as e:
            self.log(f"  [{i}/{total}]  {recipient:<30}  →  ✗ {e}")
            failures += 1

        time.sleep(1)   # be polite to Gmail

    self.log(f"\nDone. {successes} sent, {failures} failed.")

Three patterns to internalize:

• time.sleep(1) — one second between sends. Gmail’s free tier rate-limits at around 100/day; even at the limit, one-second pacing keeps you well under any per-minute throttle. Tomorrow’s AI version can use 2-3 seconds for extra safety.

• try/except per row — one bad recipient doesn’t stop the batch. The failure is logged; the next row goes out.

• successes and failures counters — the user sees a useful summary at the end, not a wall of green checkmarks.

Understanding Reusing send_email from Day 1

We don’t rewrite send_email. We import it:

# At the top of mail_merge.py
from email_sender import send_email

This is the architecture lesson of the week. Day 1’s send_email() was a pure function — no Tkinter, no I/O outside of the SMTP call — exactly so this would work. Day 3 will import it again.

If you wrote send_email() as a method on the Day 1 EmailSenderApp class, today’s loop would be much harder. Pure functions compose; methods on classes don’t.

Understanding the Architecture

The whole app is built on three layers:

1. send_email() — the SMTP primitive (imported from Day 1)

2. personalize() — template + dict → final string

3. send_all() — the loop that ties them together

Each layer does one thing. Each layer is reusable. Day 3’s AI assistant will reuse the same three — Gemini produces the body, personalize() fills in any placeholders, send_email() sends it. That’s the discipline that makes week-by-week projects compose.

Practical Use Cases

1. Notify a class / team / customer list:

Drop the names into a CSV, write your email once, hit Send All.

2. Send personalized invoices, certificates, receipts:

{first_name}, {amount}, {invoice_id} - one email per row.

3. Course communications:

Reminders, deadline alerts, assignment feedback — anywhere you'd otherwise BCC.

4. Event RSVPs and follow-ups:

"Hi {name}, thanks for attending {event}! Your certificate is at {link}."

5. Foundation for Day 3:

Tomorrow's AI assistant uses the same send_email() function and the same personalize() trick.

Coming Tomorrow

Tomorrow we add AI. The AI Email Assistant takes a quick rough draft — “tell my boss I’ll be out tomorrow morning, doctor’s appointment” — and Gemini rewrites it as a polished email in your chosen tone (polite / firm / concise / apologetic). One click and it’s ready to send via the same send_email() you built on Day 1.

View Code Evolution

Compare today’s mail merge tool with Day 1’s single sender — and see how a small for loop, a templating helper, and per-row error handling turn one email into a personalized hundred.

Read more

This week, we build an Email Productivity Suite — a complete set of tools for one of the most universal time-sinks in modern life: writing and sending email. By Friday, you’ll have a desktop app that sends single emails, mail-merges to a CSV list, and uses AI to rewrite rough drafts in any tone you want.

Why build this? Because email isn’t going away — AI made it easier to write but more of it gets sent. Python automation around email is one of those rare skills that pays off in every job, every side project, every freelance gig.

What you’ll learn: This series teaches you Python’s smtplib and email modules, Gmail authentication via app passwords, sending plain and HTML email, attaching files, CSV-driven personalization, and integrating LLMs into a real desktop workflow.

Why this matters: By Day 3, you’ll paste a rough message — “tell my professor I cant make tomorrow’s office hours” — and the AI rewrites it as a polite, professional email, ready to send. That’s the kind of tool you’ll actually use.

• Day 1: Email Sender Desktop App (Today)

• Day 2: Mail Merge from CSV

• Day 3: AI Email Assistant (Gemini)

View All Projects This Week

Today’s Project

We start with the foundation of everything else this week: a desktop app that sends one email. Type the recipient, subject, and body, click Send, and your email goes out via Gmail. By the end you’ll understand how Python actually talks to a mail server — the same code powers everything from cron-job notifications to 100-recipient mail merges.

This is the engine. Days 2 and 3 wrap new features around it.

Project Task

Build a Tkinter desktop email sender that:

• Provides input fields for: From (your Gmail), App Password, To, Subject, Body

• Sends the email through Gmail’s SMTP server using smtplib

• Shows clear status: connecting, authenticating, sending, success/failure

• Hides the password field with asterisks

• Validates input (no empty recipient, no missing password) before sending

• Reports detailed error messages when authentication or sending fails

• Has a “Clear” button to reset the form

• Doesn’t crash on network errors — failures are caught and reported

This project gives you hands-on practice with smtplib, the email.message module, Gmail’s STARTTLS authentication, Tkinter form layout, password fields, multi-line text widgets, and the cleanest pattern for separating GUI from logic in a desktop app.

Expected Output

Running the app:

python email_sender.py

That will open the following app in your computer. Below, I already filled in the sender email address and its app password (see further below on how to get an app password for your Gmail account). I also wrote an email.

Pressing the SEND EMAIL button will send out the email from the sender’s address to the receiver’s address. Here is what I received in my gmail account:

Setup Instructions

Step 1 — Install (no dependencies):

This entire project uses Python’s standard library — smtplib, email, tkinter. Nothing to install.

python email_sender.py

Step 2 — Create a Gmail App Password (5 minutes):

You can’t use your regular Gmail password anymore — Google requires an App Password for Python scripts since May 2025.

1. Go to myaccount.google.com and sign in.

2. Click Security in the left sidebar.

3. Under “How you sign in to Google,” enable 2-Step Verification if you haven’t already. (App passwords require it.)

4. Once 2FA is on, go directly to myaccount.google.com/apppasswords.

5. Type any name (e.g., “Python Email”) and click Create.

6. Google shows a 16-character password in a yellow box. Copy it.

7. Paste it into the App Password field in the app — no spaces, just the 16 characters.

Security note: for simplicity, this app keeps the password in the form text field, in memory only. It’s never saved to disk. A more secure setup uses keyring or environment variables — we’ll mention that briefly on Day 3.

Understanding SMTP and Gmail

SMTP — Simple Mail Transfer Protocol — is the protocol that’s moved email between servers since the 1980s. It’s still the protocol. Your script connects to Gmail’s SMTP server, authenticates, and hands over a message; Gmail does the rest.

Python’s smtplib handles the protocol:

import smtplib

with smtplib.SMTP("smtp.gmail.com", 587) as smtp:
    smtp.starttls()                          # upgrade to encrypted connection
    smtp.login("you@gmail.com", "your_app_password")
    smtp.send_message(msg)

Three terms worth knowing:

• smtp.gmail.com — Gmail’s SMTP server address.

• Port 587 — the standard “submission” port for outgoing mail.

• STARTTLS — starts the connection in plain text, then upgrades to TLS encryption. Required by Gmail.

There’s also a port 465 variant using SMTP_SSL (encrypted from the start). Both work for Gmail; we use 587/STARTTLS because it’s the most common pattern in production code.

Understanding the email.message Module

smtplib sends bytes. To build a proper email — with a Subject, From, To, and a body — we use email.message.EmailMessage:

from email.message import EmailMessage

msg = EmailMessage()
msg["From"] = "you@gmail.com"
msg["To"] = "recipient@example.com"
msg["Subject"] = "Hello"
msg.set_content("This is the email body.")

EmailMessage is the modern (Python 3.6+) way to construct emails. It handles headers, encoding, MIME types, and attachments cleanly. The older MIMEText / MIMEMultipart classes still work but feel clunky compared to this one.

Setting headers is dictionary-style (msg["Subject"] = ...); setting the body is method-style (msg.set_content(...)). That’s just a quirk of the API — once you know it, the rest is straightforward.

Understanding the Pure Send Function

We keep the email-sending logic in a pure function — no Tkinter, no GUI. It takes the fields, sends the email, and either returns or raises an exception. This makes it reusable for Days 2 and 3 and easy to test in isolation:

def send_email(sender, password, recipient, subject, body):
    msg = EmailMessage()
    msg["From"] = sender
    msg["To"] = recipient
    msg["Subject"] = subject
    msg.set_content(body)

    with smtplib.SMTP("smtp.gmail.com", 587) as smtp:
        smtp.starttls()
        smtp.login(sender, password)
        smtp.send_message(msg)

A simple, four-step function. The Tkinter app just calls it. Tomorrow’s mail-merge tool calls it in a loop. Day 3’s AI assistant calls it after the AI is done. One sending function, many surfaces around it.

Understanding Tkinter Form Layout

For a form-style UI, grid() is much cleaner than pack(). You think in rows and columns; labels go in column 0, inputs in column 1:

frame = tk.Frame(root)
frame.pack(padx=10, pady=10, fill=tk.BOTH, expand=True)

tk.Label(frame, text="From:").grid(row=0, column=0, sticky="w")
tk.Entry(frame, textvariable=self.from_var, width=50).grid(row=0, column=1, sticky="ew")

tk.Label(frame, text="To:").grid(row=1, column=0, sticky="w")
tk.Entry(frame, textvariable=self.to_var, width=50).grid(row=1, column=1, sticky="ew")

frame.columnconfigure(1, weight=1)   # input column stretches with the window

Three small details that earn their place:

• sticky="w" — labels left-align (”west”) instead of centering on their row.

• sticky="ew" — inputs stretch horizontally (”east-west”) to fill the column.

• columnconfigure(1, weight=1) — declares “column 1 gets the extra space when the window resizes.” Without this, the input column never grows.

Understanding Password Fields

Tkinter has a one-line way to hide a password as it’s typed — show="●" on an Entry:

tk.Entry(frame, textvariable=self.password_var, width=50, show="\u2022").grid(...)

\u2022 is the bullet character. ("*" works too; bullets just look cleaner.) The variable still holds the real password — show only affects what the user sees, not what’s stored.

Understanding the Body Text Widget

A single-line Entry won’t work for a multi-line email body — we need a Text widget:

self.body_text = tk.Text(frame, width=50, height=10, wrap="word")
self.body_text.grid(row=4, column=1, sticky="nsew")

# Reading the body later:
body = self.body_text.get("1.0", "end-1c")  # from line 1 char 0 to end, minus the trailing newline

Two Text-widget quirks you’ll see a lot:

• get("1.0", "end-1c") — Tkinter Text indexes use "line.column" format. "1.0" is “line 1, column 0” (the start). "end-1c" is “end minus 1 character” — that strips the trailing newline that Text always appends.

• wrap="word" — wraps lines at word boundaries instead of mid-word.

Understanding Catching SMTP Errors

The send function can fail in many ways: wrong password, bad recipient, network down, Gmail rate-limiting. smtplib raises distinct exceptions for each — we catch them specifically so we can give the user a useful message:

try:
    send_email(sender, password, to_addr, subject, body)
    self.set_status(f"\u2713 Email sent to {to_addr}")

except smtplib.SMTPAuthenticationError:
    self.set_status("\u2717 Authentication failed: check your "
                    "Gmail address and App Password.")

except smtplib.SMTPRecipientsRefused as e:
    self.set_status(f"\u2717 Recipient refused: {e}")

except smtplib.SMTPException as e:
    self.set_status(f"\u2717 Send failed: {e}")

except Exception as e:
    self.set_status(f"\u2717 Error: {e}")

Specific exceptions on top, generic exceptions on the bottom — the classic exception-handling pattern. SMTPAuthenticationError in particular is the one your users will hit most often, almost always because they pasted the App Password with spaces in it. A clear message saves them a lot of confusion.

Understanding Validation Before Send

The send button should refuse to fire when the form is incomplete. A small validation function returns the first error or None:

def validate(self):
    if not self.from_var.get().strip():    return "Please enter your Gmail address."
    if not self.password_var.get():        return "Please enter your App Password."
    if not self.to_var.get().strip():      return "Please enter a recipient."
    if not self.subject_var.get().strip(): return "Please enter a subject."
    body = self.body_text.get("1.0", "end-1c").strip()
    if not body:                           return "Please enter a message body."
    return None

Then in the send handler:

error = self.validate()
if error:
    self.set_status(f"\u2717 {error}")
    return

This is much better than letting the SMTP call fail with a cryptic error. The user sees the actual problem in plain English, before the script even tries to connect.

Understanding the Status Bar

A status bar at the bottom of the window gives the user feedback as the script runs. Three lines do it:

self.status_var = tk.StringVar(value="Ready. Fill in the form and click Send.")
tk.Label(self.root, textvariable=self.status_var, anchor="w",
         bd=1, relief="sunken", padx=8, pady=4).pack(fill="x", side="bottom")

And inside the send method, we update it at each step:

self.set_status("Connecting to smtp.gmail.com...")
self.root.update_idletasks()   # force the GUI to repaint NOW

self.set_status("Authenticating...")
self.root.update_idletasks()

# ... send ...

self.set_status(f"\u2713 Email sent to {to_addr}")

self.root.update_idletasks() is the magic ingredient. Without it, Tkinter batches GUI updates and the user sees nothing happen until the whole send_email call finishes. Calling update_idletasks() forces an immediate repaint — so the user sees “Connecting...” then “Authenticating...” then “Sending...” in real time, instead of a frozen window followed by “Done.”

Coming Tomorrow

Tomorrow we go from one email to many. The Mail Merge tool loads a CSV of recipients, lets you write the email once with placeholder variables like {first_name} and {course}, and personalizes one email per row — sending each with a polite delay between them. Same send_email() function, just a loop around it.

Skeleton and Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build an Image Toolkit with Pillow that lets you edit photos in a desktop app, watermark whole folders of images at once, and turn photos into shareable memes and collages.

• Day 1: Photo Filters Studio (Tkinter)

• Day 2: Batch Watermarker

• Day 3: Meme Generator & Collage Maker (Today)

View All Projects This Week

Today’s Project

Welcome to the finale — and the fun one. Today we build two image tools in a single app: a meme generator with the classic top+bottom Impact-font caption (white text, thick black outline) and a collage maker that arranges several photos into a single grid image.

Both are real, shareable image tools — output you’d actually post somewhere. And they reuse every Pillow skill from Days 1 and 2: image opening, text drawing, RGBA, positioning, batch handling. The lesson lands: the same fundamentals power every image tool you’ll ever build.

Project Task

Build a two-mode image studio with Tkinter and Pillow that:

Meme Mode:

• Loads an image

• Lets the user type top and bottom caption text

• Draws captions in the classic meme style (uppercase, white with black outline)

• Auto-fits the text to the image width (no clipping)

• Live preview as text changes

• Saves the finished meme

Collage Mode:

• Loads multiple images

• Lays them out in a configurable grid (2×2, 3×2, 3×3)

• Resizes each image consistently

• Adds a configurable colored border between images

• Saves the finished collage

This project gives you hands-on practice with text styling and stroke outlines, dynamic font sizing, grid layout math, the Image.paste compositing pattern, mode tabs in Tkinter, and reusing one image pipeline across two visual products.

Expected Output

Running the studio:

python meme_collage.py

Meme Mode:

The user can write two pieces of text in the GUI on the right side and the image will be updated instantly:

Collage Mode:

In collage mode, the user can select multiple images in their computer and then user the configuration settings on the right to create a collage view of those images:

Each mode is its own clean tool, sharing a preview area and Save workflow.

Setup Instructions

Install Pillow:

pip install Pillow

(One dependency, same as Days 1 and 2.)

Run it:

python meme_collage.py

For meme mode, use any image (the sample.jpg from Day 1 works great). For collage mode, you’ll want 2–9 images — drop a few photos into a folder and load them.

Understanding the Two-Mode Architecture

The app is structured around tabs: meme mode and collage mode each get their own controls and their own logic, but share the same preview area, save workflow, and font-loading helper.

class ImageStudio:
    def __init__(self, root):
        # shared state lives on self
        self.preview_image = None
        self.tk_image = None
        # ...

        self.notebook = ttk.Notebook(root)
        self.meme_tab = MemeTab(self.notebook, self)
        self.collage_tab = CollageTab(self.notebook, self)
        self.notebook.add(self.meme_tab.frame, text="Meme")
        self.notebook.add(self.collage_tab.frame, text="Collage")

Two small structural choices that matter:

• Each tab gets its own class. Meme and collage are different enough that mixing their state on one giant class gets ugly. Splitting them keeps each tool readable.

• The shared parent (self) holds the preview machinery. Both tabs call parent.set_preview(image) to update the central preview — that way the GUI bit lives in one place.

ttk.Notebook is the standard Tkinter tabbed-pane widget. Switch between Meme and Collage with a click; each tab carries its own state without stomping on the other.

Understanding the Meme Style

The classic meme look has three rules:

1. Bold sans-serif font (Impact traditionally; we use a system bold).

2. Uppercase text.

3. White fill, thick black outline — readable against any background.

Pillow’s draw.text(...) supports outlines directly via two parameters: stroke_width and stroke_fill. The whole meme caption is one call:

def draw_meme_text(draw, text, position, font, fill="white", outline="black", stroke=4):
    draw.text(
        position,
        text.upper(),
        font=font,
        fill=fill,
        stroke_width=stroke,
        stroke_fill=outline,
        anchor="mm",   # center the text at the anchor point
    )

Two parameters earn the magic:

• stroke_width=4 — pixels of outline thickness. 3 is too thin; 6 is too cartoony. Around 4 hits the sweet spot for most image sizes.

• anchor="mm" — the text is middle-horizontal, middle-vertical-anchored at the position you give it. This makes centering trivial: pass the center point of where the text should be, not its top-left corner.

Understanding Auto-Fitting Text Width

A meme caption that overruns the image looks broken. We auto-shrink the font until the rendered text fits the image width:

def fit_font_to_width(text, max_width, target_size):
    """Find the largest font size <= target that fits in max_width."""
    size = target_size
    while size > 10:
        font = load_meme_font(size)
        bbox = ImageDraw.Draw(Image.new("RGB", (10, 10))).textbbox(
            (0, 0), text.upper(), font=font, stroke_width=4
        )
        text_width = bbox[2] - bbox[0]
        if text_width <= max_width:
            return font
        size -= 4
    return load_meme_font(10)

The pattern: start at the user’s requested size and shrink in steps until the text fits. The loop only kicks in for long captions — short text gets full size, no measurement overhead.

Note we include stroke_width=4 in the measurement. The outline adds to the text’s actual width, so we have to account for it — or the loop returns a size that fits the unstroked text but overflows once outlined.

Understanding the Meme Builder

The full meme function ties everything together. Top text goes near the top, bottom text near the bottom, both center-aligned:

def make_meme(image, top_text, bottom_text, target_size=60):
    img = image.convert("RGB").copy()
    draw = ImageDraw.Draw(img)

    # Margin from each edge
    margin = max(20, img.height // 20)

    if top_text:
        font = fit_font_to_width(top_text, img.width - 2 * margin, target_size)
        bbox = draw.textbbox((0, 0), top_text.upper(), font=font, stroke_width=4)
        text_h = bbox[3] - bbox[1]
        center = (img.width // 2, margin + text_h // 2)
        draw_meme_text(draw, top_text, center, font)

    if bottom_text:
        font = fit_font_to_width(bottom_text, img.width - 2 * margin, target_size)
        bbox = draw.textbbox((0, 0), bottom_text.upper(), font=font, stroke_width=4)
        text_h = bbox[3] - bbox[1]
        center = (img.width // 2, img.height - margin - text_h // 2)
        draw_meme_text(draw, bottom_text, center, font)

    return img

The pattern repeats for top and bottom — fit the font, measure the rendered height, compute the center point, draw. Pure function, no side effects: hand it an image and some text, get back the meme. Same as Day 2’s apply_watermark — and that consistency is the point.

Understanding Grid Math for Collages

The collage problem boils down to: given N images and a grid (R rows × C columns), how big should each cell be, and where does each image go?

The cleanest approach is to fix a target cell size and compute the canvas from there, rather than the other way around:

def make_collage(images, rows, cols, border, border_color, cell_size=400):
    cell_w = cell_h = cell_size
    canvas_w = cols * cell_w + (cols + 1) * border
    canvas_h = rows * cell_h + (rows + 1) * border

    canvas = Image.new("RGB", (canvas_w, canvas_h), border_color)

    for index, img in enumerate(images[:rows * cols]):
        row = index // cols
        col = index % cols
        x = border + col * (cell_w + border)
        y = border + row * (cell_h + border)
        canvas.paste(fit_into_cell(img, cell_w, cell_h), (x, y))

    return canvas

A few moves worth understanding:

• (cols + 1) * border — there’s a border on each side and between every column. For 3 columns, that’s 4 border-widths total. Same logic vertically.

• row = index // cols, col = index % cols — integer division and modulo turn a flat list index into a 2D grid position. Image 0 → (0, 0); image 1 → (0, 1); image 3 in a 3-col grid → (1, 0). Classic pattern.

• canvas.paste(img, (x, y)) — Pillow’s basic compositing: draw img onto canvas at the given top-left corner.

Image.new("RGB", (w, h), color) creates the canvas filled with the border color, so empty space between the pasted images is automatically the right color. We never have to “draw the borders” — they’re just the parts of the canvas that don’t get pasted over.

Understanding the fit_into_cell Helper

Real photos come in different aspect ratios — landscape, portrait, square. Just resizing them all to the same dimensions distorts them. The right move is fit and center: shrink each photo to fit inside the cell preserving its aspect ratio, then center it on a square cell-sized canvas:

def fit_into_cell(img, cell_w, cell_h):
    """Resize `img` to fit in a cell_w x cell_h cell, centered, no distortion."""
    img = img.copy()
    img.thumbnail((cell_w, cell_h), Image.Resampling.LANCZOS)

    # Center it on a cell-sized canvas
    cell = Image.new("RGB", (cell_w, cell_h), (240, 240, 240))
    offset = ((cell_w - img.width) // 2, (cell_h - img.height) // 2)
    cell.paste(img, offset)
    return cell

thumbnail preserves aspect ratio; the centering math handles whatever’s left over with a light gray pad. The result: every cell is exactly cell_w × cell_h, no distortion, every photo placed sensibly. The collage layout math from above doesn’t need to care about individual photo sizes.

Understanding Saving Both Modes

Each mode runs through a Save button that triggers a save dialog and writes the current preview-equivalent image (full resolution, not the screen-shrunk version):

def save_meme(self):
    if not self.image:
        return
    final = make_meme(self.image, self.top_var.get(), self.bottom_var.get(),
                      self.size_var.get())
    path = filedialog.asksaveasfilename(defaultextension=".jpg", ...)
    if path:
        final.save(path, quality=92)

Note we rebuild the meme at full resolution before saving — the preview was a shrunken version. The user saw the layout; the save call produces the high-quality version of exactly that layout. Preview is a window onto truth, not its own truth.

Understanding the Full Picture

Three days of image work, one shape:

• Day 1 took one image and applied filters interactively.

• Day 2 took one folder and looped over it.

• Day 3 takes one image (meme) or many images (collage) and produces a single new image.

The same Pillow primitives — Image.open, copy, convert, paste, ImageDraw.Draw, ImageFont, ImageEnhance — assembled in different shapes. There’s no special trick for “memes” or “collages” — only for combining the primitives sensibly. That’s the actual lesson of the week.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete Pillow image toolkit:

• Day 1: Live photo filters and adjustments in a Tkinter studio

• Day 2: Batch-watermark whole folders with live preview

• Day 3: Meme generator and collage maker in a tabbed app

You now have:

✅ Pillow fundamentals — Image, ImageDraw, ImageFont, ImageOps, ImageEnhance, ImageFilter ✅ The PIL ↔ Tkinter bridge — ImageTk.PhotoImage, with the GC reference rule baked in ✅ Real-world habits — EXIF orientation, RGBA compositing, JPEG quality ✅ Live-preview architecture — same function powers preview and save, so what users see is what they get ✅ Multiple working tools — filters, watermarker, meme generator, collage maker

Real-world applications:

• 🎨 Social media content — branded posts, memes, collages, watermarked photos

• 📷 Photography workflow — batch process exports, add copyright, generate previews

• 🛒 E-commerce — watermark product photos at scale

• 📚 Documentation and tutorials — annotate screenshots, generate diagrams

• 🤖 Automation — feed any of these into a script that runs on every uploaded image

Next steps:

• Add a real font picker (browse local fonts)

• Support emoji and unicode characters (Pillow can do it with a unicode font)

• Add an undo stack (keep a list of past edited images)

• Build a CLI version so it’s scriptable

• Try Pillow’s ImageDraw.rounded_rectangle for fancier collage borders

You’ve built the foundation for a real image-processing pipeline. 🚀

View Code Evolution

Compare today’s two-mode studio with Day 1’s filters and Day 2’s batch watermarker — and see how the same Pillow primitives assemble into completely different tools when you change the shape of the loop around them.

Read more

This week, we build an Image Toolkit with Pillow that lets you edit photos in a desktop app, watermark whole folders of images at once, and turn photos into shareable memes and collages.

• Day 1: Photo Filters Studio (Tkinter)

• Day 2: Batch Watermarker (Today)

• Day 3: Meme Generator & Collage Maker

View All Projects This Week

Today’s Project

Yesterday we edited one photo at a time. Today we process hundreds at once.

The Batch Watermarker takes a folder of images, applies a text watermark with the position, opacity, and color you pick, and writes the watermarked copies to an output folder. Live preview while you tune the settings, then one click processes the whole batch — leaving every original untouched.

This is where image processing gets useful. Photographers protecting their work, content creators branding their screenshots, ecommerce sellers labeling product photos — the workflow is the same, and you can build it in 200 lines of Python.

Project Task

Build a batch text watermarker with Tkinter and Pillow that:

• Lets the user pick an input folder of images and an output folder

• Lets the user type the watermark text

• Lets the user choose the position: one of 5 anchor points (corners + center)

• Lets the user adjust opacity (0–100%) and font size with sliders

• Shows a live preview of the watermark on the first image in the folder

• Processes the whole folder when “Apply to All” is clicked

• Preserves EXIF orientation and original quality

• Reports progress per file and continues on errors

• Never overwrites originals — writes to a separate output folder

This project gives you hands-on practice with Pillow’s ImageDraw and ImageFont, RGBA compositing for proper transparency, working with image folders, anchor-based positioning, and turning a single-image operation into a batch pipeline.

Expected Output

Running the watermarker:

python batch_watermarker.py

Application Window:

The user can pick an input folder where the images are and an output folder to save the watermarked images.

Once the user has selected the folder the program already applies watermarks and displays the first image in the GUI.

The user can click the APPLY TO ALL button to process all images and a new output folder with the generated images will be generated. Here is a snapshot of that folder:

Setup Instructions

Install Pillow:

pip install Pillow

(Same single dependency as Day 1.)

Run it:

python batch_watermarker.py

You’ll need a folder of images to point at. To try it quickly, copy sample.jpg from Day 1 into a folder a few times — even one image works.

Understanding the Two-Step Architecture

The whole app is built around one simple split:

apply_watermark(image, settings)  →  the pure image operation
process_folder(input, output, settings)  →  the batch loop

The first function takes a single PIL image plus a dict of settings and returns a watermarked copy. The second function walks the input folder, calls the first function for each image, and saves the result.

This split is the whole point of today’s lesson. Once you have a clean single-image function, batching it is just a for loop with error handling around it. Days 1, 2, and 3 of this week all reuse the same shape: do one thing well, then loop.

Understanding RGBA Compositing

A watermark isn’t just text drawn on top of an image — it’s text with transparency. If you set opacity to 60%, that means “60% white text, 40% whatever pixel is underneath.” That’s RGBA compositing, and it’s the foundation of every transparency effect in Pillow.

The pattern:

1. Convert the photo to RGBA (so it can mix with transparent things).

2. Create a separate transparent overlay the same size as the photo.

3. Draw the text onto the overlay with an alpha channel set by opacity.

4. Composite the overlay onto the photo with Image.alpha_composite.

def apply_watermark(image, text, position, font_size, opacity, color):
    # 1. Force RGBA so compositing works
    base = image.convert("RGBA")

    # 2. Transparent overlay
    overlay = Image.new("RGBA", base.size, (0, 0, 0, 0))
    draw = ImageDraw.Draw(overlay)

    # 3. Compute the alpha (0-255) from opacity (0.0-1.0)
    fill_rgb = (255, 255, 255) if color == "white" else (0, 0, 0)
    fill_rgba = (*fill_rgb, int(opacity * 255))

    # 4. Position the text and draw it
    font = ImageFont.truetype(...)  # see below
    x, y = compute_position(text, font, base.size, position)
    draw.text((x, y), text, font=font, fill=fill_rgba)

    # 5. Composite and return
    return Image.alpha_composite(base, overlay)

The critical move is drawing the text onto a separate transparent overlay, not directly onto the photo. Drawing directly works for fully opaque text, but the moment you want partial transparency, the alpha values in fill get ignored unless the canvas itself supports alpha. The overlay does.

Understanding ImageDraw and ImageFont

ImageDraw is Pillow’s drawing API. You wrap a canvas image and call drawing methods on it:

from PIL import ImageDraw, ImageFont

draw = ImageDraw.Draw(overlay)
draw.text((50, 50), "Hello", font=font, fill=(255, 255, 255, 200))

For text, you need a font — and this is where things get a little fiddly across platforms. The cleanest approach: try a likely system font, fall back to Pillow’s built-in if it isn’t found:

def load_font(size):
    candidates = [
        "DejaVuSans-Bold.ttf",                       # Linux
        "/Library/Fonts/Arial Bold.ttf",             # macOS
        "C:/Windows/Fonts/arialbd.ttf",              # Windows
    ]
    for path in candidates:
        try:
            return ImageFont.truetype(path, size)
        except (OSError, IOError):
            continue
    # Last resort: a tiny built-in font (size is not adjustable on this one)
    return ImageFont.load_default()

The truetype paths are platform-specific, so we try a few. load_default() always works but the size is fixed — only acceptable as a fallback.

Understanding Text Measurement and Positioning

To anchor a watermark to a corner, you need to know how big the text actually is in pixels. The modern way in Pillow is textbbox:

# Measure the rendered size of the text
bbox = draw.textbbox((0, 0), text, font=font)
text_w = bbox[2] - bbox[0]
text_h = bbox[3] - bbox[1]

textbbox((x, y), text) returns (left, top, right, bottom) — the bounding box of where the text would be drawn if you called draw.text((x, y), ...). Subtract to get width and height.

With that, anchor positioning is just math:

def compute_position(text_w, text_h, img_w, img_h, position, pad=20):
    if position == "top-left":     return (pad, pad)
    if position == "top-right":    return (img_w - text_w - pad, pad)
    if position == "bottom-left":  return (pad, img_h - text_h - pad)
    if position == "bottom-right": return (img_w - text_w - pad, img_h - text_h - pad)
    if position == "center":       return ((img_w - text_w) // 2, (img_h - text_h) // 2)
    return (pad, pad)

The pad of 20 px keeps the text off the very edge — a small touch that makes a huge visual difference. Without padding, watermarks look glued to the corner; with padding, they look intentionally placed.

Understanding Folder Iteration

Listing images in a folder needs care. Folders contain hidden files, non-image files, subfolders. pathlib makes it tidy:

from pathlib import Path

IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".bmp", ".gif", ".webp"}

def list_images(folder):
    return sorted(
        p for p in Path(folder).iterdir()
        if p.is_file() and p.suffix.lower() in IMAGE_EXTS
    )

Three small choices that matter:

• Set lookup for extensions — O(1) membership and easy to extend.

• p.suffix.lower() — match .JPG as well as .jpg.

• sorted(...) — predictable order in the progress log, instead of whatever the filesystem coughs up.

Understanding Saving and Format

The watermarked image is in RGBA mode. That’s fine for PNG, but JPEG can’t store transparency. So before saving JPEG, we flatten the alpha:

def save_image(img, path):
    if path.suffix.lower() in {".jpg", ".jpeg"}:
        # Flatten to RGB so JPEG can save it
        img = img.convert("RGB")
        img.save(path, quality=92)
    else:
        img.save(path)

convert("RGB") drops the alpha channel by blending against black — fine for our case because the photo underneath has no transparent areas. (If you needed to preserve transparency with JPEG-style compression, you’d save WebP or PNG instead.)

The quality=92 is genuinely worth setting — Pillow’s JPEG default is 75, which is enough for thumbnails but visibly lossy on photos. 92 keeps the output looking essentially identical to the input.

Coming Tomorrow

Tomorrow we make things fun. The Meme Generator & Collage Maker uses today’s text drawing skills for meme captions (the classic top + bottom text in white-with-black-outline) and adds a collage builder that arranges multiple photos into a single grid image. Two image tools, one app, ready to share.

View Code Evolution

Compare today’s batch watermarker with yesterday’s filters studio and see how the single-image transformation pattern scales to whole folders with one loop and a little error handling.

Read more

This week, we build an Image Toolkit with Pillow that lets you edit photos in a desktop app, watermark whole folders of images at once, and turn photos into shareable memes and collages.

Why build this? Because image manipulation is one of the most fun and visible domains in Python. Click a button, see your photo transform — instant feedback you can’t get with text-based tools. Plus the skills transfer everywhere: any time you need to resize, watermark, filter, or generate images, this is the foundation.

What you’ll learn: This series teaches you Pillow (the modern PIL), Tkinter GUI design with image previews, color and pixel manipulation, image filters and convolutions, batch processing, EXIF orientation handling, and the classic bridges between PIL and Tkinter.

Why this matters: By Day 3, you’ll have built a meme generator and collage maker that produces images you’d actually post. That’s portfolio-level fun.

• Day 1: Photo Filters Studio (Tkinter) (Today)

• Day 2: Batch Watermarker

• Day 3: Meme Generator & Collage Maker

View All Projects This Week

Today’s Project

We start with the visual heart of any image toolkit: a filter studio. Load a photo, see it on screen, click filter buttons or drag sliders, watch the image transform in real time, and save the result. By the end you’ll have a polished little Photoshop-lite — and the Pillow fundamentals to do anything image-related in Python.

The classic six one-click filters (Grayscale, Sepia, Blur, Sharpen, Edge Detect, Invert) plus three live sliders (Brightness, Contrast, Saturation). Reset returns to the original. Save writes a new file — your original is never touched.

Project Task

Build a photo filter desktop app with Tkinter and Pillow that:

• Loads an image via a file dialog (JPG, PNG, etc.)

• Displays the image in a preview area, resized to fit

• Correctly handles phone-photo rotation (EXIF orientation)

• Applies one-click filters: Grayscale, Sepia, Blur, Sharpen, Edge Detect, Invert

• Provides live sliders for Brightness, Contrast, and Saturation

• Re-renders the preview every time the user changes something

• Has a Reset button to return to the original

• Saves the edited image as a new file (Save As…)

• Never modifies the original — every edit happens on a copy

This project gives you hands-on practice with Pillow’s Image, ImageFilter, ImageEnhance, and ImageOps modules, Tkinter file dialogs, scales/sliders, the ImageTk bridge for previewing PIL images in a GUI, and the small architecture choices that keep an image editor sane.

Expected Output

Running the app:

python photo_filters.py

Application Window:

Here is a snapshot of how the finished app looks like:

Setup Instructions

Install Pillow:

pip install Pillow

That’s it — Tkinter ships with Python. (On Linux: sudo apt-get install python3-tk if it isn’t already there.)

Get a test image:

A sample.jpg is provided below — a colorful generated scene that shows off every filter beautifully. You can also use any photo from your computer.

Run the app:

python photo_filters.py

Understanding the Image-Editing Architecture

Before any code, the architecture matters. There are three images alive at all times:

self.original   →  The image as loaded from disk. NEVER touched again.
self.edited     →  The current edited version. This is what gets saved.
self.preview    →  A small, screen-sized copy of `edited` for display only.

Two rules fall out of this:

1. Filters and slider changes modify edited — they never touch original.

2. Every change is followed by a refresh_preview() — which creates a fresh preview from edited and updates the GUI.

Reset is then trivial: self.edited = self.original.copy() — and refresh. This separation prevents the most common bugs in image editors (irreversible damage, slow re-renders, getting “stuck” on a filter).

Understanding Pillow Basics

Pillow’s central object is Image. Opening, copying, and saving look like this:

from PIL import Image

img = Image.open("sample.jpg")   # lazy load — actual pixels read on demand
edited = img.copy()              # always work on a copy
edited.save("output.jpg")        # save anywhere with any supported extension

Three habits worth forming on day one:

• Always .copy() before editing. Pillow’s filters often return new images, but it’s safer to think of the original as immutable.

• Don’t worry about format on open. Pillow detects JPG/PNG/BMP/GIF/WebP/HEIC automatically.

• The format on save is taken from the extension — output.png writes PNG, output.jpg writes JPEG.

Understanding EXIF Orientation

Here’s a real-world trap: photos taken on a phone often look correct in Photos but appear rotated 90° when opened with Pillow. That’s because the camera saves the image in its native sensor orientation and stores the intended rotation as EXIF metadata — and Pillow doesn’t apply it automatically.

The fix is one line:

from PIL import ImageOps

img = Image.open("photo.jpg")
img = ImageOps.exif_transpose(img)   # apply the EXIF rotation

exif_transpose reads the EXIF orientation tag and physically rotates/flips the image to match what the user expects. Without it, your “perfectly working” app will mysteriously fail on the first phone selfie someone tries. Call it once, right after opening — done.

Understanding ImageTk: PIL → Tkinter Bridge

Tkinter’s widgets can’t display a PIL Image directly. The bridge is ImageTk.PhotoImage, which wraps a PIL image into a Tkinter-compatible image object:

from PIL import ImageTk

self.tk_image = ImageTk.PhotoImage(self.preview)
self.preview_label.config(image=self.tk_image)

The classic gotcha: if you don’t keep a reference to self.tk_image, Python garbage-collects it and the image vanishes from the screen with no error. Always store it on self. — never as a local variable inside a method.

This is the #1 most common “my image won’t show” bug in Pillow+Tkinter code. Now you’ll never write it.

Understanding Resizing for Preview

A 4000×3000 photo can’t be shown at full size — and resizing on every slider tick would be sluggish. The pattern: resize once, when the image changes, to a screen-friendly version.

Pillow’s thumbnail() method resizes in place and preserves aspect ratio:

preview = self.edited.copy()
preview.thumbnail((600, 600), Image.Resampling.LANCZOS)
self.preview = preview

Two details that matter:

• thumbnail is in-place. It doesn’t return a new image — it modifies the one you call it on. So we copy() first.

• Image.Resampling.LANCZOS is the modern constant for the high-quality downsampling filter. Older code uses Image.LANCZOS (deprecated) or Image.ANTIALIAS (removed). LANCZOS is the right choice for shrinking photos.

Understanding ImageFilter: One-Click Filters

For Blur, Sharpen, and Edge Detect, Pillow ships built-in convolution kernels in ImageFilter:

from PIL import ImageFilter

self.edited = self.edited.filter(ImageFilter.BLUR)
self.edited = self.edited.filter(ImageFilter.SHARPEN)
self.edited = self.edited.filter(ImageFilter.FIND_EDGES)

A “convolution” is just: for each pixel, look at its neighbors, compute a weighted average. Blur averages neighbors. Sharpen subtracts a blurred version from the original. Edge detect amplifies differences between neighbors. You don’t need to write the math — ImageFilter has clean, named presets.

Understanding ImageOps: Grayscale and Invert

For pure color transformations, ImageOps is the right toolbox:

from PIL import ImageOps

self.edited = ImageOps.grayscale(self.edited).convert("RGB")
self.edited = ImageOps.invert(self.edited)

A subtlety with grayscale: ImageOps.grayscale returns a single-channel (”L” mode) image. Converting back to “RGB” keeps the rest of the pipeline happy — later filters and the ImageTk display expect 3-channel images.

invert simply flips every pixel value (255 − v), giving you that classic film-negative look.

Understanding Sepia: Custom Per-Channel Math

Sepia isn’t built in — and that’s a good thing, because it’s the perfect excuse to teach per-pixel/per-channel manipulation. The standard sepia formula transforms each pixel’s RGB values:

new_R = 0.393·R + 0.769·G + 0.189·B
new_G = 0.349·R + 0.686·G + 0.168·B
new_B = 0.272·R + 0.534·G + 0.131·B

Implemented in Pillow:

def apply_sepia(img):
    img = img.convert("RGB")
    pixels = img.load()         # an indexable pixel grid

    for y in range(img.height):
        for x in range(img.width):
            r, g, b = pixels[x, y]
            tr = int(0.393 * r + 0.769 * g + 0.189 * b)
            tg = int(0.349 * r + 0.686 * g + 0.168 * b)
            tb = int(0.272 * r + 0.534 * g + 0.131 * b)
            pixels[x, y] = (min(tr, 255), min(tg, 255), min(tb, 255))

    return img

img.load() returns a pixel-access object you can read and assign with pixels[x, y]. The min(..., 255) clamps values that overflow the 0–255 range. Slow for huge images but perfectly fine for the screen-sized previews we’re editing.

Aside: for large images and production code, you’d use numpy or Image.point() for vectorized math. But the pixel loop is the canonical way to teach what every image filter is doing under the hood, so we use it here. Worth the few seconds.

Understanding ImageEnhance: Adjustable Effects

Brightness, Contrast, and Saturation aren’t on/off filters — the user dials them. That’s what ImageEnhance is for:

from PIL import ImageEnhance

enhancer = ImageEnhance.Brightness(self.edited)
self.edited = enhancer.enhance(1.4)   # 1.0 = unchanged; >1 brighter, <1 darker

The number is a factor, not an amount:

• 1.0 — leave as is

• 1.5 — 50% brighter / more contrast / more saturated

• 0.5 — 50% darker / flatter / desaturated

• 0.0 — completely black / no contrast / fully grayscale

This factor pattern is uniform across Brightness, Contrast, Color (saturation), and Sharpness. Once you know one, you know all four.

Understanding the Slider Refresh Pattern

Sliders fire continuously as the user drags. To keep the UI snappy, each slider re-applies all three enhancements from the original, every time:

def update_adjustments(self, _event=None):
    img = self.original_for_adjust.copy()

    img = ImageEnhance.Brightness(img).enhance(self.brightness_var.get())
    img = ImageEnhance.Contrast(img).enhance(self.contrast_var.get())
    img = ImageEnhance.Color(img).enhance(self.saturation_var.get())

    self.edited = img
    self.refresh_preview()

The key choice: we start each update from self.original_for_adjust, a snapshot taken when the user last clicked a filter. That way the sliders work on top of “grayscale”, “sepia”, etc., but moving a slider doesn’t accumulate enhancements every frame (which would amplify them out of control).

This snapshot-based pattern is the small architectural trick that makes the studio feel like a real editor instead of a glitchy demo.

Understanding Saving

Pillow infers the output format from the file extension:

def save_as(self):
    path = filedialog.asksaveasfilename(
        defaultextension=".jpg",
        filetypes=[("JPEG", "*.jpg *.jpeg"),
                   ("PNG",  "*.png"),
                   ("All",  "*.*")],
    )
    if not path:
        return
    self.edited.save(path)

Two important details:

• JPEG can’t store transparency. If the user has applied filters that introduce an alpha channel (rare here, but possible), save to PNG. Otherwise force RGB before saving JPEG: self.edited.convert("RGB").save(path).

• JPEG quality defaults to 75. For photos worth sharing, add quality=92 to the save call.

Our app saves whatever PIL deems best for the extension, which works for the 95% case.

Coming Tomorrow

Tomorrow we go from one photo to a whole folder. The Batch Watermarker adds a text or logo watermark to every image in a folder, with position controls, opacity, and a live preview — turning today’s manual workflow into a one-click batch operation.

Skeleton and Solution

Below you will find both a downloadable skeleton.py file to help you code the project with comment guides and the downloadable solution.py file containing the correct solution.

Get the code skeleton here:

View Code Skeleton

Get the code solution here:

View Code Solution

This week, we build a Web Scraping with BeautifulSoup suite that extracts a complete product catalog from a real website, scales up to thousands of items across many pages, and turns the scraped data into a searchable mini-database.

• Day 1: Scrape a Single Page

• Day 2: Multi-Page Scraping with Pagination

• Day 3: Search & Analyze the Scraped Data (Today)

View All Projects This Week

Today’s Project

Welcome to the finale. You have a dataset of 1,000 scraped books. Now what? Now you query it. Today we turn the CSV into an interactive command-line search engine — keyword search, filters by category, price, and rating, sorting, and on-demand statistics. The scraping pipeline becomes a real tool you can use.

This is what makes scraping worth doing. Data sitting in a CSV is just rows. The same data behind a query interface becomes a mini-database — yours to ask questions of.

Project Task

Build an interactive book search and analysis tool that:

• Loads all_books.csv from Day 2 into a pandas DataFrame

• Offers a friendly REPL — type a command, see results

• Supports keyword search across titles and descriptions

• Filters by category, price range, rating, and stock status

• Combines filters (category=Fiction min_price=20 rating>=4)

• Sorts results by any field (sort=price desc)

• Shows summary statistics on the full catalog or any filter

• Exports the current view to a CSV

• Has a clear help command and gracefully handles bad input

This project gives you hands-on practice with pandas filtering, boolean masks, string matching, building a small CLI loop, parsing command arguments, and turning a dataset into a tool people can actually use.

Expected Output

Running the tool:

python search_books.py

Interactive session:

The user can interact with the program in the terminal. All the text in red color are commands the user has submitted:

You drop a search command and your dataset answers. Combine filters and it slices in real time. That’s a tool.

Setup Instructions

Install pandas:

pip install pandas

(You also need all_books.csv — the output of Day 2. If you haven’t run that yet, run it first, or use the sample file provided below)

Download all_books.csv

Run the program:

python search_books.py

The tool drops you into an interactive prompt. Type help to see the commands; type quit to exit.

Understanding the REPL Loop

A REPL — Read, Evaluate, Print, Loop — is just a while loop around input() and a dispatch on what the user typed. It’s the simplest way to build an interactive tool, and it suits this kind of dataset exploration perfectly:

def run_repl(df):
    current = df  # the current "view" - starts as the whole dataset

    while True:
        line = input("> ").strip()
        if not line:
            continue
        if line == "quit":
            break

        # parse the command and dispatch
        current = handle_command(line, df, current)

Three habits worth noticing:

• An empty line just continues — common enough to handle explicitly.

• The user typed string is .strip()ped — trailing whitespace from a paste shouldn’t break commands.

• current holds the filtered view, separate from df (the full catalog) — so reset always has the full data to return to.

Understanding Command Parsing

Each command starts with a word (search, filter, sort, stats...) followed by arguments. The simplest parse: split on whitespace, take the first token as the command name:

def handle_command(line, df, current):
    parts = line.split()
    cmd = parts[0].lower()
    args = parts[1:]

    if cmd == "search":
        return command_search(current, " ".join(args))
    if cmd == "filter":
        return command_filter(df, args)
    if cmd == "stats":
        command_stats(current)
        return current
    # ...

Each command gets its own function. command_search and command_filter return the new view (so the REPL can update current); command_stats just prints and returns the view unchanged. This keeps the dispatch readable — each command does one thing in one place.

Understanding pandas String Search

The search command should match across titles and descriptions, case-insensitively. pandas’ .str.contains() does the heavy lifting:

def command_search(df, query):
    if not query:
        print("  Usage: search <keyword>")
        return df

    title_match = df["title"].str.contains(query, case=False, na=False)
    desc_match = df["description"].str.contains(query, case=False, na=False)

    matches = df[title_match | desc_match]
    show_results(matches, f"Found {len(matches)} matches for '{query}'")
    return matches

A few important details:

• case=False — case-insensitive matching. "python" finds "Python".

• na=False — treats missing values as “no match” instead of NaN. Without it, you get errors when the column has any blanks.

• | — boolean OR between the two masks. A book matches if either the title or the description contains the keyword.

The result is a new DataFrame — a filtered view of the catalog — that becomes the new current. The next command operates on those matches.

Understanding Boolean Masks

Filtering in pandas is built on boolean masks: a column of True/False the same length as the DataFrame, used to keep only the True rows.

# Books over £30
mask = df["price"] >= 30
expensive = df[mask]

# Books over £30 AND rated 4+
mask = (df["price"] >= 30) & (df["rating"] >= 4)
expensive_and_good = df[mask]

Two non-obvious rules every pandas user hits:

• Use & and |, not and and or. The Python keywords don’t work on column-level booleans.

• Wrap each condition in parentheses. & has higher precedence than >=, so without parens the order is wrong and you get errors.

(condition) & (condition) & (condition) is how every multi-filter query in pandas looks. Get used to it.

Understanding the Filter Command

The filter command takes key-value pairs like category=Fiction min_price=20 rating>=4. We parse each piece, build a list of conditions, then combine them:

def command_filter(df, args):
    mask = pd.Series(True, index=df.index)   # start: keep everything
    applied = []

    for arg in args:
        if arg.startswith("category="):
            value = arg.split("=", 1)[1].lower()
            mask &= df["category"].str.lower().str.contains(value, na=False)
            applied.append(f"category={value}")

        elif arg.startswith("min_price="):
            value = float(arg.split("=", 1)[1])
            mask &= df["price"] >= value
            applied.append(f"price>={value}")

        elif arg.startswith("max_price="):
            value = float(arg.split("=", 1)[1])
            mask &= df["price"] <= value
            applied.append(f"price<={value}")

        elif arg.startswith("rating>="):
            value = int(arg.split("=", 1)[1])
            mask &= df["rating"] >= value
            applied.append(f"rating>={value}")

    result = df[mask]
    print(f"  Filter applied: {', '.join(applied)}")
    show_results(result, f"{len(result)} books match.")
    return result

Two patterns worth taking away:

• Build masks incrementally with &= — starting from True everywhere, each condition narrows the result. Whether the user passes one filter or four, the same loop handles it.

• Always re-filter from the full catalog, not the current view. That way filter category=Fiction replaces the previous filter rather than narrowing within it — usually what you actually want.

Understanding the Sort Command

sort price desc reorders the current view. The parse picks up the column and direction:

def command_sort(df, args):
    if not args:
        print("  Usage: sort <field> [asc|desc]")
        return df

    field = args[0]
    ascending = not (len(args) > 1 and args[1].lower() == "desc")

    if field not in df.columns:
        print(f"  Unknown field: {field}")
        return df

    sorted_df = df.sort_values(field, ascending=ascending).head(10)
    direction = "ascending" if ascending else "descending"
    print(f"  Sorted by {field} ({direction}). Showing top 10:")
    show_results(sorted_df, "")
    return df  # return original; don't permanently sort the view

A subtlety: sort should display the sorted top-10 but not change the filter state. So we show the sorted view, but return the unsorted current view. The user expects “show me the top by price” to be a view, not a state change — the next filter shouldn’t be operating on a sliced sorted list.

Understanding Formatting Aligned Output

A search result is just a DataFrame — but print(df) looks ugly. We format each row manually so titles align, prices line up by the decimal, and ratings use stars:

def show_results(df, header_message):
    if header_message:
        print(f"\n  {header_message}")
    if df.empty:
        print("  (no results)")
        return

    print()
    print(f"    {'Title':<40} {'Category':<18} {'Price':>8} {'Rating':>10}")
    print("    " + "─" * 80)

    for _, row in df.head(20).iterrows():
        title = row["title"]
        if len(title) > 40:
            title = title[:37] + "..."
        stars = "★" * int(row["rating"])
        print(f"    {title:<40} {row['category']:<18} "
              f"£{row['price']:>6.2f}   {stars:>8}")

.head(20) caps the output — nobody scrolls through 1,000 rows in a terminal. The column widths (:<40, :<18, :>8) and right-aligned price with two decimals (>6.2f) make the table read like a real product, not raw data.

Understanding the Stats Command

stats runs on the current view, so the same command answers both “stats for the full catalog” and “stats for what I just filtered.” It’s all pandas aggregation:

def command_stats(df):
    if df.empty:
        print("  No books in current view.")
        return

    print(f"\n  Books:            {len(df)}")
    print(f"  Categories:       {df['category'].nunique()}")
    print(f"  Average price:    £{df['price'].mean():.2f}")
    print(f"  Price range:      £{df['price'].min():.2f} – £{df['price'].max():.2f}")
    print(f"  Avg rating:       {df['rating'].mean():.2f} / 5")
    in_stock = (df["stock_count"] > 0).sum()
    print(f"  In stock:         {in_stock} ({in_stock / len(df) * 100:.0f}%)")

    print("\n  Rating breakdown:")
    for rating in range(1, 6):
        count = (df["rating"] == rating).sum()
        stars = "★" * rating
        print(f"    {stars:<13} {count:>3}")

    print("\n  Top 5 categories by book count:")
    top = df["category"].value_counts().head(5)
    for cat, count in top.items():
        print(f"    {cat:<22} {count:>3}")

The reusable trick: df['category'].value_counts() — counts how often each value appears, sorted descending, in one line. It’s the fastest way to build a “top categories” or “top X” list from any column.

Understanding the Export Command

save results.csv writes the current view to a file. One line:

def command_save(df, args):
    if not args:
        print("  Usage: save <filename>")
        return df
    path = args[0]
    df.to_csv(path, index=False)
    print(f"  ✓ Exported {len(df)} books to {path}")
    return df

This is the closing of the loop: scrape → enrich → query → export the answer. The user can take their filtered subset into pandas, Excel, or another tool. The tool becomes a gateway, not just an endpoint.

Understanding Graceful Error Handling

Users mistype things. A REPL that crashes on a typo is unusable, so each command wraps its risky parts in try/except:

try:
    value = float(arg.split("=", 1)[1])
except ValueError:
    print(f"  Could not parse: {arg}")
    continue

The unknown-command fallback is just as important — help is a single press away, and bad input is just a printed message:

print(f"  Unknown command: {cmd}. Type 'help' for the list.")

Quiet, clear, forgiving. A tool you’d actually use.

Understanding Why This Is the Right Finale

Look at the three days together:

• Day 1 taught you to extract from one page.

• Day 2 taught you to scale across many pages with pagination and detail-page enrichment.

• Day 3 taught you to use the result — turning raw scraped data into answers.

The arc matters: scraping by itself isn’t valuable, it’s the dataset and what you do with it that matters. Every real scraping project ends with a query interface, a dashboard, an analysis — something that turns rows into decisions. Today’s REPL is the smallest, simplest version of that, and it makes the whole week click into place.

What You’ve Accomplished This Week

🎉 Congratulations! You’ve built a complete web scraping pipeline:

• Day 1: Extract structured data from a single web page

• Day 2: Scale across pagination + detail pages, with polite delays

• Day 3: Turn the scraped catalog into an interactive search tool

You now have:

✅ Scraping fundamentals — requests, BeautifulSoup, CSS class selection, attributes ✅ Pagination and multi-level scraping — listing pages + detail pages, combined cleanly ✅ Polite scraping habits — User-Agent, timeouts, delays, error handling ✅ Querying skills with pandas — boolean masks, string search, sorting, aggregation ✅ A real tool — your scraped data, queryable in real time

Next steps:

• Add a charts module: matplotlib distribution plots for price and rating

• Persist the dataset to SQLite for fast queries on larger catalogs

• Build a Streamlit version of the query interface

• Add a watcher: nightly rescrape, diff against the previous CSV

• Handle login-required sites with requests.Session

You’ve built the foundation for a real scraping-and-analysis pipeline. 🚀

View Code Evolution

Compare today’s query tool with Day 1’s single-page scraper and Day 2’s full-catalog scraper — and see how a clean three-step pipeline (extract → scale → query) is the shape of every real scraping project.

Read more

This week, we build a Web Scraping with BeautifulSoup suite that extracts a complete product catalog from a real website, scales up to thousands of items across many pages, and turns the scraped data into a searchable mini-database.

• Day 1: Scrape a Single Page

• Day 2: Multi-Page Scraping with Pagination (Today)

• Day 3: Search & Analyze the Scraped Data

View All Projects This Week

Today’s Project

Yesterday we scraped 20 books from one page. Today we scrape 1,000 books from 50 pages — the full catalog of books.toscrape.com. Then we go deeper: for each book, we also visit its detail page to pull richer information you can’t get from the listing — the exact UPC, real stock count, full description, and category.

This is what a real scraper looks like: it follows links, scales across pages, and handles failures politely. By the end of today, you’ll have a single CSV holding every book on the site, ready for Day 3’s search tool.

Project Task

Build a multi-page book scraper that:

• Scrapes every book across all 50 catalog pages

• Detects the “next page” link to follow pagination automatically

• Visits each book’s detail page to pull richer fields (UPC, stock count, description, category)

• Adds a polite delay between requests

• Shows clear progress (page X of Y, book X of Y)

• Continues running if one page or detail page fails

• Saves everything to a single CSV

• Reports a final summary with counts and totals

This project gives you hands-on practice with pagination, two-level scraping (listing → detail page), polite rate limiting, robust error handling, progress reporting, and combining data from multiple HTTP requests — the techniques behind every production-grade scraper.

Expected Output

Running the full-catalog scraper:

python scrape_all_books.py

Console Output:

The script will run and visit different pages (50 in total) of the books.toscrape.com website and print out info for each book in the terminal:

Generated all_books.csv:

In addition to being printed in the terminal, the data will also be saved in a CSV file. Here is a snapshot of the file:

We will have 1,000 rows, 9 columns, every book on the site. Ready for tomorrow’s search tool.

Setup Instructions

Install Required Packages:

pip install requests beautifulsoup4

(Same as Day 1 — no new dependencies.)

Run it:

python scrape_all_books.py

The full run takes roughly 5-10 minutes depending on your internet — that’s 50 listing pages + 1,000 detail pages = 1,050 requests with a 0.2 second delay between each. The script prints progress so you’ll know it’s working.

Tip: if you just want to see it work without waiting, set MAX_PAGES = 3 near the top of the script for a quick three-page test.

Understanding Pagination

The first scaling problem is finding all the pages. We could hardcode page-1.html through page-50.html, but that breaks the moment the site adds page 51. The robust way is to let the page tell us where to go next.

Look at the bottom of any catalog page and you’ll find:

<li class="next">
  <a href="page-2.html">next</a>
</li>

So our strategy is: scrape the current page, look for <li class="next">, follow it. When that link doesn’t exist, we’re done.

def find_next_url(soup, current_url):
    """Return the absolute URL of the next page, or None if this is the last."""
    next_li = soup.find("li", class_="next")
    if next_li is None:
        return None  # we've reached the end of the catalog
    next_href = next_li.find("a")["href"]
    return urljoin(current_url, next_href)

That urljoin(current_url, ...) is doing real work: pages 2 through 50 use relative links like page-3.html, and urljoin resolves each one against the current page’s URL. The result is always a correct absolute URL.

Understanding the Main Pagination Loop

With find_next_url in hand, the main scrape becomes a small while loop:

url = BASE_URL
page_number = 0
all_books = []

while url:
    page_number += 1
    print(f"Page {page_number}:  {url}")

    html = fetch_page(url)
    if html is None:
        break  # can't fetch this page, stop the run

    soup = BeautifulSoup(html, "html.parser")
    books = scrape_listing_page(soup)
    all_books.extend(books)

    url = find_next_url(soup, url)   # None when we've reached the end
    time.sleep(REQUEST_DELAY)

The whole loop reads in plain English: fetch this page, extract its books, find the next URL, sleep, repeat. When find_next_url returns None, the loop ends naturally. No page counting, no hardcoded limit, no fragility.

Understanding Polite Scraping

Hitting a server with 1,050 requests as fast as Python can send them is rude — and a great way to get rate-limited or banned. The fix is a small delay between requests:

import time

REQUEST_DELAY = 0.2   # seconds between requests

# ...inside the loop, after each request:
time.sleep(REQUEST_DELAY)

0.2 seconds is invisible to you but a huge break for the server — five requests per second is gentle for any well-running site. The fact that books.toscrape.com explicitly invites scraping doesn’t change the principle: every scraper you write should pause between requests. The habit matters more than this particular site’s tolerance.

Understanding Two-Level Scraping

A listing page tells you a book exists. The detail page tells you everything about it — the UPC, the precise stock count, the full description, the category. To get those, we visit each book’s detail URL.

The pattern is two-level: scrape the listing, then for each book in that listing, fetch and scrape its detail page.

def enrich_with_details(book):
    """Fetch the book's detail page and add the extra fields to its dict."""
    html = fetch_page(book["url"])
    if html is None:
        # detail fetch failed; keep the listing data, add empty extras
        book.update(stock_count=None, category=None, upc=None, description=None)
        return book

    soup = BeautifulSoup(html, "html.parser")
    book["upc"] = extract_upc(soup)
    book["stock_count"] = extract_stock_count(soup)
    book["category"] = extract_category(soup)
    book["description"] = extract_description(soup)
    return book

The function takes a book dict (from the listing scrape) and returns the same dict with the extra fields added. This composes cleanly — listing data and detail data end up in one place.

Understanding find_next_sibling

find_next_sibling("p") is worth a closer look — it’s the move for “find me the next paragraph after this element.” That’s how the description is structured on the site: a header div followed by the description paragraph at the same level.

header = soup.find("div", id="product_description")
description_p = header.find_next_sibling("p")

This pattern shows up constantly in scraping when content is logically related but lives in separate tags. Sibling navigation lets you say “the thing right after this one,” without having to know its position in the document. You’ll reach for it again and again.

Understanding Saving the Combined Data

The output schema is the union of listing data and detail data — nine columns total:

FIELDNAMES = [
    "title", "price", "rating", "availability",
    "stock_count", "category", "upc", "description", "url",
]

with open("all_books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=FIELDNAMES)
    writer.writeheader()
    writer.writerows(all_books)

Because every book is a dict with the same keys, DictWriter aligns the columns correctly even when a few rows have empty description fields. The result is a single, clean CSV — 1,001 lines including the header — that becomes the input for Day 3.

Coming Tomorrow

Tomorrow we put the dataset to work. The Search & Analyze tool takes all_books.csv and turns it into a query interface: search by title or keyword, filter by category, price range, and rating, and generate summary statistics — average prices per category, distribution of ratings, top-N by any field. Your scraped data becomes a small but real, useful database.

View Code Evolution

Compare today’s full-catalog scraper with yesterday’s single-page version and see how a small pagination loop and a detail-page step scale 20 books into 1,000 — without rewriting the extraction logic.

Read more