Measured

AI, tested and proven.

Three questions. Guess a number, then watch it get measured.

Question 1 of 3From chapter 1 of the book

One prompt, sent to the same AI model 20 times at its default settings. How many distinct answers came back?

The three measurements

Each number below comes from a run in the book's own repository.

  1. Question 1 of 3From chapter 1 of the book

    One prompt, sent to the same AI model 20 times at its default settings. How many distinct answers came back?

    Measured: 10 distinct answers out of 20.

    A scale from 1 to 20. The measured value, marked with a gold tick, is 10 distinct answers out of 20.

    What happened
    10 distinct answers in 20 calls. The most common one came back 8 times.
    Why the gap
    A model picks its next words by sampling, so by default the same prompt does not give the same text twice. All 20 answers agreed in meaning: only the wording moved.
    How it was measured
    20 calls with one prompt to claude-opus-4-6. Identical replies were grouped and the groups counted.
    In the book
    Chapter 1 of the book runs the same 20 calls under a second sampling setting and counts again.
    Run file:
    runs/ch1-sampling-2026-09-07.txt
    Script:
    scripts/ch1_sampling.py
  2. Question 2 of 3From chapter 8 of the book

    An AI assistant with three tools gets one short question. The instructions for its tools travel with the question. What share of that first message was instructions?

    Measured: 98.0 % of the first message.

    A scale from 0 to 100. The measured value, marked with a gold tick, is 98.0 % of the first message.

    What happened
    The first message was 748 tokens long (a token is a piece of a word). 733 of them were instructions for the tools. The question itself was 8.
    Why the gap
    The instructions for every tool are sent in full with each message, whether or not a tool gets used. A short question does not make them shorter.
    How it was measured
    Counted with the model provider's own token counter on claude-opus-4-6, by comparing the message with and without each part.
    In the book
    Chapter 8 of the book measures what happens to this cost when one assistant is split into several.
    Run file:
    runs/ch5-context-composition.txt
    Script:
    scripts/ch5_context_composition.py
  3. Question 3 of 3From chapter 14 of the book

    An AI assistant handles 12 conversations, one after the other. Then it handles the same 12 with 8 going at once. How many times faster did it finish?

    Measured: 5.75 times faster.

    A scale from 1 to 12. The measured value, marked with a gold tick, is 5.75 times faster.

    What happened
    One after the other, the 12 conversations took 123.4 seconds. With 8 going at once, they took 21.5 seconds: 5.75 times faster, not 8.
    Why the gap
    With 12 conversations and 8 going at once, there are two rounds, and the second is only half full. Each conversation also took about as long as before: the waiting overlaps, it does not shrink.
    How it was measured
    The same 12 conversations were run on claude-opus-4-6 with 1, 2, 4 and 8 going at once, and the total time on the clock was recorded for each.
    In the book
    Chapter 14 of the book looks at what running conversations side by side costs, and when not to do it.
    Run file:
    runs/ch14-async-2026-10-28.txt
    Script:
    scripts/ch14_async_run.py

What Measured is

Most AI content shows a demo. Measured shows the number: what was tested, what happened, why, and how it was measured.

The book

Front cover of the book AI Engineering, Measured
Launching soon

AI Engineering, Measured

by Hassan Mansour

The book. Every number in it came from a run. Launching soon.

About the bookJoin the waitlist

One step leads to the next

  1. Free

    The three questions on this page. Free tools will be added here as each one is built.

  2. Courses

    Measured Academy, with a test in every module. No course is open yet.

  3. For teams

    A hands-on workshop, 2 hours or 3.5 hours, quoted for your team after a free 20-minute fit call.

The Measured family

  • Launching soon

    Measured Academy

    by Hassan Mansour

    Courses with a test in every module. For learners aged 18 and over.

  • Live

    AI at Work, Measured

    A hands-on team workshop, 2 hours or 3.5 hours. The first pilot has not run yet.

  • Live

    Proofline

    An open-source multi-agent pipeline with evals and a human approval gate, built in public.

  • Live

    Measured Lab

    Check AI work against rules you can read, in your browser: Evaluate, Observe and Secure, with result links anyone can run again. Open now.

  • Live

    Measured Index

    by Hassan Mansour

    A model comparison on Hassan's own test sets: score, cost per run and speed, each number linked to its raw run file. Issue 1 is out.

Who measures

Hassan Mansour

Senior Applied AI Developer

Works on agentic AI and RAG in a regulated enterprise.

  • 6+ years at Bell Canada building production systems, about the last 3 in AI
  • Bell Hackathon winner; two-time recipient of the Bell Canada Excellence and Innovation Award
  • NSERC and FRQNT research scholarship recipient
  • BSc Computer Science, Concordia; MSc candidate, Multilingual Digital Communication, McGill
  • Works in English and French; native Arabic

Waitlist

Waitlist opens soon

There is no form here yet, so this page collects nothing about you.

Pick your way in