The Brier score, explained with examples

The Brier score measures how good a probability forecast was. It is the squared difference between the probability you gave and what actually happened, averaged over all your forecasts. Lower is better: 0 is perfect, 0.25 is what you get by always saying 50%, and 1 is the worst possible.

It is also how you check any claim that AI can predict the future. It was introduced by the meteorologist Glenn W. Brier in 1950 to grade weather forecasts (Brier, 1950), and it is now the standard way to score forecasters, from weather services to AI superforecasters.

Try it: score your own forecasts with our free Brier score calculator. Paste probabilities and outcomes, and get your Brier score, skill score and a calibration check.

The formula

For a yes-or-no question, the Brier score of one forecast is:

Brier score = (forecast − outcome)²

• The forecast is the probability you gave, written from 0 to 1. So 70% is 0.7.
• The outcome is 1 if the event happened and 0 if it did not.

For many forecasts, add up the scores and divide by the number of forecasts.

Worked examples

Two patterns stand out:

• Confident and right is rewarded. A 90% call that happens scores 0.01.
• Confident and wrong is punished hard. A 90% call that misses scores 0.81, more than three times a coin flip.

Because the error is squared, a few overconfident misses can wreck an otherwise good record.

A four-forecast example

Say you made four forecasts:

1. 80% that a central bank cuts rates. It did. Score (0.8 − 1)² = 0.04
2. 30% that a launch happens this year. It did not. Score (0.3 − 0)² = 0.09
3. 60% that a bill passes. It did not. Score (0.6 − 0)² = 0.36
4. 95% that a company reports on time. It did. Score (0.95 − 1)² = 0.0025

Your Brier score is the average: (0.04 + 0.09 + 0.36 + 0.0025) ÷ 4 = 0.123. That is clearly better than the 0.25 you would get by saying 50% every time.

What is a good Brier score?

It depends heavily on how hard the questions are, but these rules of thumb help:

• 0.25 is the coin-flip line: saying 50% on everything. Anything above it is worse than not trying.
• Around 0.2 means you are adding some information, but not much.
• Below 0.15 is solid on genuinely uncertain questions.
• Below 0.10 on hard, real-world questions is excellent, the territory of top forecasters.

Never compare Brier scores across different question sets. Easy questions ("Will the sun rise tomorrow?") make everyone look brilliant. Only compare forecasters on the same questions.

Why you cannot game it

The Brier score is a proper scoring rule: your expected score is best when you report exactly what you believe. If you think something is 70% likely, saying 90% to sound bold makes your expected score worse, and so does hedging down to 50%.

This is what makes it fair. It rewards honesty, not confidence and not caution. It is also why the score pairs so naturally with calibration: a forecaster who wants a good Brier score has every reason to make their probabilities mean what they say.

What the score is made of

In 1973 Allan Murphy showed that the Brier score splits into three parts (Murphy, 1973):

• Reliability: how well your probabilities match reality. This is calibration. Lower is better.
• Resolution: how well your forecasts separate the events that happen from those that don't. Higher is better.
• Uncertainty: how unpredictable the questions were in the first place. You cannot control this.

A forecaster who always says the average rate can be perfectly calibrated and still useless, because they have no resolution. Good forecasters are calibrated and decisive when the evidence allows.

Brier skill score: comparing against a baseline

A raw Brier score is hard to judge on its own, because it depends on how hard the questions were. The Brier skill score (BSS) fixes that by comparing your score with a simple baseline on the same questions:

BSS = 1 − (your Brier score ÷ baseline Brier score)

The baseline is usually a strategy anyone could follow without skill, such as always forecasting the historical base rate (in weather forecasting this is called climatology), or always saying 50%.

How to read it:

• 1 is perfect: you removed all of the baseline's error.
• Above 0 means you beat the baseline. 0.5 means you removed half of its error.
• 0 means you did no better than the baseline.
• Below 0 means you did worse than the baseline: your forecasts added noise.

Worked example

Take the four forecasts above. Two of the four events happened, so a forecaster who always said 50% would score 0.25 on every question: a baseline Brier score of 0.25. Your score was 0.123, so:

BSS = 1 − (0.123 ÷ 0.25) = 0.51

You removed about half of the baseline's error. The Brier score calculator computes the skill score against the base rate, a coin flip or a prediction market for you.

Why it matters for rare events

For rare events, a low Brier score can flatter you. If an event happens 10% of the time, always forecasting 10% already scores 0.1 × 0.9 = 0.09. A forecaster with a Brier score of 0.08 looks excellent next to the 0.25 coin-flip line, but against the right baseline the skill score is only 1 − (0.08 ÷ 0.09) = 0.11. That is why serious comparisons report skill against a baseline, not raw Brier scores alone. It is also how AI forecasting benchmarks compare AI forecasters with humans and prediction markets; see AI vs. superforecasters.

Brier score versus log loss

Log loss is another proper scoring rule. It punishes confident misses even more severely: a 100% forecast that turns out wrong gets an infinite penalty. The Brier score is bounded, easier to explain, and more forgiving of a single disaster, which is why most forecasting tournaments use it.

A note on the original formula

Brier's 1950 paper summed the error over both possible outcomes, which gives a range of 0 to 2 for a yes-or-no question. Most forecasting work today uses the simpler version on this page, from 0 to 1. Rankings are the same; only the scale differs.

Key takeaways

• Brier score = (forecast − outcome)², averaged over many forecasts. Lower is better.
• 0 is perfect, 0.25 is always saying 50%, 1 is certain and wrong.
• Confident misses are punished hard, because the error is squared.
• It is a proper scoring rule: honesty gives the best expected score.
• Only compare Brier scores on the same questions, ideally with a skill score.

FAQ

Is a lower or higher Brier score better?

Lower. A Brier score of 0 means every forecast was perfect. Always forecasting 50% scores 0.25.

What is the Brier score of a coin flip?

0.25. Saying 50% scores (0.5 − 1)² = 0.25 if the event happens and (0.5 − 0)² = 0.25 if it does not.

What is a good Brier skill score?

Anything above 0 beats the baseline, and 1 is perfect. There is no universal "good" value, because it depends on the questions and on which baseline you use, but on hard real-world questions even a modest positive score shows real skill. A negative score means the forecasts did worse than the baseline.

What is the difference between the Brier score and calibration?

Calibration asks whether your 70% forecasts come true about 70% of the time. The Brier score measures overall accuracy and includes calibration as one of its three parts, together with resolution and uncertainty.

Sources

• Brier, G. W. (1950): Verification of forecasts expressed in terms of probability, Monthly Weather Review
• Murphy, A. H. (1973): A new vector partition of the probability score, Journal of Applied Meteorology
• Wikipedia: Brier score
• UVA Library: A Brief on Brier Scores