Building a Prediction Market Trading Journal: Reviewing Decisions, Not Just Outcomes

Show people the exact same decision twice — once where it worked out, once where it didn't — and they rate the quality of the reasoning completely differently. A 2023 replication of Baron and Hershey's classic outcome-bias experiment put 692 participants through it and measured a Cohen's d of 1.10 — a very large effect. Rating an identical decision on a seven-point scale running from −3 to +3, participants gave it a mean of 1.81 when the outcome was good and 0.45 when it was bad: a gap of 1.36 points for reasoning that was word-for-word the same (Aiyer et al., International Review of Social Psychology, 2023).

That's an enormous effect, and it's aimed straight at how most traders review their week. You scroll the P&L, feel good about the greens, and quietly re-litigate the reds. In a market where every contract is an explicit probability, that's the worst possible review method — because a well-priced 70% bet is supposed to lose three times out of ten.

**Key Takeaways** - Outcome bias is measured at d = 1.10 — people rate an identical decision 1.36 points higher on a 7-point scale when it happens to win ([Aiyer et al.](https://pmc.ncbi.nlm.nih.gov/articles/PMC12372742/), 2023) - The one column that makes a prediction market journal different from a stock journal: **your probability estimate at entry vs the market price at entry** - A ~45-minute probability-training module measurably improved forecaster Brier scores and calibration in a 2,246-person tournament ([Mellers et al.](https://sydneyscott.nfshost.com/pubs/Psychological_Strategies_for_Winning_a_G.pdf), *Psychological Science*, 2014) - Superforecasters scored a Brier of 0.035 vs 0.13 for untrained individuals, against 0.25 for saying "50%" every time — converted from the paper's 0–2 scale ([Mellers et al.](https://sydneyscott.nfshost.com/pubs/Psychological_Strategies_for_Winning_a_G.pdf), 2014) - Your calibration numbers are noise until you have well over 100 resolved trades ([Metaculus](https://www.metaculus.com/notebooks/28552/comparing-forecasting-track-records-for-ai-benchmarking-and-beyond/), 2024)

Prediction market volume went from under $5 billion a month in September 2025 to roughly $24 billion a month by April 2026 — nearly 5x in seven months, against roughly $14 billion a month for all legal US sportsbooks in 2025 (Pew Research Center, May 2026). The two figures aren't strictly comparable — notional market volume counts both sides and every resale, sportsbook handle counts money wagered — but the direction is unmistakable. A lot of new money is about to learn the same lesson the hard way. This post is the review process that shortens it.

We've written before about validating a strategy *before* you deploy it. This is the other half: what to record, and how to score it, once real decisions are hitting the book.

Your P&L Cannot Tell You Whether You Traded Well

Annie Duke calls it "resulting" — judging a decision by how it turned out. In continuous markets it's a bad habit. In binary contracts it's a category error, because the payoff is a coin flip weighted by a number you can actually check.

Here's the structural problem. If you buy YES at 30¢ and it resolves NO, you lost. But you lost on a bet that was supposed to lose 70% of the time. A single trade carries almost no information about whether you priced it correctly. You need the distribution, and the distribution only exists if you wrote down what you believed at the moment you clicked.

**Why individual outcomes mislead:** Baron and Hershey's original 1988 study found an outcome-bias effect of d = 0.53 on a sample of 20. The 2023 replication with 692 participants found **d = 1.10** — twice as strong. Participants shown a physician's identical decision to operate rated its reasoning **1.81** when the patient survived and **0.45** when the patient died, on a scale running −3 to +3 ([Aiyer et al.](https://pmc.ncbi.nlm.nih.gov/articles/PMC12372742/), 2023). The scenario is medical, but the mechanism is the one that runs your Sunday P&L review: the outcome leaks backward into your judgment of the reasoning.

The stakes here aren't academic. Of Brazilians who started day trading between 2013 and 2015 and stuck with it for more than 300 days, 97% lost money. Only 1.1% earned more than minimum wage (Chague, De-Losso & Giovannetti, 2020). ESMA's 2018 product-intervention analysis found 74–89% of retail CFD accounts lose money, with average losses between €1,600 and €29,000 (ESMA, 2018). Those populations weren't short on screen time. The plausible missing ingredient is structured feedback — neither study measured journaling, but neither population had an obvious way to tell a bad process from a bad run.

What to Log: The 14-Field Trade Record

The trading-journal templates we reviewed — StockBrokers.com's Excel template and the standard Edgewonk/TraderSync/TradeZella column sets — all track roughly the same things: entry price, exit price, P/L, days held, stop, target, R:R, and a free-text "lesson learned" box. That's a stock trader's journal. None of them has a field for what you thought the probability was, which means none of them can score anything.

This is the schema we settled on. Fourteen fields, four of which don't appear on any of the templates above:

Four of these do the real work.

my_probability is the whole ballgame. Without it you have a P&L, not a journal. With it, every resolved trade becomes a scored forecast. It has to be captured before entry — a probability you reconstruct afterward is contaminated by hindsight and worth nothing.

edge_bps makes your edge measurable ex ante. You believed 71%, the book said 63%, so you claimed 800 basis points of edge. Over a hundred trades you can ask a question that's normally unanswerable: does my claimed edge actually predict my realized returns? If trades where you claimed 1,500bps don't outperform trades where you claimed 300bps, your estimates aren't carrying information — your P&L is coming from somewhere else.

thesis_tag must be a closed enum. Free-text theses are unanalyzable. Pick eight to twelve tags (spot_lead, stale_quote, news_lag, resolution_drift, sports_lineup, mean_revert, vol_crush, arb_leg) and force every trade into one. Then you can compute a Brier score per thesis and find out that six of your eight ideas are noise.

exit_trigger written before entry is what makes exits reviewable. "I got out because it felt wrong" can't be scored. price>=0.85 OR t<5m can — you can check whether you actually followed it.

If you're doing this by hand, that's a fourteen-column spreadsheet and under a minute per trade. If a bot is trading, it should emit the record itself:

{
  "trade_id": "KXBTCD-26AUG31-T112000-Y-0417",
  "logged_at": "2026-08-31T14:02:11Z",
  "market": "KXBTCD-26AUG31-T112000",
  "side": "YES",
  "market_price": 0.63,
  "my_probability": 0.71,
  "edge_bps": 800,
  "liquidity_band": "100k_250k",
  "thesis_tag": "spot_lead",
  "thesis_note": "Coinbase spot broke strike 4m ago, contract has not repriced",
  "size_rule": "kelly_quarter",
  "size_usd": 120.00,
  "exit_trigger": "price>=0.85 OR t<5m OR spot_reverts",
  "resolved_outcome": null
}

resolved_outcome gets patched at settlement. That's it — that's the entire data structure a calibration review needs. For how the size_rule field should be populated, see our guide to position sizing and risk management on prediction markets.

Calibration Scoring: Were Your 70% Calls Right 70% of the Time?

Calibration is the simplest question you can ask of a forecasting record, and almost nobody asks it. Bucket every resolved trade by your stated probability. Then check what fraction actually resolved YES in each bucket. If the trades where you said 70% resolved YES about 70% of the time, you're calibrated. If they resolved YES 55% of the time, you're overconfident by 15 points and every one of those bets was mispriced by you.

The shape above is the single most common pattern in forecasting research, and it has a name: overconfidence. Note that it's a flattening, not a uniform shift — the curve sits above the diagonal at the low end and below it at the high end. You're pushing your estimates toward the extremes in both directions. In the Good Judgment Project's geopolitical tournament, every calibration point fell below the identity line on the 50–100% confidence range they measured, and "overconfidence was worse among forecasters without training" (Mellers et al., Psychological Science, 2014).

The encouraging finding from the same study: a ~45-minute probability-training module measurably improved both Brier scores and calibration across two tournament years, with 2,246 participants in year one and 1,648 in year two. Calibration improvement was statistically significant in both (Y1 F(2,1586) = 3.16, p = .04; Y2 F(1,938) = 3.78, p < .05). Forty-five minutes. This is fixable.

Brier Scores: One Number, and What Counts as Good

The Brier score is the mean squared error of your probability forecasts. For each trade, take (your probability − outcome)², where outcome is 1 or 0. Average across all trades. Lower is better, and 0.25 is the score you get by saying "50%" to everything.

In a spreadsheet it's one column: =(my_probability - resolved_outcome)^2, then average it.

The important thing is knowing what number to aim at, because "0.18" means nothing without reference points.

Every tier in that chart beat chance. That's the first thing worth noticing — ordinary volunteers with no training scored 0.13, roughly half the squared error of a coin flip. The interesting part is the spread above them: teams landed at 0.08, superforecasters at 0.035, about 7x better than chance (Mellers et al., 2014). Structure, not raw talent, is most of that gap.

A scaling trap worth knowing, because it's silently wrong in a lot of forecasting write-ups: Mellers et al. report Brier on the 0–2 two-vector scale, which is exactly double the 0–1 convention your spreadsheet formula produces. Their published figures are 0.26 / 0.24 / 0.16 / 0.16 / 0.07; we've halved them above so they sit on the same scale as everything else in this post. If you compare a raw GJP number to a Kalshi Brier without converting, you'll conclude ordinary forecasters are worse than a coin flip. They aren't.

Every Brier figure from here on is on the 0–1 scale. For a different question set, the Forecasting Research Institute has superforecasters leading the ForecastBench leaderboard at 0.086, with the best LLMs at 0.103 (Forecasting Research Institute, Mar 2026). That's 0.086 versus 0.035 for what is broadly the same population — which tells you almost nothing about the forecasters and almost everything about how much question difficulty drives the number.

One warning that matters enormously and that nobody states clearly. Kalshi's own research, reported by Semafor, found its markets score a Brier of about 0.09 three months before close, dropping to 0.02 on the day of close, across 2.3 million bets placed since 2021 (Semafor, Aug 2026). Do not benchmark yourself against 0.02. That number is dominated by contracts that were nearly resolved already — predicting a game with 30 seconds left is trivially easy. A Brier score is only comparable against forecasts of similar difficulty and similar time-to-resolution. Compare yourself to your own prior quarter, on the same series, at the same horizon. Nothing else is apples to apples.

**Brier decomposition, briefly:** Murphy and Winkler showed the score splits into variability, calibration, and resolution. **Calibration** is the mean squared gap between your stated buckets and realized frequencies — zero means perfectly calibrated. **Resolution** measures whether you discriminate at all, i.e. whether your 80% calls really do resolve YES more often than your 30% calls ([Mellers et al.](https://sydneyscott.nfshost.com/pubs/Psychological_Strategies_for_Winning_a_G.pdf), 2014). You need both. A trader who says "50%" to everything is perfectly calibrated and completely useless.

So how good is the market you're scoring yourself against? A Federal Reserve working paper co-authored by a Board economist found Kalshi macro markets "yield well-calibrated, rapidly updating density forecasts" (Diercks, Katz & Wright, FEDS 2026-010, Feb 2026).

Independent academic work is less flattering. Clinton and Huang analyzed 2,500+ political markets and $2.4 billion of volume in the final five weeks of the 2024 election. PredictIt markets predicted outcomes better than chance 93% of the time — but only 78% of Kalshi markets and 67% of Polymarket markets did, with identical contracts diverging in price across exchanges (Clinton & Huang, Vanderbilt, 2025).

And Good Judgment — which sells superforecaster subscriptions, so read accordingly — reported its forecasters beat Polymarket on 76% of forecast days across 25 central bank meetings, 0.102 vs 0.126 at the meeting level (Good Judgment, Jan 2026).

The market is beatable. It is not beatable by accident.

How Many Trades Before Your Numbers Mean Anything?

This is where most self-review quietly goes wrong, and it's the part every trading-journal article skips.

David Aldous worked out the arithmetic. In a 100-question tournament, a contestant who is genuinely 5% more accurate than a rival has only about a 75% chance of actually scoring better. Being 10% more accurate raises that to roughly 90% (Aldous, The American Statistician, 2021; preprint 2019). Metaculus, reviewing its own scoring, put it plainly: there is "considerable statistical noise even with 100+ unweighted scored questions," and "more questions are needed to draw significant conclusions than most people's intuition tells them" (Metaculus, Sep 2024).

Rules of thumb I'd draw from that, though neither source states these cutoffs:

• Under 30 resolved trades: you have anecdotes. Log them, score nothing.
• 30–100: you can spot gross miscalibration — a 20-point gap at the high end is real signal. Small gaps aren't.
• 100–300: overall Brier and a coarse three-bucket calibration curve become meaningful for gross effects only.
• 300+: you can start segmenting by thesis_tag and liquidity_band without fooling yourself.

One useful piece of good news from Aldous: comparing predicted probabilities is far more statistically efficient than predicting Yes/No and counting hits. Logging my_probability instead of a directional call buys you real statistical power for free.

The same trap shows up in strategy validation, which is why we run permutation testing on parameter sweeps rather than trusting a good-looking result. Small samples produce confident nonsense in both places.

Finding the Biases Hiding in Your Own Record

Once you have a few hundred rows, the journal starts answering questions you can't answer by feel.

Segment your Brier by liquidity band

Polymarket's aggregate Brier improves with market volume — 0.0647 in $0–1k markets down to 0.0156 in $1M+ markets, a 4x spread (Dune / Alex McCullough, a community dashboard with unpublished methodology, data Jun 2024–May 2026).

Apply the comparability rule from the last section to that chart, though, and it shrinks. Thin markets hold proportionally more near-certain contracts, which are easy to price. On the same dashboard's panel that strips out extreme odds (below 0.9% or above 99.1%), the spread compresses to roughly 1.75x and stops being monotonic. Most of the headline 4x is question difficulty, not market skill — which is exactly why you segment your own record the same way instead of trusting an aggregate.

The same dashboard shows Polymarket's "Up/Down" crypto markets scoring a Brier of 0.2438 at 51.07% accuracy. A coin flip scores 0.25 at 50%. If your journal shows you trading a lot of those, that's a finding.

Check for the disposition effect

Odean's study of 10,000 brokerage accounts found investors realize gains at a rate a little over 1.5x the rate they realize losses — a position that's up is more than 50% more likely to be sold on a given day than one that's down. And the winners they sold outperformed the losers they kept by 3.4 percentage points over the following year (Odean, Journal of Finance, 1998).

Binary contracts make this bias unusually easy to detect, because the reference point isn't ambiguous. Take every trade where you exited early. Split them by whether market_price at exit was above or below your entry. If you're closing 30¢ → 70¢ winners at a much higher rate than you're closing 70¢ → 30¢ losers, you have the disposition effect in a market where holding to settlement is cheap — the contract resolves itself, and Kalshi charges no settlement fee under its current schedule. It isn't free (your capital is tied up), but it's close enough that early exits should have a reason you wrote down. One GROUP BY finds it.

Correct for the journal's own selection bias

Here's a distortion nobody warns you about: a journal of taken trades is a biased sample of your beliefs. You only log the forecasts where you thought you saw edge, which means your calibration curve is built entirely from your disagreements with the market. That's a systematically harder sample than a neutral one, and it will make you look worse-calibrated than you are.

The fix is cheap. Log the estimate even when you don't trade — a row with size_usd = 0. Then you can score two curves: one on everything you assessed, one on everything you acted on. If the traded curve is meaningfully worse than the assessed curve, the problem isn't your forecasting. It's your trade selection, which is a different bug with a different fix.

Check whether trading more is helping

Barber and Odean's 66,465-household study found the most active traders returned 11.4% annually against a 17.9% market return, with turnover above 250% a year — and critically, "very little difference in gross performance" (Barber & Odean, Journal of Finance, 2000). They weren't picking worse. They were paying more.

There's a nuance worth holding, though. Superforecasters made an average of 7.8 predictions per question versus 1.4 for individual forecasters (Mellers et al., 2014). Updating your estimate frequently is good. Turning every update into a trade is what costs money. Your journal is the only thing that can tell those two apart — log the estimate revision even when you don't trade on it.

For the market-structure side of why frequent trading gets punished on Kalshi and Polymarket specifically, see why prediction market trades get picked off and our liquidity calendar on when not to trade.

Making the Bot Keep the Journal

Hand-logging works for maybe twenty trades a week. It falls apart above that, and it falls apart completely if you're running automation — which, on Kalshi, is a large and growing share of activity.

A bot has one structural advantage over you: it isn't reconstructing anything. At the instant the order fires it knows the timestamp, the market price, the input values it acted on, and which rule fired. Humans reconstruct all of that afterward, and reconstruction is exactly where hindsight gets in.

But be clear-eyed about the split, because it's the useful part of this whole exercise. Automation gives you the mechanical half of the schema for free: trade_id, logged_at, market, side, market_price, size_usd, liquidity_band (where the venue exposes volume), and — once the contract settles — resolved_outcome. That's eight of the fourteen fields, captured with no memory involved.

It does not give you the judgment half — a different four from the high-value fields listed earlier. my_probability, thesis_tag, size_rule, and exit_trigger are claims about what you believed and why (the remaining two, edge_bps and thesis_note, fall out of those). A rules engine that fires on price < 0.40 has no opinion about the true probability — it has a threshold. Nothing infers your estimate for you. If you want a calibration curve, you have to declare the number.

Turbine Studio is a concrete example of that split. Every deployed strategy writes a structured trading journal — signal, execution attempt, order accepted, fill, position closed, settlement — with market price and the strategy's declared data inputs attached to each event, resolvable back to the settled contract and readable through the API. What it records is what the bot did. The probability you were implicitly asserting when you set that threshold is the column you still have to add yourself, and that's not a gap in the tooling so much as the whole point: the machine can log the decision, but only you can log the belief.

The review loop we recommend: paper trade first, accumulate 100+ resolved decisions, score calibration and Brier per thesis tag, retire the tags that don't beat chance, and only then scale size. Our post on paper trading before going live covers the first stage in detail.

See Turbine Studio plans if you'd rather have the execution record kept for you and spend your own effort on the four judgment columns.

Frequently Asked Questions

What's the difference between decision quality and outcome quality?

Decision quality is whether your probability estimate and sizing were correct given what you knew. Outcome quality is whether the contract resolved your way. In a market of explicit probabilities they come apart constantly — a correctly priced 70% bet loses 30% of the time. The 2023 outcome-bias replication measured d = 1.10: participants rated an identical decision 1.36 points higher on a 7-point scale purely because it worked out (Aiyer et al., 2023).

What is a good Brier score for a prediction market trader?

Context matters more than the number. On the 0–1 scale, 0.25 is what you'd score by saying "50%" to everything. Untrained forecasters in the Good Judgment tournament averaged 0.13 and superforecasters 0.035, converted from the 0–2 scale that study uses (Mellers et al., 2014). Don't benchmark against Kalshi's 0.02 day-of-close figure — near-resolution forecasts are far easier. Compare against your own prior quarter on the same series.

How many trades do I need before my calibration numbers mean anything?

More than you'd guess. Aldous showed that in a 100-question tournament, a forecaster who is genuinely 5% more accurate only outscores a rival about 75% of the time (Aldous, 2021), and Metaculus reports considerable noise even past 100 scored questions. Treat under 30 as anecdote, 100–300 as a coarse read, and 300+ as segmentable.

Can I just use a normal stock trading journal template?

No. Standard templates track entry, exit, P/L, stop, target, and a free-text lesson. None of them has a field for your probability estimate at entry, which is the only field that makes calibration scoring possible. Add my_probability, market_price, edge_bps, and a closed-enum thesis_tag and the same spreadsheet becomes scoreable.

Should I log trades I considered but didn't take?

Yes, for probability estimates specifically. Superforecasters made 7.8 predictions per question versus 1.4 for individual forecasters (Mellers et al., 2014). Frequent re-estimation improves accuracy; frequent trading destroys returns. Logging estimates you didn't act on is how you get the first without the second.

The Bottom Line

A prediction market journal is not a diary. It's a dataset, and it exists to answer four questions your P&L structurally cannot:

• Am I calibrated? Bucket your stated probabilities, check realized frequencies, and look for the curve flattening — above the diagonal at the low end, below it at the high end.
• Do I discriminate? Resolution, not just calibration — your 80% calls should resolve YES more often than your 30% calls.
• Is my claimed edge real? Sort by edge_bps and check whether high-edge trades actually paid more.
• Which of my ideas are noise? Brier per thesis_tag, and retire everything that doesn't clear chance.

Fourteen columns. Two separate findings are worth keeping straight here: a ~45-minute probability-training module produced a small but statistically significant calibration gain, and — independently of training — structured team forecasting moved Brier from 0.13 to 0.08 (both converted from the paper's 0–2 scale). Neither is talent. Both are process, and the second is much larger than the first. The edge here isn't clever. It's just that almost nobody does it, because reviewing decisions feels worse than reviewing wins.

Start logging before your next trade, not after. A probability you write down beforehand is data. A probability you remember afterward is a story. If you're running automation, Turbine Studio keeps the execution side of that record for you — leaving you the four columns that actually require a view.

---

This article is for educational purposes only. Trading prediction markets and automated strategies involves substantial risk of loss. Past performance does not guarantee future results. Always validate strategies in paper trading before risking real capital.