Friday, November 20, 2026

Calibrated Thinking: Best Practices and How to Get Better at It

Most of us judge our thinking by whether we turned out to be right. That is the wrong scorecard. A person who is right 60% of the time but claims to be certain every time is a poor guide. A person who is right 60% of the time and says, "I'd put this at about 60%," is a far more trustworthy one. You can rely on the second person because their confidence tells you something. (Whether they are also a useful guide depends on something more: how well they can tell likely outcomes from unlikely ones. We'll come to that.)

That is the core of calibrated thinking: making your confidence match the evidence, and making your effort match the stakes. When you are well calibrated, the things you call 80% likely happen about 80% of the time. You spend more care on decisions that are complex or costly, and less on decisions that are routine or easy to reverse.

The goal is not to doubt everything or to agonize over every choice. It is to become more accurate and more efficient. And calibration is not a fixed trait. It is a skill that can be measured, and what can be measured can be practiced. This article explains what calibration is, why we usually lack it, and the habits that can improve it.

What Calibration Means

Strictly speaking, calibration is a statistical property: whether your stated probabilities match how often things actually happen. This article treats that as the center of a broader practice with three parts. The first is calibration in the strict sense; the other two are the habits that make it possible.

1. Confidence matches accuracy
Your stated certainty lines up with how often you are actually right. You avoid both overconfidence (claiming more than you know) and underconfidence (hedging on things you know well).
2. Effort matches stakes
How hard you think, check and research scales with how difficult the decision is and how much it costs to get it wrong.
3. Beliefs update with feedback
You track outcomes, notice where you were off, and adjust both your beliefs and your confidence as new evidence comes in.

Confidence is a forecast, not a feeling

Imagine a manager who says, "I'm 90% sure we'll hit the deadline." If projects given that rating finish on time only six times out of ten, the manager's confidence overstates their reliability. Now imagine a second manager whose 90% calls succeed about nine times out of ten. That manager will still miss a deadline now and then, and that miss does not prove poor judgment. A 90% forecast openly leaves room for failure. Calibration is about the pattern across many judgments, not any single one. The practical question becomes: When I feel this sure, how often am I actually right?

Calibration is not the same as being right, or being cautious

Forecasters distinguish two qualities. Calibration asks whether your probabilities mean what they say. Resolution asks whether you can tell likely events from unlikely ones. Someone who says "50%" about everything can be perfectly calibrated on coin-flip questions and still be useless, because they never commit to anything. Nor is calibration the same as softening every claim with "maybe" and "possibly." Underconfidence is a miscalibration too. The goal is strong confidence when the evidence is strong, real uncertainty when it is weak, and honesty about which is which.

Pattern What it sounds like Track record
Overconfident "I'm 90% sure." Right about 65% of the time
Underconfident "Maybe 60%?" Right about 85% of the time
Calibrated "About 75%." Right about 75% of the time

Why We Are Usually Miscalibrated

Overconfidence is one of the most studied biases in judgment research. Psychologists Don Moore and Paul Healy showed that the word actually covers three different errors: overestimation (thinking you performed better than you did), overplacement (thinking you are better than others), and overprecision (being too sure your beliefs are accurate). Of the three, overprecision was the most persistent.1 It shows up whenever people give a range they are "90% sure" contains the answer and the true value falls outside it far more often than one time in ten.

A few forces keep us miscalibrated:

  • We rarely score ourselves. Most predictions are vague ("it'll probably work out") and are never checked, so there is no feedback loop.
  • Hindsight rewrites memory. Once we know the outcome, we tend to remember having expected it.
  • The inside view feels convincing. We picture how our own plan will unfold and neglect how similar plans have gone for other people.
  • Confidence is socially rewarded. Sounding certain often wins the room, even when certainty isn't earned. In many workplaces, in politics and in the media, the person who says "I'm about 70% sure" can come across as weaker than the one who says "I'm certain," so calibration sometimes takes a little courage.

Evidence That Calibration Can Be Learned

The best evidence that calibration is a skill comes from fields where people make many probability judgments and get fast, honest feedback.

Weather forecasters are the classic example. Meteorologists have long been scored with the Brier score, a measure Glenn Brier proposed in 1950 for judging probability forecasts (it rewards accurate probabilities and penalizes confident misses most heavily).2 When American forecasters began attaching probabilities to their forecasts, researchers Allan Murphy and Robert Winkler found that they could make those forecasts reliably, meaning their stated chances of rain closely matched how often it actually rained.3 Forecasters are not smarter than everyone else. What sets them apart is the conditions they work in: many repeated probability forecasts, clear outcomes, and a score on every one.

Laboratory training points the same way. In a 1980 study, Sarah Lichtenstein and Baruch Fischhoff gave people repeated rounds of probability judgments with detailed feedback on their calibration, and most participants' calibration improved.4

Forecasting tournaments are the most famous recent case. In a U.S. intelligence-funded geopolitical forecasting tournament, the Good Judgment Project reported that a brief probability training module, working in teams, and grouping top performers together improved forecast accuracy.5 A follow-up analysis by some of the same researchers found that training helped reduce overconfidence over several years of the tournament.6 This research is not fully settled, however. A reanalysis published in late 2024 argued that because forecasters chose their own questions and timing, the measured benefits of teams and training may have been overstated.7 In plain terms: if some groups tended to pick easier questions, or to forecast closer to the outcome when more was known, their scores could look better for reasons that had nothing to do with training or teamwork. The original researchers dispute the reanalysis, so this question remains open.8

Feedback is not a guaranteed cure. Recent training studies show mixed results. One study of automated, individualized feedback found reduced overconfidence and better calibration, with the clearest effect in the more controlled of its two experiments.9 Another, with nearly 1,500 participants across two experiments, found no improvement in calibration from either of the feedback methods it tested.10 The fair summary is that sustained, feedback-rich practice can improve calibration, but a quick training session is no guarantee, and the size of the gains is still debated. That is a good reason to build the habits below into your routine rather than treating calibration as a one-time lesson.

If you do only three things

  1. Keep a prediction log, with a confidence number and a revision trigger for each entry.
  2. Start every estimate from the base rate: how do situations like this usually turn out?
  3. Score yourself once a month against what actually happened.

The other practices below are tools that make these three work better.

Part 1: Make Your Confidence Measurable

1. Put numbers on your beliefs

Words like "likely," "possible" and "a real chance" mean very different things to different listeners. One person's "likely" is 60%; another's is 90%. Absolute words are worse: "definitely," "impossible" and "always reliable" claim a certainty that the evidence rarely supports. Replacing these words with a rough number forces you to decide what you actually believe, and it gives you something to check later. The number doesn't need to be precise. "About 70%" is far more useful than "probably."

A simple scale is enough to start:

Number Plain meaning Wrong about
50% Genuine toss-up 1 time in 2
70% Leaning yes About 1 time in 3
90% Confident 1 time in 10
99% Near certain 1 time in 100

The right-hand column is the reality check. If you say 90%, you are claiming you would be wrong only once in ten such calls. Ask yourself honestly whether your track record supports that.

2. Keep a prediction log

Calibration cannot improve without feedback, and most of life does not give you clean feedback on its own. A prediction log creates it. Write each claim so that a stranger could settle it: "The project will go well" can't be scored, but "The project ships by March 31" can. Record your confidence before you know the result. A complete entry looks like this:

Claim
The agreed scope will be delivered by the end of the month.
Confidence
70%
Main reasons
Most of the work is done, but one outside dependency remains.
Revision trigger
The dependency is still unresolved one week before the deadline.
Outcome and review (filled in later)
What happened, and was the reasoning sound or just lucky?

The revision trigger is the most valuable line. Deciding in advance what would change your mind makes it much harder to explain away bad news when it arrives. A spreadsheet or a notebook works fine; the habit matters more than the tool.

3. Score yourself honestly

After you have logged a few dozen predictions, group them by confidence level and compare. Of all the claims you rated 70%, how many came true? If it was closer to 50%, you are overconfident at that level. If it was closer to 90%, you are underconfident. Treat about 20 entries in a group as the point where a pattern starts to show, not as proof. Small samples can drift well away from their true rate by chance, so keep the count visible next to the success rate, be patient, and trust the picture more as the numbers grow. With small samples, treat what you see as a hint, not a verdict; eight predictions at 70% can't tell you much. The same caution applies, even more strongly, when you judge someone else's record.

If you want a single number, use the Brier score: for each prediction, take your probability (as a decimal), subtract the outcome (1 if it happened, 0 if not), square the result, and average across all predictions. Zero is perfect. Saying 50% about everything earns 0.25, so a score worse than that means your forecasts did worse on that set of questions than saying 50% to everything would have. But beating 0.25 is a low bar. A better benchmark is the base rate: if only 20% of the things you predict tend to happen, simply saying "20%" every time would score 0.16. Your score shows real skill only when it beats a sensible benchmark like that. One more caution: the Brier score blends calibration with your ability to tell likely events from unlikely ones, so use it alongside the confidence-level check, not instead of it. And resist the easy fix of simply lowering every estimate. That can reduce overconfidence while making your judgments less informative.

4. Drill with confidence intervals

Because overprecision is so stubborn, practice estimating ranges. Take questions with known numerical answers (the year a building was finished, the population of a city, the length of a river) and give a range you are 90% sure contains the answer. Check your results after twenty questions. Most beginners find that far fewer than 18 of their 20 ranges contain the truth. The fix is to widen your ranges until the hit rate matches what you claimed. Douglas Hubbard's How to Measure Anything builds an entire calibration-training approach around exercises like this.

Part 2: Test Your Reasoning

5. Separate facts, interpretations and speculation

A common source of poor judgment is treating three different kinds of statements as if they were equally solid. Consider a news report about trade:

  • Fact: The country's exports fell 8% over the past year.
  • Interpretation: The decline suggests weakening foreign demand.
  • Speculation: The decline will eventually cause a financial crisis.

The first may rest on official statistics. The second requires economic judgment. The third is a prediction. Each deserves a different level of confidence. When you evaluate an argument, ask: What is actually established? What conclusions are being drawn from it? What assumptions connect the two? Which statements are predictions rather than observed outcomes? And what evidence would strengthen or weaken the case?

6. Start with the base rate

Before you think about the details of your situation, ask how situations like it usually turn out. Daniel Kahneman and Dan Lovallo described the tendency to treat each problem as unique, focusing on our own plans and scenarios (the "inside view") instead of the statistics of similar cases (the "outside view"). They argued that this is a major source of overly optimistic forecasts.11

In practice: if you're estimating how long a renovation will take, first find out how long similar renovations usually take and how often they run over. Then adjust for what is genuinely different about your case. The base rate doesn't override every detail. It sets a starting point that the details must justify moving away from.

7. Deliberately consider the opposite

Once you form a view, your mind tends to gather support for it. A simple counter is to ask, "What evidence would convince me I'm wrong?" and then argue that side seriously. If sales are falling, don't settle on the first explanation that comes to mind. List the alternatives (new competition, pricing, changing customer tastes, the wider economy) and ask which one the evidence actually supports. In classic experiments by Charles Lord, Mark Lepper and Elizabeth Preston, telling people to consider the opposite reduced biased judgment more than simply telling them to be fair and unbiased.12 Good intentions aren't enough. A specific mental move works better.

Go one level deeper, too. Many errors begin before any particular forecast, in the big assumptions that generate it: "history moves in one direction," "this time is different," "technology will solve it," "the experts are always wrong." Once a sweeping assumption like that is in place, new evidence tends to get fitted to it rather than tested against it. Name the background assumption behind an important forecast, and ask how well forecasts drawn from that view have actually held up in the past.

8. Run a premortem on important plans

Psychologist Gary Klein's premortem turns "What might go wrong?" into "It's a year from now and this failed. What happened?" Team members assume the project has already failed and write down every plausible reason.13 Imagining the failure as a fact makes it easier to voice concerns that people would otherwise keep to themselves, and it tends to bring an overconfident plan back down to earth. It works just as well for personal decisions as for team projects.

The research on premortems is thin but encouraging. In one study, 178 students evaluated a hypothetical plan for responding to a flu epidemic; those who ran a premortem lowered their confidence in the plan more than those who critiqued it or listed its pros and cons, and their confidence rose again once they worked out fixes for the problems they had found.14 In another, teams using a premortem identified higher-quality risks, including more hard-to-predict "black swan" risks, than teams that brainstormed.15 Neither study shows that premortems make projects succeed. What they suggest is narrower and still useful: premortems bring more problems to the surface and temper overconfidence in a plan.

9. Judge sources by their track records

A source's credentials, reputation or polished style is not the same as its reliability. An articulate commentator can be consistently wrong; an unfamiliar analyst can be consistently right. Where you can, look at the record. For a geopolitical commentator, for example: Did they assess military capabilities accurately? Were their territorial predictions close? Did they anticipate major developments? When they were wrong, did they admit it and revise?

To answer those questions fairly, pin down what was actually predicted: the specific claim, when it was made, and what would have counted as being wrong. Without that, vague statements ("tensions will rise," "the economy faces headwinds") can be reinterpreted afterward as successful predictions. This is especially common in geopolitics, where broad assertions are hard to disprove. A forecast that could never have failed tells you nothing about the forecaster.

Two errors sit on either side. Don't keep trusting a source that is repeatedly wrong, and don't dismiss a good source over a single miss. Judge the overall pattern, and adjust your trust by demonstrated reliability, not by how much you agree. Also watch for a quieter trap: several sources that agree may all be repeating the same underlying report. Five outlets citing one wire story are one source, not five.

Ask, too, what a source is rewarded for. Commentators, salespeople, activists and media figures are often paid in attention for conviction and drama, not for accuracy. When being right matters less than sounding sure, overconfidence is the predictable result, so weigh confident claims from such sources accordingly. And remember that a forecasting record tests one skill, not all of them. Someone with a weak record of predictions can still explain a subject well, and someone can make an accurate prediction for the wrong reasons.

10. Use AI as a tool, not an authority

AI tools can speed up research, surface overlooked considerations and organize competing arguments. They can also produce fluent, confident text that is wrong, including made-up citations. Treat AI like a capable junior research assistant: useful for speed and ideas, but verified before you rely on any fact, figure or quotation that matters. The best use is to test your reasoning rather than reinforce it:

  • Ask it to list the assumptions behind an argument.
  • Ask which claims need independent verification, and which are facts versus interpretations.
  • Instead of "Why am I right about this?" ask, "What are the strongest arguments against my conclusion, and which are supported by evidence?"
  • Check anything important against primary sources before you act on it.

The higher the stakes, the more you check. The goal is to improve your judgment, not to outsource it.

Part 3: Match Your Effort and Keep Learning

11. Match your effort to the stakes, and know when to stop

Calibration applies to how hard you think, not only to how sure you are. Overthinking a lunch order wastes time, and underthinking a career move or a large purchase can be costly. Research by Daniella Kupor and colleagues found that observers judged decisions as better, and were more open to being influenced, when the decision maker appeared to have given the right amount of thought for the situation—neither too much nor too little.16 Note what that study measured: how thoughtful someone appeared and how others reacted, not whether their decisions were actually better. Looking appropriately thoughtful and thinking effectively are different things. The case for matching effort to stakes rests mainly on common sense about costs: careful checking is worth the most where mistakes are expensive and hard to undo.

Decision type Example Right level of effort
Low stakes, reversible Which book to read next Decide quickly, adjust later
Moderate stakes Hiring a contractor Check references, compare 2–3
High stakes, hard to undo Signing a long-term contract Base rates, premortem, advisers

It helps to keep three questions apart: What do I believe? How strong is the evidence? What should I do, given the consequences? They don't always line up. Sometimes you must act before the evidence is conclusive. Sometimes a belief you hold firmly still deserves checking because acting on it would be costly and hard to reverse.

A simple example shows why a probability is not a decision. Suppose you think there's a 20% chance of a serious leak in your house. If an inspection costs $50 and would prevent $10,000 of damage, inspecting is an easy call: a one-in-five chance of avoiding $10,000 in damage is worth far more than $50. If the inspection costs $5,000 and the most it could save is $100, it isn't worth it. The probability is the same in both cases; the right action changes with the costs and consequences. The aim is not to reach some level of certainty before acting. It is to use your uncertainty well.

More research is not always better. At some point, another hour of reading stops changing anything. The standard isn't "Have I eliminated uncertainty?" but "Have I investigated enough for this decision?"

Stopping rule: Before digging further, ask, "What could I realistically learn next, and could it change my choice?" If the honest answer is no, it's time to decide. Two supporting checks:

  • Have I gathered enough reliable information and considered the main alternatives?
  • Is more research worth the time it would take?

12. Update in steps, not leaps

When new evidence arrives, ask how much it should move you, not just which way. Strong, direct evidence deserves a big move. Weak or ambiguous evidence deserves a small one. In the Good Judgment Project, forecasters who updated their beliefs more often tended to be more accurate.17 Two errors sit on either side: refusing to budge because you're attached to your first view, and swinging wildly with every headline. Writing down your new number after each piece of evidence makes your updating visible and easier to audit later.

Calibration also means holding your ground when the evidence still supports you. Intellectual humility is not perpetual uncertainty. Reopening a well-supported conclusion again and again without new evidence is not sophistication; it is wasted effort. And changing your mind every time you hear a persuasive-sounding counterargument is not open-mindedness; it is letting rhetoric do the work that evidence should do. The rule is the same in both directions: the size of a change in your view should match the strength of the new evidence.

The same applies to knowledge in general. Even well-supported scientific theories have been refined or replaced as evidence accumulated. That doesn't make all knowledge equally uncertain, but it is a reminder not to confuse "the best current explanation" with "the final word."

13. Seek feedback from other people

Other people can see your blind spots more easily than you can. Share your estimates with a trusted colleague or friend and ask them to challenge your reasoning. Ask open questions ("What makes you think that?" "How would we know if we were wrong?") rather than yes-or-no ones. When a group is making a forecast, have each person write down an estimate privately before discussion so that the loudest or most senior voice does not anchor everyone else.

A Language for Honest Uncertainty

Calibration also changes how you talk. You don't need to attach a percentage to every sentence. You need to make clear what is known, what is inferred and what is unresolved. Phrases like these help:

  • "This is my best estimate, but the evidence is thin."
  • "I'm confident about the overall trend, less confident about the timing."
  • "These sources agree, but they may all rely on the same original report."
  • "I'd change my view if this particular thing happened."

Common Pitfalls

  • Judging decisions only by outcomes. A good decision can turn out badly through bad luck, and a bad decision can turn out well. Annie Duke calls the habit of grading decisions solely by their results "resulting." Grade the reasoning and the information you had, not just what happened.
  • Drawing conclusions from too few predictions. Ten predictions tell you almost nothing about your calibration. Dozens begin to show a pattern; hundreds give a clear picture.
  • Hiding in the middle. Saying 50% to avoid ever being "wrong" doesn't make you calibrated (that works only if the events really happen about half the time), and it makes your forecasts useless either way. Commit when the evidence supports it.
  • Logging only easy questions. If every claim in your log is one you're sure about, you'll look well calibrated and learn nothing. Include hard calls.
  • Misplaced deference. Trusting genuine experts on their own subject is usually sound. The error is deferring to a famous name outside its field, or letting status outweigh the evidence.
  • Emotional reasoning. Frustration, fear or loyalty can make a weak argument feel strong. Notice the feeling, then look at the evidence again.
  • Analysis paralysis. Continuing to research long after more information could change the decision.

A Quick Checklist

For any judgment that matters, run through these steps:

  1. Define the question clearly enough that the answer can be checked.
  2. Weigh the stakes and decide how much effort the question deserves.
  3. Sort the evidence into facts, interpretations and speculation.
  4. Check the base rate and the track records of your sources.
  5. Consider alternatives, including what would prove you wrong.
  6. Assign a number and a revision trigger.
  7. Stop when more research is unlikely to change your decision.
  8. Review the outcome later and adjust.

A 30-Day Plan to Get Started

Week 1: Speak in numbers
Each time you catch yourself saying "probably" or "I doubt it," attach a rough percentage, even if only in your head. Start a prediction log and add at least one entry a day.
Week 2: Drill intervals
Do two sets of twenty 90% confidence-interval questions. Count your hits and widen your ranges if you fall well short of 18 out of 20.
Week 3: Add the outside view and the opposite
For every new log entry, write down the base rate first, then your adjusted number, then one reason you might be wrong. Run a premortem on one real plan.
Week 4: Score and adjust
Score the predictions that have resolved so far (many won't have yet, and that's normal). Group them by confidence level, look for where you're over- or underconfident, and write down one specific adjustment for the next month. Then keep the log going.

A Closing Thought on Humility

Calibrated thinking is, at bottom, a disciplined form of intellectual honesty. It asks you to know what you know, admit what you don't, and keep score so that your confidence stays tied to reality. Scripture has long commended this posture: "The simple believe anything, but the prudent give thought to their steps" (Proverbs 14:15, NIV), and "In a lawsuit the first to speak seems right, until someone comes forward and cross-examines" (Proverbs 18:17, NIV). Paul's counsel to think of oneself "with sober judgment" (Romans 12:3, NIV) is a fitting summary of the whole practice.

Nobody becomes perfectly calibrated. But the aim is simple enough to remember: be confident when the evidence justifies it, stay uncertain when it doesn't, change your mind when better information arrives, and spend extra effort only where learning more could change what you do. Anyone willing to put numbers on their beliefs and check them against reality can become steadily more trustworthy, both to others and to themselves.

Recommended Reading

  • Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (2015). The most accessible account of the Good Judgment Project and the habits of its best forecasters.
  • Philip E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (2005). The long study of expert predictions that preceded the tournaments.
  • Douglas W. Hubbard, How to Measure Anything: Finding the Value of "Intangibles" in Business, 3rd ed. (2014). Includes practical calibration-training exercises.
  • Annie Duke, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (2018). On separating decision quality from outcomes.
  • Julia Galef, The Scout Mindset: Why Some People See Things Clearly and Others Don't (2021). On wanting to see the truth rather than defend a position.
  • Daniel Kahneman, Thinking, Fast and Slow (2011). A broad tour of judgment biases, including the inside and outside views. Note that some findings in the book, especially the chapter on social priming, have not held up well in replication attempts.

Notes

  1. Don A. Moore and Paul J. Healy, "The Trouble with Overconfidence," Psychological Review 115, no. 2 (2008): 502–517. doi:10.1037/0033-295X.115.2.502 ↩
  2. Glenn W. Brier, "Verification of Forecasts Expressed in Terms of Probability," Monthly Weather Review 78, no. 1 (1950): 1–3. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 ↩
  3. Allan H. Murphy and Robert L. Winkler, "Reliability of Subjective Probability Forecasts of Precipitation and Temperature," Journal of the Royal Statistical Society, Series C (Applied Statistics) 26, no. 1 (1977): 41–47. doi:10.2307/2346866 ↩
  4. Sarah Lichtenstein and Baruch Fischhoff, "Training for Calibration," Organizational Behavior and Human Performance 26, no. 2 (1980): 149–171. doi:10.1016/0030-5073(80)90052-5 ↩
  5. Barbara Mellers, Lyle Ungar, Jonathan Baron, et al., "Psychological Strategies for Winning a Geopolitical Forecasting Tournament," Psychological Science 25, no. 5 (2014): 1106–1115. doi:10.1177/0956797614524255 ↩
  6. Don A. Moore, Samuel A. Swift, et al., "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science 63, no. 11 (2017): 3552–3565. doi:10.1287/mnsc.2016.2525 ↩
  7. Clifford E. Hauenstein, Rick P. Thomas, David A. Illingworth, and Michael R. Dougherty, "Rethinking the Role of Teams and Training in Geopolitical Forecasting: The Effect of Uncontrolled Method Variance on Statistical Conclusions," Psychological Science 36, no. 1 (2025): 3–18; first published online December 4, 2024. doi:10.1177/09567976241266481 ↩
  8. Barbara Mellers, "Adversarial Collaboration on the Forecaster Benefits of Teams and Training," Mack Institute for Innovation Management, Wharton School, University of Pennsylvania (funded research proposal responding to Hauenstein et al.). mackinstitute.wharton.upenn.edu ↩
  9. Eric R. Stone, Jason Luu, Cory K. Costello, and Annie H. Somerville, "Automated Calibration Training for Forecasters," Journal of Behavioral Decision Making 36, no. 4 (2023): e2334. doi:10.1002/bdm.2334 ↩
  10. Matthew Martin and David R. Mandel, "Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence," Futures & Foresight Science 7, no. 1 (2025): e199. doi:10.1002/ffo2.199 ↩
  11. Daniel Kahneman and Dan Lovallo, "Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking," Management Science 39, no. 1 (1993): 17–31. doi:10.1287/mnsc.39.1.17 ↩
  12. Charles G. Lord, Mark R. Lepper, and Elizabeth Preston, "Considering the Opposite: A Corrective Strategy for Social Judgment," Journal of Personality and Social Psychology 47, no. 6 (1984): 1231–1243. doi:10.1037/0022-3514.47.6.1231 ↩
  13. Gary Klein, "Performing a Project Premortem," Harvard Business Review 85, no. 9 (September 2007): 18–19. hbr.org ↩
  14. Beth Veinott, Gary A. Klein, and Sterling Wiggins, "Evaluating the Effectiveness of the PreMortem Technique on Plan Confidence," in Proceedings of the 7th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2010), ed. Simon French, Brian Tomaszewski, and Christopher Zobel (Seattle, 2010). idl.iscram.org ↩
  15. David Gallop, Chris Willy, and John Bischoff, "How to Catch a Black Swan: Measuring the Benefits of the Premortem Technique for Risk Identification," Journal of Enterprise Transformation 6, no. 2 (2016): 87–106. doi:10.1080/19488289.2016.1240118 ↩
  16. Daniella M. Kupor, Zakary L. Tormala, Michael I. Norton, and Derek D. Rucker, "Thought Calibration: How Thinking Just the Right Amount Increases One's Influence and Appeal," Social Psychological and Personality Science 5, no. 3 (2014): 263–270. doi:10.1177/1948550613499940 ↩
  17. Barbara Mellers, Eric Stone, Pavel Atanasov, et al., "The Psychology of Intelligence Analysis: Drivers of Prediction Accuracy in World Politics," Journal of Experimental Psychology: Applied 21, no. 1 (2015): 1–14. doi:10.1037/xap0000040 ↩


Map 1: What Calibrated Thinking Is and Why It's Hard Calibrated thinking: what it is and why it is hard Calibrated Thinking Three Parts Confidence matches accuracy (calibration proper) Effort matches the stakes Beliefs update with feedback Not the Same As… Being right: resolution tells likely from unlikely Caution: underconfidence is miscalibration too Why We Fall Short Overprecision: the most persistent overconfidence No scoring, hindsight, the inside view Incentives reward sounding sure Can It Be Learned? Yes, with repeated, scored feedback (forecasters) Tournament training gains are disputed Quick feedback training is no guarantee Strictly, calibration is only the first part; the other two are habits that support it.

Conceptual Map 2: How to Get Better at Calibrated Thinking How to get better at calibrated thinking How to Get Better Make Confidence Measurable Put numbers on beliefs Log predictions with a revision trigger Score against the base rate; drill 90% ranges Test Your Reasoning Separate facts, interpretations and speculation Start from base rates; consider the opposite and hidden assumptions Premortems; judge sources by track record; use AI to challenge, not decide Match Effort, Keep Learning Scale effort to stakes; stop when more research can't change the choice A probability is not a decision Update in proportion to evidence, and hold your ground when it holds Get others' private estimates before discussion Pitfalls to Avoid Grading by outcomes; hiding at 50%; logging only easy calls Misplaced deference, emotion, analysis paralysis Start here: keep a prediction log, begin from base rates, and score yourself monthly.

No comments:

Post a Comment

Which Intellectual Decathlon Events Are Most Worth Preparing For? Ranking the Ten Events by Real-Life Payoff

Ranking the Ten Events by Real-Life Payoff You have taken the Intellectual Decathlon , recorded your results, and built a profile of yo...