Nick Maggiulli wrote a blog post a few weeks ago titled Respect the Base Rate , which opens with this wonderful story:
This is a companion discussion topic for the original entry at https://commoncog.com/the-base-rate-is-a-hell-of-a-thing/
This is a companion discussion topic for the original entry at https://commoncog.com/the-base-rate-is-a-hell-of-a-thing/
Huh. On one hand I have never been an active investor, because getting (index fund + reinvested dividends + index survivor bias) returns for approximately zero effort has always felt like a great deal. On the other hand, as a regular buyer of assets, I look forward to the times when prices are down so I can buy at a discount.
Itâs enough to make me wonder if I ought to experiment with a small active investment.
Oh dear, what have I done?
1 Like
This was a thought provoking post. I tried to apply some of the ideas discussed here to my own life, and discovered a few ideas that could be added to ârespect the base rateâ to help make better decisions
- Respect the base rate given your shots at goal
- Respect the base rate given the outcome payoff disparity
- Respect the base rate given the difference in reference classes
Respect the base rate given your shots at goal
If you only have one shot at goal (like Kahnemanâs team did when writing a book for 7 years), it makes a lot of sense to respect the base rate. But when you have multiple shots at goal, you can see the compounding effects of the base rate. This is particularly true in cases where you can try, fail, and iterate quickly
For example, if you try something with a 20% chance of success but only have one shot at it, and requires a significant commitment to try â it may not be worth it. But if you have say, 10 shots at goal, the probability that youâll fail in all 10 is only (1-0.2)^(10) = 10.7%
One can supercharge this process (i.e., have a higher chance of success in subsequent iterations) with the approach that Cedric had listed in his post about Better Trial and Error:
- Donât blow up (avoid risk of ruin)
- Donât select trials randomly or suboptimally. Iterate and learn from your failures. Instead, search for relevant approaches and fully reflect on your failures to figure out what to vary for future trials
- Donât irrationally repeat the same trial over and over again, expecting different results
- Donât think that solvable problems are unsolvable and stop prematurely
- Be efficient. Once youâve figured out what works and generalized your approach, you can apply it to other similar scenarios and get to an optimal solution quicker
If one is more efficient with trial and error, they can increase the chances of success after every iteration. So one should respect the base rate, but always condition it on the number of shots on goal that one has
Respect the base rate, given the outcome payoff disparity
This is obvious, but worth stating in a checklist. If something that is likely to fail has a disproportionate payoff, such that the expected value of doing is positive, you should still do it.
When flipping a fair coin, getting 3 heads in a row is unlikely (12.5% probability). But if youâre given a game where you pay $1 to play the game, and will get $50 if you get 3 heads in a row, you should absolutely play the game (if losing the $1 wonât cause ruin for you)
Respect the base rate, given differences in reference classes
Cedric talked about this in the last bit of the blog, but this was worth putting as a separate point.
I have autoimmune issues (increase in eosinophils in response to specific food items) that are quite rare. If I take the advice that applies to most people (eat a wide variety of foods) â if will be catastrophic for me. My reference class in this case is indeed different from that of most people.
So instead of blindly looking at the base rate of an event x happening (P(x)), one should look at conditional probabilities that apply to them P(x|y). Where y is the set of attributes that applies to you.
If x and y are independent, one really doesnât have to worry about this. Because in that scenario, P(x|y) will be equal to P(x). But in scenarios where they donât, missing out the conditionally can be deadly.
Most people canât be professional basketball players. If youâre a muscular 7â6â with great cardio and coordination thoughâŚ
Buffett puts this difference in reference classes really well when talking about how he evaluates opportunities
You should be like a basketball coach who runs into a seven-footer on the street. I mean, youâre interested to start with; now you have to find out if you can keep him in school, if heâs coordinated, and all that sort of thing
2 Likes
Fear not, Cedric. No temperament check is going to make me allergic to downside risk. 
1 Like
Michael Mauboussin said one of the most important things he learned is this idea of using base rates. The idea is to combine the âInside Viewâ (your analysis) with the âOutside Viewâ(base rate).
Iâm curious @cedric, from your research , have you found entrepreneurs or business people really use this idea of combining the âInside Viewâ with base rates to make better decisions and predictions?
For example Elon Musk has said he thought SpaceX had a 10% chance of success. Maybe if he combined the âInside Viewâ with base rates the odds are closer to 2-5% chance of success.
Would he really have invested most of his net worth at the time into SpaceX knowing the odds were only 2-5% chance of success?
Curious peopleâs thoughts.
1 Like
Hey @CharlieM â great question. Iâll give a short answer and a longer answer.
The short answer is, no, I havenât found that entrepreneurs or business people use this idea very much.
The longer answer:
I do think that the idea is useful, but I think this isnât as intuitive or as widespread amongst business operators as it is amongst investors.
Why is this the case? I think the underlying frame that most operators have is Action Produces Information â that is, whatever informational benefits that might be had from taking action far exceeds the benefits of analysis ⌠even base rate analysis!
That said, I have reached for base rate analysis in the past few years, mostly as a way to anchor my appraisal of some concrete situation. (I am not an investor). The most recent example I can think of was a discussion about a particular customer, where I asked for overall retention metrics for the most relevant (or comparable) customer cohort.
One parting thought: thereâs a quote from Ben Horowitzâs The Hard Thing About Hard Things that haunts me:
Startup CEOs should not play the odds. When you are building a company, you must believe there is an answer and you cannot pay attention to your odds of finding it. You just have to find it. It matters not whether your chances are nine in ten or one in a thousand; your task is the same.
(âŚ) I donât believe in statistics. I believe in calculus.
Which I think captures this bias towards taking action to generate information (or finding a solution) instead of analysing probabilistically.
1 Like
Thanks for your thoughts!
When using the inside view outside view theory, Kahneman recommends using a correlation coefficient and formula to come up with the final prediction. He talks about this in detail in Appendix C âCorrecting Predictionsâ from his book âNoiseâ.
When you used the inside view outside view theory, did you use a correlation coefficient? Or did you just pick a number closer to the outside view (base rate)? Or how did you use the inside view outside view theory?
1 Like
I havenât read Noise yet, and so I didnât use a correlation coefficient. One of the things that Iâve found challenging about the base rate approach is that picking the right reference class is actually quite difficult, especially for operating decisions.
For instance, if you want to evaluate a specific enterprise deal, you may chose to:
- Compare against other enterprise deals of the same size that your company has done.
- Compare against other enterprise deals of the same size in your industry
- Compare against other enterprise deals within the past year. Or over a five year period? But what if underlying conditions have changed over the five year period?
And so on.
The truth is youâll have to exercise some discretion and judgment when using base rate analysis â selecting an anchor for your judgment is well and good, but it isnât a panacea!
But putting base rate analysis for operations aside, there are some interesting implications of the technique in investment decisions (which we should expect, seeing as the approach has had more traction in that domain).
For instance, Michael Mauboussin has a number of publications on using base rates in investment decisions:
The second publication basically illustrates a problem with the base rate approach: Mauboussin points to a growing body of research that newer businesses have significantly different business model characteristics (due to software, network effects, and other intangibles), and thus cannot be compared to older companies as a reference class. For an approachable take on this argument, see Sean Stannard-Stocktonâs Problems with Base Rates.
So I guess my current take is that this is a powerful analytical tool, but caveat emptor.
2 Likes
According to Mauboussinsâ book âThe Success Equationâ some domains are made up of almost all skill, and some domains are made of skill and a lot of luck.
For domains where itâs almost all skill, like tennis or chess, you want to make predictions that are close to the inside view, because the correlation coefficient will be almost 1.
For domains, like business and stock picking, you want to stick closer to the outside view or base rates, because there is a lot of luck involved, so the correlation coefficient will be closer to .1 or .3.
The reason you need to add the correlation coefficient is because when something has a lot of luck, it will also have more Regression to the Mean.
In reality, I never heard anyone adding correlation coefficients to their predictions, so Iâm curious why arenât people using it?
Could this be a case of Outcome Bias? There might be 100 startup CEOs who follow this, but failed and never got to tell other people this narrative.
Annie Duke, in her book âQuit,â would argue that learning when to fold earlier, gives you the advantage of being able to start another company when the expected value looks better.
2 Likes
It could be, but I will say that I havenât (yet) met any successful operators who default to this kind of âbase rate reasoningâ mindset. Your guess as to why is as good as mine.
If I would wager two reasons:
a) If you only take bets with good base rates, youâre likely to do something obvious; sometimes, the most valuable moves to make in business are decisions where it isnât a clear reference class to use.
b) Thinking probabilistically has a way to drive you insane in operating (not investing) contexts. By this I mean, when youâre doing a startup, your default outcome is death, and constantly calculating the odds of survival is cognitively very hard on you. Instead, the more tractable thing to do â and the thing that I see most people doing, myself included â is to remind yourself that at any point in a companyâs life there are really only a small handful of things that have to go very very right, so you should just focus on those to the exclusion of all other lower level priorities.
I must repeat that I mostly think this is true in relation to the early-to-mid stage. I have less experience with latter stage companies. Anecdotally, I think the more latter stage you are, the more âcapital allocationâ-type decisions you have, and so you can get away with more of a balanced mix of âstatisticalâ and âcalculusâ type thinking. But âcalculusâ type thinking definitely dominates in the early stage.
2 Likes
I think we can meld the two approaches. Yes, focusing on the base rates is not helpful when executing. However, thereâs more to forecasting than the base rate. If the base rate is P(Outcome), then you can refine your forcast with P(Outcome | Evidence A, Evidence B, ...).
With this you can look at Evidence A, Evidence B, ⌠and work out which has the largest impact on the final odds, and which you can control. That leads you naturally to âif we do X and Y, we should see A and B, which we believe will lead to a good outcomeâ. Later, if you do X and Y and do not see A or B, then you can update your beliefs about how P(Outcome | Evidence A, Evidence B, ...) works. Once you have your model and you believe you know the âbig leversâ you go to work.
Notice how closely that tracks with SPC concepts. Youâre not measuring the outcome; youâre modelling the system and working on what you can change. You donât even need to âfullyâ model the system youâre working on, just enough to guess at the odds. That means you can do a quick check to see if adding a parameter to your model helps a lot. If not, do not include it.
This also points to another benefit: The choice of base rate doesnât matter. As you accumulate evidence your guess gets better. Even a âmaximally stupidâ initial base rate will eventually be corrected (unless you pick P(Outcome) = 1 or 0). That means you can stop worrying if you need to think about all companies, companies in your industry, etc. Just pick one (I go for the broadest personally) and get iterating. As my old crypto prof used to say: âGuess and check.â That said, very stupid & confident initial guesses may take a lot of evidence (i.e. expense) to correct, which may still kill the company.
The main drawback of this is that adding evidence can increase the uncertainty. Take the example of âshould I look only at companies in general or in my industry?â We model that as P(Outcome) or P(Outcome | Industry). P(Outcome | Industry) has fewer data points. Smaller data sets usually give you less information, leading to more uncertainty.
1 Like