Executive summary: You don’t have to worry about survivorship bias if you read business history as instantiations of business concepts. Experts in business and investing do this, but it is pretty counter-intuitive if you aren’t used to reading history. Read on to find out why.
May we all operate businesses that last long enough to motivate accusations of survivorship bias
I have some thoughts on the statements that the critics of reading history may make.
Mentioned in the original essay:
Survivorship bias and selection bias are real problems, so reading history has limited value.
But if you see bias as a mental shortcut that you take when analyzing problems and making decisions, then bias may not necessarily be a bad thing. A bias can be good (if it is useful), or it can be bad, depending on the context. This is what I learned from a presentation on bias by Gerd Gigerenzer—
—notably this part:
Let me start with the distinction between two interpretations of bias:
Bias is error; that’s the first one. Biases hamper cognition and should be eliminated. That’s, for instance, the message of most proponents of Behavioral economics and social sciences.
On the other hand, there is a different interpretation. Bias is functional. It helps us. Actually, biases are necessary for good cognition.
Here are examples of bias as an error:
There are a number of consistency theories that postulate the deviations of judgment from logical and probabilistic rules are seen as cognitive biases such as the base rate fallacy that I mentioned, the conjunction fallacy, and framing. So framing means that you should only listen to the logical structure of a message and not to its content or pragmatic use.
Correspondent theories do not compare or define bias as a violation of consistency but as something in the world. It’s about deviations of judgments from states of the world. They are seen as cognitive biases like overconfidence bias, planning fallacy.
On the other hand, we have a number of approaches where bias is seen as something functional, not something to be eliminated:
And here are just three examples:
- Biological preparedness. That means that in order to avoid for instance, dying from learning, there are certain biases in our mind prepared. So human children, like those of our nearest relatives, have no inborn fear of snakes and spiders, but are prepared to learn this more quickly than fear of other objects, often with a single trial by observing, say, a parent who has fear of spiders, and that’s enough, and you can see the logic. So the bias is built in to protect us from suffering from learning from experience.
A second example is unconscious inferences, where particular perceptions are involved, where without a bias built in we would see nothing but chaotic sensory input. On the other hand, the bias allows us to see but also determines what we see.
Finally, predictive power in AI and in other algorithms. We know that models that are biased, for instance, ignore systematically, can enable better predictions.
So here is the tension between these two interpretations.
And when a bias is functional and useful, one may give it a more good-sounding name, such as “heuristic”, or “rule of thumb”.
Some people assume that bias is a bad thing, and this is reflected in their word choices, including collocations, e.g., pairing “suffer from” with “selection bias”, as in another statement in the essay:
if you read recorded histories of both successful and failed businesses, you’re still going to suffer from selection bias due to the sampling effects around business failure.
A Commoncog reader who adopts some of the recommendations from the essay might say: “I read histories of both successful and failed businesses, and I know I will experience selection bias to some degree; however, I don’t suffer from it.” Now, when a bias doesn’t harm your goal, you don’t suffer from the bias; you experience it. Or handle it. The negative connotations of the word bias don’t mean that it is negative for your own goal.
PS,
The bias‑as‑an‑error interpretation includes two categories: consistency theories and correspondence theories. These two theories of “bias being erroneous” map neatly, respectively, to the coherence theory of truth and the correspondence theory of truth in The Four Theories of Truth As a Method for Critical Thinking. And the “bias is functional” interpretation aligns well with the pragmatic theory of truth:
The question “is it true?” can often be substituted with “does it work?” or “is it useful with regard to my goals?”
History is interesting because we become part of it while studying it, and as such affect the outcome. This interactive dynamic makes history a different sort of animal from stuff that can be isolated from its environment and studied in that way. History also factors in when considering “timing”, namely is now a good time to pursue a certain thing. Maybe it is, maybe it isn’t, but that’s a consideration I understand may be part of an investor’s thinking, for instance, when determining whether or not to back a particular venture. And, as a business gains influence in the world the decisions made may have a larger effect on history.
When dealing directly with history, such as when considering how to end a war (or start one, I suppose), this particular participatory dynamic may matter even more and thus threading history’s needle might be even more important. Specifically because one is searching for a way to make a thing that is happening not happen, and anything really is possible. It could be that parties are shamed into coming to the table. One side might secretly (or not-so-secretly) be close to exhausting their war material. Forces in the world can change quickly in such a way that a conflict no longer “feels” important to its participants. And on and on. I like the “doctor” analogy because that’s a part of this – recognizing the “symptoms” that may point to an opportune moment (and sometimes it really comes down to minutes or even seconds) where action or inaction in the right way may achieve a desired result. The Cuban Missile Crisis is a good example of inaction winning the day, and being the right decision. One narrative about that has it that essentially the Soviets were of two minds, with leadership leaning away from war while lower in the ranks the call to escalate was louder. And, that Kennedy and some of his team caught wind of this dynamic and thus were able to diffuse the situation.
So, I think the thing I want to say is we have an opinion on how history should go for ourselves, at the least, and thus we actively participate in making it happen. That is one of the things that makes history and related stuff ill-defined, the actors involved may act or do whatever they wish and thus we all interact and those interactions become history. We are paving the road as we drive.
This write-up of the case study method was really helpful, but it still leaves me in an uncertain place on how to make this truly actionable.
You specifically note that one should be using cases to understand what possible outcomes are (“The best you can do is to be alert to the range of things that can happen, which is useful in reasoning.”). But I believe I have a good intuition for likely possible universes that can emerge (I think most people most folks who’ve spent time operating build this intuition). What I most care about is trying to predict what is the likeliest outcome from different decision paths.
To make my quandary more concrete: I’m listening to the Acquired podcast on Epic, and there’s a point where Epic is in a bakeoff for Kaiser Permanente’s business against Cerner. At the eleventh hour, Kaiser approaches both and asks for warrants in the respective companies, if picked; and Judith Faulkner, Epic Founder/CEO, has to decide what to do (spoiler: she says no, but they still win the deal).
That’s pertinent to me because in my startup we’ve found ourselves in rhyming situations with potential customers that are substantially larger than us (sadly not on quite the scale of an 8-figure/year contract that Epic was looking at!). It would obviously be facile to “take the lesson” that in 100% of similar-ish situations, I should say no to that customer. So agreed, it’s inappropriate to use cases to draw hard lessons.
But is learning about this particular scenario useful to me at all? That’s not a rhetorical question: I’m genuinely ambivalent and would welcome thoughts. On the one hand, I intellectually already knew it was a real possibility that a last-minute request from a large prospect in a close deal might not actually be a dealbreaker/maker. Duh. But it could be genuinely helpful to have that embedded in my brain in the context of a very specific story, as maybe it would make me less likely to unnecessarily panic-agree to a bad deal term in the heat of a big opportunity.
Thoughts from Cedric/others?
Obviously the Epic anecdote is not a CommonCog case study, so tell me if you think it doesn’t apply to CCM; but the question seems pretty generalizable.
A couple sides notes:
I don’t love the medical analogy you use: sure, docs don’t need to read cases about standard stuff they’ll see day-to-day; but doctors usually have a good sense and sometimes very specific statistical data around the frequency of the weird stuff they read about in cases. So say there’s a 1 in 1,000,000 chance that someone presenting with a UTI actually has Ebola [1]; it’s helpful for the doc to know about that weird unlikely cause, but it’ll probably be extremely late in their attempt to come up with a differential diagnosis b/c it’s so rare. In other words: the doctor absolutely cares “about the distribution of probable outcomes” in that case.
Not directly pertinent to my thoughts above: @cedric is your take that this is a logical and intentional process, whereby a person explicitly builds the list of “fragments, and these fragments may be recombined to help with sensemaking in new, novel cases”; or do you think this is built at some subconscious intuitive level (e.g., makes me think about @Philip ‘s comment about bias as functional predictive power).
[1] To be clear, I just randomly picked a super-common symptom and a rare condition; I’m not medical so not implying any actual connection here! 
︎
My take is the one case is not sufficiently useful. What is useful would be a handful of cases of equity transactions in enterprise deals / last minute concessions in enterprise deals / similar. Then you have a sort of repertoire of “equity worked here because the equity turned into a JV which led to an acquisition” and “it didn’t work here because the stake conflicted out other enterprise customers and the company died” and “here they just said no because everyone in this industry likes asking for those but nobody takes it seriously” etc… and you can then triangulate where you are and what factors may apply to you. And if you’re not sure, it gives you a place to start gathering more information, e.g. “has this company asked for equity stakes from other vendors before” or “how common is this deal structure in our vertical” etc.
FWIW I think it’s the first and then the second. A bit into the woo, but Josh Waitzkin (who I think has been mentioned elsewhere in this forum) calls this “learning numbers to leave numbers” or “form to leave form”
Caveat I have def not gotten there with generalized business yet but have experienced it in other domains. e.g. in code review I’ll often skim a changeset, immediately go “no this feels wrong”… but it takes a few min to logically lay out what is wrong + why I thought so + maybe past cases where I’ve seen a similar approach bug out. Or in food, “there’s something off about the way this dough is behaving,” etc…
Sorry for the late replies — your post needed a good, thoughtful response, and I was sleep compromised and somewhat burnt out at the end of the year. (And then we sleep trained our baby, and now I’m in a better place! 
)
Yes absolutely a logical and intentional process.
@Philip, I should note that Gigerenzer’s work, while useful as pushback to the cognitive biases & heuristics literature, isn’t actually useful more broadly (for outcomes that we care about in our lives) because it doesn’t give us a mechanism to test and improve. Compare the heuristics concept and contrast with Gary Klein’s Recognition Prime Decision making model (RPD) — RPD gives us levers to pull to improve our intuition. Much of NDM is intended to uncover similar reasoning models, which we can operationalise in our lives. Heuristics, on the other hand, are too impoverished a concept for us to use as a model for improvement.
I know this opinion is shared by some other members of the NDM community, though they do regard Gigerenzer as an ally against the cognitive biases / heuristics tradition.
At any rate, Gigerenzer’s ideas are not directly relevant to this discussion, because survivorship bias as I’ve treated it in the essay is NOT a cognitive bias so much as a reasoning and scientific methodological error, as @Brian_Knoles has noted elsewhere. (If we want to be more rigorous, we may say that the cognitive bias that’s applicable here is the availability bias, not survivorship bias, which is a special instance of the scientific reasoning error that we call ‘selection bias’. But as Klein and Gigerenzer himself have pointed out, availability bias is how our brains work; RPD requires availability for the initial sensemaking step.)
This is a longer discussion and we can have it in a separate thread, but — again — it is ultimately irrelevant to the topic at hand.
Perfectly ok to cite any and all sources of business history! As the quote from Charlie Munger in the piece goes, you want to learn from everything — business press, popular coverage, annual reports, etc.
It’s funny; this is a really good example because I listen to Acquired, and in that episode I intentionally filed away this exact anecdote as a calibration case.
Here’s how to approach this.
First, you need to decide what concept to hang this fragment on. I have, in my head, a concept labelled ‘deal structures’. When I came across this anecdote, I hung it in my head as ‘here is another way folks can use warrants — when you have deal leverage against a counterparty and want better downside protection by becoming an equity holder.’
Why do I have such a concept in my head? Because I may do business deals in the near future and I need to know what tools are available to me, or to my counter-party. This works the same way, say, if you read a nice turn of phrase in a book and file that away with a note to yourself “I can use that in my own writing!” It’s also the same way a programmer may spot a design pattern in some third-party codebase and go “oh, I can use it in my future projects when I have such-and-such a problem to solve.” What other instances do I have under this label? To pick a recent example, in the Gillette case:
By July 1989, Buffett had invested more than $600 million via his holding company, Berkshire Hathaway, and had become a director at Gillette. The way he conducted this deal was interesting: Berkshire purchased newly issued preferred convertible stock that would pay out an 8.75% annual dividend (on the face value of the stock). As was common with this sort of deal, the dividend would accrue if it was not paid out in a particular year. This slug would be convertible in two years into roughly 11% of Gillette common stock — a clause which Buffett duly triggered when the deadline arrived. In the process, the deal structure would take the company out of play for the rest of the highly acquisitive, LBO-crazy 80s.
Buffett has used warrants for different deals. Most famously, during the GFC Berkshire bought $5 billion of Goldman Sachs warrants at $115 a share. He made a killing in that deal. Why preferred convertibles in this case, and warrants in the Goldman case? Figuring this out would give you an idea of the kind of deal structures and considerations that folks have. It will then improve your ability to cut deals in the future — either when you’re on the receiving end, or if in a few years you become large enough that you want to start acquiring other companies.
Hopefully that answers your question.
Yes, ill-structuredness is a spectrum; you may say (and I also state in the essay) that medicine is less ill-structured compared to business and investing. You may say that there’s a distribution of symptoms in medicine. In some cases, medical reasoning is also taught using ‘base rates’ as a conceptual thinking tool, which is statistical in nature. Meanwhile in business it is very dangerous to think that there is a stable distribution, because new technology or new market dynamics may emerge that change things fundamentally.
However, even in medicine it is possible for distributions to be thrown out of whack — and this is the case where the patient has many diseases at the same time (heart disease and a stroke and high blood pressure and diabetes and the common cold). In such cases, statistical reasoning must be tempered with judgment — and CFT tells us that the way judgment works is rapid case comparison with fragments of cases they’ve seen in the past.
