How to Run Smart Experiments When You Just Don’t Know - Commoncog

Blog Comments
1

How all good entrepreneurs run experiments at the earliest stages of a business, and how you can use this approach for your own career.

This is a companion discussion topic for the original entry at https://commoncog.com/run-smart-experiments-dont-know
2

Enjoyed this one. Learning how to systematically navigate the unknown and unknowable – a paradox, because as mentioned, you must commit to starting and figure it out along the way – but certainly a fundamental source of value creation. This essay (and the other two referenced writings) made me think of a 2006 paper I read recently called Investing in the Unknown and Unknowable by Richard Zeckhauser (Harvard professor, and interestingly, bridge champion): https://scholar.harvard.edu/files/rzeckhauser/files/investing_in_unknown_and_unknowable.pdf

The context may be slightly different, i.e., investing in rather than running businesses, but the question it answers is the same - how to rationally navigate uncertain situations? The essay offers some framework for doing so.

Some of the major ideas/observations by Zeckhauser:

  • People, by their nature and due to the unavailability of prior relevant experience to guide them, are turned off by situations that are UUU (Unknown, Unknowable, Unique). This can create mispriced opportunities for the few who can navigate UUUs
  • Identifying people with complementary skills who therefore tend to successfully navigate unknown and unknowable situations can create a lot of value. One such complementary skill can even be the “unusual judgment” of an uncertain and unknowable situation itself. Obvious, but a takeaway here being…find these people
    • Zeckhauser’s Maxim A: “Individuals with complementary skills enjoy great positive excess returns from UU investments. Make a sidecar investment alongside them when given the opportunity”
  • Understand human behavior - your own tendencies/susceptibilities and those on the other side of the table
    • “Similarly, the more difficult a field is to investigate, the greater will be the unknown and unknowables associated with it, and the greater the expected profits to those who deal sensibly with them. Unknownables can’t be transmuted into sensible guesses – but one can take one’s positions and array one’s claims so that unknowns and unknowables are mostly allies, not nemeses. And one can train to avoid one’s own behavioral decision tendencies, and to capitalize on those of others.”
  • Understand information asymmetries (i.e., behavioral economics, game theory, competitive knowledge, ignorance, etc.) – such as with competitors or those on another side of a deal
    • What do I know? What do they know? What do each of us know that the other knows and/or doesn’t know? How hard is it for me to estimate? How hard is it for them to estimate

Curious about any reactions and how this framework plays with Cedric’s essay.

2 Likes
4

Vaughn believes that many problems we face today come from misapplying the ideas that work in risk (such as Annie Duke’s Thinking in Bets, a book he hates) to the areas of business and life that are purer expressions of uncertainty.

Seconded. When risk managers attempt to convert the indefinite (an uncertain situation) into the definite by placing bets (bets need defined parameters) things can get ugly

My experience with this is part of my career history but off topic for the forum, so rather than go into that I’ll note that there are several instantiations of this phenomenon that come from Hollywood’s ongoing failure to compete with or engage profitably with YouTube

Quibi is one instance, Disney’s acquisition of Maker Studios is another

2 Likes
5

Are there any credible people talking about this further? I suspect Hank Green would be one such person - launches all the time and talks about ideas that are in this essay.

6

The question “what are possible outcomes?” throws me off a little bit. It makes me think this is a question that I would ask before I do an experiment. If I ask it afterward, am I asking about outcomes that I now think might happen that I didn’t think before? Not the outcomes that actually happened?

2 Likes
7

I take it to mean both things! Sometimes some new outcomes have occurred that you couldn’t have imagined. And sometimes those outcomes opens up a new set of possible outcomes you didn’t think was on the table, which provokes further experimentation.

1 Like