You’re going to need to read the previous essay — titled Becoming Data Driven, From First Principles — to understand this one. Do that first.
Thanks for the essay, I really found it helpful.
In case this is helpful for others, I first came across the charts in the Working Backwards book and loved them. We’ve published some code on GitHub that others can use that’ll work with pandas + matplotlib to give you the same style: nooda/example.ipynb at main · rvu-tech/nooda · GitHub for an example.
This was a great piece! Seriously, getting to the heart of the WBR is no simple task, and it was achieved. Wanted to share a few additional thoughts / observations for discussion:
Interesting connection to AI Training / Research: The overall approach / objectives of the WBR seem very similar to how Ilya Sutskever described how they approach AI training / research. The focus is less on “new ideas,” and more on understanding the behavior of the system. Here’s the quote:
“Coming up with new ideas is actually just a modest part of the work. Even more important is to understand the results…to understand what’s going on…figuring out the next experiment to run. A lot of the time is spent on that. Understanding what could be wrong, what could have caused the system (neural net) to produce a result which was not expected.”
The real driver of progress / execution / performance/ superior results comes less from the generation of the new ideas, more on the ability to understand WHY the outcomes are happening. To put it another way: the better you understand the system, the better your innovation and performance will be.
EUREKA: The essay provided a Eureka moment for me for something I haven’t been to fully articulate: I like to argue that “less data provides more information about the business.” And the epistemology section here articulates a big part of why I think that is true, but hadn’t realized before. What I’m really trying to say is more like: “Better to have “less” data that actually meets good epistemological criteria vs. having all available metrics that only may be relevant”.
The metrics for the WBR go through a rigorous process to refine down what is actually useful to be reviewed on a weekly basis. So in this sense, “Less metrics” doesn’t mean to just focus on a small number metrics, but to go through a WBR-type process to determine which metrics to track. (And to take it a step further, it is better to have a smaller set of metrics that have gone through this process, even if they don’t capture the full scope of the business, than to have a lot of metrics that haven’t gone through such a process).
To tie this back to the first point: in order to identify those metrics you need to have an understanding of how the business operates–of how the system in which your company exists operates. And it is that understanding that is required to get to the “less metrics.” And it is that understanding that delivers the results.