One of W. Edwards Deming’s favourite questions was “how would you know?”
He would often ask this question when presented with some process improvement, by some hapless manager, at some industrial company. “How would you know?” he would ask — that is, how would you know that your idea has worked?
I didn’t look at it too closely at the time, but this article reminded me to go find it. I think I originally thought he was doing something a bit more sophisticated (looking at it now I think it’s a pretty simple t test), and I’m not sure whether there’s PMF fit yet (based on the low price and free 30 minute calls I kind of doubt it) but relevant to n=1 and cool someones trying to monetize that.
I’m not sure if anyone noticed, but at the end of Becoming Data Driven, From First Principles, I gave an answer in the form of the three questions of Continuous Improvement:
Well, I’ve just shown you how. In this essay I’ve given you:
A desired end state: when a company is data driven, they are culturally set up to pursue knowledge.
A proven, 90-year-old mechanism for achieving that end state: process behaviour charts, which will inculcate a deep understanding of variation and will train people to adopt the ‘process control worldview’. An important consequence of this worldview is that insight is derived from action, not analysis: you only learn to improve your business when you test control factors, not when you discover them.
A test for how you know you’ve succeeded: when your organisation actively pursues knowledge in everything that they do.
And if you squint a little more, the three questions of continuous improvement actually maps to the three parts of an Operational Definition:
If the terms ‘criterion, test procedure and decision rule’ are a little difficult to remember, here’s the way I try to remember it:
Thing — as in what Thing do you want to measure (and why?)
Procedure — how are you going to measure it?
Rule — what counts as that Thing?
Every metric that we track in Commoncog is defined using an OD format, so I think of the three questions of continuous improvement a lot.