I think one wonderful thing about the idea of ‘knowledge’ is that you can just ask yourself “what can give me more marginal knowledge about the customer?” And sometimes (often!) this is ‘talking to the customer’, and sometimes this is ‘instrumenting the product so we can see what the customer actually finds valuable.’
So I don’t see any reason you should give one up when you can also do the other.
This is one point I wanted to make but I guess I would save it for a later essay. During our podcast together, I asked Colin if the methods behind the WBR would work for a pre-product market fit product. And he said something to the effect of “Of course! Often you need to get a whole bunch of things right to get a successful product. (Cedric’s note: like if you’re launching a streaming service you need to ensure the video selection is large enough and the streaming is fast enough and latency of the services behind it is low etc etc). If you don’t instrument those things, then how do you know if your product failed because the idea was bad or if your execution was bad?”
And I think that’s a damn good point. It’s worth recalling that while Amazon Prime itself was an intuitive bet, they instrumented the hell out of it just to make sure they had actual knowledge of consumer behaviour + program behaviour + financial performance over the life of the entire bet, up to and beyond the point it was proven two years later.
And all throughout that process they were subjecting program costs to process control: constantly iterating to see if they could bring overall costs of Prime down to a controllable, predictable level, before it bankrupted the company.
A huge deal. I think one of the more depressing things we learnt investigating this body of work is that if you don’t have the power to structure incentives, you aren’t really going to be able to execute the full dream scenario described in the ‘What If It Doesn’t Have To Be This Way?’ section. Every example I gave at the end of the essay (Amazon, Koch, Ford, etc) was of a CEO-led initiative, which enabled (forced?) the various departments to work together.