This is an investigative note, and only nominally part of the Becoming Data Driven in Business series. A fair warning — this essay is not intended to be as useful as prior essays in the series.
The books I’m currently digging into, in rough order:
I exclude Eastern or specifically Japanese companies from the statement above because I don’t know enough about them to comment; Deming’s influence is largest in Japan. Perhaps they’re operationally rigorous, wonderful places to work?
Having spent roughly 15 years in two companies that took continuous improvement very seriously, my main takeaway is that it is relentless.
I’ve started paying a lot more attention to celebration of successes and failures in continuous improvement because of it. Because if you never stop to celebrate successful changes or learning from the unsuccessful changes it really becomes a treadmill without an end. Which is demotivating.
It really helps to think about a rhythm for continuous improvement with pauses / low activity periods for the same reason. Currently thinking through what a continuous planning setup for our teams would look like and this has been on my mind a lot. Quarterly planning rhythm has its advantages too 
(never thought I would say that out loud)
So I am a huge fan of Deming and his concepts. I honestly think his concepts and many of the business strategies you write about are very deeply linked. The interesting thing is that many of the business processes used by American companies today like Lean, six sigma, and just-in-time production have links back to Japan, which links back to Deming and from him to Shewhart.
If I could recommend a book to really grasp a bit of the human element of Deming, it would be the book “The World of W. Edward Deming”. It is half biography, half journal, and all around a sort of meandering journey of his life. Not a book you would read to gain business knowledge. However, if you were interested in the geisha girls who served him in Japan while he visited, or the sashimi he ate, and his thoughts that he wrote down about statistics, and where he gained his knowledge, it is a very interesting read.
On the topic of the dehumanizing elements of the data-driven environment vs the humanistic elements espoused by Deming, I think that it is much like the history of the office cubicle by Propst. His intention was to create a customized workspace element to improve comfort and work output. Instead, his concept was taken and bastardized to produce a cheap dehumanizing uniform office for maximum production at the lowest cost.
Thanks @kazuo.todd! I’m adding that to my toread list (it sounds like exactly what I’m looking for).
As a funny aside, this article ended up on the Hacker News front page, which in turn ended up on Twitter, which in turn caught the attention of a veritable Deming scholar. Who then wrote up a response:
While all excellent books, I’d suggest a re-ordering and some additions that can help accelerate the learning (outside of reading this newsletter, of course…):
- The New Economics , Deming. Read, re-read, read again. This is one of those rare books that will change as you grow and evolve your own thinking over time.
- The Deming Dimension , Dr. Henry Neave - excellent companion that can help buttress the concepts.
- Out of the Crisis , Deming. I’d refer to this as-needed to elucidate concepts referred to in The New Economics and The Deming Dimension, diving into the 14 Points and their expansion.
- The Symphony of Profound Knowledge , Dr. Ed. Baker. This is a deep book that took Baker years to write, based on his learning of Deming’s philosophy while working with him at Ford. Not everyone’s cup of tea and I’d put it in the “Advanced” reading category.
- Four Days with Dr. Deming , William J. Latzko, David M. Saunders. This is a good book to get a feel of what a classic Deming Four Day Seminar was like, stepping you through the learnings on each day. Latzko was a long-time friend and colleague to Deming, contributing a portion of Chapter 7 of Out of the Crisis on applying Deming’s thinking in banking and financial institutions.
- The Essential Deming , Dr. Joyce Orsini. Orsini is a PhD student of Deming’s, and also contributed significantly to developing his theory. This book nets together a number of his lectures, papers, and correspondence that give some context.
I fear this rabbit hole is a lot deeper than expected …
The rabbit hole is a lot deeper, but it’s worth jumping into. I have a jumble of thoughts that I’ll aim to make somewhat coherent regarding this, as this is another topic I spend a lot of time thinking about.
The label I currently have on the paradox you are describing is “the agent-principal problem”. The principal-agent problem is familiar to many of us, and there is a lot of formal material on the topic of how to ensure that an agent stay in line with a principal working on their behalf. However, I find almost no material addresses the issue you describe the tendency of a principal to capture all the increased value that comes from improvements where the agent plays an essential role.
The disparity of literature itself strikes me as an indicator of why the problem you describe exists. One cause of this is the notion that a business is primarily (if not exclusively) intended to serve shareholders, even at the expense of other participants. Thus, shareholders have the right to take more value from the system whenever they see an opportunity to do so. Since those opportunities primarily occur when things change, a system of continuous improvement provides continual opportunities for value extraction, which creates the “relentless” pressure that @ramon described.
Individuals and teams can counteract this pressure locally, but since the broader system reinforces this extraction, this is why Deming’s ideas, Goldratt’s Theory of Constraints, and other systems that would create better workplaces are difficult to sustain. One exercise that tends to make me sad is to look at examples cited of organizations resisting this pressure and following up on them to find that they are no longer operating that way due to acquisitions, changes in ownership, death of the founder, etc.
Incidentally, I view this as one overlooked reason why crypto/web3 is generating so much resistance. The different model of value distribution is threatening to the extractive model. Thus the centralizing forces that have arisen in the space and the nature of regulations/enforcements coming from the US and its allies.
In short, the issue you describe seems inextricably linked to much broader societal forces that are resistant to these kinds of changes. That may seem depressing, but there is a hidden benefit. Since the resistance to this change is also holding back many other vital changes we sorely need, pushing on this is one way to help us all create a better future. That’s probably too cryptic without more context but I don’t want to abuse the attention of this forum more than I already have. 
Is it just me, or was this response rather patronizing, even if also a bit helpful? It’s hard not to think “Feel free to keep your enlightened state if it’s going to make you sound like a pompous blowhard”, but perhaps I was just feeling really uncharitable when I read that.
Heh, to be honest, one of the reasons I’ve been so reluctant to dig into the Deming canon (despite being exposed to at least some of his ideas over the past 5 years) is precisely due to such vibes in the literature. I think I’ll say two things: 1) I’m going to ignore the tone for now, because I think the intention is still positive, and 2) it’s probably no surprise that I find SPC a gentler way into the broader Deming literature. I think I much prefer Wheeler as an introduction to Deming’s ideas, albeit with a focus on just the ‘theory of variation’ pillar.
Oh god, I think this is a way better, more elegant way of describing what I was getting at in this blog post. So well said. And whilst I didn’t respond directly to @ramon, his comment is actually the one I think about the most.
God.
I’ve been thinking on the question of why an operationally rigorous company may grind people down and I have a few hypotheses:
People used to have comparatively more buying power from their income, so even if a company was relentlessly driving you to work you wouldn’t mind as much. This seems implausible as Amazon was your first example, and generally programming salaries have stayed fairly consistent (as opposed to most other professions).
Work used to take you through more levels of the “power process” of setting goals and achieving them through your own planning and actions. Optimizing businesses has removed this aspect of jobs so the work itself is less satisfying than it used to be. Other companies with less rigor give slack to employees, so they can spend their time on other activities they would prefer to do over work which is not available at rigorous companies.
The underlying value system in people has changed. People used to gain satisfaction from even menial work as a good in and of itself. With that value system, doing more ultimately productive work would only be more satisfying for an employee. The value system has shifted to wanting to gain meaning from other things.
The value system is the same, but most companies today provide less tangible value then most historic companies. Amazon’s value add today is fairly abstract to the average person compared to a nearby locally run store and an employee intrinsically feels that the value they are providing to “their world” (friends, families and neighbors) is almost nothing compared to the equivalent job they would have previously held. An example of this is contractors today building houses owned by a property development company vs. a contractor that is building a house for a specific family. The work could be the same and operationally rigorous, but I’d expect being able to see the tangible value being created from your work makes you not mind the relentless pace so much.
I think 4 is my most compelling narrative and have believed for years that job satisfaction comes in part from how few layers of indirection there are between your work, and the value you provide. I also think 3 is a major factor though, as it does seem like the value system has shifted even in my lifetime.
It seems to me what you call “agent-principal” problem has been described by Karl Marx already in the mid-1800s If the memory of my socialist upbringing serves me right (I was born in the socialist part of Germany before the Berlin Wall came down) he called it the problem of “surplus value”. Surplus value is the difference between the value that the company owner receives for a good minus the cost of labor they paid for. The desire to maximise that surplus value by lowering labor costs is the “capitalist cardinal sin” in Marx’ economic theory and the cause of the misery of the working class.
In the tradition of socialist economists, there is a lot of material on this topic. The phenomenon @cedric describes fits what Marx and his successors in the socialist lineage call “alienation”, which is the phenomenon that unlike traditional artisans, factory workers (and by extension, office workers) are not reaping the benefits of their own labor and lack autonomy and meaning in their work because they don’t “own the means of production”.
Socialist economic theory might not be the most obvious place to look for answers on this particular problem, but it is a very rich source if you want to go there. (Obviously, with the huge caveat that it is only worth looking into their problem analysis, the proposed solutions have not stood the test of time very well, not the least in my own birth country!)
Spent some time thinking about this thread.
I think there’s a couple things that stand out to me.
SPC/Deming’s work is relatively value-free, or maybe more correctly purpose agnostic, it’s a management technique for manufacturing that also can be applied to software development, closely linked to kanban. If you apply that technique on a business with a primary goal of value extraction for shareholders you will get very different results than if you apply it with a primary goal of quality improvement of whatever you are manufacturing.
if you look at the knowledge that exists about the japanese implementation of Deming, what stands out is the degree of agency for employees at every level in the system. This is what stands out as missing in the AMZ implementation, that makes it so soulcrushing. Agency has been almost completely removed from the lower levels of the hierarchy. Compare that to the implementation of andon cords in Toyota where any line employee can stop the entire manufacturing line by pulling the cord.
I would recommend reading the Toyota kata for a good introduction on how Deming’s work was transformed to match japanese culture/operating conditions. Deming’s method is an abstract framework that has many practical instantiations, so the same restrictions that @cedric wrote about previously apply.
Deming is clear that all processes have normal variation and the method focuses on removing exceptional variation from the process. A lot of what seems to make AMZ such a horrible soulcrushing place to work appears to stem from attempts to remove all variation from the process. This is very much tied to a Taylor style worldview with the associated mechanical view on the role of people in the process. As @cedric already pointed out, this is not the way Deming looked at people in the process, he mostly viewed them as the source of creative solutions to reimagine the process once exceptional variation was driven out. See also the series on goodhart’s law.
I keep coming back to the non-intuitive insight from that series that as long as you have exceptional variation in the process, you need to continue tweaking the process. As soon as you’ve removed exceptional variation, your only option for improvement is re-imagining/redesigning the process
Not so sure on the principal-agent dynamic, that entire paradigm is based on Friedman’s doctrine of maximizing shareholder value, which takes one perspective on business value, elevated it to the only perspective that matters and proceeded to ignore all other perspectives.
That has always struck me as extremely lazy thinking or a deliberate attempt to cram a complex system into the obvious/complicated area of cynefin, with all the usual disastrous consequences.
This is great and makes me reevaluate how my hypotheses I describe above fit together.
The primary mechanism of worker dissatisfaction comes from the business removing agency (my second hypothesis above). I’d posit that the resulting corporate cultures actually train the workforce out of agency, resulting in a changed value system. One of my favorite articles states this well: “The attempt to steer a person can make it hard for them to move, because it inactivates their own guidance system.”
If we have two companies with the same business model, one that gives employees from this background agency and the other that doesn’t, I would bet that the former loses to the latter, because they designed their processes for a different workforce. The only way to overcome this is to probably hire with agency as the top priority, then the former company would probably have an advantage. There is probably additional cost associated with that though as there are fewer workers that are used to functioning in a high agency organization.
Even if your goal is to build the highest quality product, you still could very well do this better with a low agency, high authoritarian environment (my mental model of AMZ).
Not sure, mostly due to a lack of data to compare and more importantly a large amount of disagreement over what it means when a company “does better” or has “an advantage”.
What’s pretty clear from data is that employees tend to need very little to no training on agency.
See commoncog podcast with Lia DiBello on the Foundry workers that essentially taught themselves JIT manufacturing or her research on the NYC public transport. Similar examples exist in Europe such as buurtzorg and Mondragon group in Spain, with an EBITDA result of >1.3B Euros over 2021.
I believe this is “wholesale transfer pricing”
Tren Griffin writes a lot on this topic
@ellen and @cwong, thank you both for your responses and the pointers to additional materials. I just had not been looking in the right places or thinking about this as broadly as I should, so this is helpful. Wholesale transfer via alienation is a much better way to describe this phenomenon, and also has the benefit of tying this into much broader lines of thinking.
While I agree with you @ellen that Marxist solutions do not seem that helpful here, the fact that we can tie this into fundamental societal concerns reinforces my thinking that addressing the Deming Paradox @cedric has recognized goes well beyond metric selection and method design.