This is an investigative note, and only nominally part of the Becoming Data Driven in Business series. A fair warning — this essay is not intended to be as useful as prior essays in the series.
The books I’m currently digging into, in rough order:
I exclude Eastern or specifically Japanese companies from the statement above because I don’t know enough about them to comment; Deming’s influence is largest in Japan. Perhaps they’re operationally rigorous, wonderful places to work?
Having spent roughly 15 years in two companies that took continuous improvement very seriously, my main takeaway is that it is relentless.
I’ve started paying a lot more attention to celebration of successes and failures in continuous improvement because of it. Because if you never stop to celebrate successful changes or learning from the unsuccessful changes it really becomes a treadmill without an end. Which is demotivating.
It really helps to think about a rhythm for continuous improvement with pauses / low activity periods for the same reason. Currently thinking through what a continuous planning setup for our teams would look like and this has been on my mind a lot. Quarterly planning rhythm has its advantages too 
(never thought I would say that out loud)
So I am a huge fan of Deming and his concepts. I honestly think his concepts and many of the business strategies you write about are very deeply linked. The interesting thing is that many of the business processes used by American companies today like Lean, six sigma, and just-in-time production have links back to Japan, which links back to Deming and from him to Shewhart.
If I could recommend a book to really grasp a bit of the human element of Deming, it would be the book “The World of W. Edward Deming”. It is half biography, half journal, and all around a sort of meandering journey of his life. Not a book you would read to gain business knowledge. However, if you were interested in the geisha girls who served him in Japan while he visited, or the sashimi he ate, and his thoughts that he wrote down about statistics, and where he gained his knowledge, it is a very interesting read.
On the topic of the dehumanizing elements of the data-driven environment vs the humanistic elements espoused by Deming, I think that it is much like the history of the office cubicle by Propst. His intention was to create a customized workspace element to improve comfort and work output. Instead, his concept was taken and bastardized to produce a cheap dehumanizing uniform office for maximum production at the lowest cost.
Thanks @kazuo.todd! I’m adding that to my toread list (it sounds like exactly what I’m looking for).
As a funny aside, this article ended up on the Hacker News front page, which in turn ended up on Twitter, which in turn caught the attention of a veritable Deming scholar. Who then wrote up a response:
While all excellent books, I’d suggest a re-ordering and some additions that can help accelerate the learning (outside of reading this newsletter, of course…):
- The New Economics , Deming. Read, re-read, read again. This is one of those rare books that will change as you grow and evolve your own thinking over time.
- The Deming Dimension , Dr. Henry Neave - excellent companion that can help buttress the concepts.
- Out of the Crisis , Deming. I’d refer to this as-needed to elucidate concepts referred to in The New Economics and The Deming Dimension, diving into the 14 Points and their expansion.
- The Symphony of Profound Knowledge , Dr. Ed. Baker. This is a deep book that took Baker years to write, based on his learning of Deming’s philosophy while working with him at Ford. Not everyone’s cup of tea and I’d put it in the “Advanced” reading category.
- Four Days with Dr. Deming , William J. Latzko, David M. Saunders. This is a good book to get a feel of what a classic Deming Four Day Seminar was like, stepping you through the learnings on each day. Latzko was a long-time friend and colleague to Deming, contributing a portion of Chapter 7 of Out of the Crisis on applying Deming’s thinking in banking and financial institutions.
- The Essential Deming , Dr. Joyce Orsini. Orsini is a PhD student of Deming’s, and also contributed significantly to developing his theory. This book nets together a number of his lectures, papers, and correspondence that give some context.
I fear this rabbit hole is a lot deeper than expected …
The rabbit hole is a lot deeper, but it’s worth jumping into. I have a jumble of thoughts that I’ll aim to make somewhat coherent regarding this, as this is another topic I spend a lot of time thinking about.
The label I currently have on the paradox you are describing is “the agent-principal problem”. The principal-agent problem is familiar to many of us, and there is a lot of formal material on the topic of how to ensure that an agent stay in line with a principal working on their behalf. However, I find almost no material addresses the issue you describe the tendency of a principal to capture all the increased value that comes from improvements where the agent plays an essential role.
The disparity of literature itself strikes me as an indicator of why the problem you describe exists. One cause of this is the notion that a business is primarily (if not exclusively) intended to serve shareholders, even at the expense of other participants. Thus, shareholders have the right to take more value from the system whenever they see an opportunity to do so. Since those opportunities primarily occur when things change, a system of continuous improvement provides continual opportunities for value extraction, which creates the “relentless” pressure that @ramon described.
Individuals and teams can counteract this pressure locally, but since the broader system reinforces this extraction, this is why Deming’s ideas, Goldratt’s Theory of Constraints, and other systems that would create better workplaces are difficult to sustain. One exercise that tends to make me sad is to look at examples cited of organizations resisting this pressure and following up on them to find that they are no longer operating that way due to acquisitions, changes in ownership, death of the founder, etc.
Incidentally, I view this as one overlooked reason why crypto/web3 is generating so much resistance. The different model of value distribution is threatening to the extractive model. Thus the centralizing forces that have arisen in the space and the nature of regulations/enforcements coming from the US and its allies.
In short, the issue you describe seems inextricably linked to much broader societal forces that are resistant to these kinds of changes. That may seem depressing, but there is a hidden benefit. Since the resistance to this change is also holding back many other vital changes we sorely need, pushing on this is one way to help us all create a better future. That’s probably too cryptic without more context but I don’t want to abuse the attention of this forum more than I already have. 
Is it just me, or was this response rather patronizing, even if also a bit helpful? It’s hard not to think “Feel free to keep your enlightened state if it’s going to make you sound like a pompous blowhard”, but perhaps I was just feeling really uncharitable when I read that.
Heh, to be honest, one of the reasons I’ve been so reluctant to dig into the Deming canon (despite being exposed to at least some of his ideas over the past 5 years) is precisely due to such vibes in the literature. I think I’ll say two things: 1) I’m going to ignore the tone for now, because I think the intention is still positive, and 2) it’s probably no surprise that I find SPC a gentler way into the broader Deming literature. I think I much prefer Wheeler as an introduction to Deming’s ideas, albeit with a focus on just the ‘theory of variation’ pillar.
Oh god, I think this is a way better, more elegant way of describing what I was getting at in this blog post. So well said. And whilst I didn’t respond directly to @ramon, his comment is actually the one I think about the most.
God.
I’ve been thinking on the question of why an operationally rigorous company may grind people down and I have a few hypotheses:
People used to have comparatively more buying power from their income, so even if a company was relentlessly driving you to work you wouldn’t mind as much. This seems implausible as Amazon was your first example, and generally programming salaries have stayed fairly consistent (as opposed to most other professions).
Work used to take you through more levels of the “power process” of setting goals and achieving them through your own planning and actions. Optimizing businesses has removed this aspect of jobs so the work itself is less satisfying than it used to be. Other companies with less rigor give slack to employees, so they can spend their time on other activities they would prefer to do over work which is not available at rigorous companies.
The underlying value system in people has changed. People used to gain satisfaction from even menial work as a good in and of itself. With that value system, doing more ultimately productive work would only be more satisfying for an employee. The value system has shifted to wanting to gain meaning from other things.
The value system is the same, but most companies today provide less tangible value then most historic companies. Amazon’s value add today is fairly abstract to the average person compared to a nearby locally run store and an employee intrinsically feels that the value they are providing to “their world” (friends, families and neighbors) is almost nothing compared to the equivalent job they would have previously held. An example of this is contractors today building houses owned by a property development company vs. a contractor that is building a house for a specific family. The work could be the same and operationally rigorous, but I’d expect being able to see the tangible value being created from your work makes you not mind the relentless pace so much.
I think 4 is my most compelling narrative and have believed for years that job satisfaction comes in part from how few layers of indirection there are between your work, and the value you provide. I also think 3 is a major factor though, as it does seem like the value system has shifted even in my lifetime.
It seems to me what you call “agent-principal” problem has been described by Karl Marx already in the mid-1800s If the memory of my socialist upbringing serves me right (I was born in the socialist part of Germany before the Berlin Wall came down) he called it the problem of “surplus value”. Surplus value is the difference between the value that the company owner receives for a good minus the cost of labor they paid for. The desire to maximise that surplus value by lowering labor costs is the “capitalist cardinal sin” in Marx’ economic theory and the cause of the misery of the working class.
In the tradition of socialist economists, there is a lot of material on this topic. The phenomenon @cedric describes fits what Marx and his successors in the socialist lineage call “alienation”, which is the phenomenon that unlike traditional artisans, factory workers (and by extension, office workers) are not reaping the benefits of their own labor and lack autonomy and meaning in their work because they don’t “own the means of production”.
Socialist economic theory might not be the most obvious place to look for answers on this particular problem, but it is a very rich source if you want to go there. (Obviously, with the huge caveat that it is only worth looking into their problem analysis, the proposed solutions have not stood the test of time very well, not the least in my own birth country!)
Spent some time thinking about this thread.
I think there’s a couple things that stand out to me.
SPC/Deming’s work is relatively value-free, or maybe more correctly purpose agnostic, it’s a management technique for manufacturing that also can be applied to software development, closely linked to kanban. If you apply that technique on a business with a primary goal of value extraction for shareholders you will get very different results than if you apply it with a primary goal of quality improvement of whatever you are manufacturing.
if you look at the knowledge that exists about the japanese implementation of Deming, what stands out is the degree of agency for employees at every level in the system. This is what stands out as missing in the AMZ implementation, that makes it so soulcrushing. Agency has been almost completely removed from the lower levels of the hierarchy. Compare that to the implementation of andon cords in Toyota where any line employee can stop the entire manufacturing line by pulling the cord.
I would recommend reading the Toyota kata for a good introduction on how Deming’s work was transformed to match japanese culture/operating conditions. Deming’s method is an abstract framework that has many practical instantiations, so the same restrictions that @cedric wrote about previously apply.
Deming is clear that all processes have normal variation and the method focuses on removing exceptional variation from the process. A lot of what seems to make AMZ such a horrible soulcrushing place to work appears to stem from attempts to remove all variation from the process. This is very much tied to a Taylor style worldview with the associated mechanical view on the role of people in the process. As @cedric already pointed out, this is not the way Deming looked at people in the process, he mostly viewed them as the source of creative solutions to reimagine the process once exceptional variation was driven out. See also the series on goodhart’s law.
I keep coming back to the non-intuitive insight from that series that as long as you have exceptional variation in the process, you need to continue tweaking the process. As soon as you’ve removed exceptional variation, your only option for improvement is re-imagining/redesigning the process
Not so sure on the principal-agent dynamic, that entire paradigm is based on Friedman’s doctrine of maximizing shareholder value, which takes one perspective on business value, elevated it to the only perspective that matters and proceeded to ignore all other perspectives.
That has always struck me as extremely lazy thinking or a deliberate attempt to cram a complex system into the obvious/complicated area of cynefin, with all the usual disastrous consequences.
This is great and makes me reevaluate how my hypotheses I describe above fit together.
The primary mechanism of worker dissatisfaction comes from the business removing agency (my second hypothesis above). I’d posit that the resulting corporate cultures actually train the workforce out of agency, resulting in a changed value system. One of my favorite articles states this well: “The attempt to steer a person can make it hard for them to move, because it inactivates their own guidance system.”
If we have two companies with the same business model, one that gives employees from this background agency and the other that doesn’t, I would bet that the former loses to the latter, because they designed their processes for a different workforce. The only way to overcome this is to probably hire with agency as the top priority, then the former company would probably have an advantage. There is probably additional cost associated with that though as there are fewer workers that are used to functioning in a high agency organization.
Even if your goal is to build the highest quality product, you still could very well do this better with a low agency, high authoritarian environment (my mental model of AMZ).
Not sure, mostly due to a lack of data to compare and more importantly a large amount of disagreement over what it means when a company “does better” or has “an advantage”.
What’s pretty clear from data is that employees tend to need very little to no training on agency.
See commoncog podcast with Lia DiBello on the Foundry workers that essentially taught themselves JIT manufacturing or her research on the NYC public transport. Similar examples exist in Europe such as buurtzorg and Mondragon group in Spain, with an EBITDA result of >1.3B Euros over 2021.
I would argue it only seems like a paradox if you assume that Operationally Rigorous Companies exclusively use Deming’s philosophy or that it is the only approach to Operational Excellence.
In the book The Halo Effect: How Managers let Themselves be Deceived
Andrew Grove (Legendary Intel CEO) is quoted
" The quality guru W. Edwards Deming advocated stamping out fear in corporations. I have trouble with the simple-mindedness of this dictum. The most important role of managers is to create an environment where people are passionately dedicated to winning in the marketplace. Fear plays a major role in creating and maintaining such passion. Fear of competition, fear of bankruptcy, fear of being wrong and fear of losing can all be powerful motivators."
[emphasis on fear is mine]
On business paradoxes, I’d like to quote this older thread.
“… A founder needs to simultaneously hold multiple strong conflicting beliefs & be able to switch back & forth between them.”
The paradox stems from the fact that
In business, conflict will arise naturally. Why? Competition is extremely fierce. In the U.S. 3 out of 4 businesses that start fail. That’s compounded in the tech world, where 5 out of 6 start ups fail. (Note - I’m not sure how accurate those figures are)
Part of the competitive nature of tech is the Innovators Dilemma. The same forces that allow tech companies to innovate and then dominate a market means they have to constantly innovate least they get “disrupted”
To quote Mark Zuckerberg
“If we don’t create things that will kill Facebook, someone else will”
Part of the conflict of winner take all nature and innovation is that it’s easier to not reinvent the wheel and make your own copy of the competition or directly compete in their lane.
We can see this in multiple cases
We can see copying the competition in social media (think Instagram vs Tik -Tok vs Youtube)
AWS which has had a historic first mover advantage in Cloud Computing is facing intense competition from Microsoft and Google and with the coming AI wars risks losing the advantage in the market for the first time.
Speaking of Microsoft and Google, in a bigger eye-bulging case, Google tried to release a cloud gaming platform Stadia that failed but would have cut into one of Microsoft’s profitable markets of Xbox and Windows - gaming.
Microsoft (teaming up with OpenAI) responded by trying to compete in Google’s biggest comparative advantage - the search Engine.
The last example, I think, really illustrates a hard fact of life. Tech companies at the highest level are not just competitive, they are cut-throat competitive.
To quote Andrew Grove
A corporation is a living organism; it has to continue to shed its skin. Methods have to change. Focus has to change. Values have to change. The sum total of those changes is transformation.
Only the paranoid survive in fiercely competitive markets. Which brings me to one of the least discussed parts of both business and tech - emotion. Both tech and business have a tendency to talk about logic when speaking about knowledge, but in the brain the parts that are responsible for logic have their foundation in the emotional side of the brain. (Don’t have a source off the top of my head) Emotions can get placed into a false dichotomy of positive or negative. Some of this is justified - nobody likes to feel fear. However, fear in of itself is not negative, our reaction to fear is what can be positive or negative. We can fear criticism from our superior, which demotivates our working capacity. Fear of failure can paralyze us from taking meaningful action because we might incorrectly assume that our efforts will not be rewarded even if we work hard. Even more insidiously, we can fear success because even if we believe we can fix a problem with sustained effort, that we would disqualify our efforts because it might be due to luck and that we will fail at the next challenge.
On the flip side, in tech there are real winners or losers. Companies that take years to build and rake in billions on razor-thin margins can be upended by a competitor with a new innovative advantage that can be scaled. With this reality in mind, positive fear response can help sustain the temporary discomfort you feel to get in a “deep work” session to improve your skills or pry for the small details to save the company money or innovate a performance optimization.
Business knowledge is “tacit” it’s more than what you read, it’s the whole cycle of what you do and the feedback you receive. You might get yelled at for breaking a downstream system in software, but you will literally “feel” not to do that next time. Ideally, we don’t like to yell and make people uncomfortable, and we do avoid that most of the time. However, when you are under time-constraints to deliver an important feature or have a large ticket for a pipeline breaking change, empiricism shows that management can be messy when you get in the trenches.
It was written in one post that Amazon see’s “communication as defective.” That’s probably not untrue, but more positively I think the idea that we should abstract is that “actions speak louder than words (communication)” When Bezos made the mandate to open all the team’s interface to make a platform least they be fired, it did eventually spur actions to AWS though it was not pretty for developers.
It’s worth noting Andrew Grove himself was the founder of the “OKR” review and popularized “strategic inflection points.” Deming was influential in scientific management and making data driven decisions, but he was not the only one to do so. It’s also worth noting the Grove did not encourage toxic work environments, just that he realized the value of constructive confrontation and the role fear played in that.
That’s not to justify toxic corporate environments, it’s just worth noting that corporate decision-making that when working towards a profit, a company that succeeds in satisfying customers in a winner take all market (extremely difficult to do) will likely be a relatively stressful and demanding place for employees to work usually stemming from the harshness of continuous improvement but not creating a constructive environment while doing so.
A work environment can be authoritarian while still providing a good deal of agency. As a software engineer for Amazon, I have mandatory meetings such as a daily stand-up, sprint planning and occasional working groups to tackle the big picture problems but outside of that I am not “forced” to do much. In the industry, micro-managing your engineers will lead to bad employee retention. Amazon makes up for this by having a harsh review process that makes it easy to get fired if you’re not bringing results to the company. So even though my day to day gives me agency if I don’t have tangible results for that agency then I will be let go. This saves the company time and money by not having someone constantly check up on you but gives it a reputation as a “churn and burn”
I believe this is “wholesale transfer pricing”
Tren Griffin writes a lot on this topic
@ellen and @cwong, thank you both for your responses and the pointers to additional materials. I just had not been looking in the right places or thinking about this as broadly as I should, so this is helpful. Wholesale transfer via alienation is a much better way to describe this phenomenon, and also has the benefit of tying this into much broader lines of thinking.
While I agree with you @ellen that Marxist solutions do not seem that helpful here, the fact that we can tie this into fundamental societal concerns reinforces my thinking that addressing the Deming Paradox @cedric has recognized goes well beyond metric selection and method design.