How to Improve at Sensemaking AI? - Commoncog

cedric · May 3, 2026, 9:22am

Note: This is Part 3 of a short series on sensemaking. You may read Part 2 here.

This is a companion discussion topic for the original entry at https://commoncog.com/how-to-improve-at-sensemaking-ai

janrezac · May 5, 2026, 2:05pm

It reminds me of Simon Wardley again and his concepts of inertia & co-evolution of practice.

cedric · May 5, 2026, 3:25pm

Aye, Simon Wadley’s maps are built for exactly this scenario — making sense of an emerging new technology. But the self improvement approach implied by Data-Frame Theory goes way beyond technological revolutions alone! It’s how you come up with insights for business strategy, for example, to defeat your competitors. It’s actually the target of accelerated expertise programs. (It is necessary before we talk about how Cognitive Transformation Theory works, for instance!)

So while Wadley maps help you orient yourself when a new technology is emerging, it’s ultimately your frame construction that helps you thrive in whatever new world emerges.

janrezac · May 6, 2026, 10:40am

Wardley Mapping is useful even when new technology is not emerging.

Anyway my point was simpler – those two different tools match in this aspect. And I am glad you have studied and shared Data-Frame Theory with us.

What my interpretation of Wardley Mapping says will happen next:

New practice will emerge. You cannot escape it.
Both “never AIs” & ‘pragmatic AI adopters’ have inertia.
‘software dark factory folks’ are trying to create a new practice
this new practice will emerge in a few years
for me it is easier to change my habits all the time than to wait and see what new practice will emerge and then change all at once… which is why ‘pragmatic AI adopters’ should closely watch ‘software dark factory folks’ and do their own experiments in this space even when they do not believe in this relevant practice at all

(and I am not a software developer, I have masters in IT but that was 20 years ago… but same thing it happening in Design/Marketing space right now)

ellen · May 8, 2026, 3:17pm

@cedric I am just catching up with the sense-making series. First of all, it’s an amazingly lucid take on the whole topic of AI developments, especially in software development. Thanks for all the work you put into this. The 2nd part of the series was mind-expanding to me

Could you elaborate on this piece with a deeper insight into the actual frames of the “3 camps” of software engineers wrt. AI usage? I understand that constructing my own frames from the case studies is more helpful as an exercise, but I would also love to be able to understand other folks’ frames more easily as a side benefit of data-frame theory.

Frame in the sense the term is used in the Klein paper you linked so generously in the 2nd part of this series, defined there as

We use the term frame to denote an explanatory structure that defines entities by describing their relationship to other entities. A frame can take the form of a story, explaining the chronology of events and the causal relationships between them; a map, explaining where we are by showing
distances and directions to various landmarks and showing routes to destinations; a script, explaining our role or job as complementary to the roles or jobs of others; or a plan for describing a sequence of intended actions. (pg. 8)

While “AI pragmatists” and “Dark SW factory engineers” are useful shorthands, they don’t really tell me much about how these folks actually conceptualize the relationship between the typical entitities of a softare delivery process in novel ways and different from the “Never AI” folks.

cedric · May 9, 2026, 4:45pm

Thanks for the kind words, @ellen. I should be careful here and state that this is my interpretation of the frames — in reality, of course, I think the various proponents of each frame have slightly different, idiosyncratic beliefs that are unique to the individual (for obvious reasons!)

But I’ll do my best to state the safest, most conservative story that I think best represents each frame.

The pragmatic engineer frame:

AI fundamentally changes the way coding is done. We are seeing massive productivity gains in teams that use agentic coding tools, to the point where we have to worry about what this means for our profession longer term. But in the short term we should not lose sight of what’s important.

Code generation is easy and plentiful now, and perhaps even too easy. The fact that code is so cheap to generate means that we are able to do more things, but also things that we’d never consider doing before: tackle excessively tedious tasks, do low priority bug fixes, or even consider building tools (e.g. linters, tests, internal test harnesses or internal business process tools) that were never worth the effort. However, it’s clear that these tools come with serious tradeoffs. The amount of code generated far exceeds the ability of humans to evaluate code changes. Human judgement still matters. Keeping bugs out of production is becoming a unique challenge never before seen in our profession — and humans are responsible for that.

Ultimately we believe the fundamentals of software engineering still apply. We cannot let the superhuman capabilities of AI tools outstrip our ability to ship, check, and produce good code. We must adapt to these tools but not let this get away from ourselves.

The ‘dark factory folks’:

AI fundamentally changes the way coding is done. Code is now an order magnitude cheaper than it was before. In the past, when such a large change occurs in our industry, it unlocks completely different ways of doing things. For instance, when the cloud became a thing, it changed the way production engineering was done, and reorganised the way devs and server admins worked (those roles became what we now know as SWEs and SDEs). And before that, when distributed systems matured as a discipline, it unlocked the possibility of using masses of cheap, commodified consumer hardware to run large, distributed web services … because you could write your software to be resilient to individual machine failure. (This was the insight Google had when building out their infra in the 2000s — before Google, dotcom-era companies all bought expensive, powerful workstations from companies like IBM and Netscape and Sun).

AI is likely a similar change for the practice of software development. What might this change unlock for us?

We do not know, but there are clues. Experimenting with new methods of software development is worthwhile because there are clear benefits if we can get it to work — we can iterate faster or ship at a higher velocity, at vastly lower costs. (We may also discover new benefits during experimentation — akin to the discovery of chaos engineering, or serverless functions — both discoveries that came a few years into the cloud). But we will also discover new problems, and new tradeoffs, and we are unsure if we will be able to overcome all of them. The only way through is to find out.

Like prior shifts, our new approach will require changes across the board — not just how you build and ship software, but also how software organisations are built, who you hire, how you train those new hires, how to evaluate performance and productivity, and so on. We’ll get plenty of things wrong along the way. We are constantly sensemaking and trying to find out if someone, somewhere has figured out a new practice that unlocks further new developments. This is normal, because innovation is diffuse and rapid when you are at the frontier. This is so exciting! Let’s go on an adventure find out.

You can also infer specific variations of each frame! For instance, Mario Zechner, the creator of the Pi coding agent harness, is clearly more in the pragmatist camp. (See this talk, this podcast interview, ‘slow the fuck down’). This is super interesting because Pi is one of the most innovative coding agent harnesses.

There’s also Matt Pocock (“Software Engineering Fundamentals Matter More Than Ever”). Here Pocock’s angle is that “coding agents are awesome and give you crazy speed and leverage, but software engineering fundamentals matter because if you don’t pay attention to that, then your codebase will turn to slop, and coding agents will start to falter in your slop codebase, and you’ll lose all your leverage from AI.”

Simon Willison’s “Dark Factories are Coming” actually uses a very different anchor to justify that dark factories are coming. Willison points out that two decades ago, (this is at the heyday of Microsoft — when they were shipping things like the Win32 API) the best practice was that software engineers would NOT do code review. Instead, QA happened in a different department, pretty far downstream from the development process itself.

Willison observes that we’ve largely moved away from that over the past two decades, but it did work for its time. The thinking for the shift was that you want to have engineers be responsible for the code they write. (Note: I personally think that old QA model could only exist when most software shipped in shrink-wrapped boxes; it simply couldn’t hold up in a world where you’re pushing to production every single day. But there are likely other developments that pushed us away from this model of SDLC, including ideas like Agile and TDD.)

But then Willison asks: what if you can spin up an entire department of QA agents to simulate what a QA organisation of yesteryear would do? This has worked in the past, after all. So on one hand we’ve got AI agents generating huge amounts of code — and perhaps one way to keep up with that output is to have an equally large number of AI agents doing the QA. How would that work? He walks through a demo he saw from StrongDM — a security firm that has been around for a long time.

Anyway, I recommend listening to a sample of engineers and seeing how they make sense of things. The analogies they use and the arguments they make give you hints of their frames. And you’ll quickly recognise that there are in-group differences as well as within-group differences — as with all things human, I suppose.

cedric · May 11, 2026, 3:11am

I want to clarify a paragraph in this essay:

Thankfully, the Data-Frame theory already offers us one way out: you don’t have to believe their frame. You may hold on to your current frame, and elaborate a second frame in parallel. Folks who believe humans are Bayesian updaters won’t have this cognitive move available to them — they will think that the way to integrate new data is to do careful updating on their priors, with a belief score assigned to new developments. But if you take the Data-Frame theory seriously, you’ll know that you can do what many experts across different domains do: hold an alternative frame in abeyance, and use the human brain’s natural tendency for confirmation 'bias’to elaborate said frame. You may then switch to (or discard) the second frame later if you wish.

I’ve had three readers reach out to me over this. One reader thought I was making fun of the rationalists (the LessWrong folks). Well … maybe I am, but only indirectly. I was primarily thinking about the field of Judgment and Decision Making.

In the field of JDM, there are three types of theories:

Normative theories — how humans ought to think/decide. Traditionally this is probability theory, or Bayesian theory.
Descriptive theories — how humans actually think. (Kahneman and Tversky won their Nobel for this: Prospect Theory, the theory they came up with, is a descriptive theory that better predicts how humans will react to decision problems).
Prescriptive theories — how to get humans from (2) to (1).

Now I should note that the Naturalistic Decision Making critique of JDM is to reject even the concept of a ‘normative theory’ (“who died and said humans have to conform to the constraints of probability theory?”) The NDM epistemology is “let’s study experts and see how they are able to do what they do, and then use those findings to accelerate expertise / design better interfaces / design better socio-technical systems that are adaptive etc etc” It’s a very conservative epistemology; it doesn’t assume anything about optimal behaviour. So an NDM researcher would look at the setup above and be very uncomfortable.

And I happen to agree with the NDM approach, but let’s set that aside. It is nice if we have a normative theory to compare against. If nothing else this means JDM methods work in domains where probability theory usefully describes optimal behaviour (e.g. trading, poker). So let’s just assume that it’s ok to have a normative theory — and in that case even I agree that Bayesian theory is a pretty damn good normative theory.

Now on to descriptive theories. Humans are not Bayesian thinkers. The cognitive machinery of the human mind leads to all sorts of interesting deviations from probability / Bayesian theory. This is literally the life’s work of Kahneman, Tversky et al.

What I’m trying to say with this paragraph is that a) humans are not Bayesian thinkers, and b) three decades of using Bayesian thinking as a prescriptive theory has failed. It has failed! The primary critique that NDM makes (if we translate to JDM language) is that the best way to get humans to act in a Bayesian-optimal way is to not use Bayesian thinking.

In Part 2 I gesture at this stance:

I can write a longer essay about this ‘open secret’. Perhaps I will. But I want to leave you with the following observation: whenever you see a cognitive bias, you should understand that there are two ways to get better performance. You may do error reduction, or you may fix it by gaining expertise.

The NDM approach, of course, is “go gain expertise.”

Anyway, in this case, a) the reframing strategy is not taught as ‘Bayesian thinking’ (at least not in the prescriptive interventions I’ve seen, e.g. Tetlock et al), and b) you’re going to get better results if you seek out fragments of cases and causal models in the domain, since that helps with alternate frame construction.

And I guess c) data points are always constructed within the context of a frame, they do not arrive in the brain as discrete packets. So your updating strategy is bottlenecked on your ability to form new frames, not on your ability to do accurate-calibration updates.

PS: As I was writing this, I was informed by @jkoppel that you can easily adapt Bayesian theory to model the reframing strategy. He gave me a causal model to demonstrate. So I stand corrected that it is not possible to think in this way as a Bayesian thinker. Perhaps I should rework the paragraph to reflect the original intent: which is that 30 years of teaching humans Bayesian thinking/updating does not lead to superior outcomes.

Paul_Ingles · May 11, 2026, 6:17pm

I really enjoyed this update, and it made me go back and re-read the other articles; I don’t think I’d really taken framing to the core.

A slightly meta point (given the AI discussion) - agent skills give you the opportunity to have an agent help you with sense-making. I’m going to see if that helps both 1) identify framing in content I think is interesting or particularly relevant, and 2) help surface my current frame when working through decisions/strategy etc., and propose alternatives. As much as anything I think it might help coach me on applying the model, recognising when it’s at work etc.

I’ve only just created it so I’m open to seeing whether it works well or not, but was interested to see if anyone else here is already doing the same. I’ll happily stick mine up on GitHub if it’s helpful.

cedric · May 12, 2026, 4:51pm

I actually agree that an agent skill could make it possible to infer frames — both your own and others’. It seems like exactly the kind of thing a LLM would be good at. I don’t really know how to prompt for it, though, so if you have any good results, I’d be very happy to try it with my own work.

Martin_Chesbrough · May 17, 2026, 7:07am

Hi there, I have recently done 2 separate presentations based on the material from @cedric about SenseMaking AI (and received good feedback on the presentations being valuable to folks) and I am now looking to develop the material into a ½ day workshop.
As I started to think about this I went off at a tangent into Goldratt’s Thinking Processes (also discussed here). Specifically the Evaporating Conflict Cloud (https://youtu.be/TMMeRixlsSU).
A colleague of mine also uses a similar technique called Polarity Mapping (https://youtu.be/WfRcwweWxtQ?si=HF3QkPeig64FevH0) and I also try to practice Roger Martin’s Integrative Thinking, which has some similarities.
My thought was to apply Goldratt’s ECC technique to the different sense-making frames that Cedric talks about in article 3. I thought I’d put the idea out there for this forum to comment?