There’s a lot of buzz about something big happening in software engineering thanks to the latest batch of AI models. However, most knowledge workers think this is just a “coding thing” which doesn’t apply to them. They’re wrong.
Dan Shapiro, Glowforge CEO and Wharton Research Fellow, recently published a five-level framework which maps the level of AI assistance for coding from simple searches all the way to a “dark factory,” where AI is essentially just a black box that turns specs into software.
I want to walk through those five levels, because I think this pattern also applies to knowledge work, and we knowledge workers are not far behind coders in this regard.
Shapiro’s five levels of AI use in coding
(I’ve condensed but mostly used his words here)
- Level 0: AI is spicy autocomplete. You’re doing manual coding and not a character hits the disk without your approval. You might use AI as a search super search engine, or occasionally accept a suggestion, but the code is unmistakably yours.
- Level 1: AI is a coding intern. You offload discrete tasks to AI. “Write a unit test.” “Add a docstring.” You’re seeing speedups, but you’re still moving at the rate you type.
- Level 2: AI is a junior developer. You’re a “pair programmer” with AI and now have a junior buddy to hand off all your boring stuff to. You code in a flow state and are more productive than you’ve ever been. Shapiro says 90% of “AI-native” developers are here, and the danger is from level 2, and every level after it, the coder feels like they’ve maxed out and they’re done. But they’re not.
- Level 3: AI is a developer. You’re not the developer anymore. (That’s your AI’s job.) You’re the manager. You’re the human in the loop. Your coding agent is always running in multiple tabs, and you spend your days reviewing code and changes. For many people, this feels like things got worse. Almost everyone tops out here.
- Level 4: AI is an engineering team. Now you’re not even a developer manager, you’re a product manager. You write specs, argue with the AI about specs, craft skills plan schedules, then leave for 12 hours and check if the tests pass. (Shapiro says he’s here.)
- Level 5: AI is a dark software factory. You’re the engineering manager who sets the goals of the system in plain English. The AI defines implementation, writes code, tests, fixes bugs, and ships. It’s not really a software process anymore. It’s a black box that turns specs into software.
AI + coding future thinking also applies to AI + knowledge work
Nate B. Jones covers the AI-and-software engineering beat better than almost anyone. His YouTube videos are “required watching” for me. I realized recently that everything he says about how AI is impacting software engineering also applies to AI impacting knowledge workers. For example, some quotes of his from recent videos about coding which apply verbatim to knowledge worker using AI:
“The bottleneck has shifted. You are now the manager of however many agents you can keep track of productively. Your productive capacity is limited now only by your attention span and your ability to scope tasks well.”
“These are supervision problems, not capability problems. And the solution isn’t to do the work yourself. It’s to get better at your management skills.”
If we do some simple Mad Libs style find-and-replace, Nate’s also a pretty good “future of work” strategist! Just swap out:
- code → deliverables / work product / output
- engineer → knowledge worker
- technical leader → business leader
- implementation → producing deliverables
- system → outcome
- tests → success criteria
- codebase → work stream
- syntax → formatting
Let’s try that on some more quotes from his recent videos:
“It’s less time writing code deliverables. It’s much more time defining what you want. It’s much more time evaluating whether you got there.”
“Most engineers knowledge workers have spent years developing their intuitions around implementation producing deliverables and those are now not super useful. The new skill is describing the system outcome precisely enough that AI can build it, and then writing tests success criteria that capture what you actually need, and reviewing AI-generated code output for subtle conceptual errors rather than simple syntax formatting mistakes.”
“If you’re not thinking through what you want done, the speed can lead you to very quickly build a giant pile of code work product that’s not very useful. That is a superpower that everyone has been handed for better or worse and we are about to see who is actually able to think well.”
“We need to think as technical business leaders about where engineers knowledge workers should stand in relation to the code AI-generated output based on the risk profile of that codebase work stream itself.”
That last one is illustrates the power of this perfectly. That concept applies to software engineering, but I never would have thought about it in the context of knowledge work. Yet it 100% applies there as well. Which strategic deliverables require human review and which can you trust to the Dark Knowledge Factory?
The five levels of AI use in knowledge work
Now let’s take Shapiro’s five levels of AI use in coding and translate them to knowledge work. (Some of these loosely map to my own 7-stage roadmap for human-AI collaboration in the workplace from six months ago, though Shapiro’s levels address the relationship between humans and AI, whereas I focused on the mechanics of the collaboration.)
Putting Shapiro’s coding levels through our Mad Libs code-to-knowledge work translator:
- Level 0: AI is a spicy search engine. You’re doing the knowledge work and not a word hits the page without your approval. You might use AI as a super search engine, or occasionally accept a suggested sentence, but the deliverable is unmistakably yours. This is most enterprise knowledge workers today.
- Level 1: AI is a research intern. You offload discrete tasks to AI. “Summarize this document.” “Draft a response to this email.” You’re seeing speedups, but you’re still moving at the rate you type. You’re still the one producing the deliverable. This is most people’s experience with Office Copilot
- Level 2: AI is a junior analyst. You’re “pair working” with AI and now have a junior buddy to hand off all your boring stuff to. You’re in a flow state and more productive than you’ve ever been. Workers at this level use persistent AI collaboration spaces, like Google NotebookLM, Claude Projects, or Copilot Notebooks. Like their coding counterparts, starting at Level 2 and every level after it, knowledge workers feel like once they’re here that they’re done and they’ve maxed out. But they haven’t.
- Level 3: AI is an analyst. You’re not the one producing work anymore. (That’s your AI’s job.) You’re the manager. You’re the human in the loop. Your AI is always running and you spend your days reviewing and editing everything it generates. Strategy decks, market analyses, competitive intelligence, communications. Your life is tracked changes. For some workers, this feels like things got worse. Almost everyone tops out here. This is where workers using a personal AI knowledge system / “second brain” are. This is where I am.
- Level 4: AI is a strategy team. Now you’re not even a manager, you’re a director. You don’t write deliverables or even review them line by line. You write specs for deliverables. You define what a good competitive analysis looks like, what the acceptance criteria are, and what scenarios it needs to handle. You craft the prompts, system instructions, and the evaluation rubrics. Then you walk away and check if the output passes your scenarios.
- Level 5: AI is a dark knowledge factory. You are the executive who sets the goals of the organization in plain English. The AI defines the approach, produces deliverables, evaluates quality, iterates, and ships. It’s not really a work process anymore. It’s a black box that turns business intent into business outcomes. A handful of people run what used to be an entire analyst function. The verification framework is the intellectual property, not the reports themselves.
But how do you know the AI’s work is any good?
I feel like I can follow along the analogy through Level 3, but Levels 4 and 5 seem weird to me and it’s hard to see exactly how they would apply to knowledge work. (Heh, funny I’m personally at Level 3 and as Shapiro wrote, people after Level 2 think that whatever level they’re at is the top.)
The hardest question at Levels 4 and 5 is the same whether you’re writing code or strategy memos: how do you verify the output without a human reviewing every piece?
In code, the answer turned out to be end-to-end behavioral tests stored separately from the codebase (so the AI can’t cheat). For knowledge work, I think it maps to something like:
- You define what “good” looks like (for a strategy recommendation, a presentation, etc.) and you deliberately keep those separate from the AI so it can’t game the criteria. These need to be real and deep things, like “Does this account for the competitor’s likely response? Does this identify second-order effects? Would the CFO approve this?”
- Then once the main AI generates the content, a different AI uses your verification docs and is prompted to be a skeptical board member, a hostile competitor, or a regulatory lawyer and tries to find flaws. So this way, the verification loop isn’t human, it’s AI verifying AI against criteria that humans defined.
(There will be a lot of interesting work done here in the next year!)
AI’s true impact to knowledge work is just beginning
As I wrote in the opening, most talk about AI’s impact today is focused on software engineering. But coding was the beachhead, not the destination. Software was first because code has built-in verification layers, specific syntax, and billions of pages on the internet about how to write good code.
Knowledge work is next, but the timeline will be more compressed. (We have better AI now and lots of lessons from the software world.) If frontier coding teams are at Level 4-5 today while frontier knowledge workers are only at Level 1-2, a pretty good way to know what knowledge work looks like in 18 months is to look at what coders are doing right now.
As we progress towards this future, remember that the bottleneck keeps moving. At Level 1 it’s, “how fast can you produce work?” At Level 4 it’s, “how precisely can you specify what should exist?” By Level 5 it’s, “how rigorously can you verify that it’s good?” Level 5 of knowledge work will introduce a governance problem that nobody has a playbook for yet. Who owns the specs? Who defines the verifications? Who’s making sure the Dark Knowledge Factory isn’t producing hallucinated strategy recommendations that look right but fall apart under scrutiny?
Most enterprises don’t have the governance infrastructure for any of this, and it’s coming whether they’re ready or not.
Read more & connect
Join the conversation and discuss this post on LinkedIn. You can find all my posts on my author page (or via RSS).
from Citrix Blogs https://ift.tt/yGRPxQp
via IFTTT
No comments:
Post a Comment