You joke but that’s basically the conclusion of both the newer agent studies and the older Xerox park study on peripheral use. Language and logic are stored in an easier to access location in the brain than the kinds of skills used in architecture and planning.

To be clear, I’m no rube. I recognize that these are powerful tools. But I suppose my take here is that if an LLM is necessary to get rid of a lot of the boilerplate and setup for a task that indicates we should explore how we’re doing the task.

What I want to see is using these tools not to delegate our problem solving but finding ways to enhance and accelerate it and I don’t think writing spec sheets is the answer.

One, that sounds truly miserable.

Two, there is a decent body of evidence to suggest that this method does not actually speed up development unless you go full lights out software factory which… Why would you want to do that?

https://ide.mit.edu/insights/ai-productivity-and-roi/

EDIT: As an aside this actually reminds me of the Xerox park study into efficiency gains for keyboard heavy workflows vice mouse heavy workflows. Keyboards were perceived as faster by subjects but when actually measured the mouse was faster.

I genuinely question if they’re thinking very far into the future. Billions of agents doing what? We didn’t need a rogue AI to turn us into paper clips; we’re going to do that ourselves. Is he thinking local AI at everyone’s computer? People barely use the features of the machines they have now.

What is the difference between the models writing code “well” and their performance in this context? Are we referring to readability?

Genuine question. If we use agents to read, edit, and review code, why do we care about readability? That’s a human constraint. Unless attempting to do those three is not effective and thus requires human attention to correct issues which would justify readable code. If that’s the case; why use the agent to edit the code in the first place?

To quote the research paper conclusion…

Conclusion We evaluate the impact of context files on coding agent performance for four common coding agents on SWE-BENCH and the novel CTXBENCH, built from recent GitHub issues and less popular repositories containing developer-written context files. We find that all context files consistently increase the cost and number of steps required to complete tasks. LLM-generated context files have a marginal negative effect on task success rates, while developer-written ones provide a marginal performance gain, neither statistically significant. Our trace analyses show that instructions in context files are generally followed and lead to more test- ing and broader exploration; however, they do not function as effective repository overviews. Over- all, our results suggest that context files don’t improve coding agent performance, and should only contain specific additional instructions beyond what is already available in the codebase. This high- lights a concrete gap between current agent-developer recommendations and observed outcomes, and motivates future work on principled ways to automatically generate concise, task-relevant guid- ance for coding agents.

That sounds like “Agents.MD doesn’t work” to me.

Am I missing something?

This. This 1000%.

Software is not a mass-produced commodity that has to be distributed as fast as possible. People already have a limited attention span for the features of already existing software so speed running even more does not create value.

At some point we’re going to have to rekon with the fact that creating, reviewing, and provisioning aren’t the real bottlenecks.

Humans using the software is the bottle neck. Humans trusting the software is the bottleneck. Humans building communities and cultures around specific tools and methodologies is the bottleneck.

At the end of the day you’re user base isn’t going to be “AI” agents. It’s going to be people.