Don't Go Off Vibes: What the Data Actually Says About AI and Software Development
The numbers are in, and they're both exciting and humbling. AI is generating half to two-thirds of new code at some organizations, nearly doubling pull request throughput. But faster code is only part of the story. The real challenge for engineering and product leaders right now is figuring out what to do with that speed, and how to tell whether it's actually creating value.
Here are a few of the strongest takeaways from Part 2 of our "Don't Go Off Vibes" series with Span CTO Stephen Poletto. (You can read Part 1 here.)
Measure Business Value, Not Token Consumption
The most foundational warning from the session: don't let AI spend become your north star. Organizations can easily fall into the trap of optimizing for AI usage itself, a classic case of Goodhart's Law, where the metric becomes the target and loses its meaning. High token consumption or high AI spend does not automatically equal higher productivity or better customer outcomes. The organizations winning with AI are the ones that stay ruthlessly focused on what the work produces, not how much AI was involved in producing it.
Traditional Productivity Metrics Are Losing Their Relevance
PR throughput, lead time, deployment frequency … these metrics still have a place, but they were designed to measure a narrow slice of the software development lifecycle. In an AI-accelerated world, they become even less representative of whether a team is creating real value. Leaders need to shift focus upstream and downstream: are we building the right things? Are customers better off?
Speed Is Creating New Bottlenecks
AI can dramatically accelerate code generation, yes. But what's happening downstream is equally important. As code volume increases, code review, quality assurance, and validation are becoming the new constraints. The bottleneck hasn't disappeared, it's just moved. Teams that don't redesign their review and QA workflows will find that the gains from AI-generated code get swallowed by review backlogs.
The Highest Performers Are Creating Space for AI Mastery
One of the most striking anecdotes from the session was about a leader who told their team to stop writing code entirely and work exclusively through AI agents. This was not meant to stop shipping, but to force a step-change in how engineers think about their work. This kind of deliberate, structured experimentation is what separates teams that are incrementally adopting AI tools from teams that are fundamentally transforming how they operate.
Agent-Based Development Is the Next Frontier
Companies like Stripe, Ramp, and WorkOS are already experimenting with background agents that handle development tasks end-to-end: generating pull requests, running tests, validating changes, and interacting with production systems before surfacing work for human review. This isn't a distant future, it's happening now, and it will reshape what "software engineering" means for most organizations within the next few years.
Leadership Is the Limiting Factor
Tooling is the easy part. The harder challenge is organizational: creating executive support for experimentation, giving teams the time and safety to develop new ways of working, and evolving operating models as AI capabilities continue to accelerate. The organizations seeing the greatest impact aren't just the ones with access to the best models, they're the ones with leaders who are actively guiding their teams through adoption, strain, and eventual mastery.
The Bottom Line
AI is not a productivity shortcut you can plug in and walk away from. It's a capability shift that requires new metrics, new workflows, new infrastructure, and new leadership approaches. The organizations that treat it that way, measuring outcomes over outputs, investing in review and verification systems, and creating real space for their teams to learn, are the ones that will come out ahead.
The vibes are good. But the data is better.
Want to see all of the takeaways from this event, and more? Log in, or become a member for full access.
