Planning for an AI Stack That Won’t Stand Still
Philip Kiely, who leads Special Projects at Baseten and is the author of Inference Engineering, joined Enrich for an intimate dinner conversation moderated by Roshan Paiva, Enrich member and Head of Engineering at Cotality. Philip has been working in AI since before ChatGPT and has lived through 3-4 hardware generations and 12+ model generations, giving him a rare vantage point on where the stack is actually headed versus where the hype says it's headed.
In this conversation, Philip Kiely explored what engineering leaders should be planning for as AI models and infrastructure continue to evolve. His message wasn't to bet on a particular model or provider, but to build for constant change: maintain flexibility across models, develop your own evaluation capabilities, expect inference costs to drive dramatically more AI usage, and recognize that proprietary data and post-training may become some of the most important advantages companies can build.
Key Takeaways
Don't architect your AI strategy around today's "best" model.
The leading model changes too quickly to make a single-provider strategy durable. Philip recommended maintaining flexibility across closed and open models, multiple labs, and multiple providers so teams can adopt better technology as it emerges rather than being constrained by yesterday's decision. The goal is to create a unified layer that makes the underlying models interchangeable.
Your own evals may be one of the most valuable AI assets you build.
Public benchmarks often tell companies surprisingly little about whether a model will perform well on their work. Philip argued that organizations should develop proprietary evaluations based on their real tasks, inputs, outputs, and definition of success. That capability makes it possible to confidently compare models, switch providers, and determine whether customization is actually improving performance.
The future isn't necessarily bigger models—it's increasingly specialized ones.
Frontier models will continue pushing the limits of what's possible, but Philip is seeing companies build products on frontier models and then scale them using smaller, customized models. Open models can be post-trained on proprietary data and optimized for specific tasks, potentially giving companies better economics and greater control than relying exclusively on frontier APIs.
Proprietary data becomes more valuable when you can turn it into proprietary model performance.
Several companies in the room discussed moving toward open models because they don't want valuable internal data leaving their environments. Philip pointed to companies using their own data for post-training and reinforcement learning as an important emerging pattern: instead of simply feeding proprietary context into someone else's model, organizations can use that knowledge to create models that perform uniquely well on their business.
Cheaper inference probably won't lower your AI bill—it will expand what you use AI for.
Philip shared an example of a Baseten customer spending roughly $450,000 annually. After Baseten introduced an optimization that could have cut the customer's cost dramatically, the customer expanded the feature to many more users—and ultimately spent substantially more. His experience has been consistent: when inference gets cheaper, companies tend to consume significantly more of it because previously uneconomical use cases suddenly become viable.
The next wave of AI adoption depends on making inference dramatically cheaper.
AI has penetrated some of the world's highest-value knowledge work first—software engineering being the clearest example—because expensive inference can still produce an attractive ROI. Extending AI into a much broader range of jobs and workflows will require the economics to change. As tokens become cheaper, entirely new categories of automation become financially viable.
Optimize for the cost of completing the task—not the price of the model.
A cheaper model isn't necessarily cheaper if it takes more attempts, more tokens, or more human intervention to produce the right result. Philip discussed model routing as a practical way to match tasks with the appropriate level of intelligence, often using only two or three tiers rather than trying to perfectly select among dozens of models.
Sometimes a "model problem" is actually a tooling problem.
When an attendee described LLMs struggling to reliably perform calculations for real estate transactions, Philip's response was that he wouldn't expect the model itself to do the math. Just as a financial analyst uses Excel or a calculator, AI systems should be given specialized tools for tasks that require deterministic accuracy. Better agent architecture can be more important than finding a smarter model.
The hardest AI problems are increasingly the ones where "good" is difficult to measure.
Coding has proven especially well suited to AI because outputs can be tested: the software either works or it doesn't, and performance can often be quantified. Creative and qualitative work is harder because concepts like taste, style, and quality are difficult to encode into a reward signal. Philip pointed to this as an important frontier for the next generation of AI systems.
Build your AI infrastructure for optionality.
Perhaps the most transferable lesson from Philip's discussion was to avoid making irreversible bets in a market changing this quickly. His approach is to create heterogeneity at each layer—models, providers, tools, and even services such as search—then expose those choices through a unified interface. You don't need to correctly predict which technology will dominate in three years if your architecture makes it easy to change your mind.
