Listen and Subscribe
The next phase of AI adoption will depend on making intelligence more efficient, not just more capable. In this episode, Jaya is joined by Stanford Ph.D. researchers Jon Saad-Falcon and Avanika Narayan to discuss how local and hybrid AI can help bring these tools to billions more people.
Much of today’s LLM inference runs through frontier models in centralized data centers. But Jon and Avanika argue that many tasks can already be handled by smaller, open-weight models, including models running on local hardware. Their concept of “intelligence per watt” measures how much task performance a given model-and-hardware configuration delivers relative to the power it consumes.
They also discuss OpenJarvis and Minions, two projects that explore how AI systems can use local models when possible and turn to cloud models when more capability is needed.
From there, the conversation turns to what this shift could mean for OpenAI, Anthropic, and other frontier labs. As open-weight models improve and inference becomes cheaper and more distributed, Jon and Avanika argue that low switching costs and limited lock-in could put pressure on the labs’ margins. They also explain why enterprises should be careful about handing proprietary data and decision-making processes over to model providers.
They close with advice for Ph.D. students and academic researchers: rather than trying to reproduce the frontier labs’ work with fewer resources, choose problems where original ideas matter more than scale.
What we covered:
00:00 – Cold open
00:52 – Welcome and guest introductions
02:03 – Why intelligence efficiency matters
03:22 – Defining intelligence per watt
05:46 – Overview of OpenJARVIS and Minions Projects
08:00 – Industry adoption of intelligence per watt metrics and hybrid execution strategies
09:10 – Why local capabilities will shift demand away from centralized cloud infrastructure toward fragmented, specialized hardware
12:35 – Technical drivers for local AI
12:56 – How tokenmaxxing complements efficiency research
14:36 – Why proprietary AI providers face revenue risks as open-weight models compress in size and rival frontier capabilities
17:15 – The real moats in AI today
18:23 – Advice for enterprise CIOs
22:15 – Hot takes
23:48 – The importance of US public policy and aligning public benefit with AI advancements
24:45 – Advice for PhD students and researchers
25:56 – Closing thoughts
Read the transcript:
Avanika We love tokenmaxxing, and our goal is to make that feasible. We want to make intelligence so cheap to meter that people can be running intelligence 24/7 on every device that they have.
Jon: At the same time, the path towards that efficiency isn’t necessarily linear. People think the curve naturally bends towards efficiency. It doesn’t. It took insights on the efficiency side, like the work of brilliant engineers and scientists, to actually make transistors so cheap.
We think the same thing is going to happen for computing efficiency and intelligence. There are going to be several key insights along the way towards making computing that is so ubiquitous and so distributed that it’s just going to be everywhere. It’s not going to be something that’s owned by Silicon Valley or some discrete set of insiders. It’s going to be something you can see the world over.
Jaya: Thank you guys so much for doing this, and welcome to AI in the Real World. Maybe we can start off with you guys telling us a little bit about yourselves. We’ll go with Avanika first, and then we’ll go to Jon.
Avanika: Thank you so much, Jaya, for hosting us. So, a little bit about me: I am from Palo Alto, California, born and raised. I did my undergrad at Stanford, played varsity tennis there, and ended up doing a PhD at Stanford, where I was co-advised by Chris Ré and John Hennessy.
My research interests have spanned a variety of things, but as of late, Jon and I have been focusing on local AI and, more broadly, intelligence efficiency, and how we can light up the world with intelligence. So, Jon, how about you?
Jon: Yeah, I’m Jon. I am also doing my PhD at Stanford, advised by Azalea Husaini and Chris Ré. Before Stanford, I did my undergrad at Georgia Tech and was lucky to work on a lot of great teams in both industry and academia. To name a few: Google, Databricks, AI2, and Goldman Sachs.
Now, I’m very excited about intelligence efficiency, and very excited about the intersection of LLMs with ML systems and hardware. A lot of our projects have kind of combined different parts of all of the above, so I’m excited to chat today.
Jaya: Most of the attention right now in AI is focused a lot on capability and scale. What made you guys interested in efficiency? What makes it the more interesting problem?
Jon: I think efficiency is kind of a counterintuitive thing to work on. At the same time, I think it is at the core of a lot of different problems that we’re seeing in the world today, where inference demand is going parabolic, and it’s kind of becoming an issue to satiate that demand. And even as we start to satiate the demand, the demand only grows.
It’s one thing for folks in Silicon Valley to use AI in their daily lives, but it’s another thing for 8 billion people to be able to use it in their daily lives to augment what they’re doing and improve their quality of life more generally. We view this as central to making AI work for everybody.
If you look at the world more broadly, even in the United States, a vast majority of people haven’t even had their first experience with AI to date. When that happens, not only will the load on these existing systems explode, but the costs are also going to exponentially explode. So there is a fundamental need to be more efficient in terms of the way that we do inference, and in terms of the way that we actually put these models out into the world. We’re super excited about this question.
Jaya: You guys have coined a beautiful term: “intelligence per watt” What does that measure, and what exactly was the motivation behind creating this metric?
Jon: The goal of intelligence per watt was to understand how intelligence efficiency has been changing over the years and ground it in two metrics that people really care about.
To talk about the numerator—intelligence—intelligence just means accuracy or performance on any set of meaningful tasks that people care about. Intelligence can mean different things to different people, depending on whether you're doing chat, reasoning, agentic workloads, or coding workloads. But it’s meant to capture the intended capabilities that you're aiming for on any given task or set of tasks.
When it comes to the denominator—watts—we want to understand the actual wattage, or the actual power required for running the entire LLM system. That includes the hardware, the kernels, the inference engine itself, and the LLM. There are a lot of different things bundled together to get the actual wattage: hardware selection (NVIDIA, AMD, Apple, etc.), but also all sorts of decisions around how you actually serve that LLM. There are so many consequential decisions baked into how you write the kernels, pick the inference engine, decide to do batching, or do quantization.
What we aim to do with intelligence per watt and intelligence per joule is try to understand the full global picture of how all of these decisions compound together and how LLMs and hardware have been shifting over time, particularly when it comes to local AI.
Avanika: Fundamentally, what is intelligence per watt? It’s a measure for intelligence efficiency. In previous eras of computing, the thing that enabled compute to become ubiquitous was this drive towards efficiency. The fact that you could have capable and performant compute in smaller form factors is what enabled people to actually have computers everywhere and in every device.
If we believe that in the future intelligence is going to be everywhere, that is the metric that we should care about, and that’s why we coined intelligence per watt.
Jaya: Maybe to go through your two other papers pretty quickly: unlike a lot of research papers, yours actually have a lot of implications for right now. Tell us a little bit about OpenJarvis and Minions, and then we’ll dive into the real-world stuff.
Jon: With OpenJarvis, we wanted to examine and operationalize a lot of our insights from the intelligence per watt work. One of the key takeaways is that local AI running on local hardware is actually more performant than people would expect. There are a lot of queries and workloads that don’t necessarily need frontier-level intelligence in the cloud; they can actually just run on-device with open-source LLMs. We wanted to create a software stack for actually putting that in people’s hands.
A lot of people are excited about local AI, but it was harder for them to boot up their Mac or Windows PC and actually get to work with it. So we built OpenJarvis to put as much of the full LLM stack on people's devices.
There is a lot of other great work with Open AI agents or Hermes Agent that build LLM agents in the cloud, but they rely on the cloud for intelligence or memory, where you're constantly paying a rental fee and sending all your private info. With OpenJarvis, we tried to flip the switch and ask, “How many of these day-to-day workloads can you run on-device for free with complete ownership?” That’s what led to the work.
Avanika: Minions was similar and dissimilar in many ways. Can we use a small open-source model operating on a cheaper device, working in coordination with a cloud model, to finish a task in the most cost-effective way possible?
Now, these hybrid inference systems—we coined the term a year and a half ago—are everywhere. Every single company is doing this sort of thing, so we're very happy to see that work being widely used, and we also pulled it into the OpenJarvis framework. We like to say that OpenJarvis is trying to make inference “local by default, hybrid by design,” and that is one of the core ethos that we take with us today. That’s the way inference is going to be moving forward.
Jaya: And NVIDIA has cited your IPW framing in their marketing materials, right?
Jon: Yeah, quite a few companies at this point! NVIDIA has talked about intelligence per watt, performance per watt, and performance per joule. Different chip companies like SambaNova and Apple are talking about how they maximize intelligence per joule for their systems. It’s exciting to see engineers thinking about this as they build the next generation of chips. It's really flattering.
Avanika: And it’s not just at the chip level; it's also at the data center level. Google Cloud just released a report talking about the main tentpoles for their vision going forward, and intelligence efficiency and energy efficiency were P0. They also talk about hybrid execution and how to use on-prem deployments and big cloud resources in coordination. So yeah, it’s good to see all our ideas in the real world.
Jaya: Today, the overwhelming majority of queries are processed by frontier models in centralized cloud infrastructure, and a lot of capital buildout and pricing models assume that pattern continues. The combination of a lot of your work argues that a share of that demand could be redistributed. If you’re right, what assumptions break about the markets, buildouts, and pricing models?
Jon: First, on the demand side, people presuppose that for all workloads, we’ll always want frontier-level intelligence—like GPT-4, Claude, etc. I think that’s fundamentally incorrect. There is a huge amount of workloads for which you don’t need frontier-level intelligence to solve. It’s actually a significant majority that grows over time as the frontier gets pushed out. You can solve a lot of those handily with open-source LLMs and local hardware.
At the same time, there’s going to be a lot of different silicon coming out. NVIDIA has had a dominant position in building the state-of-the-art training stack, but there will be a lot of different silicon from NVIDIA, AMD, Apple, SambaNova, and others that optimize for different metrics—whether that's intelligence per watt, intelligence per joule, or cost per token.
Avanika: Everyone is quick to say, “The winner on the model layer is going to be X, and the winner on the hardware stack is going to be Y,” but markets are becoming more fragmented. Silicon is becoming multi-silicon; models and agents are becoming multi-model. You have this fracturing and redistribution of power and control. It’s all very exciting.
Get insights directly to your inbox.
Jaya: What about the privacy angle?
Jon: Privacy is an increasingly important problem, especially for enterprises. There’s going to be huge value placed on IP and expertise. It’s funny that today people are willing to send all their traces, logs, and decisions to black-box LLMs without wondering who is using, training on, or reselling that data. You’re ceding your entire decision-making process away.
Jaya: What has to be true technically for local AI to take off?
Jon: On the technical side, current trends just need to continue. Open-source LLMs need to keep getting better, more compact, and more efficient in delivering intelligence with fewer parameters and less energy. Accelerators also need to keep getting better. The intelligence you can put on your laptop or desk needs to keep improving, and we don't see that slowing down anytime soon. The intelligence needed for many tasks already exists today.
Jaya: Transitioning to a hot topic born over at Meta: “tokenmaxxing.” Is this the philosophical opposite of intelligence per watt, and how does it connect to your research?
Avanika: We love tokenmaxxing! We want to make intelligence so cheap to meter that people can run intelligence 24/7 on every device they have. That’s our goal. We want everybody to have superintelligence in their pockets and on their desks.
Jon: At the same time, the path towards that efficiency isn’t linear. People think the curve naturally bends towards efficiency, but it doesn’t. Historically, whether through Dennard scaling or Moore’s Law, there were key technical insights along the way. It took brilliant engineers and scientists to make transistors so cheap that we now have transistors at the scale of single atoms.
We think the same thing will happen for computing efficiency in intelligence. There will be several key insights along the way, and we hope to help drive computing to become ubiquitous, decentralized, and distributed worldwide—not just owned by Silicon Valley or a discrete set of insiders.
Jaya: If you were Dario Amodei or Sam Altman right now, what would you be most scared about reading your research?
Jon: That’s easy: they'd be scared that open source is getting too good. They were expecting recursive self-improvement or a runaway game where, once they reached a certain degree of intelligence, the loop would train and accelerate away. They’re worried that hasn’t happened or won't happen. That’s spurring concerns around open source, model customization, fine-tuning, and post-training. Margins are good today, but competitors will threaten that pie.
Avanika: Honestly, I would be afraid their revenue is going to go to zero! There are two trends: open models reaching frontier capability, and the compression of compute. GPT-4 quality is now accessible in an 8-billion-parameter model. Everything they sell is based on size, capability, and centralization. Once that becomes distributed and accessible outside of them, they have no moat.
Large enterprises are realizing that instead of spending $1M on Claude, they only need to spend $20,000 and shift the rest to something cheaper that they own and control.
Jon: Especially because switching costs are so low. You can just switch the API endpoint and pay an entirely different company. You don’t need long-term contract lock-in like traditional cloud. Their revenue is much less recurring than they anticipated.
Avanika: Once momentum builds around hybrid architectures, everyone follows. Every company from DoorDash to Harvey is doing this. Once that momentum is there, the frontier labs have lost the game.
Jaya: What lock-in do they have right now?
Jon: Right now, it’s too easy to pick up your info and go elsewhere. They bet on memory and personalization, but I haven’t felt it yet. Their only real moat right now is brand.
Avanika: If a team isn’t AI-native, going to them is the safe bet—a white-glove service where person X gets promoted for bringing AI to the organization. But for any organization with smart AI people in the room, they really don’t have a moat.
Jaya: Alex Karp at Palantir called the token-rental model “irresponsibly oversold,” saying 90% of queries don’t need frontier capability. He argues model, app, and compute need to be aligned. Are you making the same bet at the personal level that Palantir is making at the enterprise level?
Jon: Completely. The model isn’t the product; it needs to sit around other components to generate value long-term. With OpenJarvis, you can’t just drop an LLM in a harness; you need to align the model, harness, hardware, memory, and tools to generate real value.
Jaya: What advice would you give to an enterprise CIO today?
Jon: Don't undervalue your expertise! It’s mind-boggling that companies pay someone to take their internal documents, train on them, and sell them back. It's a bizarre inversion of incentives and leverage.
Avanika: Start building in-house capabilities as soon as possible. Identify “exploit” versus “explore” workflows. Anything in the exploit phase should shift toward open-weight models. Point your APIs to Fireworks or Together to see how much accuracy you recover, then transition to owning your compute and running inference yourself. Your knowledge is your brand—guard it with your life.
Jon: Most enterprise tasks don’t require frontier intelligence. If you served a 30B-parameter open model alongside a top frontier model, most people couldn’t tell the difference for everyday tasks.
Jaya: What’s a view you hold that most people in AI would disagree with?
Jon: Scaling laws won’t necessarily break, but they won't be enough to get where people want to go under the current LLM paradigm.
Jaya: If you woke up as Elon Musk tomorrow morning, what’s the first thing you’d do?
Jon: Strap a GPU to every Tesla car and charger to build the biggest distributed cloud in the world. You could take older GPUs, load the largest parameter model possible, and instantly gain gigawatts of inference compute.
Avanika: The US won’t fail in the AI race due to lack of technical expertise; it will fail if backing AI becomes politically unsavvy. Elon is uniquely positioned to make the broader public the chief beneficiaries of AI, and we need public policy that makes the country pro-AI.
Jaya: If you could have 1,000 new teams working on something, what would it be?
Jon: Intelligence efficiency! Driving intelligence per watt and per joule requires re-examining the full stack: training, deployment, kernels, inference engines, hardware, and firmware. We need several orders of magnitude improvement to make tokenmaxxing a reality.
Jaya: What is your piece of advice for PhD students?
Jon: There is still so much meaningful work to be done outside conventional scaling laws—AI for biology, physics, and chemistry is very underappreciated. Take unconventional, thought-out bets.
Avanika: My tagline is: “Think different, bet on yourself.” A PhD is the best way to do that. Keep building what you believe should exist.
Jaya: Independent thinking is scarce! Thank you guys so much for doing this.
Avanika and Jon: Thanks for having us!