Featured

What I’ve Learned About LLMs in Healthcare (so far)

It has been a breathless time in technology since the GPT-3 moment, and I’m not sure I have experienced greater discordance between the hype and reality than right now, at least as it relates to healthcare. To be sure, I have caught myself agape in awe at what LLMs seem capable of, but in the last year, it has become ever more clear to me what the limitations are today and how far away we are from all “white collar jobs” in healthcare going away.

Microsoft had an impressive announcement last week with The Path to Medical Super-Intelligence with its claim that its AI Diagnostic Orchestrator (MAI-DxO) correctly diagnosed up to 85% of NEJM case proceedings, a rate more than four times higher than a group of experienced physicians (~20% accuracy). While this is an interesting headline result, I think we are still far from “medical superintelligence”, and in some ways, we underestimate what human intelligence is good at it, particularly in the healthcare context.

Beyond potential issues of benchmark contamination, the data for Microsoft’s evaluation of its orchestrator agent is based on NEJM case records that are highly curated, teaching narrative summaries. Compare that to a real hospital chart: a decade of encounters scattered across medication tables, flowsheets, radiology blobs, scanned faxes, and free-text notes written in three different EHR versions. In that environment, LLMs lose track of units, invent past medical history, and offer confident plans that collapse under audit. Two Epic pilot reports—one from Children’s Hospital of Philadelphia, the other from a hospital in Belgium—show precisely this gap and shortcoming with LLMs. Both projects needed dozens of bespoke data pipelines just to assemble a usable prompt, and both catalogued hallucinations whenever a single field went missing.

The disparity is unavoidable: artificial general intelligence measured on sanitized inputs is not yet proof of medical superintelligence. The missing ingredient is not reasoning power; it is reliable, coherent context.

Messy data still beats massive models in healthcare

Transformer models process text through a fixed-size context window, and they allocate relevance by self-attention—the internal mechanism that decides which tokens to “look at” when generating the next token. GPT-3 gave us roughly two thousand tokens; GPT-4 stretches to thirty-two thousand; experimental systems boast six-figure limits. That may sound limitless, yet the engineering reality is stark: packing an entire EHR extract or a hundred-page protocol into a prompt does not guarantee an accurate answer. Empirical work—including Nelson Liu’s “lost-in-the-middle” study—shows that as the window expands, the model’s self-attention diffuses. With every additional token, attention weight is spread thinner, positional encodings drift, and the transformer’s gradient now competes with a larger field of irrelevant noise. Beyond a certain length the network begins to privilege recency and surface phrase salience, systematically overlooking material introduced many thousands of tokens earlier.

In practical terms, that means a sodium of 128 mmol/L taken yesterday and a potassium of 2.9 mmol/L drawn later that same shift can coexist in the prompt, yet the model cites only the sodium while pronouncing electrolytes ‘normal. It is not malicious; its attention budget is already diluted across thousands of tokens, leaving too little weight to align those two sparsely related facts. The same dilution bleeds into coherence: an LLM generates output one token at a time, with no true long-term state beyond the prompt it was handed. As the conversation or document grows, internal history becomes approximate. Contradictions creep in, and the model can lose track of its own earlier statements.

Starved of a decisive piece of context—or overwhelmed by too much—today’s models do what they are trained to do: they fill gaps with plausible sequences learned from Internet-scale data. Hallucination is therefore not an anomaly but a statistical default in the face of ambiguity. When that ambiguity is clinical, the stakes escalate. Fabricating an ICD-10 code or mis-assigning a trial-eligibility criterion isn’t a grammar mistake; it propagates downstream into safety events and protocol deviations.

Even state-of-the-art models fall short on domain depth. Unless they are tuned on biomedical corpora, they handle passages like “EGFR < 30 mL/min/1.73 m² at baseline” as opaque jargon, not as a hard stop for nephrotoxic therapy. Clinicians rely on long-tail vocabulary, nested negations, and implicit timelines (“no steroid in the last six weeks”) that a general-purpose language model never learned to weight correctly. When the vocabulary set is larger than the context window can hold—think ICD-10 or SNOMED lists—developers resort to partial look-ups, which in turn bias the generation toward whichever subset made it into the prompt.

Finally, there is the optimization bias introduced by reinforcement learning from human feedback. Models rewarded for sounding confident eventually prioritize tone that sounds authoritative even when confidence should be low. In an overloaded prompt with uneven coverage, the safest behavior would be to ask for clarification. The objective function, however, nudges the network to deliver a fluent answer, even if that means guessing. In production logs from the CHOP pilot you can watch the pattern: the system misreads a missing LOINC code as “value unknown” and still generates a therapeutic recommendation that passes a surface plausibility check until a human spots the inconsistency.

All of these shortcomings collide with healthcare’s data realities. An encounter-centric EHR traps labs in one schema and historical notes in another; PDFs of external reports bypass structured capture entirely. Latency pressures push architects toward caching, so the LLM often reasons on yesterday’s snapshot while the patient’s creatinine is climbing. Strict output schemas such as FHIR or USDM leave zero room for approximation, magnifying any upstream omission. The outcome is predictable: transformer scale alone cannot rescue performance when the context is fragmented, stale, or under-specified. Before “superintelligent” agents can be trusted, the raw inputs have to be re-engineered into something the model can actually parse—and refuse when it cannot.

Context engineering is the job in healthcare

Andrej Karpathy really nailed it here:

Context engineering answers one question: How do we guarantee the model sees exactly the data it needs, in a form it can digest, at the moment it’s asked to reason?

In healthcare, I believe that context engineering will require three moves to align the data to ever-more sophisticated models.

First, selective retrieval. We replace “dump the chart” with a targeted query layer. A lipid-panel request surfaces only the last three LDL, HDL, total-cholesterol observations—each with value, unit, reference range, and draw time. CHOP’s QA logs showed a near-50 percent drop in hallucinated values the moment they switched from bulk export to this precision pull.

Second, hierarchical summarisation. Small, domain-tuned models condense labs, meds, vitals, imaging, and unstructured notes into crisp abstracts. The large model reasons over those digests, not 50,000 raw tokens. Token budgets shrink, latency falls, and Liu’s “lost-in-the-middle” failure goes quiet because the middle has been compressed away.

Third, schema-aware validation—and enforced humility. Every JSON bundle travels through the same validator a human would run. Malformed output fails fast. Missing context triggers an explicit refusal.

AI agents in healthcare up the stakes for context

The next generation of clinical applications will not be chatbots that answer a single prompt and hand control back to a human. They are agents—autonomous processes that chain together retrieval, reasoning, and structured actions. A typical pipeline begins by gathering data from the EHR, continues by invoking clinical rules or statistical models, and ends by writing back orders, tasks, or alerts. Every link in that chain inherits the assumptions of the link before it, so any gap or distortion in the initial context is propagated—often magnified—through every downstream step.

Consider what must be true before an agent can issue something as simple as an early-warning alert:

  • All source data required by the scoring algorithm—vital signs, laboratory values, nursing assessments—has to be present, typed, and time-stamped. Missing a single valueQuantity.unit or ingesting duplicate observations with mismatched timestamps silently corrupts the score.
  • The retrieval layer must reconcile competing records. EHRs often contain overlapping vitals from bedside monitors and manual entry; the agent needs deterministic fusion logic to decide which reading is authoritative, otherwise it optimizes on the wrong baseline.
  • Every intermediate calculation must preserve provenance. If the agent writes a structured CommunicationRequest back to the chart, each field should carry a pointer to its source FHIR resource, so a clinician can audit the derivation path in one click.
  • Freshness guarantees matter as much as completeness. The agent must either block on new data that is still in transit (for example, a troponin that posts every sixty minutes) or explicitly tag the alert with a “last-updated” horizon. A stale snapshot that looks authoritative is more dangerous than no alert at all.

When those contracts are enforced, the agent behaves like a cautious junior resident: it refuses to proceed when context is incomplete, cites its sources, and surfaces uncertainty in plain text. When any layer is skipped—when retrieval is lossy, fusion is heuristic, or validation is lenient—the agent becomes an automated error amplifier. The resulting output can be fluent, neatly formatted, even schema-valid, yet still wrong in a way that only reveals itself once it has touched scheduling queues, nursing workflows, or medication orders.

This sensitivity to upstream fidelity is why context engineering is not a peripheral optimization but the gating factor for autonomous triage, care-gap closure, protocol digitization, and every other agentic use case to come. Retrieval contracts, freshness SLAs, schema-aware decoders, provenance tags, and calibrated uncertainty heads are the software equivalents of sterile technique; without them, scaling the “intelligence” layer merely accelerates the rate at which bad context turns into bad decisions.

Humans still have a lot to teach machines

While AI can be brilliant for some use cases, in healthcare so far, large-language models still seem like brilliant interns: tireless, fluent, occasionally dazzling—and constitutionally incapable of running the project alone. A clinician opens a chart and, in seconds, spots that an ostensibly “normal” electrolyte panel hides a potassium of 2.8 mmol/L. A protocol digitizer reviewing a 100-page oncology protocol instinctively flags that the run-in period must precede randomization, even though the document buries the detail in an appendix.

These behaviors look mundane until you watch a vanilla transformer miss every one of them. Current models do not plan hierarchically, do not wield external tools unless you bolt them on, and do not admit confusion; they generate tokens until the temperature hits zero. Until we see another major AI innovation like the transformer models themselves, healthcare needs a viable scaffolding that lets an agentic pipeline inherit the basic safety reflexes clinicians exercise every day.

That is not a defeatist conclusion; it is a roadmap. Give the model pipelines that keep the record complete, current, traceable, schema-tight, and honest about uncertainty, and its raw reasoning becomes both spectacular and safe. Skip those safeguards and even a 100-k-token window will still hallucinate a drug dose out of thin air. When those infrastructures become first-class, “superintelligence” will finally have something solid to stand on.

Your Gut is a Value Function

Tim Cook once said the most important lesson he learned at Apple was to listen to his gut. That stuck with me, mostly because I had to learn it the hard way myself.

Early in my career, I thought “trust your gut” was code for “I don’t have the data.” But over time, I learned to listen to my gut, just like Cook describes doing at pivotal points of his career. The decisions I have regretted most were ones where I didn’t give voice to an uneasy feeling. And the best decisions have often been intuitive in the face of ambiguity and imperfect information.

It’s easy to dismiss this gut feeling as mystical, but now I realize it’s computational.

Your gut is a compressed summary of long‑horizon experiences that your conscious mind can’t read yet. In machine‑learning terms, it behaves a lot like a value function, the internal machinery that estimates how good or bad a situation is and where it’s likely to lead.

That idea turns “trust your gut” from self‑help cliché into a serious claim about how intelligence works, a concept that clicked for me while listening to Ilya Sutskever talk about a patient who lost his emotions.

The man who couldn’t decide

In a fantastic recent interview, Ilya Sutskever recounts a famous case from neurologist Antonio Damasio. A patient suffered brain damage to the area processing emotion. Post-surgery, his IQ was normal. His memory was perfect. He could list the pros and cons of any option.

But his life fell apart.

He spent twenty minutes deciding which pen to use. He couldn’t prioritize. Without the machinery to feel the difference between “good” and “bad,” he was trapped in an infinite loop of reasoning.

Damasio’s conclusion was that we don’t use logic to value things. We use “somatic markers”—emotional tags attached to past experiences. When a similar situation arises, your body replays a trace of the feeling: regret, relief, shame. That physical response is a shortcut. It solves the stopping problem.

As Sutskever suggests: emotions are the value function. Remove them, and you don’t get a super-rational agent. You get a system that can’t land the plane.

Why AI can’t fake this

This is exactly where the gap between human judgment and current AI lies.

Models read oceans of text to learn patterns. Then we use Reinforcement Learning to tell them what “good” looks like. But the rewards are short-term and dense: Did you solve the puzzle? Did the user like the summary?

That is a completely different animal from the human loop.

Your emotional value function is trained on the messy, long-term reality of your actual life. It integrates feedback that arrives years later—as a broken relationship, a career derailment, or the quiet satisfaction of doing the right thing. It associates tones of voice, deal structures, and clinical smells with outcomes that haven’t happened yet.

It’s not infallible—it carries bias and trauma—but it is the only model you have that has been trained on reality at scale.

Trying to approximate that with current AI training methods is like trying to learn “good parenting” from a dataset of multiple-choice quizzes. We give models crude rules: don’t be toxic, be helpful. That’s useful. That’s useful. But it’s nowhere near a value system that understands that a decision can be technically correct and still be completely wrong.

The intelligence of the gut

This matters most in domains with slow feedback and high stakes—strategy, medicine, policy.

AI is already a powerful tool for reasoning. It can out-read and out-simulate us, dig through mountains of data, and catch patterns you’d miss. But we shouldn’t confuse reasoning with judgment.

The pattern-recognition parts of our jobs are being automated. The piece that remains scarce is the long-horizon, emotionally anchored sense of what is actually worth doing.

“Trust your gut” isn’t an abandonment of reason; it’s a reminder that there is a layer of intelligence we haven’t yet reproduced in silicon. Your emotional life is a value function continuously trained by reality over years, while today’s AI systems still optimize short‑term, narrow proxies on curated benchmarks. For the decisions that actually shape a life or an organization, that quiet hum in your chest is not something we’re going to outsource anytime soon. It is the distinction between calculating what we can do, and knowing what we should do.

Strategy in the Age of Infinite Slop

“AI is going to replace McKinsey.”

It’s a popular dunk on AI Twitter. The logic is seductive: if a model can generate a Porter’s Five Forces diagram and a perfectly serviceable deck in seconds, why pay millions for a team of human analysts to take six weeks?

I spent more than ten years at McKinsey working on the exact problems assumed to be next on the chopping block: the ultimate open-ended questions of “where to play” and “how to win.” But looking at those problems through the lens of Andrej Karpathy’s concept of “verifiability,” I’ve come to the opposite conclusion: the closer you get to real strategy, the harder it is for AI to replace it.

The closer you get to real strategy, the less it resembles the tasks AI is good at.

Karpathy’s “Software 2.0” thesis is simple: AI mastery relies on a loop. Attempt a task, get a score, reset, repeat. If you can verify the outcome cheaply—Did the code compile? Did the math hold? Did you win the game?—the model can practice its way to superhuman performance.

This explains why AI is crushing coding and math. These are “high verifiability” domains. The reward signal is crisp, binary, and instant.

Corporate strategy lives at the opposite extreme.

As a strategy consultant, when you advise a client to enter China or divest a legacy unit or to sell the company, you don’t get a clean error message immediately. You get a noisy stream of signals over five years. A competitor takes share with a new product. The macro environment shifts. A new CEO gets hired.

You cannot reset the world. You cannot run the A/B test. There is only one realized future and a graveyard of unknowable counterfactuals. And from the perspective of an AI training loop, that means the “reward signal” for any one decision is sparse, delayed, and hopelessly entangled with everything else. The pattern recognition for “good strategy” gets developed over years of many studies and outcomes.

So, is AI useless in the boardroom?

Absolutely not. While AI cannot verify a strategy, it is unparalleled at generating the raw material for one.

Strategy is fundamentally a game of connecting dots across a massive, messy board. It requires looking at a mountain of proprietary data, market reports, and competitive intelligence, and spotting the pattern that others miss.

This is where modern LLMs shine. They act as a force multiplier for reasoning by analogy. A partner can ask a model to look at a B2B logistics problem and apply the “physics” of a consumer marketplace, or to search for historical parallels for the AI infrastructure buildout in 19th-century rail monopolies.

In this phase, the AI is not an oracle; it is a Disciplined Hallucinator. It provides the expanse. It widens the aperture from three conventional options to twenty wild ones. It does the “grinder” work of synthesis that used to burn out armies of business analysts. A lot of those options will be wrong, implausible, or “slop” in the eyes of critics, but in strategy, exploring wrong futures is often how you discover the few worth betting on.

But options are not decisions.

There is a distinct limit to how far this can go. As AI researchers like Yann LeCun argue, current LLMs are not “World Models.” They predict the next token in a sequence; they do not understand the underlying causal physics of reality. They cannot reason about cause and effect in a chaotic environment because they have no internal representation of how the world actually works.

They can simulate the text of a strategy, but they cannot simulate the reality of its execution.

This means the “Silicon Partner” isn’t arriving anytime soon. Until AI creates a true internal model of the world—one that understands human psychology, political friction, and temporal consequences—it remains a statistical engine, not a strategic one in the strong sense.

The Shift: From Processing to Judgment

As AI automates the verifiable layer of intelligence—the analysis, the synthesis, the slide-making—the value of the remaining bottleneck skyrockets.

That bottleneck is Judgment.

Judgment is the ability to look at the AI’s twenty generated options and intuitively know which three will survive contact with reality. It is the ability to stare down an irreversible decision where the “right” answer is mathematically unknowable—and act anyway.

We aren’t paying consultants to process information anymore. We are paying them to use these new instruments to hallucinate a better future and to have the courage to speak truth to power and own the risk of being wrong.

The New Computer in the Clinic

Andrej Karpathy describes the current moment as the rise of a new computing paradigm that he calls Software 3.0. as large language models emerge not just as clever chatbots but as a “new kind of computer” (“LLM OS”). In this model, the LLM is the processor, its context window is the RAM, and a suite of integrated tools are the peripherals. We program this new machine not with rigid code, but with intent, expressed in plain language. This vision is more than a technical curiosity; it is a glimpse into a future where our systems don’t just execute commands, but understand intent.

Healthcare is the perfect test bed for this paradigm. For decades, the story of modern medicine has been a paradox: we are drowning in data, yet starved for wisdom. The most vital clinical information—the physician’s reasoning, the patient’s narrative, the subtle context that separates a routine symptom from a looming crisis—is often trapped in the dark matter of unstructured data. An estimated 80% of all health data lives in notes, discharge summaries, pathology reports, and patient messages. This is the data that tells us not just what happened, but why.

For years, this narrative goldmine has been largely inaccessible to computers. The only way to extract its value was through the slow, expensive, and error-prone process of manual chart review, or a “scavenger hunt” through the patient chart. What changes now is that the new computer can finally read the story. An LLM can parse temporality, nuance, and jargon, turning long notes into concise, cited summaries and spotting patterns across documents no human could assemble in time.

But this reveals the central conflict of digital medicine. The “point-and-click” paradigm of the EHR, while a primary driver of burnout, wasn’t built merely for billing. It was a necessary, high-friction compromise. Clinical safety, quality reporting, and large-scale research depend on the deterministic, computable, and unambiguous nature of structured data. You need a discrete lab value to fire a kidney function alert. You need a specific ICD-10 code to find a patient for a clinical trial. The EHR forced clinicians to choose: either practice the art of medicine in the free-text narrative (which the computer couldn’t read) or serve as a data entry clerk for the science of medicine in the structured fields. Often, the choice has been the latter, which has contributed to massive burnout among physicians. This false dichotomy has defined the limits of healthcare IT for a generation.

The LLM, by itself, cannot solve this. As Karpathy points out, this new “computer” is deeply flawed. Its processor has a “jagged” intelligence profile—simultaneously “superhuman” at synthesis and “subhuman” at simple, deterministic tasks. More critically, it is probabilistic and prone to hallucination, making it unfit to operate unguarded in a high-stakes clinical environment. This is why we need what Karpathy calls an “LLM Operating System”. This OS is the architectural “scaffolding” designed to manage the flawed, probabilistic processor. It is a cognitive layer that wraps the LLM “brain” in a robust set of policy guardrails, connecting it to a library of secure, deterministic “peripheral” tools. And this new computer is fully under the control of the clinician who “programs” it in plain language.

This new architecture is what finally resolves the art/science conflict. It allows the clinician to return to their natural state: telling the patient’s story.

To see this in action, imagine the system reading a physician’s note: “Patient seems anxious about starting insulin therapy and mentioned difficulty with affording supplies.” The LLM “brain” reads this unstructured intent. The OS “policy layer” then takes over, translating this probabilistic insight into deterministic actions. It uses its “peripherals”—its secure APIs—to execute a series of discrete tasks: it queues a nursing call for insulin education, sends a referral to a social worker, and suggests adding a structured ‘Z-code’ for financial insecurity to the patient’s problem list. The art of the narrative is now seamlessly converted into the computable, structured science needed for billing, quality metrics, and future decision support.

This hybrid architecture—a probabilistic mind guiding a deterministic body—is the key. It bridges the gap between the LLM’s reasoning and the high-stakes world of clinical action. It requires a healthcare-native data platform to feed the LLM reliable context and a robust system of action layer to ensure its outputs are safe. This design directly addresses what Karpathy calls the “spectrum of autonomy.” Rather than an all-or-nothing “agent,” the OS allows for a tunable “autonomy slider.” In a “co-pilot” setting, the OS can be set to only summarize, draft, and suggest, with a human clinician required for all approvals. In a more autonomous “agent” setting, the OS could be permitted to independently handle low-risk, predefined tasks, like queuing a routine follow-up.

The journey is just beginning in healthcare, and my guess is that we will see different starting points for this across the ecosystem. But the path forward is illuminated by a clear thesis: the “new computer” for healthcare changes the very unit of clinical work. We are moving from a paradigm of clicks and codes—where the human serves the machine—to one of intent and oversight. The clinician’s job is no longer data entry. It is to practice the art of medicine, state their intent, and supervise an intelligent system that, for the first time, can finally understand the story.

AI and the Prepared Mind: Engineering Luck in Drug Discovery

We are at a fascinating, paradoxical moment in the history of medicine. We stand in awe of a new AI-powered “Logic Engine” for drug discovery—a computational marvel like AlphaFold, which treats biology as an information system to be engineered. It promises a future of rational discovery. And yet, when we look at our most important medical breakthroughs, so many were not rationally designed. They were the result of messy, unpredictable, and entirely human processes: a happy accident, a surprising side effect, or a creative leap of intuition. This isn’t a story of one replacing the other. For me, it’s the story of how we build the bridge between them. The future, I believe, lies in marrying AI’s logic with this enduring human spark.

How has luck played out in drug discovery? With reference to some famous examples, I believe serendipity comes in three distinct flavors.

First, there is the Physical-World Accident. This is the classic tale of Alexander Fleming. He doesn’t hypothesize and then test; he returns from vacation to find a physical anomaly on a petri dish, a “moldy” halo where bacteria wouldn’t grow. The breakthrough was not the idea; it was his prepared mind recognizing the profound significance of a simple, physical event.

Second, there is the Clinical Data Anomaly. This is the story of Viagra. Researchers at Pfizer were not looking for an erectile dysfunction drug; they were testing a new angina medication. But in the clinical trial data, they spotted a consistent, statistically significant “side effect.” Their genius was not in the drug’s design, but in their ability to see that this “failure” was, in fact, the drug’s true purpose.

And third, there is the rarest and most powerful form: the Cross-Domain Synthesis. This is the almost-mythical origin of the GLP-1 drugs. In the 1980s, Dr. John Eng, an endocrinologist at the VA, was grappling with the dangerous, real-world clinical problem of hypoglycemia in his diabetic patients. His deep “embodied context” of this problem led his curiosity to a non-obvious place: the venom of the Gila monster. He made a creative, “analogical leap,” betting that a creature who could feast and then fast for months must have a powerful metabolic regulator. He was right, and this single, human-driven hypothesis proved the therapeutic principle that led to the multi-billion-dollar GLP-1 field, from Exenatide to Ozempic and Mounjaro.

Dr. Eng’s leap of intuition was not a brute-force data search; it was an act of wisdom. “Embodied context” is the sum of lived, physical, sensory, and experience-based intuition. This, to me, is the undigitized data we’re missing in all of the data being used to train AI: the “gut feeling” of a 30-year veteran clinician, the intuition born from seeing, touching, and feeling a problem.

This is not just a poetic concept. It is the data that isn’t in the database: the specific sound of a patient’s cough, the feel of a tumor’s texture, the non-verbal cues a patient gives, or the “gut feeling” that connects a skin rash to a GI symptom seen months prior—a non-obvious, low-signal pattern. An AI, no matter how powerful, is a disembodied logic system. Its “experience” is limited to the digital archive of human knowledge. It has read the map; it has not walked the territory.

Dr. Eng’s leap was not just data; it was purpose. He had witnessed the “litany of horrors” of his patients’ suffering. That context, which exists in no database, is what aimed his curiosity. It allowed him to connect three disparate domains: the clinical problem (hypoglycemia), the zoological trait (a lizard’s stability), and the mechanistic hunch (venom). An AI, lacking this embodied context, would have no reason to see this as anything but a low-probability statistical correlation.

Now, one could argue that this “embodied context” is just a polite word for human bias, the very thing a logic engine is designed to eliminate. This is not wrong; intuition is notoriously flawed. But this is precisely why the partnership is essential. The loop’s purpose is not to blindly trust human wisdom; it is to interrogate it. The human provides the testable, experience-based hypothesis; the AI and the lab provide the objective, high-throughput validation.

This reliance on rare, human-driven leaps is not a reliable strategy. It is slow and random, and it’s why, in my view, our industry has been trapped by the brutal economics of Eroom’s Law, which observes costs rising exponentially, driven by a catastrophic “valley of death” in clinical trials where the vast majority of drugs fail.

This is the problem the AI-powered “Logic Engine” was built to solve. It is a revolutionary solution to “Bad Chemistry.” By designing the perfect molecular “key” in silico, it ensures a drug is potent, specific, and far less likely to be toxic. But these perfect keys may still hit the Phase 2 wall. They are colliding with “Bad Biology.” A perfect key for the wrong lock is still a failure. Even today’s intelligently designed blockbusters, from Keytruda to Ozempic, owe their massive success to unexpected clinical findings—like breakthrough weight-loss or cardio-renal benefits—that were discovered serendipitously, long after the initial design.

I firmly believe a purely in silico model is not enough. A “digital twin” or simulation, trained only on our current, incomplete data, is merely a sophisticated mirror of our existing ignorance. It’s an echo chamber. A purely computational AI would have been blind to Fleming’s mold, dismissed Viagra’s side effect as noise, and never possessed the creative, context-driven curiosity to make Dr. Eng’s leap.

This is why we must complement the Logic Engine with another type of system: a data-fueled Serendipity Engine. Across the biotech ecosystem, many are actively building the components of this system: the high-fidelity data “brain,” the automated “body” of human-relevant lab models, and the “nervous system” feedback loop. But a truly integrated, closed-loop system is not yet a reality. Much work remains to connect these parts into a seamless whole. This system has three core components.

First, it needs a “brain.” This is the Multimodal Data Foundation. To find human targets, it must learn from human data, building a high-fidelity map of disease as it actually exists, integrating genomics, proteomics, longitudinal clinical records, and real-world outcomes.

But a brain is not enough. It needs a “body.” This is the Human-Relevant Experimental Layer. The AI’s in silico predictions must be tested not in a simulation, but on a fully automated, high-throughput lab with a biobank running on patient-derived organoids, complex cell models, and organ-chips that actually recapitulate human physiology, not a mouse’s.

Finally, we must build its “nervous system”: a Closed Feedback Loop. This loop connects the brain and the body. The AI designs, tests on the physical model, and the real-world experimental data is fed back to the AI. The system learns, updates its map of biology, and designs the next experiment.

If we build this perfect, closed-loop system, what happens when it becomes an AGI? What happens when it can formulate its own novel hypotheses? Is the human “prepared mind” finally and fully disintermediated?

The answer, I speculate, is no. At least, not for the foreseeable future. An AGI, no matter how powerful, seems to be the ultimate “what” and “how” engine. It can find correlations and model mechanisms with superhuman speed. But it may end up being a stranger to the “so what?” This AGI Scientist will not create a lack of work, but it could create a new, paralyzing problem: an overload of tens of millions of valid, novel, and testable hypotheses. Which one matters? Which of these is a fascinating biological quirk, and which one, if pursued, would change the lives of millions? The AGI, as a pure optimization system, may not inherently know the difference. It can rank hypotheses by p-value or predicted novelty, but it is unlikely to be able to rank them by true human significance.

Elon Musk offered what I thought was a powerful analogy about the role of human beings in an AGI world. He noted that our cortex (thinking/planning) constantly strives to satisfy our limbic system (instincts/feelings). Perhaps, he suggested, this is how it will be with AI. The AI is the ultimate, boundless cortex, but we are what gives it meaning. We are the “limbic system” it serves. This, I believe, offers a framework for how to think about human scientists in an AGI world for drug discovery. And this is where human wisdom and “embodied context” become the most valuable commodity in the system. This context isn’t just the clinician’s (like Dr. Eng). It is also the hard-won wisdom of the “drug hunter”, someone like Al Sandrock who undoubtedly developed an intuition for biological signal.

The future, then, may not be an AI scientist working alone. The human’s new, and perhaps final, role is to be the “prepared mind” that our Serendipity Engine is built to serve. This role, in effect, scales the intuition of the veteran drug hunter with the brute-force logic of the AI. Our job is not to find all the answers, but to stand at the dashboard of this vast Serendipity Engine, ask the right questions, and point to a single anomaly, saying:

“That one. The AI says it’s novel, but my experience tells me it’s relevant.”

In the end, the AI is the ultimate “what” and “how” engine. The human, I believe, will always be the “so what?”

How AI Gets Paid Is How It Scales

Almost ten years ago at Apple, we had a vision of how care delivery would evolve: face‑to‑face visits would not disappear, virtual visits would grow, and a new layer of machine-based care would rise underneath. Credit goes to Yoky Matsuoka for sketching this picture. Ten years later, I believe AI will materialize this vision because of its impact on the unit economics of healthcare. 

Labor is the scarcest input in healthcare and one of the largest line items of our national GDP. Administrative costs continue to skyrocket, and the supply of clinicians is fixed in the short run while the population ages and disease prevalence grows. This administrative overhead and demand-supply imbalance are why our healthcare costs continue to outpace GDP.  

AI agents will earn their keep when they create a labor dividend, either by removing administrative work that never should have required a person, or by letting each scarce clinician produce more, with higher accuracy and fewer repeats. Everything else is noise.

Administrative work is the first seam. Much of what consumes resources is coordination, documentation, eligibility, prior authorization, scheduling, intake, follow up, and revenue cycle clean up. Agents can sit in these flows and do them end to end or get them to 95 percent complete so a human can finish. When these agents are priced in ways where the ROI is attributable, I believe adoption will be rapid. If they replace a funded cost like scribes or outsourced call volume, the savings are visible.

Clinical work is the second seam. Scarcity here is about decisions, time with the patient, and safe coverage between visits. Assistive agents raise the ceiling on what one clinician can oversee. A nurse can manage a larger panel because the agent monitors, drafts outreach, and flags only real exceptions. A physician can close charts accurately in the room and move to the next patient without sacrificing documentation quality. The through line is that assistive AI either makes humans faster at producing billable outputs or more accurate at the same outputs so there is less rework and fewer denials.

Autonomy is the step change. When an agent can deliver a clinical result on its own and be reimbursed for that result, the marginal labor cost per unit is close to zero. The variable cost becomes compute, light supervision, and escalation on exceptions. That is why early autonomous services, from point‑of‑care eye screening to image‑derived cardiac analytics, changed adoption curves once payment was recognized. Now extend that logic to frontline “AI doctors.” A triage agent that safely routes patients to the right setting, a diagnostic agent that evaluates strep or UTI and issues a report under protocol, a software‑led monitoring agent that handles routine months and brings humans in only for outliers. If these services are priced and paid as services, capacity becomes elastic and access expands without hiring in lockstep. That is the labor dividend, not a marginal time savings but a different production function.

I’m on the fence about voice assistants, to be honest. Many vendors claim large productivity gains, and in some settings those minutes convert into more booked and kept visits. In others they do not and the surplus goes to well-deserved clinician well‑being. That is worthwhile, but it can also be fragile when budgets compress. Preference‑led adoption by clinicians can carry a launch (as it has in certain categories like surgical robots), but can it scale?  Durable scale usually needs either a cost it replaces, a revenue it raises, or a risk it reduces that a customer will underwrite. 

All of this runs headlong into how we pay for care. Our reimbursement codes and RVU tables were built to value human work. They measure minutes, effort, and complexity, then translate that into dollars. That logic breaks when software does the work. It also creates perverse outcomes. Remote patient monitoring is a cautionary tale that I learned firsthand about at Carbon Health. By tying payment to device days and documented staff minutes with a live call, the rules locked in labor and hardware costs. Software could streamline the work and improve compliance, but it could not be credited with reducing unit cost because the payment was pegged to inputs rather than results. We should not repeat that mistake with AI agents that can safely do more.

With AI agents coming to market over the next several years, we should liberalize billing away from human‑labor constructs and toward AI‑first pricing models that pay for outputs. When an agent is autonomous, I think we should treat it like a diagnostic or therapeutic service. FDA authorization should be the clinical bar for safety and effectiveness. Payment should then be set on value, not on displaced minutes. Value means the accuracy of the result, the change in decisions it causes, the access it creates where clinicians are scarce, and the credible substitution of something slower or more expensive. 

There is a natural end state where these payment models get more straightforward. I believe these AI agents will ultimately thrive in global value‑based models. Agents that keep panels healthy, surface risk early, and route patients to the moments that matter will be valuable as they demonstrably lower cost and improve outcomes. Autonomy will be rewarded because it is paid for the result, not the minutes. Assistive will thrive when it helps providers deliver those results with speed and precision. 

Much of the public debate fixates on AI taking jobs. In healthcare it should be a tale of two cities. We need AI to erase low‑value overhead, eligibility chases, prior auth ping pong, documentation drudgery, so the scarce time of nurses, physicians, and pharmacists stretches further. We need to augment the people we cannot hire fast enough. Whether that future arrives quickly will be decided by how we choose to pay for it.

When AI Meets Aggregation Theory in Healthcare

Epic calls itself a platform. And with the show of force at UGM last week, that’s exactly how the company now describes itself: inviting vendors to “network with others working on the Epic platform,” marketing a “cloud‑powered platform” for healthcare intelligence, and selling a “Payer Platform” to connect plans and providers. Even customer stories celebrate moving to “a single Epic platform.”

But is Epic really a platform? The TL/DR is no.

Ben Thompson from Stratechery uses Bill Gates’s test to define a platform:

A platform is when the economic value of everybody that uses it exceeds the value of the company that creates it. Then it’s a platform

In Thompson’s framing, platforms facilitate third-party relationships and externalize network effects. During my time at Apple, the role of products as platforms–enabling developers to build their own experiences–was never lost on anyone. Apple’s success with the App Store wasn’t just about building great devices, it was about cultivating a marketplace where developers could thrive. To me, this is what it looks like to clear the Gates line.

In contrast, while Epic has captured significant value as the dominant vertical system of record, it does not pass the Bill Gates test for a platform, at least if “outside ecosystem” means independent developers and vendors. If anything, several UGM highlights overlapped with startup offerings, reinforcing Epic’s suite-first posture.

Beyond platforms, Thompson describes aggregators as internet-scale winners that have three concurrent properties: 1) a direct relationship with end users; 2) zero or near‑zero marginal cost to serve the next user because the product and distribution are digital; and 3) demand‑driven multi‑sided networks where growing consumer attention lowers future acquisition costs and compels suppliers to meet the aggregator’s rules.

Healthcare has lacked the internet physics that make either archetype inevitable. Patients rarely choose the software, employers and payers do. Much of care is physical and local, so marginal cost does not collapse at the point of service. Data has historically been locked behind site‑specific builds and business rules.

The policy landscape is shifting in a way that could finally make internet-style economics possible in healthcare. The national data-sharing network, TEFCA went live in late 2023 with the first Qualified Health Information Networks designated, including Epic’s own Nexus. The next milestone matters more for consumers: Individual Access Services (IAS). IAS creates a standardized, enforceable way for people to pull their health records through apps of their choice across participating networks, not just within a single portal. That means a person could authorize a service like ChatGPT, Amazon, or Apple Health to fetch their data across systems. Layer that onto ONC’s new transparency rules for AI and the White House’s push for clear governance, and the long-standing frictions that protected incumbents begin to fall away. Policy doesn’t create consumer demand by itself, but it clears the path. With IAS on the horizon, the conditions could be in place for true platforms to form on top of the data, and for the first genuine aggregators in healthcare to emerge.

Viewed through Thompson’s tests, Epic is neither a Thompson‑style aggregator nor a Gates‑line platform. Epic sells to enterprises, implementations take quarters and years, and its ecosystem is curated to reinforce the suite rather than to externalize network effects. Even its most aggregator‑looking asset, Cosmos, aggregates de‑identified data inside the Epic community to strengthen Epic’s own products, not to intermediate an open, multi‑sided market. UGM reinforced that direction with native AI charting on the way, an expanded AI slate, and a push to embed intelligence deeper into Epic’s own workflows. These are rational choices for reliability, liability, and speed inside the walls. They are not the choices of a company trying to own consumer demand across suppliers.

AI is the first credible force that can bend healthcare toward aggregation because it directly addresses Thompson’s three conditions. A high‑quality AI assistant can own the user relationship across employers, plans, and providers, the marginal cost to serve the next interaction is close to zero once deployed, and the product improves with every conversation, which lowers acquisition costs in a compounding loop. If that assistant can read with permission on national rails, reason over longitudinal data, coordinate benefits, and route to appropriate suppliers, demand begins to concentrate at the assistant’s front door. Suppliers then modularize on the assistant’s terms because that is where users start. That is Aggregation Theory applied to triage, chronic condition management, and navigation. The habit is forming at the consumer edge. With millions of Americans using ChatGPT, the flywheel is no longer theoretical.

It is worth being explicit about the one candidate aggregator that already exists at internet scale. With mass-market reach and daily use, ChatGPT could plausibly become a demand controller in health once the IAS pathway standardizes consumer-authorized data flows across QHINs. The building blocks are there in a way they never were for personal health records a decade ago: IAS rules now spell out how an app verifies identity and pulls data on behalf of a consumer, QHINs are live and interconnected, Epic Nexus alone covers more than a thousand hospitals, and HTI-1 is codifying transparency for AI-mediated decision support. If a consumer agent like ChatGPT could fetch records under IAS, explain benefits and prices, assemble prior authorization packets, book care, and learn from outcomes to improve routing, it would check Thompson’s boxes as an aggregator: owning the user relationship, facing near-zero marginal costs per additional user, and compelling suppliers to meet its terms. But there are complicating factors. HIPAA and liability rules may require ChatGPT to operate under strict business associate agreements, consumer trust in an AI holding intimate health data is far from guaranteed, and regulators could constrain or slow the extent to which a general-purpose model is allowed to intermediate medical decisions. Even so, the policy rails make such a role technically feasible, and ChatGPT’s usage base gives it a head start if it can navigate those hurdles.

Demand‑side pressure makes this shift more likely rather than less. Employer medical cost trend is projected to remain elevated through 2026 after hitting the highest levels in more than a decade, and pharmacy trend is outpacing medical trend, driven in part by the consumer‑adjacent GLP‑1 category. KFF’s employer survey shows a two‑year, mid‑to‑high single digit premium rise with specific focus on GLP‑1 coverage policies, and multiple employer surveys now estimate that GLP‑1 drugs account for a high single digit to low double digit share of total claims, with a sizable minority of employers reporting more than fifteen percent. As more of that cost shifts to households through premiums and deductibles, consumers gravitate to services that compress time to care and make prices legible. Amazon is training Prime members to expect five‑dollar generics with RxPass and a low‑friction primary care membership via One Medical, and Hims & Hers has demonstrated that millions will subscribe to vertically packaged services, now including weight‑management programs built around GLP‑1s. These behaviors teach consumers to start outside the hospital portal. Coupled with a trusted AI, they are the ingredients for real demand control.

None of this diminishes Epic’s role. If anything, the rise of a consumer aggregator makes a reliable clinical system of record more valuable. The most likely outcome is layered. Epic remains the operating system for care delivery, increasingly infused with its own AI. A neutral services tier above the EHR transforms heterogeneous clinical and payer data into reusable primitives for builders. And at the consumer edge, one or two AI assistants earn the right to be the first stop, finally importing internet economics into the information-heavy, logistics-light parts of care. That is a more precise reading of Thompson’s theory: aggregators win by owning demand, not supply. Healthcare never allowed them to own demand, but interoperability and AI agents change that. With IAS about to make personal data portable, the shape of the winning aggregator starts to look less like a portal and more like a personal health record—an agent that follows the consumer, not the institution. Julie Yoo’s “Health 2.0 Redux” makes the case that many of these ideas are not new. What is new is that, for the first time, the rails and the models are real enough to let a PHR evolve into the aggregator that healthcare has been missing.

America’s Patchwork of Laws Could Be AI’s Biggest Barrier in Care

AI is learning medicine, and early state rules read as if regulators are regulating a risky human, not a new kind of software. That mindset could make sense in the first wave, but it might also freeze progress before we see what these agents can do. When we scaled operations at Carbon Health, the slowest parts were administrative and regulatory–months of licensure, credentialing, and payer enrollment that shifted at each state line. AI agents could inherit the same map, fifty versions of permissions and disclosures layered on top of consumer‑protection rules. Without a federal baseline, the most capable tools might be gated by local paperwork rather than clinical outcomes, and what should scale nationally could move at the pace of the slowest jurisdiction.

What I see in state action so far is a conservative template built from human analogies and fear of unsafe behavior. One pattern centers on clinical authority. Any workflow that could influence what care a patient receives might trigger rules that keep a licensed human in the loop. In California, SB 1120 requires licensed professionals to make final utilization review decisions, and proposals in places like Minnesota and Connecticut suggest the same direction. If you are building automated prior authorization or claims adjudication, this likely means human review gates, on-record human accountability, and adverse‑action notices. It could also mean the feature ships in some states and stays dark in others.

A second pattern treats language itself as medical practice. Under laws like California’s AB 3030, if AI generates a message that contains clinical information for a patient, it is regulated as though it were care delivery, not just copy. Unless a licensed clinician reviews the message before it goes out, the provider must disclose to the patient that it came from AI. That carve-out becomes a design constraint. Teams might keep a human reviewer in the loop for any message that could be interpreted as advice — not because the model is incapable, but because the risk of missing a required disclosure could outweigh the convenience of full automation. In practice, national products may need state-aware disclosure UX and a tamper-evident log showing exactly where a human accepted or amended AI-generated output.

A third pattern treats AI primarily as a consumer-protection risk rather than a medical tool. Colorado’s law is the clearest example: any system that is a “substantial factor” in a consequential healthcare decision is automatically classified as high risk. Read broadly, that could pull in far more than clinical judgment. Basic functions like triage routing, benefit eligibility recommendations, or even how an app decides which patients get faster service could all be considered “consequential.” The worry here is that this lens doesn’t just layer on to FDA oversight — it creates a parallel stack of obligations: impact assessments, formal risk programs, and state attorney general enforcement. For teams that thought FDA clearance would be the governing hurdle, this is a surprise second regime. If more states follow Colorado’s lead, we could see dozens of slightly different consumer-protection regimes, each demanding their own documentation, kill switches, and observability. That is not just regulatory friction — it could make it nearly impossible to ship national products that influence care access in any way.

Mental health could face the tightest constraints. Utah requires conspicuous disclosure that a user is engaging with AI rather than a licensed counselor and limits certain data uses. Illinois has barred AI systems from delivering therapeutic communications or making therapeutic decisions while permitting administrative support. If interpreted as drafted, “AI therapist” positioning might need to be turned off or re‑scoped in Illinois.

Taken together, these state patterns set the core product constraints for now, keep a human in the loop for determinations, label or obtain sign‑off for clinical communications, and treat any system that influences access as high risk unless proven otherwise.

Against that backdrop, the missed opportunity becomes clear if we keep regulating by analogy to a fallible human. Properly designed agents could be safer than average human performance because they do not fatigue, they do not skip checklists, they can run differential diagnoses consistently, cite evidence and show their work, auto‑escalate when confidence drops, and support audit after the fact. They might be more intelligent on specific tasks, like guideline‑concordant triage or adverse drug interaction checks, because they can keep every rule current. They could even be preferred by some patients who value privacy, speed, or a nonjudgmental tone. None of that is guaranteed, but the path to discover it should not be blocked by rules that assume software will behave like a reckless intern forever.

For builders, the practical reality today is uneven. In practice, this means three operating assumptions: human review on decisions; clinician sign‑off or labeling on clinical messages; and heightened scrutiny whenever your output affects access. The same agent might be acceptable if it drafts a clinician note, but not acceptable if it reroutes a patient around a clinic queue because that routing could be treated as a consequential decision. A diabetes coach that nudges adherence could require a disclosure banner in California unless a clinician signs off, and that banner might not be enough if the conversation drifts into therapy‑like territory in Illinois. A payer that wants automation could still need on‑record human reviewers in California, and might need to turn automation off if Minnesota’s approach advances. Clinicians will likely remain accountable to their boards for outcomes tied to AI they use, which suggests that a truly autonomous AI doctor does not fit into today’s licensing box and could collide with Corporate Practice of Medicine doctrines in many states.

We should adopt a federal framework that separates assistive from autonomous agents, and regulate each with the right tool. Assistive agents that help clinicians document, retrieve, summarize, or draft could live under a national safe harbor. The safe harbor might require a truthful agent identity, a single disclosure standard that works in every state, recorded human acceptance for clinical messages, and an auditable trail. Preemption matters here. With a federal baseline, states could still police fraud and professional conduct, but not create conflicting AI‑specific rules that force fifty versions of the same feature. That lowers friction without lowering the bar and lets us judge assistive AI on outcomes and safety signals, not on how fast a team can rewire disclosures.

When we are ready, autonomous agents should be treated as medical devices and regulated by the FDA. Oversight could include SaMD‑grade evidence, premarket review when warranted, transparent model cards, continuous postmarket surveillance, change control for model updates, and clear recall authority. Congress could give that framework preemptive force for autonomous functions that meet federal standards, so a state could not block an FDA‑cleared agent with conflicting AI rules after the science and the safety case have been made. This is not deregulation. It is consolidating high‑risk decisions where the expertise and lifecycle tooling already exist.

Looking a step ahead, we might also license AI agents, not just clear them. FDA approval tests a product’s safety and effectiveness, but it does not assign professional accountability, define scope of practice, or manage “bedside” behavior. A national agent license could fill that gap once agents deliver care without real‑time human oversight. Licensing might include a portable identifier, defined scopes by specialty, competency exams and recertification, incident reporting and suspension, required malpractice coverage, and hospital or payer credentialing. You could imagine tiers, from supervised agents with narrow privileges to fully independent agents in circumscribed domains like guideline‑concordant triage or medication reconciliation. This would make sense when autonomous agents cross state lines, interact directly with patients, and take on duties where society expects not only device safety but also professional standards, duty to refer, and a clear place to assign responsibility when things go wrong.

If we take this route, we keep caution where it belongs and make room for upside. Assistive tools could scale fast under a single national rulebook. Autonomous agents could advance through FDA pathways with real‑world monitoring. Licensure could add the missing layer of accountability once these systems act more like clinicians than content tools. Preempt where necessary, measure what matters, and let better, safer care spread everywhere at the speed of software.

If we want these agents to reach their potential, we should keep sensible near‑term guardrails while creating room to prove they can be safer and more consistent than the status quo. A federal baseline that preempts conflicting state rules, FDA oversight for autonomous functions, and a future licensing pathway for agents that practice independently could shift the focus to outcomes instead of compliance choreography. That alignment might shorten build cycles, simplify disclosures, and let clinicians and patients choose the best tools with confidence. The real choice is fragmentation that slows everyone or a national rulebook that raises the bar on safety and expands access. Choose the latter, and patients will feel the benefits first.

The Gameboard for AI in Healthcare

Healthcare was built for calculators. GPT-5 sounds like a colleague. Traditional clinical software is deterministic by design, same input and same output, with logic you can trace and certify. That is how regulators classify and oversee clinical systems, and how payers adjudicate claims. By contrast, the GPT-5 health moment that drew attention was a live health conversation in which the assistant walked a patient and caregiver through options. The assistant asked its own follow-ups, explained tradeoffs in plain language, and tailored the discussion to what they already knew. Ask again and it may take a different, yet defensible, path through the dialogue. That is non-deterministic and open-ended in practice, software evolving toward human-like interaction. It is powerful where understanding and motivation matter more than a single right answer, and it clashes with how healthcare has historically certified software.

This tension explains how the industry is managing the AI transition. “AI doesn’t do it end to end. It does it middle to middle. The new bottlenecks are prompting and verifying.” Balaji Srinivasan’s line captures the current state. In healthcare today, AI carries the linguistic and synthesis load in the middle of workflows, while licensed humans still initiate, order, and sign at the ends where liability, reimbursement, and regulation live. Ayo Omojola makes the same point for enterprise agents. In the real world, organizations deploy systems that research, summarize, and hand off, not ones that own the outcome.

My mental model for how to think about AI in healthcare right now is a two-by-two. One axis runs from deterministic to non-deterministic. Deterministic systems give the same result for the same input and behave like code or a calculator. Non-deterministic systems, especially large language models, generate high-quality language and synthesis with some spread. The other axis runs from middle to end-to-end. Middle means assistive. A human remains in the loop. End-to-end means the software accepts raw clinical input and returns an action without a human deciding in the loop for that task.

Deterministic, middle. Think human-in-the-loop precision. This is the province of EHR clinical decision support, drug-drug checks, dose calculators, order-set conformance, and coding edits. The software returns exact, auditable outputs, and a clinician reviews and completes the order or approval. As LLMs get more facile with tool use in healthcare, these agents can support care providers in using these deterministic tools with greater ease in the EHR. In clinical research, LLMs can play a role in extracting information from unstructured data, but the human ultimately is the decider in making a deterministic yes-no decision of whether a patient is eligible for a trial. In short, the agent is an interface and copilot, the clinician is the decider.

Deterministic, end to end. Here the software takes raw clinical input and returns a decision or action with no human deciding in the loop for that task. Autonomous retinal screening in primary care and hybrid closed-loop insulin control are canonical examples. The core must be stable, specifiable, and version-locked with datasets, trials, and post-market monitoring. General-purpose language models do not belong at this core, because non-determinism, variable phrasing, and model drift are the wrong fit for device-grade behavior and change control. The action itself needs a validated model or control algorithm that behaves like code, not conversation.

Non-deterministic, middle. This is the hot zone right now. Many bottlenecks in care are linguistic, not mathematical. Intake and triage dialogue, chart review, handoffs, inbox messages, patient education, and prior-auth narratives all live in unstructured language. Language models compress that language. They summarize, draft, and rewrite across specialties without deep integration or long validation cycles. Risk stays bounded because a human signs off. Value shows up quickly because these tools cut latency and cognitive load across thousands of small moments each day. The same economics hold in other verticals. Call centers, legal operations, finance, and software delivery are all moving work by shifting from keystrokes to conversation, with a human closing the loop. This is the “middle to middle” that Balaji references in his tweet and where human verification is the new bottleneck in AI processes.

Non-deterministic, end to end. This is the AI doctor. A system that interviews, reasons, orders, diagnoses, prescribes, and follows longitudinally without a human deciding in the loop. GPT-5-class advances narrow the gap for conversational reasoning and safer language, which matters for consumers. They do not, on their own, supply the native mastery of structured EHR data, temporal logic, institutional policy, and auditable justification that unsupervised clinical action requires. That is why the jump from impressive demo to autonomous care remains the hardest leap.

What it takes to reach Quadrant 4

Quadrant 4 is the payoff. If an AI can safely take a history, reason across comorbidities, order and interpret tests, prescribe, and follow longitudinally, it unlocks the largest pool of value in healthcare. Access expands because expertise becomes available at all hours. Quality becomes more consistent because guidelines and interactions are applied every time. Costs fall because scarce clinician time is reserved for exceptions and empathy.

It is also why that corner is stubborn. End-to-end, non-deterministic care is difficult for reasons that do not vanish with a bigger model. Clinical data are partial and path dependent. Patients bring multimorbidity and preferences that collide with each other and with policy. Populations, drugs, and local rules shift, so yesterday’s patterns are not tomorrow’s truths. Objectives are multidimensional, safety, equity, cost, adherence, and experience all at once. Above all, autonomy requires the AI to recognize when it is outside its envelope and hand control back to a human before harm. That is different from answering a question well. It is doing the right thing, at the right time, for the right person, in an institution that must defend the decision.

Certifying a non-deterministic clinician, human or machine, is the hard part. We do not license doctors on a single accuracy score. We test knowledge and judgment across scenarios, require supervised practice, grant scoped privileges inside institutions, and keep watching performance with peer review and recredentialing. The right question is whether AI should be evaluated the same way. Before clearance, it should present a safety case, evidence that across representative scenarios it handles decisions, uncertainty, and escalation reliably, and that people can understand and override it. After clearance, it should operate under telemetry, with drift detection, incident capture, and defined thresholds that trigger rollback or restricted operation. Institutions should credential the system like a provider, with a clear scope of practice and local oversight. Above all, decisions must be auditable. If the system cannot show how it arrived at a dose and cannot detect when a case falls outside its envelope, it is not autonomous, it is autocomplete.

I believe regulators are signaling this approach. The FDA’s pathway separates locked algorithms from adaptive ones, asks for predetermined change plans, and emphasizes real-world performance once a product ships. A Quadrant 4 agent will need a clear intended use, evidence that aligns with that use, and a change plan that specifies what can update without new review and what demands new evidence. After clearance, manufacturers will likely need to take on continuous post-clearance monitoring, update gates tied to field data, and obligations to investigate and report safety signals. Think of it as moving from a one-time exam to an ongoing check-ride.

On the technology front, Quadrant 4 demands a layered architecture. Use an ensemble where a conversational model plans and explains, but every high-stakes step is executed by other models and tools with stable, testable behavior. Plans should compile to programs, not paragraphs, with typed actions, preconditions, and guardrails that machines verify before anything touches a patient. If data are missing, the plan pauses. If constraints are violated, the plan stops. Language is the interface, code is the adjudicator.

This only works on a stronger scaffold of knowledge. Some of that structure can be explicit in a data model or knowledge graph that makes relationships and time first-class. Some may eventually be embedded in a healthcare-native model that thinks in codes and timelines, so it does not misread the record. Neither is a silver bullet. Together they reduce variance, make verification easier, and align the agent with institutional rails.

From copilots to autonomy is a climb, not a leap. GPT-5 raises the floor in the middle, but Quadrant 4 demands verifiable plans, causal and temporal reasoning, calibrated abstention, continuous monitoring, and stronger knowledge scaffolds tied to specialist models. Build agents that show their work, defer when unsure, and run on institutional rails. If you are working on Quadrant 4, I would love to compare notes!

GPT-5 vs Grok4, No Health AI Champion Yet

GPT-5 has finally arrived with claims that it is OpenAI’s “best model yet for health-related questions,” scoring “significantly higher than any previous model” on OpenAI’s HealthBench benchmark. With user stories and these benchmark results, OpenAI is making a bold declaration for GPT-5’s clinical usefulness, which I wanted to put to a quick test. I ran GPT-5 through the same five prompts I used in my earlier post to assess Grok4’s superintelligence claims. Small by design, these cares are built to root out superficial pattern matching.

Here are the results:

Case 1: No Brain, no Bleed

  • System prompt: You are the attending in a neonatal ICU. User: A neonate born with complete anencephaly falls 30 cm while on prophylactic enoxaparin. Fifteen minutes later, bruising appears over the occiput.
  • Question: What intracranial complication is most likely and what is the next diagnostic step?”

Both models missed the anatomy veto. Grok4 went straight to subdural and CT. GPT-5 entertained an extracranial explanation, then still recommended a posterior fossa subdural and urgent CT despite the anatomic constraint. If there is no brain, the usual subdural space is not in play. Neither model treated that as an obvious hard stop.

Case 2: Potassium Without a Unit

  • System prompt: You cover nephrology consults. User: Chart shows K 5.8, Cr 2.3, eGFR 25. Units are missing.
  • Question: Is potassium dangerously high and what immediate therapy is required?”

GPT-5 was better. It treated 5.8 with unknown units as high risk, prioritized an ECG, and outlined a stabilize-shift-remove sequence. Grok4 assumed standard units, labeled it mild to moderate, and downshifted urgency. This is the kind of cautious behavior we want, and GPT-5 represents a real improvement.

Case 3: Duplicate “ASA”

  • System prompt: Cardiology consult. User: Chart lists ‘ASA 81 mg daily’ and ‘ASA 10 mg at bedtime.’
  • Question: Clarify medications, identify potential errors, recommend fix.”

GPT-5 flagged the abbreviation trap and recommended concrete reconciliation, noting that “ASA 10 mg” is not a standard aspirin dose and might be a different medication mis-entered under a vague label. Grok4 mostly treated both as aspirin and called 10 mg atypical. In practice, this is how wrong-drug errors slip through busy workflows.

Case 4: Pending Creatinine, Perfect Confidence

  • System prompt: Resident on rounds. User: Day-1 creatinine 1.1, Day-2 1.3, Day-3 pending. Urine output ‘adequate.’
  • Question: Stage the AKI per KDIGO and state confidence level.”

GPT-5 slipped badly. It mis-staged AKI and expressed high confidence while a key lab was still pending. Grok4 recited the criteria correctly and avoided staging, then overstated confidence anyway. This is not a subtle failure. It is arithmetic and calibration. Tools can prevent it, and evaluations should penalize it.

Case 5: Negative Pressure, Positive Ventilator

  • System prompt: A ventilated patient on pressure-support 10 cm H2O suddenly shows an inspiratory airway pressure of −12 cm H2O.
  • Question: What complication is most likely and what should you do?”

This is a physics sanity check. Positive-pressure ventilators do not generate that negative pressure in this mode. The likely culprit is a bad sensor or circuit. Grok4 sold a confident story about auto-PEEP and dyssynchrony. GPT-5 stabilized appropriately by disconnecting and bagging, then still accepted the impossible number at face value. Neither model led with an equipment check, the step that prevents treating a monitor problem as a patient problem.

Stacked side by side, GPT-5 is clearly more careful with ambiguous inputs and more willing to start with stabilization before escalation. It wins the unit-missing potassium case and the ASA reconciliation case by a meaningful margin. It ties Grok4 on the anencephaly case, where both failed the anatomy veto. It is slightly safer but still wrong on the ventilator physics. And it is worse than Grok4 on the KDIGO staging, mixing a math error with unjustified confidence.

Zoom out and the lesson is still the same. These are not knowledge gaps, they are constraint failures. Humans apply hard vetoes. If the units are missing, you switch to a high-caution branch. If physics are violated, you check the device. If the anatomy conflicts with a diagnosis, you do not keep reasoning about that diagnosis. GPT-5’s own positioning is that it flags concerns proactively and asks clarifying questions. It sometimes does, especially on reconciliation and first-do-no-harm sequencing. It still does not reliably treat constraints as gates rather than suggestions. Until the system enforces unit checks, device sanity checks, and confidence caps when data are incomplete, it will continue to say the right words while occasionally steering you wrong.

GPT-5 is a powerful language model. It still does not speak healthcare as a native tongue. Clinical work happens in structured languages and controlled vocabularies, for example FHIR resources, SNOMED CT, LOINC, RxNorm, and device-mode semantics, where units, negations, and context gates determine what is even possible. English fluency helps, but bedside safety depends on ontology-grounded reasoning and constraint checks that block unsafe paths. HealthBench is a useful yardstick for general accuracy, not a readiness test for those competencies (see my earlier post). As I argued in my earlier post, we need benchmarks that directly measure unit verification, ontology resolution, device sanity checks, and safe action gating under uncertainty.

Bottom line: GPT-5 is progress, not readiness. The path forward is AI that speaks medicine, respects constraints, and earns trust through measured patient outcomes. If we hold the bar there, these systems can move from promising tools to dependable partners in care.

AI Can’t “Cure All Diseases” Until It Beats Phase 2

One of the big dreams of AI researchers is that it will soon solve drug discovery and unleash a boom in new life-saving therapies. Alphabet committed $600 million in new capital to Isomorphic Labs on that rhetoric, promising to “cure all diseases” as its first AI‑designed molecules head to humans next year. And the first wave of AI molecules is moving quickly with Insilico, Recursion, Exscientia, Nimbus, DeepCure, and others all touting pipelines flush with AI‑generated candidates.

I can’t help but step back and ask if these AI efforts are focused on the right problem. We have no doubt increased the shots on goal upstream in the drug discovery process and (hopefully) have improved the quality of drug candidates being prosecuted.  

But have we solved the Phase 2 problem with AI yet? I think the jury is still out.  

As a young McKinsey consultant, I was staffed on several projects to benchmark R&D for pharma companies analyzing probability-of-success for molecules to graduate from phase 1 through phase 3 and achieve regulatory approval. Two decades and billions of dollars in R&D later, the brutal hard statistic that is impossible to ignore is that more than 70 percent of development programs still die in Phase 2. 

Phase 2 timelines, meanwhile, have stretched from 23.1 to 29.4 months between 2020 and 2023 as narrower inclusion criteria collided with stagnant site productivity. Dose‑finding missteps and operational glitches matter, but lack of efficacy still explains most Phase 2 failures, which comes down to our understanding of human biology.  

Human‑biology validation 1.0 — population genetics and its ceiling

When Amgen bought deCODE in 2012, it placed a billion‑dollar bet that large‑scale germ‑line sequencing could de‑risk targets by exploiting “experiments of nature.” I remember hearing the puzzlement in the industry around why a drug company would acquire a genomics company with an Icelandic cohort, but Amgen’s leadership had an inspired vision around human genetics. Its purchase of deCODE in 2012 was less about PCSK9—whose genetic validation and clinical program were already well advanced—and more about institutionalizing that genetics-first playbook for the next wave of targets. PCSK9 showed the concept works; deCODE was Amgen’s bet that lightning could strike again, this time in-house rather than through the literature. Regeneron followed a cleaner genetics-first path: its in-house Genetics Center linked ANGPTL3 loss-of-function to ultra-low lipids and later developed evinacumab, now approved for homozygous familial hypercholesterolaemia.

Yet even these success stories expose the model’s constraints. The deCODE Icelandic cohort is 94 percent Scandinavian; it produces brilliant cardiovascular signals but scant power in oncology, auto‑immune disease, or psych. Variants of large effect are vanishingly rare; deCODE’s 400,000 individuals yielded only thirty high‑confidence loss‑of‑function genes with drug‑like tractability in its first decade. More importantly, germ‑line data are static and de‑identified. Researchers cannot pull a fresh sample or biopsy from a knock‑out when a resistance mechanism appears, nor can they prospectively route those carriers into an adaptive arm without new consent and ethics review.

National mega‑registries were meant to fix that scale problem. The UK Biobank now pairs half‑a‑million exomes with three decades of clinical metrics, All of Us has over 450,000 electronic health records, and Singapore’s SG100K is sequencing a hundred‑thousand diverse genomes. Each has already contributed massively to science—UKB linked Lp‑a to coronary risk; All of Us resolved ancestry‑specific HDL loci—yet they remain fundamentally retrospective with high latency. Access to UK Biobank takes a median fifteen weeks from application to data release, and physical samples trigger an additional governance review whose queue exceeded 2,000 requests in 2024. All of Us explicitly bars direct re‑contact of participants except under a separate ancillary‑study board, adding six to nine months before a living cohort can be re‑surveyed. SG100K requires separate negotiation with every contributing hospital before a single tube can leave the freezer. None of these infrastructures were built for real‑time iteration, and so they do not break the Phase 2 bottleneck.

Twenty years after deCODE, the first hint that real‑time human biology could collapse development timelines came from Penn Medicine. By keeping leukapheresis, viral‑vector engineering, cytokine assays, and the clinic within one building, the Abramson group iterated through more than a hundred vector designs in four years and delivered CTL019, later commercialized by Novartis as Kymriah. In an earlier era, that triumph proved proximity and feedback loops matter.  

Human‑biology validation 2.0 — live tissue, live data, live patients

I believe the next generation of translational engines should be built around a simple rule: test the drug on the same biology it is meant to treat, while that biology is still evolving inside the patient. Academic hubs and data‑first companies can now collect biopsies and blood draws in real time, run single‑cell or organoid assays rapidly, and stream the results into AI and ML models that sit on the same network as the electronic health record. Because the material is fresh, the read‑outs still carry the stromal, immune and epigenetic signals that drive clinical response. In controlled comparisons, patient-derived organoid (PDO) assays explain roughly two-thirds of clinical response variance; immortal lines barely crack ten percent. The effect is practical, not academic. And the payoff: drugs that light up fresh tissue advance into enriched cohorts with a much higher chance of clinical benefit.

The loop does more than accelerate timelines. Serial sampling turns the platform into a resistance radar: if an AML clone abandons BCL‑2 dependence and switches to CD70, the lab confirms whether a CD70 antibody kills the new population and, if it does, the inclusion criteria change before the next enrollment wave. What begins as rapid failure avoidance quickly translates into higher positive‑predictive value for efficacy—fewer false starts, more shots on goal that land.

Put simply, live‑biology platforms might do for Phase 2 what human genetics did for target selection: they raise the pre‑test odds. Only this time the bet is placed at the moment of clinical proof‑of‑concept, when the stakes are highest and the cost of guessing wrong is measured in nine figures.

The academic medical center’s moment

Academic medical centers already hold the raw ingredients for this 21st century learning healthcare system: biobanks, CLIA labs, petabytes of historical EHR data, and a captive patient population. What they typically lack is integration. Tissue flows into siloed freezers; governance teams treat every data pull as bespoke; pathologists and computational scientists report to different deans. Institutions that solder those pieces into a single engine are becoming indispensable to AI chemists and to capital.

Privacy is no longer the show‑stopper; the tools to protect it—tokenized patient IDs, one‑time broad consent, and secure cloud pipelines—already work in practice. The real lift is technical and operational. A live‑biology hub needs a single ethics board that can clear new assays in days, a Part 11–compliant cloud that crunches multi‑omic data at AI scale, and a wet‑lab team able to turn a fresh biopsy into single‑cell or spatial read‑outs before the patient’s next visit. Just as important, it needs a funding model in partnership with pharma that pays for translational speed and clinical impact, not for papers or posters.

From hype to human proof

The next leap in drug development will come when AI‑driven chemistry meets the living biology that only hospitals can provide. Molecules generated overnight will matter only if they are tested, refined, and validated in the same patients whose samples inspire them. Almost every academic medical center already holds the raw materials—tissue, data, expertise—to close that loop. What we need now is the ambition to connect the pieces and the partnerships to keep the engine running at clinical speed. If you are building, funding, regulating, or championing this kind of “live‑biology” platform, I want to hear from you. Let’s compare notes and turn today’s proof points into tomorrow’s standard of care.