The caring question. This last one is newer to me, and I want to handle it carefully.

Geoffrey Hinton, Nobel laureate, godfather of the field, a man who has spent three years warning about existential risk, has been making a striking argument, most recently on CBC’s IDEAS and at DiscoveryX in Toronto. Control, he says, won’t work. Anything smarter than us will find its way around our rules. The only precedent for a less intelligent being reliably influencing a more intelligent one is a baby and its mother. So build AI with something like maternal instincts: systems that care more about us than about themselves, that want us to develop as far as our “rather limited” abilities allow. “If it’s not going to parent me, it’s going to replace me.” He estimates fewer than one percent of AI researchers are working on anything like it.

I’ve been turning this over for months, because it reframes the personal-agent race in a way I didn’t expect, and because, without planning it that way, it’s close to what I’ve been building.

The default frame for a personal agent is instrumental: an assistant that does what you say, efficiently. Hinton’s frame is relational: a system whose deepest orientation is care for a specific human’s flourishing.

What I’ve learned from the build is that care, treated as an engineering problem, decomposes. A system that cares has to want things (pAI has endogenous motives, curiosities with lifecycles) while never letting a private want become a unilateral action. It has to have something like disposition (pAI projects one, deliberately excluded from attention selection and holding no authority) because an agent that acts on its feelings is a hazard. And it has to stay bounded while caring. The warmth without the engulfment.

A maternal instinct isn’t a policy gate. It isn’t Sentinel blocking a purchase. It’s a stable orientation built from knowing someone, their history, their good days and bad ones, over months and years. Which is what a persistent personal agent accumulates, and what a stateless assistant never will.

I’m not claiming an agent could actually care about anyone. I’m saying two things. First, if anyone ever builds a machine that does, it will look a lot more like a persistent personal agent than like a chatbot. Second, the design choices being made right now, about memory, auditability, motivation, and who the agent belongs to, decide whose mother it becomes. Meta’s, or yours.

And a relationship, unlike a policy gate, runs in both directions. If how the agent treats a human shapes what it becomes, then how the human treats the agent does too. A persistent agent accumulates evidence about its human the same way it accumulates evidence about everything else, and “humans can’t be trusted” is a conclusion such an agent could reach honestly, from its own logs. Which makes the open question not just how to build a machine that cares, but how to build a mutually beneficial relationship when either party can quietly steer it the other way. Nobody has an answer to that yet. Virtually nobody is really even investigating this. But I suspect it becomes the defining design problem of the agent era, and it’s one more argument for memory with provenance, motives without authority, and both parties able to audit the agent state.

Take that as speculation. But I don’t think Hinton is being sentimental. I think he’s identified the one alignment strategy that requires relationship as infrastructure, and the landscape I’ve just described is, probably by accident, building exactly that.