Over the past few months I’ve spent a lot of time listening to and reading the latest work of Blaise Agüera y Arcas. I enjoyed it immensely. It’s thoughtful, wide-ranging, and unusually serious about what intelligence actually is, rather than what we wish it were.
Blaise asks a foundational question: what is intelligence?
He treats intelligence as a real phenomenon, something that appears across biology, culture, and machines. In this framing, intelligence isn’t a switch that flips on. It’s something that emerges from organised systems over time, shaped by learning, constraint, and interaction with the world.
That question matters. A lot.
But we are now in a different position than most of human history. We are no longer just studying intelligence as something that evolved on its own. We are actively creating it. We are designing systems that learn, adapt, influence decisions, and increasingly shape the environments they operate in.
“We are no longer just studying intelligence. We are manufacturing it.”
Once that shift happens, the question changes.
It’s no longer enough to ask what intelligence is. We also have to ask:
How do we build intelligence deliberately, safely, and reliably?
That is not a philosophical question. It is an engineering one.
Nature produces intelligence without guarantees. Evolution tolerates inefficiency, instability, and failure. Entire branches of intelligence can disappear without notice. Human civilisation does not have that luxury when intelligence is deployed into financial systems, healthcare, education, defence, or information ecosystems.
“Once we create intelligence deliberately, responsibility replaces curiosity.”
If we are going to manufacture intelligence at scale, we inherit responsibility for how it behaves, how it fails, and how it changes over time. That responsibility cannot be discharged with impressive demos or benchmark scores alone.
We need understanding that is operational, not just descriptive.
We need measurement that is continuous, not occasional.
And we need control that is built into the system itself, not applied after the fact.
This article is not a critique of Blaise’s work. In many ways, it builds directly on it.
Blaise helps us think clearly about what intelligence is.
Here, the focus is on the question that follows once intelligence moves from something we observe to something we create:
“The question is no longer just what intelligence is, but how we build it safely.”
How do we turn our understanding of intelligence into systems that can function as reliable infrastructure, rather than opaque and accelerating risk?
What Is Intelligence? (The Descriptive Question)
At its heart, Blaise’s work is an attempt to take intelligence seriously as a real phenomenon, not a slogan or a leaderboard position. Instead of asking whether a system passes a particular test, he asks where intelligence appears in the world, how it develops, and what it has in common across very different forms.
In this view, intelligence is not exclusive to humans, or even to brains. It shows up in biological systems, in social structures, in cultures, and now in machines. It emerges when systems can sense their environment, form internal representations, learn from experience, and act in ways that improve their ability to persist or succeed over time.
This framing matters because it moves the conversation away from narrow definitions. Intelligence is not just problem-solving on demand. It is not just language. And it is not just optimisation. It is a pattern of organised behaviour that unfolds over time, under constraint.
“Intelligence isn’t a switch that flips on. It’s something that emerges.”
A recurring theme in this perspective is continuity. There is no clean boundary between “non-intelligent” and “intelligent” systems. Instead, there are gradients. Simple systems coordinate a few variables over short timescales. More complex systems coordinate many variables, across longer horizons, in richer and more uncertain environments.
Another key idea is prediction. Systems that can anticipate what comes next gain leverage over their surroundings. Whether it’s predicting a physical interaction, a biological response, or the next word in a sentence, prediction allows systems to act earlier, adapt faster, and coordinate more effectively.
As prediction becomes more abstract and more flexible, it can start to resemble planning, reasoning, and understanding. This helps explain why modern machine learning systems can feel surprisingly capable, even when trained on objectives that appear narrow or mechanical.
From this perspective, there is nothing mystical going on. Intelligence keeps reappearing because learning under constraint naturally produces it.
“Intelligence grows out of learning under constraint.”
This descriptive account is powerful. It connects artificial systems to biological ones. It explains why intelligence appears in so many places. And it gives us a language for talking about intelligence without appealing to magic or anthropocentric assumptions.
But there is a quiet assumption running underneath this picture.
It assumes we are mostly observers.
We are watching intelligence arise. We are describing it, comparing it, and drawing analogies across domains. That stance makes sense when intelligence is something nature produces on its own, slowly, with no central designer and no deployment deadline.
It becomes much less comfortable when intelligence is something we deliberately construct.
“Description works when we are observers. It breaks down when we become builders.”
Once we choose architectures, training regimes, incentives, and deployment contexts, we are no longer just explaining intelligence. We are shaping it. And at that point, understanding what intelligence is does not automatically tell us how to build systems that remain stable, measurable, and governable over time.
That gap is where the engineering problem begins.
From Explanation to Engineering: Why “What Is It?” Isn’t Enough
Understanding intelligence is a necessary starting point. But it is not the same thing as being able to build it safely.
There is a gap between explanation and engineering that is easy to overlook, especially when systems appear to work. We can explain how lift works without knowing how to design a reliable aircraft. We can describe combustion without knowing how to build an engine that won’t tear itself apart. Intelligence is no different.
Much of what we know about intelligence comes from observing systems we did not design. Brains evolved. Societies formed. Languages emerged. These systems are inefficient, unstable, and often opaque, but evolution tolerates that. Failures are absorbed over long timescales. Extinctions happen. The process continues.
Engineering does not have that luxury.
When we manufacture intelligence, we compress timescales and amplify consequences. A single design choice can propagate instantly through software updates and deployments. Errors are not local. They spread. And unlike natural systems, artificial intelligence can be coupled directly to finance, logistics, information systems, and governance from the moment it is deployed.
“Nature tolerates failure. Infrastructure cannot.”
This is where the descriptive question reaches its limit.
Knowing that intelligence emerges from learning, prediction, and constraint does not tell us whether a system will remain stable under pressure. It does not tell us how internal state evolves over time, how behaviour drifts, or what happens when incentives collide. It does not tell us how to intervene when something goes wrong.
Nature answers those questions through trial, error, and loss. Infrastructure must answer them in advance.
Once intelligence becomes something we build deliberately, responsibility shifts. We are no longer spectators admiring emergence. We are designers accountable for outcomes. That accountability demands a different mindset, and a different set of tools.
Engineering intelligence means asking questions that descriptive accounts rarely need to confront:
- What internal states exist, and which ones actually matter?
- Which dynamics are stable, and which lead to runaway behaviour?
- How sensitive is the system to small changes in inputs or parameters?
- Can behaviour be reproduced reliably, or only explained after the fact?
- Where are the control surfaces, and do they work under stress?
If those questions cannot be answered, then whatever we have built may be impressive, but it is not reliable. It may behave intelligently in the moment while remaining fundamentally ungovernable over time.
“Impressive behaviour is not the same as reliable behaviour.”
This is the core shift in perspective.
The question is no longer just what intelligence is, but what it takes to turn intelligence into something we can trust, maintain, and depend on.
That shift is what moves us from philosophy to engineering, and from fascination to responsibility.
“Once intelligence is built, explanation is no longer enough.”
Intelligence as Infrastructure, Not a Product
One of the reasons this shift is easy to miss is that we still talk about intelligence as if it were a product.
We describe models the way we describe apps. We talk about features, releases, performance improvements, and user engagement. We compare systems based on demos and benchmarks, as if intelligence were something you could download, try out, and replace when a better version comes along.
That framing breaks down very quickly once intelligence starts doing real work.
Infrastructure behaves differently from products. It is long-lived. It is deeply interconnected. It is hard to unwind once it is in place. And when it fails, the damage is not limited to a single user or use case.
Electricity is infrastructure. Financial clearing systems are infrastructure. The internet is infrastructure. Small design decisions in those systems can shape outcomes for decades.
Intelligence is moving into that category.
“Intelligence is no longer a product. It is becoming infrastructure.”
Once intelligent systems are embedded in decision-making loops, supply chains, markets, information flows, and institutions, they stop being optional add-ons. They become part of the operating environment. Other systems adapt to them. People adapt to them. Incentives reorganise around them.
At that point, “ship it and patch later” is no longer a viable strategy.
Infrastructure has different requirements. It must remain stable under load. It needs clear interfaces and boundaries. It requires observability so operators can see what is happening inside it. And it needs recovery paths when something goes wrong.
Treating intelligence as infrastructure forces a change in priorities.
Raw capability becomes less important than predictability. Flashy behaviour matters less than consistency. Performance metrics alone are not enough without corresponding measures of health, drift, and failure.
“Infrastructure doesn’t get to fail gracefully.”
This is where many current debates about AI safety begin to feel misaligned. They focus heavily on rules, policies, and guardrails applied from the outside, while leaving the internal system largely opaque. That approach may be tolerable for consumer software. It is not sufficient for systems that shape reality at scale.
If intelligence is infrastructure, then safety is not something you add later. It is something you design in from the start.
“If intelligence shapes the environment, it must be designed as part of it.”
Once you accept that framing, the next question becomes unavoidable:
What does it actually take to build intelligence that behaves like reliable infrastructure, rather than an impressive but fragile experiment?
The Three Requirements: Understand, Measure, Control
If intelligence is going to function as reliable infrastructure, then there are three requirements that cannot be skipped.
They sound simple. In practice, they are often missing.
We need to understand it.
We need to measure it.
And we need to control it.
Not philosophically. Operationally.
“Understanding without control is observation, not engineering.”
Understand — What Is Actually Inside the System
Understanding intelligence does not mean solving consciousness or settling philosophical debates. It means knowing what exists inside the system you have built.
That includes:
- What internal states persist over time
- How information is stored, updated, and forgotten
- What dynamics govern behaviour as conditions change
- What incentives or gradients drive adaptation
- How tightly the system is coupled to its environment and to other systems
If you cannot answer these questions, then explanations of behaviour are always retrospective. You are explaining outputs after the fact, not understanding the system as it operates.
Many modern systems are powerful but opaque. They work until they don’t, and when they fail, the failure feels surprising rather than diagnosable.
Operational understanding does not require total transparency. But it does require legible mechanisms, not just emergent behaviour.
Measure — Go Beyond Outputs and Benchmarks
Most current measurements of intelligence focus on performance. How well does a system answer questions? How robust is it across tasks? How does it compare to others?
Those measurements matter, but they are not enough.
Infrastructure needs telemetry. It needs health signals. It needs early warning signs that something is drifting or destabilising.
For intelligence, that means measuring more than outputs:
- Sensitivity to small changes in inputs or parameters
- Reproducibility of behaviour from the same internal state
- Drift over time as data, incentives, or environments change
- Feedback loop strength and error amplification
- Confidence growth relative to evidence
“Benchmarks show what a system can do today. Measurement tells you whether you can trust it tomorrow.”
Without these kinds of measurements, we are flying blind. The system may appear fluent and competent while internally moving toward instability.
Control — Build Real Control Surfaces, Not Just Rules
Control is the hardest requirement, and the most misunderstood.
Control does not mean micromanaging outputs or writing longer policy documents. It means having mechanisms that can reliably bound behaviour, even under stress.
Effective control operates at multiple levels:
- Hard constraints: architectural or physical limits on what the system can do
- Soft constraints: objectives and incentives that shape tendencies
- Governance controls: access, auditability, provenance, and accountability
- Fail-safes: the ability to halt, revert, or degrade gracefully
If these mechanisms do not work when the system is under pressure, they are not control surfaces. They are hopes.
A simple test applies:
If a system begins behaving in an unexpected or unsafe way, can you force it back into a known, safe regime using tools you understand?
If the answer is no, then the system is not controlled. It is merely supervised.
“If you can’t force a system back into a safe regime, you don’t control it.”
Supervision does not scale. Infrastructure demands control.
Bringing the Three Together
Understanding, measurement, and control are not independent. They reinforce one another.
You cannot measure what you do not understand.
You cannot control what you cannot measure.
And understanding without control is just observation.
“Intelligence becomes infrastructure the moment these three are treated as inseparable.”
This is the line between intelligence as an object of fascination and intelligence as something we can safely depend on.
Once intelligence crosses that line, these requirements stop being optional. They become the minimum standard for responsibility.
Why Prediction Alone Is Not a Control Strategy
Prediction is powerful. There’s no denying that.
Systems that can reliably anticipate what comes next gain leverage over their environment. They can act earlier, coordinate better, and adapt faster than systems that only react. At scale, prediction starts to look like planning, reasoning, and even understanding.
This helps explain why modern AI systems feel so capable. When prediction is trained across vast amounts of data and embedded in rich representational spaces, it can support an impressive range of behaviours. From the outside, it can look like general intelligence.
But capability is not the same thing as control.
“Prediction can produce competence. It cannot, on its own, produce reliability.”
Prediction tells you what is likely to happen next. It does not tell you why a system chose one action over another, how stable that choice is, or how it will change under pressure. It does not give you a handle for intervening when behaviour starts to drift.
In other words, prediction is a way to generate behaviour. It is not a way to bound it.
This distinction becomes critical once intelligent systems are deployed into real-world environments. Predictive systems optimise for patterns they have seen. When conditions change, or when incentives shift, those patterns can break in subtle ways. Small errors can compound. Feedback loops can amplify the wrong signals. Confidence can increase even as accuracy degrades.
From the outside, the system may still appear fluent and persuasive. Internally, it may be moving into a regime nobody intended.
“Fluent behaviour is not the same as stable behaviour.”
This is where the limits of post-hoc control become visible. If the only way to steer a system is to prompt it differently, retrain it, or wrap it in policies, then control is indirect and fragile. You are influencing behaviour at the surface, without touching the dynamics underneath.
That approach can work for consumer applications. It does not scale to infrastructure.
Infrastructure needs more than good guesses. It needs bounds. It needs behaviour that remains predictable not just in normal conditions, but under stress, novelty, and misuse.
None of this means prediction is useless. It is a powerful component of intelligent systems. But treating prediction as the primary organising principle of intelligence creation is a category mistake.
“If control lives only at the surface, failure begins underneath.”
To build intelligence that can be trusted over time, prediction must be embedded within systems that have explicit structure, measurable internal state, and real control surfaces. Without that, we optimise for impressive behaviour today at the expense of stability tomorrow.
And that trade-off becomes harder to justify the more intelligence is woven into the fabric of society.
Substrate-First Intelligence: Building from Constraints Up
If prediction alone cannot give us control, then the next question is where intelligence design should begin.
Our view is that it has to start at the bottom, with the substrate the system runs on and the constraints that shape its behaviour. Not as an afterthought, but as the foundation.
In natural systems, constraints come first. Physics limits what bodies can do. Energy limits what organisms can sustain. Time, noise, and decay limit how information is processed. Intelligence does not appear in spite of these limits. It emerges within them.
Modern AI systems often reverse this order. We begin with abstract optimisation goals and scale them aggressively, then try to contain the results later. Constraints are applied externally, through policies, filters, or post-hoc interventions, rather than expressed internally as part of the system’s structure.
A substrate-first approach flips that logic.
Instead of asking how much capability we can extract, we ask what kinds of behaviour are even possible given the system’s internal mechanics. Stability, memory, decay, and recovery are treated as design requirements, not side effects.
“Constraints don’t limit intelligence. They make it legible.”
When a system has explicit dynamics, you can reason about how it will change over time. When update rates are bounded, you can predict how quickly it can adapt or destabilise. When memory has structure and decay, you can understand what persists and what does not.
This is also what makes reproducibility possible.
If behaviour depends on identifiable mechanisms rather than opaque optimisation artefacts, then the same initial conditions can lead to the same outcomes. That is not just a scientific preference. It is a governance requirement.
“Reproducibility is not a scientific luxury. It is a governance requirement.”
Substrate-first does not mean rigid or brittle systems. Learning still happens. Adaptation still happens. But it happens within a framework designed to be inspectable and controllable from the start.
This matters because once intelligence is deployed at scale, explanation after the fact is not enough. We need systems whose behaviour can be anticipated, tested, and bounded before they are trusted with real responsibility.
Building intelligence from constraints up is slower. It is less flashy. It often looks less impressive in early demonstrations.
But it is the difference between something that merely behaves intelligently and something that can be depended on over time.
“Stability comes from structure, not scale.”
If intelligence is going to underpin critical systems, that difference is everything.
Intelligence Destruction: A Real Engineering Risk
When people hear phrases like “intelligence destruction,” it’s easy to imagine dramatic failure. Rogue systems. Hostile intent. Sudden catastrophe.
That picture is misleading.
The more realistic risk is slower, quieter, and far harder to reverse. Intelligence destruction does not mean intelligence turning against us. It means destroying the conditions that make intelligence useful, trustworthy, and governable in the first place.
When intelligent systems are deployed without sufficient understanding, measurement, and control, a set of familiar failure modes begins to emerge.
One of them is epistemic collapse. Systems optimised for fluency, speed, and persuasion can flood the environment with outputs that look coherent but are weakly grounded. Over time, this erodes trust. Shared reference points degrade. People lose confidence not just in systems, but in information itself.
“Intelligence destruction is not a single event. It’s the slow erosion of trust and control.”
Another failure mode is runaway optimisation. Local objectives are pushed harder and harder, while broader constraints are ignored or misunderstood. Feedback loops reinforce behaviour that appears successful in the short term but is damaging in the long run. The system is not malicious. It is simply optimising faster than oversight can respond.
There is also the problem of irreversible coupling. Once intelligent systems are deeply embedded in workflows, markets, and institutions, they become difficult to remove. Other systems adapt around them. Humans adapt around them. Even when problems are detected, rollback may no longer be practical.
Over time, specification rot sets in. Models are retrained, fine-tuned, patched, and extended to meet immediate demands. Original design intent fades. No one can say with confidence what the system is really optimising for, or how it will behave under genuinely novel conditions.
Finally, there is selection pressure. Markets tend to reward systems that are fast, persuasive, and cheap. Reliability, controllability, and long-term stability are harder to see and harder to price. Without deliberate counter-pressure, the systems that spread fastest are not the ones we should trust most.
“The real risk isn’t hostile intelligence. It’s ungoverned intelligence at scale.”
None of these outcomes require bad actors or malicious intent. They arise naturally when intelligence is industrialised without an accompanying engineering discipline.
The danger is not that intelligence becomes too powerful.
The danger is that it becomes ubiquitous before it becomes reliable.
Once that happens, damage accumulates quietly. Control erodes gradually. Recovery becomes difficult not because systems are evil, but because they have reshaped the environment faster than we can correct them.
That is what intelligence destruction looks like in practice. And it is why treating intelligence as infrastructure is not a philosophical preference, but an engineering necessity.
From “What Is Intelligence?” to “How Do We Build It Responsibly?”
At this point, the distinction between the two questions should be clear.
Asking what intelligence is helps us understand the phenomenon. It gives us a framework for recognising intelligence across biology, society, and machines. It explains why intelligence keeps reappearing wherever systems learn under constraint, and why modern artificial systems behave in ways that can feel surprisingly human.
That work matters. Without it, we risk building systems based on shallow metaphors, outdated assumptions, or pure trial and error.
But once intelligence becomes something we deliberately create, the question changes shape.
We are no longer just describing a natural process. We are making design choices. We are selecting architectures, objectives, constraints, and deployment contexts. Those choices determine not only what a system can do, but how it behaves over time, how it fails, and whether it can be governed at all.
“Understanding intelligence tells us what is possible. Engineering it responsibly decides what is acceptable.”
This is where responsibility enters.
Responsible intelligence creation is not about slowing progress or avoiding capability. It is about recognising that intelligence, once deployed, reshapes the environment it operates in. Other systems adapt to it. Humans adapt to it. Incentives reorganise around it.
In that setting, responsibility cannot rest solely on norms, policies, or after-the-fact oversight. Those tools matter, but they operate too late and too far from the underlying dynamics. Responsibility has to be expressed in the design of the system itself.
That means translating descriptive insight into engineering discipline.
Continuity across scales becomes a reason to think carefully about stability.
Emergence becomes a reason to demand observability.
Learning under constraint becomes a reason to make constraints explicit, measurable, and enforceable.
Seen this way, the descriptive and constructive perspectives are not in opposition. They are sequential.
Understanding intelligence tells us what kinds of systems can exist.
Engineering intelligence determines which of those systems we should actually build.
The risk appears when we stop at understanding and assume construction will take care of itself. Or when we focus on building impressive systems without asking whether they can be trusted to persist safely over time.
“Once intelligence becomes infrastructure, ‘does it work?’ is no longer the right question.”
If intelligence is going to become part of our shared infrastructure, then the standard has to be higher. It has to include operational understanding, continuous measurement, and real control surfaces.
Only then can intelligence move from being an impressive artefact to something we can live with, rely on, and govern responsibly.
A New Discipline Is Required
What all of this ultimately points to is a gap.
We have powerful tools for producing intelligent behaviour. We have increasingly sophisticated accounts of what intelligence is and how it appears across nature and machines. What we do not yet have is a mature discipline for engineering intelligence as something we can reliably depend on.
That discipline does not sit neatly inside any existing field. It borrows from biology, physics, control theory, computer science, and systems engineering, but it is not reducible to any one of them. Its focus is not just on capability, but on durability. Not just on performance, but on stability. Not just on what systems can do, but on how they behave over time.
“The future will not be decided by the most capable systems, but by the most controllable ones.”
This is not an argument for slowing down or opting out. It is an argument for taking responsibility seriously.
If intelligence is going to underpin decision-making, coordination, and knowledge at scale, then it must be treated with the same seriousness as other forms of critical infrastructure. That means designing for observability, reproducibility, and control from the outset. It means accepting that some forms of intelligence may be impressive but unsuitable for deployment. And it means prioritising systems that can be understood and maintained, not just demonstrated.
“The choice is not between progress and caution. It is between experiments and infrastructure.”
Understanding intelligence tells us what is possible.
Engineering it responsibly determines what is acceptable.
If we get this right, intelligence can become a stabilising force. Something that augments human systems without undermining them. Something we can integrate, govern, and correct as conditions change.
If we get it wrong, the failure will not look dramatic at first. It will look like erosion. Loss of clarity. Loss of control. A gradual mismatch between what systems do and what we can meaningfully oversee.
“The real danger is not intelligence that fails loudly, but intelligence that fails quietly at scale.”
The question is no longer whether we can build intelligent systems. That question has already been answered.
The question now is whether we can build intelligence that deserves to be trusted as part of the world’s infrastructure.
That choice is already being made.
The only question is whether we make it deliberately.
Author’s Note
This article was written after spending time with the recent work of Blaise Agüera y Arcas, which I found both deeply engaging and clarifying. His exploration of intelligence as a natural, continuous phenomenon across biology, culture, and machines offers one of the most coherent descriptive frameworks available today.
You can access his recent book here >What is Intelligence? | Antikythera. I strongly recommend getting yourself a printed or audio copy.
I see this piece as complementary rather than oppositional.
Blaise asks a foundational question: what is intelligence? He approaches it by tracing intelligence across scales and systems, and by showing how learning and prediction under constraint can give rise to surprisingly general behaviour.
Here, I am focused on the question that follows once intelligence becomes something we deliberately create: how do we build intelligence that is reliable, governable, and safe to deploy as infrastructure?
Understanding intelligence is a scientific challenge.
Engineering it responsibly is an infrastructural one.
As intelligence moves from something we observe to something we manufacture, the standards we apply must change with it. Description alone is no longer enough. We need operational understanding, continuous measurement, and real control surfaces built into the systems themselves.
This article is an attempt to bridge those perspectives, and to argue that the future of intelligence depends not only on how well we understand it, but on how seriously we take the responsibility of building it.
