What Are We Governing When We Govern AI?

In August 2025, the United Nations General Assembly established a Global Dialogue on Artificial Intelligence Governance. Every UN member state would have a place in a conversation about how one of the most consequential technologies of our time should be governed.

The resolution describes what the dialogue will discuss, how it will meet, and what it hopes to achieve. There is something curious about it, though.

Nowhere does it define AI governance.

It isn't the only major text to leave the term undefined.

International agreements often leave broad concepts open, and that is not necessarily a failure. Still, the omission raises an interesting question. Governments, companies and international organizations have spent years developing AI policies, standards and oversight arrangements. They have created institutions to govern AI, published principles for doing so, and increasingly made compliance a legal obligation.

Yet what exactly is the activity they have agreed to undertake?

Perhaps the answer seems obvious. We govern AI by establishing rules for how it is designed, trained, deployed and used. We require companies to test their systems, protect people's rights, explain important decisions, and take responsibility when things go wrong.

All of that matters.

Consider a government that has adopted an ambitious AI law. It establishes a regulator and gives it authority to investigate violations. The law is comprehensive, the institution has a mandate, and the country can reasonably say it has established AI governance.

Now imagine that the regulator has almost no technical staff, cannot independently evaluate the systems it oversees, and depends on information supplied by the companies it is supposed to regulate.

What has the law actually changed?

The government has acquired the legal authority to govern AI. Whether it can exercise that authority is a different question.

The distinction becomes even more important when we move beyond regulation.

A country might have access to advanced AI systems without controlling the infrastructure they depend on. A company might adopt a safety policy while rewarding the very behavior that undermines it.

And a health ministry might carefully evaluate an AI tool for its hospitals before buying it, only to discover years later that replacing it has become prohibitively expensive.

These are different problems, but they point toward a shared concern. Governing AI may involve more than establishing rules for technology. It may require shaping the conditions under which people and institutions can make decisions about that technology, challenge those decisions, and change course when necessary.

That possibility is what led me to look more closely at the meaning of AI governance.

How did one term come to mean so many things?

Part of the difficulty lies in the history of the field.

AI governance did not begin with a single definition that everyone subsequently applied. Researchers and institutions approached it from different directions, often responding to different concerns.

In 2017, legal scholar Urs Gasser and computer scientist Virgilio Almeida proposed a layered model of AI governance. They considered how technical standards, ethical principles and social and legal arrangements might develop alongside the technology. Their model recognized that governance changes as technologies mature, and that different interventions become appropriate at different times.

A year later, political scientist Allan Dafoe described AI governance, in a 2018 research agenda, as a field studying how humanity could best navigate the transition to advanced AI systems. His concern was not primarily how a company should manage a product already in use. It was how societies might shape a technological transition whose long-term consequences remained uncertain.

Both perspectives were concerned with governance, but they approached it at different scales.

As AI moved into businesses and public institutions, another understanding became increasingly important. Organizations needed practical ways to decide which systems to use, who was responsible for them, and how to manage their risks.

Management scholars began describing AI governance through organizational rules, practices, processes and tools. In 2023, the International Organization for Standardization and the International Electrotechnical Commission published ISO/IEC 42001, a standard setting requirements for an AI management system.

This was a different kind of answer to the governance question. It treated governance as something an organization could establish, operate, evaluate and improve.

Meanwhile, governments and international bodies were developing their own approaches. The Organisation for Economic Co-operation and Development, or OECD, published AI principles in 2019. UNESCO, the UN agency concerned with education, science and culture, adopted its Recommendation on the Ethics of Artificial Intelligence in 2021.

Both placed considerable emphasis on the public purposes that AI governance should serve.

The story was not confined to Europe and North America.

In 2021, the African Commission on Human and Peoples' Rights, a human rights body of the African Union, called for AI governance grounded in human and peoples' rights, African values and meaningful human control. That same year, Smart Africa, an alliance working to advance digital development across the continent, published an AI blueprint that favored flexibility and voluntary guidance before early regulation, a deliberate choice about when to act that this essay returns to.

Different institutions were addressing different parts of a growing problem.

Some focused on rights, some on organizational accountability, others on international coordination, technological development or the capacity of societies to participate in shaping AI.

What is striking is what happened as these approaches began to acquire legal and institutional form.

The European Union's AI Act, adopted in 2024, contains a chapter titled Governance. The chapter establishes the bodies and arrangements responsible for administering and enforcing the law. Yet the Act's extensive list of defined terms does not include a definition of governance itself.

The Council of Europe's Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, also opened for signature in 2024, takes another approach. The word governance does not appear in its English or French treaty text.

The absence of a definition does not diminish either legal instrument. Laws can establish precise obligations without defining every broad concept associated with them. Still, the contrast is revealing. One instrument gives governance a chapter. Another establishes a binding international framework concerning AI without using the word.

In 2024, the OECD submitted input to the UN's High-level Advisory Body on AI that asked the body to clarify what AI governance refers to. The advisory body's final report, Governing AI for Humanity, did not provide a formal definition.

Instead, it emphasized what international governance should accomplish, particularly addressing risks and making AI's benefits more widely available.

By the time the UN established its Global Dialogue in 2025, AI governance had become an established field of international activity. Yet its meaning was still being assembled through different purposes, institutions and practices.

There is nothing inherently wrong with a concept developing this way.

Many political and legal ideas acquire meaning through use, disagreement and institutional experience.

The difficulty arises when we assume that everyone using the same term is addressing the same problem.

An organization seeking compliance, a regulator protecting rights, and a government concerned about dependence on foreign computing infrastructure may all be engaged in AI governance.

Their concerns overlap, but none captures the whole field.

What the definitions reveal, and what they leave out

To understand this more systematically, I examined a collection of historical and institutional documents, then undertook a wider review of academic writing on AI governance.

The historical collection contained 38 documents. Most came from governments, intergovernmental organizations and international standards bodies. Researchers accounted for seven, while three came from industry.

None of the documents in that particular collection came from civil-society organizations, workers' groups or communities directly affected by AI.

That absence says something about the limits of the collection, not about the absence of contributions from those groups.

It matters because a history constructed mainly from institutions already recognized as governance authorities can overlook people who have been questioning how technology is governed from outside those institutions.

The wider review began with a structured search of OpenAlex, an index of scholarly publications. Nearly 5,000 records were screened, and approximately 150 papers were randomly sampled for closer examination.

The question was deliberately narrow. When authors defined AI governance, what did their definitions actually name?

A little over half of the sampled definitions emphasized governance instruments, such as policies, rules, standards and procedures. About half named AI systems themselves as what governance acts upon.

Other elements appeared less often.

Institutions were mentioned in roughly a third of the definitions, safeguards in about one in five, infrastructure in fewer than one in six, and incentives in approximately one in twenty or fewer.

Among definitions that could be coded with confidence, none explicitly combined all four of the latter categories.

Timing was another telling case. Many papers discussed the importance of anticipating technological change or acting before problems become entrenched. Yet only five definitions clearly incorporated timing, with approximately a dozen more doing so less explicitly.

The review was AI-assisted and has not been checked by an independent researcher, so these figures are exploratory rather than established measurements. They also describe what definitions explicitly say, not everything their authors discuss elsewhere.

Even with those limitations, the findings suggest a distinction worth examining.

Definitions frequently describe the instruments of governance or the technology being governed. They less often explicitly identify the circumstances that determine whether those instruments can work.

Consider a company that promises to evaluate an AI system before releasing it.

A definition centered on rules and safeguards directs us toward the evaluation procedure. We might ask whether the test is rigorous, which risks it examines, and what evidence it requires.

Those are necessary questions.

But what happens if the company faces strong pressure to release the system before a competitor does?

The pressure is real.

In 2025, OpenAI's updated Preparedness Framework said it might adjust its safety requirements, under stated conditions, if a rival released a high-risk system without comparable safeguards. Who can require the company to delay? Can independent evaluators obtain the information they need? What happens if concerns emerge after millions of people have begun relying on the system?

The initial evaluation is still important. It is simply part of a larger set of relationships that determine whether the company can be held to its commitment.

There was also a finding that challenged my own approach.

Social and professional norms appeared more often in the reviewed definitions than incentives did. Yet the framework I was developing did not give norms a category of their own.

That remains an unresolved weakness.

Professional norms can shape behavior even when no law or financial reward requires it. Any account of AI governance that emphasizes formal mechanisms and economic pressures must leave room for these less formal influences.

The literature also made clear that many of the ideas I was bringing together had already been developed by others.

Domínguez Hernández and colleagues, writing in 2025, examined AI governance through policies, norms, institutions, incentives and power.

UNESCO had asked for governance that includes anticipation, monitoring and redress. Jack Stilgoe and colleagues had explored how innovation could be made more anticipatory, reflective, inclusive and responsive.

Lawrence Lessig had long argued that behavior is shaped not only by law but also by markets, social norms and technological architecture.

Kolt and colleagues had called for early intervention and institutions designed to adapt, and Aslam had defined effective governance as keeping the authority to question, correct and, when necessary, stop an AI system.

Seán Ó hÉigeartaigh and colleagues had separated ethics, which asks what is right, from governance, which asks how to make it happen in society, and Shin-Shin Hua and Haydn Belfield had described AI governance as both the study and the practice of systems that govern AI.

The question, then, was not whether these elements had been discovered before.

They had.

It was whether putting them together could offer a clearer way to examine the choices societies face.

What, then, should AI governance mean?

The different strands of this research point toward a common problem.

Rules matter, but their effectiveness depends on the incentives surrounding them.

Institutions matter, but their authority is limited by their expertise, resources and independence. Infrastructure matters because it determines which choices are available and who controls them. Safeguards matter because they turn general commitments into protections that people can actually use.

And all of these change over time.

An arrangement that is easy to establish today may be difficult to undo tomorrow. A decision that appears technical may gradually redistribute economic power or limit the choices available to public institutions.

I have come to think that these relationships belong nearer the center of how we understand AI governance.

Here is the definition I propose.

AI governance is the field of inquiry and practice concerned with how societies should shape the incentives, institutions, infrastructures and safeguards around AI to share its benefits broadly, prevent and address harm, and maintain the capacity to change course as today's choices become tomorrow's conditions.

A shorter version is easier to carry into ordinary conversation.

AI governance is how societies shape the incentives, institutions, infrastructures and safeguards around AI, so its benefits are broadly shared and its harms addressed while they are still choices.

The definition is not intended to replace the many established approaches to governing AI. It attempts to bring several of their concerns into a common view.

The four elements are also not independent boxes.

Think again about the health ministry. Its safeguards might require a clinician to approve every AI recommendation, but that safeguard could fail if the hospital rewards speed at the expense of careful review. The regulator might possess authority to investigate, but lack access to the necessary information. The ministry might have a contractual right to change suppliers, but find that the technical infrastructure makes doing so unaffordable.

A failure in one part of the system can undermine the others.

Power is present throughout.

It appears in who sets the terms of a contract, who can demand evidence, who controls essential infrastructure, whose objections receive attention, and who can secure a remedy when something goes wrong.

This also means governance itself can produce unequal consequences. Requiring expensive independent testing may protect the public while making it harder for smaller organizations to participate. A governance arrangement that gives one regulator broad authority may improve coordination while creating new concerns about accountability or political interference.

There is no reason to assume that more governance mechanisms always produce better outcomes. Their design and the circumstances in which they operate matter.

Nor does the proposed definition settle which outcomes societies should pursue. It makes a normative claim that benefits should be shared broadly, harms should be addressed, and people should retain the ability to correct decisions. Reasonable people can disagree about how these aims should be balanced.

That disagreement should be made visible rather than hidden inside a supposedly neutral definition.

The problem of acting while we still can

The final part of the definition, concerning today's choices becoming tomorrow's conditions, deserves special attention.

In 1980, David Collingridge described a dilemma that remains familiar in debates about emerging technologies.

Early in a technology's development, we often lack enough evidence to understand its consequences. By the time those consequences become clear, the technology may be so deeply embedded in institutions and everyday life that changing it is costly and difficult.

We face uncertainty when intervention is comparatively easy, and resistance to change when knowledge has improved.

Collingridge did not suggest that governments should simply predict every harm before allowing a technology to develop. His work emphasized the importance of preserving the ability to correct decisions and keeping future options open.

That distinction matters.

Calls to govern AI early can sound straightforward. But early intervention is not automatically wise. Governments may misunderstand a technology, impose inappropriate requirements, or protect established companies from competition.

Premature rules can create the very institutional rigidity they were intended to prevent.

Waiting has costs too.

Infrastructure can become concentrated.

Organizations can grow dependent on particular suppliers. Practices introduced as temporary experiments can become permanent features of public services.

The central challenge is to learn without surrendering the ability to act on what we learn.

In practice, that may involve procurement contracts that allow governments to change suppliers, safeguards that can be revised as evidence improves, independent evaluation, periodic review, or rules that expire unless their continued usefulness is demonstrated.

These approaches do not remove uncertainty.

They make it possible to respond to uncertainty without treating every initial decision as a permanent commitment.

The question applies to governance mechanisms themselves as much as to AI systems.

A regulatory framework introduced with good intentions can turn out to be ineffective, unnecessarily burdensome or poorly suited to new circumstances. Its designers should not assume that creating a rule completes the work of governing.

Good governance must remain open to correction, including correction of its own methods.

Who has a place in these decisions?

There is a further difficulty with the word societies in the proposed definition.

It can sound inclusive while leaving unanswered who actually makes decisions on society's behalf.

In 2024, the UN's High-level Advisory Body on AI found that 118 countries were parties to none of the seven prominent international AI governance initiatives it examined.

The finding does not mean those countries lacked AI policies, institutions or views about governance. It shows that participation in prominent international arrangements was uneven.

This matters because international AI governance is not only a conversation about preventing harm. It is also a conversation about who will shape the terms under which AI develops and spreads.

Countries that depend on foreign AI infrastructure may have little influence over the technical choices made by its suppliers. Those without strong research institutions may struggle to evaluate systems independently. And countries absent from international negotiations may eventually find themselves implementing standards they had limited opportunity to shape.

The difference between receiving the benefits of AI and participating in decisions about its future is substantial.

This is especially important for countries in Africa and other regions where the conversation about AI can become narrowly focused on adoption.

Access matters.

A farmer using AI-supported weather information, a teacher adapting learning materials, or a doctor receiving diagnostic assistance may experience genuine benefits.

But access alone does not determine who controls the technology, where its economic value accumulates, which languages and communities it serves well, or whether local institutions can challenge the terms of its use.

The question is not whether every country must build every component of an AI system. It is whether countries have enough knowledge, institutional authority and practical alternatives to make decisions consistent with their own priorities.

Similar questions arise within countries.

The health ministry may authorize the AI system, but clinicians and patients know things that may never appear in a procurement assessment. Their experiences can reveal problems that laboratory testing or administrative reporting misses.

A 2019 study by Ziad Obermeyer and colleagues provides an example of why this matters. The researchers examined a widely used US healthcare algorithm that relied on predicted medical spending as a substitute for healthcare need.

Because healthcare spending did not accurately reflect underlying need across racial groups, the system underestimated the needs of Black patients. Correcting the disparity would have increased the share of Black patients selected for additional help from 17.7% to 46.5%.

The problem was not simply that the algorithm made inaccurate predictions.

It was predicting a measurable outcome that did not adequately represent the human need the system was meant to address.

This is a reminder that technical performance cannot settle every question about what a system should do. Someone must decide which outcomes matter, whose circumstances are reflected in the evidence, and who can contest the consequences.

Those decisions are part of governance too.

A way to examine real decisions

The value of a definition becomes clearer when it helps someone facing an actual decision.

A public official reviewing an AI procurement proposal, a company assessing its deployment practices, or a community questioning the use of automated systems does not necessarily need a new theoretical vocabulary.

They need a way to notice what might otherwise be overlooked.

It helps to ask a few questions of any decision. What is it meant to improve, and what rewards shape the people involved? Who can investigate problems, and do they have the means to? What does the system depend on, and who controls it? Do the safeguards work in practice, not only on paper? Who gains, who bears the costs, and what options will remain if things change?

The same questions apply to a single purchase, to national legislation, to an organization's safety policy and to an international agreement.

They do not guarantee the correct decision.

Different institutions may reach different conclusions even after examining the same evidence, particularly when rights, resources and acceptable risks are contested.

They do, however, help locate the disagreement.

Instead of debating AI governance only in terms of whether there should be more or fewer rules, we can examine which decisions are being made, who has the power to make them, what those decisions depend on, and what happens when they turn out to be wrong.

The AIGovSpec framework develops this approach into eleven questions that can be applied to real decisions.

What remains unresolved

I began with a question about what we mean when we speak of governing AI.

The research has not convinced me that the field needs a single definition everyone must adopt.

Its different approaches reflect genuine differences in purpose. A company's AI management system, a national regulatory framework and an international agreement do not need to perform identical functions.

Nor is the definition proposed here beyond criticism.

It gives explicit attention to incentives, institutions, infrastructures and safeguards, but does not give social norms a category of their own. It emphasizes widely shared benefits without fully resolving how societies should address conflicting interests or extraordinary risks. And it does not, by itself, tell us how to balance innovation, security, rights and economic development.

Those questions require evidence, political judgment and legitimate institutions. A definition can help organize the inquiry, but it cannot make those decisions for us.

The proposed framework also needs more testing.

It must be applied to real cases and compared with other approaches to see whether it reveals problems, trade-offs or possible interventions that might otherwise be missed.

That is part of the continuing work at AIGovSpec.

What the research has changed is where I think the conversation should begin.

When we ask whether an AI system is governed, it is tempting to look for the visible evidence.

Is there a policy?

Has a regulator been appointed?

Are there standards and oversight procedures?

These are reasonable places to look, but they tell us only part of the story.

We also need to understand the pressures operating on those responsible for the system, the institutions capable of intervening, the infrastructure that shapes their choices, and the protections available to the people affected.

We need to know not only whether a decision can be made, but whether it can later be questioned and changed.

The health ministry that begins by deciding whether to buy an AI system may eventually find itself deciding whether it can afford to stop using it. What once looked like a choice has become part of the environment in which every subsequent choice must be made.

That possibility extends far beyond hospitals. It applies to the ways companies build AI, governments regulate it, countries depend on it, and societies gradually reorganize around it.

The most consequential governance failures may not be the rules we failed to write. They may be the choices we allowed to become irreversible before recognizing that they were choices at all.

Further reading

This essay draws on a longer research project published at AIGovSpec, including the historical record, the full discussion of the definition, and an eleven-question framework for examining real AI governance decisions.

For readers interested in exploring the wider questions of AI, power, judgment and society, CentPol's AI and Humanity is a five-week facilitated course.

No coding or policy background is required.

Research note

The historical review drew on a collection of 38 institutional and scholarly documents. The wider literature review used a structured search of OpenAlex conducted in October 2026. Initial screening and coding were AI-assisted and have not been independently validated, so reported frequencies should be read as exploratory observations from the sampled material, not definitive estimates of the field.