Coded Consequences: Why AI Governance Matters, How It Fails, and Why Your Future Depends on It.

When a system helps rank applicants, allocate support or inform a consequential decision, its errors become somebody else's problem. Who chose the objective? Who checked the data? Who can challenge the outcome? Those are governance questions, even when the decision arrives dressed as a technical recommendation.

AI governance is the arrangement of responsibilities, safeguards and oversight around an AI system. A useful starting point is NIST's voluntary AI Risk Management Framework, which organises work around governing, mapping, measuring and managing risk. It is a framework for organising action, not proof that an organisation's system is trustworthy.¹

The distinction matters. A committee can exist without effective oversight. A model can pass a test without being appropriate for the task. A policy can promise human review while giving the reviewer neither time nor authority to disagree. My central argument is that governance must be judged by what people can actually do when something goes wrong.

Four levels of responsibility

The following four levels are an analytical lens for understanding governance, not an official classification. They help connect international choices with the individual decisions hidden inside a model.

1. Global and strategic governance

At the strategic level, societies decide which uses of AI to encourage, constrain or reject. International cooperation matters because developers, users and affected communities can be located in different jurisdictions. Shared principles can help, but they do not eliminate disagreements about surveillance, security, individual rights or economic competition.

The European Union's AI Act illustrates a risk-based legal approach. Its framework distinguishes prohibited practices, high-risk uses and other obligations, including rules for general-purpose models. The Commission's current overview also describes amendments and differing application dates. It is important to distinguish a policy's existence from the date and scope of a particular enforceable duty.²

For leaders, the strategic choice is therefore more specific than being for or against AI. What public interest does a proposed use serve? What limits should survive commercial pressure? Who gets a say before the deployment becomes difficult to reverse? Those are questions this article recommends asking; they are not a claim that every jurisdiction answers them the same way.

2. Institutional and regulatory governance

Institutions turn commitments into authority. Inside an organisation, somebody must be able to approve deployment, insist on additional testing, allocate resources for remediation and stop a system that exceeds its mandate. A governance body without those powers risks becoming an advisory layer that nobody has to heed.

A concrete regulatory example is New York City's Local Law 144. For covered automated employment decision tools, it requires a bias audit within the preceding year, public information about the audit and specified notices. The city's enforcement began on July 5, 2023. This is a defined requirement for covered tools, not a rule that every hiring-related algorithm must undergo the same audit.³

An audit report should also be read for what it tested. Meeting a procedural requirement does not, by itself, establish that every affected person was treated fairly. My recommendation is to ask who commissioned the assessment, which outcomes and groups it examined, and what action the organisation took on its findings.

3. Operational and technical governance

This is where responsibility enters the daily workflow. Data selection, model validation, access controls, monitoring and incident response all need identifiable owners. For a hypothetical organisation considering a decision-support tool, I would require a written account of its intended use, permitted inputs, known limitations and escalation route before approving a pilot.

That pilot should be bounded. Define which decisions the tool may assist, which decisions remain outside its scope and who can override it. Give reviewers enough information to challenge an output. Record errors in a form that can be investigated, rather than asking users to send complaints into an anonymous mailbox.

These are proposed safeguards, not a claim that they guarantee a good outcome. Their value is that they make failures easier to identify and responsibilities harder to evade. A documented stop condition is more useful than a broad promise to monitor responsibly.

4. Ethical and democratic governance

Technical objectives embody choices about what counts as success. Predicting spending, predicting illness and identifying people who would benefit from additional support are different objectives. Selecting one because it is easy to measure can move the system away from the human purpose it was supposed to serve.

The question of fairness cannot be outsourced to a single metric. People can reasonably disagree about which outcomes matter and which burdens are acceptable. That does not make evidence optional. It makes the assumptions behind an objective, the people included in deliberation and the consequences of a decision especially important.

My recommendation is to involve affected people early enough to change the design. Consultation after deployment should not be used to imply that the community approved choices it never had a realistic opportunity to influence.

What the documented failures show

In 2018, Reuters reported that Amazon abandoned an experimental recruiting project whose rankings exhibited bias against women. Recruiters had looked at its recommendations but did not rely solely on them. This was a development and evaluation failure; the reporting does not establish a company-wide autonomous hiring system or a count of people denied jobs by it.⁴

The lesson is about the objective and the evidence used to train it. Patterns in historical applications can reflect existing inequalities. Learning those patterns does not establish that the resulting ranking is appropriate. Nor should we assume that every failure arose because nobody conducted any oversight: discovering and abandoning a defective prototype is itself different from knowingly deploying it unchecked.

A 2019 study by Obermeyer and colleagues provides another precise example. At a given risk score, Black patients were sicker than White patients because the studied algorithm predicted healthcare costs rather than illness. The authors estimated that remedying the disparity would increase the proportion of Black patients receiving additional help from 17.7% to 46.5%.⁵

That finding is powerful without exaggeration. It is not a measured death toll or proof that every healthcare algorithm fails identically. The study demonstrates how an apparently useful proxy can reproduce inequality. The important governance question is whether the organisation checks that the target it can predict is the outcome it actually wants to improve.

What better governance would look like

Consider a hypothetical lender assessing a new decision-support model. The institution should first state the decision the tool will influence and the evidence needed to justify that use. It should test errors and outcomes across relevant groups, document exceptions and assign responsibility for complaints and remediation. This is an implementation example, not a reported success story about an unnamed bank.

Similarly, documenting how information enters a recommendation system can help an organisation investigate mistakes and assess its own data practices. It does not establish, without further evidence, that customers became more trusting or that the company gained a competitive advantage. Governance claims deserve the same scrutiny as performance claims.

For any consequential system, I would ask for a decision record: what was approved, by whom, on which evidence and subject to which conditions? I would then ask who has authority to reconsider that approval. If nobody can answer, the organisation has not yet made accountability concrete.

What this means for you

If an automated recommendation affects you, ask what role the system played and who is responsible for the final decision. Request information about the evidence used, the available review process and how to report an error. Keep a record of the explanation you receive and the route offered for contesting it.

Specific rights depend on jurisdiction, sector and circumstances. This article does not promise a universal right to inspect a model or receive a complete technical explanation. Even where a particular legal entitlement is unclear, an organisation should be able to explain its own process and identify a responsible human contact.

The work continues after approval

Governance becomes meaningful when an institution can learn from a failure and act on it. That requires more than an ethical statement. It requires people with authority, evidence that bears on the actual use, and a route by which affected people can be heard.

AI can influence important decisions. The responsibility for those decisions remains with people and institutions. The question to keep asking is simple: when the system gets it wrong, who can recognise the error, who can correct it and who answers to the person who carries the consequences?

Sources and notes

1. NIST, Artificial Intelligence Risk Management Framework, AI RMF 1.0 (January 2023), especially the core functions. A voluntary framework; not a compliance certificate. ↩
2. European Commission, AI Act overview. Current risk framework, amendment information and application timeline; specific duties require consultation of the applicable legal text. ↩
3. New York City DCWP, Automated Employment Decision Tools. Local Law 144 requirements and enforcement date. ↩
4. Jeffrey Dastin, Reuters, Insight: Amazon scraps secret AI recruiting tool that showed bias against women (October 9, 2018). Original reporting on the experimental project; no unsupported victim count is inferred. ↩
5. Obermeyer et al., Dissecting racial bias in an algorithm used to manage the health of populations, Science 366 (2019), 447–453, DOI 10.1126/science.aax2342. The percentage change is the study estimate, not a general population count. ↩