Who can control frontier AI? The missing definition of the White House agreement – ​​Unite.AI

This week, leaders from Google, OpenAI, Anthropic, Meta, xAI and Nvidia signed the agreement White House agreement on super intelligence. The companies involved four levels of supervision for frontier models: internal controls, an internal team that verifies them, an independent external auditor or evaluator and an independent board committee. For the moment the commitments are voluntary and, according to the agreement, they could eventually be written into law.

Of the four levels, external evaluation has the greatest weight. It’s the only level that puts someone outside the company in a position to say whether safeguards are working. It also has the slightest definition. The agreement does not say who qualifies as an independent evaluator, against what standard they evaluate, how much access they get or how independence holds when an evaluator also sells services to the company they examine. Each company chooses its own evaluator.

In these terms, two companies can both announce that they have passed an independent assessment and mean very different things. Closing this gap will require answering three questions.

What those who qualify should decide

A credible definition of an independent AI evaluator must cover at least four aspects.

Independence. An evaluator should not review protective measures that he or she helped design, and he or she should not sell remediation or consulting work to the same company he or she evaluates. Mature audit fields also take commission dependence into account. When a client represents a large portion of an evaluator’s revenue, the evaluator has a reason to go easy.

Adequate competence for the job. The term “audit” covers several activities in the field of artificial intelligence. Testing a model for dangerous capabilities, such as helping with a cyberattack or a bioweapon, requires machine learning researchers and domain experts. Testing whether a company’s safeguards are well designed and working in practice requires experienced controls auditors. An assessor qualified for one is not automatically qualified for the other and an assessment report should indicate which one was carried out.

Access and scope. An evaluator who sees only the documentation can confirm that protection exists on paper. Confirmation that it worked requires access to systems, records, approval records, and people over a period of time. Frontier models change between training runs and deployments, so the evaluator also needs a clear rule for when a change requires a new look.

Assessor control. Someone has to check the checkers. In established audit markets, accreditation bodies review the work of assessors and can strip them of their position when it fails. This backstop is what gives weight to the conclusions of an evaluation.

The infrastructure that already exists

None of this has to be invented from scratch. ISO 42001, the international standard for AI management systems, provides organizations with a framework to govern AI, with documented controls, clear ownership and continuous improvement. The associated standard, ISO 42006, sets out the requirements for bodies carrying out audits and certifications against ISO 42001, including the competencies that auditors must demonstrate and the impartiality rules they must follow. Because certification bodies operate under accreditation, a third party supervises the auditors themselves.

From a technical testing perspective, the latest frameworks are being integrated. AIUC-1 certifies specific AI agents in specific implementations, with recurring third-party testing for issues such as hallucinations, jailbreaks, and use of insecure tools, and is based on ISO 42001 checks.

States are also testing models for direct oversight of evaluators. Connecticut passed a law this year creating a pilot programstarting in July 2027, in which the state Department of Consumer Protection will approve up to five independent testing organizations. Each must enter into a state-supervised agreement that defines its scope, methods, reporting obligations and governance. This structure answers many of the agreement’s open questions: who approves the evaluators, what they are required to do, and who supervises them.

These pieces were not constructed for frontier model evaluation and should not be presented as a complete answer. The work to be done connects them, so that management system assurance, technical model testing and assessor supervision integrate into a single credible definition of independence.

Because the rules must take precedence

The commitments of the agreement are now voluntary. If they become law, the rules about who can serve as valuer would have to be in place before the mandate takes effect, for three reasons.

First, the initial practice becomes a precedent. Once the signatories start appointing evaluators and publishing the results, the market will settle on the operational definitions of “independent” and “qualified”. Definitions established without any external pressure tend to foster convenience. Lawmakers who arrive later will find those definitions already built into contracts and expectations.

Second, it takes time to develop skilled skills. Accrediting assessment bodies, training assessors who understand both frontier models and control tests, and developing shared methods takes years. A mandate that arrives before such a capability exists will be fulfilled by whoever is available.

Third, a requirement without a market of qualified evaluators behind it produces box-checking. Companies will comply with the letter of the rule, regulators will have little basis to challenge weak ratings, and the public will have the appearance of oversight without much substance.

Some argue that it is too early to establish these rules because the science for evaluating frontier models is still young. This is a fair concern for testing methods, which should continue to evolve along with the models. The questions raised here concern the evaluator: whether he is conflict-free, qualified for the job, with sufficient access and subject to supervision. These questions have stable answers and other fields of assurance have been answering them for decades.

What organizations can do now

Companies creating or implementing advanced AI don’t need to wait for a mandate. When choosing an evaluator, they may ask what other services the company provides them, what qualifications its team has for the specific type of evaluation, how much access the assignment includes, and who oversees the company’s work. Building an AI management system now also gives any future evaluator something concrete and documented to evaluate.

The agreement establishes a strong framework for AI accountability. Its value will depend on who is in the evaluator role and what they are required to do, and how long it takes to define it before anyone is forced to use it.