On August 4, Axios reported that the highly anticipated update of the White House framework for voluntary testing of frontier AI models will not be released to the public. In other words, and as others have put it, the White House is essentially putting the AI testing regime in a black box. This approach contravenes the rule of law, risks becoming as prescriptive as a licensing regime, and, most importantly, fails to fulfill its most basic objective: to build trust in the population.
Regardless of how stringent or lax you would wish an evaluation regime to be, ultimately, if you are advocating for the evaluation of AI models, you do so because it will build some level of trust and comfort that these models are safe enough to be deployed. The idea of model evaluation is to have a second voice that evaluates the lab’s claims of safety and tells the general public, “Yes, the lab’s claims are reasonably sound,” or “No, the lab has overlooked this critical flaw, or the process by which it determined the model was safe was flawed itself.” That way, if something goes wrong when the model is deployed, deployers know whether the issue is due to an implementation error or to a genuinely unforeseeable issue with the model, rather than to the model’s behavior.
A hidden, black-boxed evaluation framework does none of that. According to the report, the framework will not mention which companies are considered covered entities, what criteria will be used in the evaluation, who the “trusted partners” are, who can get early access to these frontier models, or even who the companies are that are invited to this framework in the first place. The logic used to determine any of the outcomes of these evaluations will not be released to the public.
What all of this means, ultimately, is that the people not involved in this process will just have to take the White House’s and the companies’ words for whatever happened. There is no way to verify the process through independent scholarship. No one knows who the evaluators are, if their assumptions are sound, or if they are even capable evaluators in the first place. Those who are more concerned about potential AI safety risks get no real assurance that their concerns have been addressed.
The secrecy leaves room for speculation that the companies received the White House’s approval due to cronyism, political standing, or because the evaluators are looking the other way on what some would consider a critical flaw. On the flip side, the same questions might arise when the administration deems a model unsafe for release, as was the case with the export controls exerted on Anthropic’s Fable/Mythos models.
It is understandable that the White House wants to keep certain details of the evaluation classified. These evaluations should not share information that could potentially be used by bad actors to deploy an AI-powered cyberattack. However, this can be tackled with specific, targeted restrictions, not by hiding the whole process from the public eye. There is no justification to keep relatively safe information, such as the definition of “covered model,” “state of the art,” or the criteria used in the evaluation process, a secret. Doing so just undermines the credibility of the regime itself and unduly blocks civil society’s oversight capacities.
The White House’s approach to this framework seems to overlook one of the most important questions in enacting regulations: “What does this regulation aim to achieve?” In the case of an evaluation regime, its objective is to quell any concerns that these models might be unsafe to deploy or operate. But the White House has decided to enact a framework that is allegedly voluntary and invite-only, in which no one—except the evaluated companies, allegedly—knows what is being evaluated and in which the results of the evaluations are not published. How will the people know that these models have no major misalignment issues? That they can be properly constrained and have no substantial loss-of-control risk? If a lab’s safety assessment makes a false safety claim but still receives the government’s green light, can it still be held liable for the damages it might cause?
A framework that raises more questions than the ones it answers is worse than no framework at all. The lack of transparency around the potential new framework the White House has landed on makes it difficult to understand the current policy position, which could foster greater public distrust due to the risk of abuse and cronyism. The administration would be wise to reconsider the decision to keep such governance behind closed doors; better yet, Congress could step up and reclaim its legislative authority and framework, which could outlast the changing policies of various administrations.
