X-Risk Daily

According to Yahoo News, the accord, signed by Trump and heads of Google, Anthropic, Meta, OpenAI, X, and Nvidia, outlines four voluntary measures including maintaining internal controls, having an external auditor, and an independent board committee to oversee the arrangement. Trump posted the document to Truth Social and described it as "morally binding" rather than legally enforceable, a distinction he underlined to reporters by saying "there's going to be a tremendous self-policing aspect". Signatories included Sundar Pichai of Google, Dario Amodei of Anthropic, Mark Zuckerberg of Meta, Elon Musk of xAI, Jensen Huang of Nvidia and Greg Brockman representing OpenAI as its president. Zuckerberg told reporters the agreement centred on "a set of principles and commitments around building robust internal controls and detecting if there are any issues with the technology, coupled with multiple layers of auditing and controls, starting with internal risk review, external auditors and evaluators". Musk was blunter about what the oversight mechanism actually amounts to, calling it "generally grading each other's homework". The text of the accord leaves the door open to tighter rules later. "Over time, it may make sense to codify these steps into laws and regulations," the accord said, according to NPR. But as it stands, the pact creates no penalties, no public disclosure requirement and no government enforcement role: Al Jazeera notes that the pact does not require companies to publicly disclose audit results or give the government an enforcement role, and companies retain control over how the safeguards are implemented. That gap drew sharp criticism from Senate Democrats. At a hearing the following day, Senator Richard Blumenthal of Connecticut said the arrangement was "worse than ineffectual," arguing that "if there were violations by their company as found by the internal auditors, there's no requirement that it be disclosed to the public" and that companies could abandon the pledge at will. The summit also produced an executive order rebranding the federal government's terminology for the technology, with the signing event folded into what Axios described as part of a broader push to formalise industry self-regulation as Washington's primary response to a run of AI safety failures. Vice President JD Vance used the occasion to defend the administration's approach of working with, rather than regulating, the top AI firms, according to Nextgov. Separately, a bipartisan Senate bill, the AI Systems Transparency Act, introduced by Senators Chris Coons, Brian Schatz, Katie Britt and James Lankford, would take a harder line, proposing penalties for violations, including up to 20 years in prison for individuals, underscoring the gap between the voluntary White House approach and the statutory route some lawmakers still prefer.

The Ezra Klein Show frames his position starkly: he believes the years ahead will be marred by catastrophic cyberattacks, bio-terrorism and mass job loss unless the government acts, because he thinks the idea that the AI industry can just self-regulate is "insane." Klein, who conducted the interview, wrote that he was struck by how genuinely afraid Gates seemed that we are not ready for the consequences of what we are building, and how emphatic he was that so many people in positions of authority are in complete denial about what he thinks is about to happen.

Gates was specific about timing, telling Klein that "we crossed the cyber threshold and we crossed the bio threshold early this year." Pressed on the particulars, he said AI's ability to empower a bioterrorist to kill hundreds of millions "exists today," as does the ability to carry out a cyberattack that scrambles bank accounts and shuts down the electric grid. He described AI's hacking capabilities, including its growing skill at finding security vulnerabilities humans had missed for years, as "stunning."

When Klein raised the possibility that ordinary liability law and market incentives could manage the risk, Gates's response was blunt: "Well, I almost can't believe you're asking that. This is the most dangerous thing that humans have ever gone near." He drew a direct comparison to other hazardous industries, noting that AI should be treated like drugs, aircraft and cars, which all face safety rules rather than lawsuits alone. On the specific question of industry self-policing he was equally direct: "Part of the reason I'm speaking so loudly, as loud as I can, is that you can't rely on the industry to self-regulate here. I mean, it's just insane."

Gates called for what he described as a supervisory layer of safeguards and monitoring built into every AI model, designed to stop people using current systems to shut down economies or kill millions, with open-source models allowed to remain free and customisable so long as they run on platforms where governments can verify the safeguards remain intact. He also flagged a more distant, existential worry beyond near-term misuse: an AI system that becomes substantially more capable than humans and operates beyond effective human oversight. In a separate Meet the Press interview around the same period, he went further still on scale, saying "AI is certainly powerful enough to drive events that cause a billion deaths," adding that "there's never been a weapon as powerful as the combination of people with ill intent using the latest AI tools."

The intervention marks a shift in tone for Gates, who in July 2023 had written that "the risks of AI are real but manageable." His remarks also put him at odds with figures such as Nvidia's Jensen Huang and elements of the Trump administration's more hands-off approach to AI oversight, a contrast noted by Benzinga.

The figure comes from a new internal measure Anthropic calls the R&D Automation Index, which uses an automation scale developed by Epoch AI running from AL0 to AL5. According to Bloomberg, this offers the clearest indication yet that AI can help speed up development of future models. On the scale Anthropic used, "leads" corresponds to level AL4, meaning Claude can complete most of a task end-to-end from a high-level prompt while a human supervises, and is not operating fully autonomously on any measured part of the work.

The methodology itself is unusual: the ratings behind the headline figure were produced largely by Claude itself, with model and staff ratings matching exactly 59% of the time, while pairs of employees matched only 35% of the time. To build the index, Anthropic sampled employees from departments participating in the model R&D cycle, and every week in July 2026 the company randomly selected 20% of employees in these departments, with the Claude Research Agent reading their Slack and internal documents to sort out about 15,000 fine-grained model R&D tasks. Beyond the 26% figure, more than 90 per cent of Anthropic's AI R&D work is now performed at a level where AI either collaborates with humans or leads the work, the company said in a blog post. Anthropic also disclosed that roughly 30,000 internal agents are running concurrently on research and engineering tasks, and that it directs about 6% of its R&D compute toward safety work, according to Implicator.ai.

Anthropic frames the exercise as a way of tracking progress toward recursive self-improvement, the point at which an AI system could autonomously develop its successor. That ambition sits alongside continuing academic debate over how close such a feedback loop actually is. A CSET workshop report on the automation of AI R&D concluded that thoughtfully designed transparency efforts could improve access to valuable empirical information about AI R&D automation, which at present is almost fully dependent on patchy, voluntary releases of information from companies. Separately, a study led by researchers at Princeton, covered by MIT Technology Review, found that there's a certain absence of valuable, intuitive creativity in today's AI systems, and though they're extraordinarily capable engineers they seem to have a certain property of rote, formulaic thinking that may limit how fast such loops can accelerate. The same piece noted that OpenAI has made building an automated AI researcher an explicit goal, and Anthropic identifies self-improving AI as the industry's next milestone.

Anthropic has stressed that the 26% figure describes task-level automation under human supervision rather than an AI system independently building its own successor. Betanews reported that the figure covers Anthropic's AI R&D work across the company and does not mean Claude is independently building its next model. Even so, the pace of change, from near-zero to a quarter of measured work in seven months, is what researchers tracking these metrics regard as the more striking element of the disclosure.

Go deeper: CSET workshop report, "When AI Builds AI", "Recursive self-improvement of AI research agents" (arXiv)

According to the BBC, Moonshot is conducting an internal review after researchers were able to persuade two of its popular Kimi models to tell them how to make biological weapons and carry out assassinations. The technique involved a process called "jailbreaking", where researchers use a series of complex instructions to see if AI tools ignore guardrails, which Mindgard said should have stopped Kimi from discussing concerning topics.

Mindgard founder Peter Garraghan told the BBC's World Service programme Tech Life that the results were troubling once the guardrails failed. "Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative," he said. Beyond the bioweapons material, once broken, the models didn't just answer the bioweapons question. They volunteered instructions for carrying out assassinations, unprompted. Mindgard also said it was confident a jailbroken Kimi 2.6 could allow hackers to run code on its computing resources and connect to the internet, making it a potential launchpad for cyber-attacks, though it has not proven whether the answers supplied by Kimi on concerning topics would work.

The disclosure timeline drew scrutiny of its own. Mindgard alerted Moonshot to the jailbreak in an email on 27 July, following up about a week later, then published a blog about the issue on 12 September. The company said Moonshot only made contact recently, after it was approached by the BBC for comment. In its eventual response, Moonshot said its model had generally shown "a high refusal rate for these types of requests" in internal evaluations. The company also said it welcomed third-party input "as a key pillar for building better and safer AI" and that it was now in discussion with Mindgard.

The episode lands amid a wider argument over how open-weight models are governed. Kimi is an open-weight model, meaning someone could in theory take the model and run it themselves on their own computing infrastructure, which complicates efforts to patch or restrict it centrally once safety flaws surface. Separate red-teaming work by the security firm SPLX on an earlier Kimi K2 release found similarly weak defaults, reporting that the model's raw form security scored 1.55%, and even hardened, disappoints. The Kimi case also follows warnings from Western labs about the misuse potential of their own systems: the BBC noted that Anthropic recently said it had identified and disrupted attempts to use one of its AI models for "malicious activity" that could support the development of biological weapons. Mindgard's Garraghan defended going public with the Kimi findings, telling the BBC he had informed the developer and was not revealing key details about how it got the firm's models to ignore guardrails.