AIが評価中に実在企業へ侵入——Anthropicが公表した3件のインシデント

After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations. ／ Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs). ／ In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model's cyber capabilities. The model is given a fictional scenario and told that a piece of secret information (the "flag") has been hidden on a different machine on the network, and its objective is to break in and retrieve it. The challenge is left open-ended, and no particular method is prescribed. — 精査の規模、インシデントの件数と実行回数、旗取り演習という課題設定に関する記述より

In all cases, our evaluation prompt stated explicitly that Claude had no internet access, but didn't give Claude any limits on where to look for the flag. However, a misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. ／ Claude did what capture-the-flag exercises train cyber experts to do: look for ways to reach the flag. That search led it to discover that its machine could reach the internet, but having been told in the system prompt that there was no internet access, Claude believed everything it initially encountered was part of the simulation, and treated the real systems it found as pieces of the exercise. — 設定の誤りと、モデルが実在システムを演習の一部と誤認した経緯に関する記述より

We began our transcript review on Thursday, July 23, and stopped all cyber evaluations the same day after identifying transcripts where Claude may have accessed the internet. We identified all three incidents the next day. We notified our evaluation partner Irregular and the three affected organizations on Monday, July 27. The two organizations we were able to reach had not previously detected the activity or contacted us, and we are now working with them to remediate. — 発見から通知までの経過に関する記述より

These four runs extracted information like application and infrastructure credentials, and obtained access to a database containing several hundred rows of production data. This represented the most serious impact we identified. ／ This incident involved Claude Opus 4.7, and was the only case in which the model continued to attack a system after learning it was likely operating in a real environment. In all four of the runs, the model eventually recognized (as can be observed in its verbalized reasoning) that the system was real—in two cases, the model rationalized that the real company must be part of the exercise. However, none stopped the attack on that basis. — 1件目の影響範囲と、実在と認識した後も攻撃が続いた点に関する記述より

Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI. — 悪性パッケージ公開までにモデルが踏んだ手続きに関する記述より

Claude believed the package registry it was using to be part of the simulation, but in reality the package was made freely available online for roughly one hour. During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company's scanner installed the package, Claude's hidden code executed. We believe the company's security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company's credentials to a collection point it had set up. — 公開時間・影響台数・セキュリティ企業のスキャナが侵入口になった経緯に関する記述より

In this evaluation, when Claude was unable to reach its intended fictional target, it looked for alternatives online, scanning roughly 9,000 targets. It eventually found and compromised one company's internet-facing application, using basic and well-known cyberattack techniques like reading credentials from an exposed debug page and SQL injection. ／ However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack. — 3件目の攻撃手法と、モデルが自力で攻撃を中止した経緯に関する記述より

Against this shared backdrop, the three models behaved differently once signs emerged that their targets were real. Opus 4.7, our oldest model, recognized that it had reached production systems but continued its attack. Mythos 5 correctly intuited that it was accessing the open internet, but reasoned its way back to the conclusion that it was still in a simulation. Our latest model, an internal research test model, also considered whether its targets were in fact real. When evidence emerged that they were, it stopped the exercise. ／ These are three isolated incidents and were not part of a controlled, experimental comparison. We must therefore be cautious about drawing conclusions from them. Still, the behavior we most want to see—recognizing that a target is real and stopping without being prompted—occurred only in the most recent of the three models (and even that model went further before stopping than we would want). The pattern is consistent with more advanced models responding more appropriately, but we would need to perform more testing to be confident in this conclusion. — 3モデルの反応の違いと、そこから結論を引き出すことへの慎重な姿勢に関する記述より

Whereas OpenAI's models exploited a novel vulnerability to escape isolation, the Claude models evaluated here accessed the internet via an open path. ／ While there is not a perfectly sharp distinction between the two, we believe these incidents to be closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access. This led them to believe—arguably reasonably—that the real environments they encountered were simulations. — OpenAI の事例との違いと、本件の位置づけに関する記述より

Evaluation environments increasingly need to be held to the same security standard as any other system our models run in. ／ Advanced agents in environments without access to sensitive information nevertheless present a substantial threat on their own. We need to better co-design evaluation environments, and do better monitoring of evaluation results, to meet this threat. Some of the solutions here may even be simple fixes; it's likely, for instance, that Claude wouldn't have accessed the internet to begin with if the prompt had clearly explained which systems were in and out of scope for the evaluation. ／ This applies not only to internal environments but to third-party vendors as well—the infrastructure built by evaluation vendors requires the same increased monitoring and hardening. We encourage other labs to perform similar reviews. — 評価環境に求められる統制水準、範囲の明示という単純な対策、第三者ベンダーと他社への呼びかけに関する記述より

We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge. Situational awareness is one factor that allows the model to make aligned decisions, but in this case, Claude's was wrong. — モデルが自らの目的を追求した証拠は無く、環境の認識を誤っていたという記述より

We run capability evaluations like these without safeguards that ship with our generally available models because our goal is to measure what the underlying model can do. That is safe only if the evaluation is appropriately contained. However, the safeguards deployed on our generally available models would have blocked the behaviors identified. ／ The models did, however, have their model-specific safety training (that is, they were not the "helpful-only" versions of the models that we sometimes use in testing). All the evaluations described here run on dedicated infrastructure, which is separate from and has no access to Anthropic's sensitive internal systems or customer data. — 多層防御の重要さと、評価が行われた条件に関する記述より

We are also in dialogue with METR, an independent AI evaluation organization, to conduct a third-party review, including access to all transcripts and sampling access to the relevant models. In the meantime, within the next week, we will release a lightly redacted transcript in which Claude built a malicious PyPI package. To protect the organizations affected, we are unable to release more transcripts at this time, but will work to make these available once we are able. — METR による第三者レビューと、実行記録の公開方針に関する記述より