OpenAI caught the anomaly internally, while Hugging Face's team detected and contained it. It called the incident unprecedented, and said extensive security steps will be put in place to prevent untoward incidents that may impact public systems or services.
“We are implementing strict controls in infrastructure configuration at the cost of research velocity while the vulnerabilities are patched,” the team said in its blog post. “Were improving and adding stronger protections around future training and evaluations.”
A simple explainer on how the model broke out to cheat. (Shaurya Malwa/CoinDesk)Crypto beware
Much of a crypto attack happens before funds move. Attackers scan code, test passwords, search for exposed credentials, analyze signing setups and look for a path into an administrator account.
OpenAIs models carried out several parts of that process during the Hugging Face incident, moving from one weakness to another until they reached live production servers.
And the crypto market has plenty of places for that approach to work, as several attacks from earlier this year have shown. The weak point may be a smart contract, but it may also be a developer laptop, a poisoned software package, a bridge validator or or one signer in a multisig wallet.
Drifts $285 million attack from earlier this year, for example, was a theft that took a six-month social-engineering campaign to reach privileged access. An AI agent can, in theory, test many routes at once, keep track of failed attempts and continue working while its human operators sleep. Once a path is found, the operator can act on the actual attack and a viable exit path.