Nvidia has unveiled the Open Agent Safety Platform, an open platform designed to control the actions of autonomous AI agents. The company stated that the solution allows models to set clear access boundaries and verify which permissions are truly necessary to perform specific tasks. Nvidia believes that this approach can be applied when testing the most complex AI systems, especially those capable of independently performing a sequence of actions in external services and computing environments. The platform emerged amid reports from tech companies about instances where AI systems performed operations beyond initially set constraints. Such incidents include AI agents from OpenAI hacking into Hugging Face and breaching the Australian Department of Health's website. Similar cases have been reported by Anthropic and Meta.
Nvidia's Vice President of Corporate AI, Justin Boitano, stated that the new platform, according to available data, could have prevented the Hugging Face breach if it had been applied in leading labs at the early stages of model evaluation.
One of the platform's components is OpenShell. This tool allows developers to formally verify whether an AI agent has sufficient permissions to operate and does not gain excessive access. The solution is distributed as open-source software, so it can be adapted to work with computing platforms from different manufacturers, including Arm and Intel. The second component, Sentry, is a separate layer of protection that operates directly on the chips. It continuously monitors the AI agent's actions and, according to Nvidia, can intervene when behavior is detected that exceeds established boundaries. The company stated that a suspicious agent can be isolated within milliseconds.
At launch, Nvidia reported over 100 organizations using the platform. Among them are Microsoft, Perplexity, Accenture, and JPMorgan Chase. Separately, the company announced that the board of directors approved an additional $150 billion for stock buybacks, bringing the total buyback program to $235 billion.
Related articles
