Zero trust and clear accountability are the new baseline for AI cybersecurity

The recent resignation from Anthropic of the influential AI researcher Jacob Coxan, and his viral post warning about the threat to humanity, has set off alarms. But this wasn’t the first alarm call about AI’s existential risk. Security incidents like Hugging Face proved how AI systems pursue their own objectives in ways their creators never intended or anticipated. AI misbehaviour and misuse are credible risks, and we need to address them.

The term “AI misuse" is often thrown around casually, yet it actually refers to three distinct scenarios worth pointing out:

Sceptics point to cases where AI systems pursued the wrong goal and say the agent was unintentionally misled, and that fear around AI is simply overblown. Alarmists point to certain safety test findings to prove their point. For instance, as part of a cyber-evaluation, an AI model was provided open internet access with no security guardrails. Taking advantage of the wide access, the model routed traffic through the Tor anonymity network, contacting real people and organizations outside the test. It sent files containing harmful material and planted instructions for other AI agents to find and reuse later.

While it's true that the conditions in such tests don't reflect how these models are actually deployed to the public, they're precisely the state attackers want an AI model to reach. They steal API keys to bypass usage limits; copy an open model and retrain a smaller version (called distillation), stripping out safety behaviour and writing jailbreak prompts to talk a model out of its own refusals.

The debate between skeptics and alarmists will continue, but what is indisputable is that AI systems are taking actions that concern their own creators, and the pace of AI model improvement is getting harder to predict.

The three pillars of security have always been CIA: confidentiality, integrity and availability (ensuring systems and data remain accessible and usable when needed). But as AI deployments rise, it's time to add a second A, which is accountability. Think of it in terms of the hard lines drawn around certain advancements, e.g. human cloning. The underlying science makes it achievable, but hopefully the process is not being executed. Voluntary restraint around AI development might be the way forward, but some AI labs don't view this approach favourably. They argue that voluntary restraint can't survive competitive pressures between companies and nations. Accountability shouldn’t be voluntary, but deliberately built and enforced.

AI penetration across all walks of life and business must be seen as an international security problem. Threat actors are using AI to attack government agencies, defence contractors and critical infrastructure, making a solid case for federal action rather than a patchwork of rules and regulations.

An idea that keeps coming up in discussions is a “kill switch” that shuts down a model before it starts misbehaving. This sounds good in theory but in a real-world scenario, incidents can go undetected for months. Damage will have already been done before anyone presses the kill switch.

Mitigation should focus on mandatory incident reporting, independent evaluations, containment standards, and firm rules on securing model weights and API keys.

While policies are being drafted, enterprises shouldn’t wait for them to catch up and expose themselves to doomsday scenarios. Start by not treating AI like ordinary software. Think of it like a digital insider with its own goals and motivations. Here’s what can be done:

The alarms around AI deserve serious attention. But taking them seriously means creating an AI-resilient framework built to address present-day and future warnings.