What would ‘rogue AI’ look like?
Organizations everywhere want to embrace artificial intelligence, or perhaps more accurately, they’re feeling pressure to embrace AI as quickly as they can.
That raises an important governance and risk management question: As you race forward with AI adoption, are you prepared to address the risks of your AI systems suddenly going haywire?
Such “rogue AI” could manifest in all sorts of ways, probably including many ways that haven’t even crossed human minds yet. Nevertheless, compliance, risk management, and internal audit teams need to think about this issue now – and try to define the governance, internal control, and disaster recovery processes you’ll need.
Let’s assume rogue AI does not mean that your AI becomes self-aware, decides humans are the true enemy, and sends a killer cyborg back in time to stop the leader of the resistance from ever being born.
Compliance and risk teams still have plenty of other tricky scenarios to consider, too. In a rough scale of severity:
- AI systems providing you with bad information
- AI systems making bad decisions for you
- Model drift or collapse, so your AI becomes useless
- AI agents misbehaving, sometimes known as acting beyond their “decision envelope”
- AI agents taking actions faster than humans can detect or prevent
- AI agents taking actions humans can’t undo
Right now, most organizations are probably stuck somewhere in the first three bullet points, worried that AI might give you bad answers or work from bad data to make bad judgments (on pricing or product recommendations to customers, for example).
More sophisticated organizations are already edging into the lower three bullet points: either using AI agents in your own corporate IT or allowing other AI agents onto your network. The challenge here is that barriers to people coding their own AI agents (including people who don’t fully understand what they’re building) are falling every day; that’s exactly what Anthropic’s new Claude coding tool allows.
Ultimately, businesses will need an ability to prevent any of the above bullet points, because they’ll all be risks that you face.
The fundamentals: risk tolerance, governance, and accountability
In theory, the first step to tame this risk is to define your tolerance for the risk. In practice that’s going to be hard to do, because right now nobody is fully clear on what the damage from rogue AI might be.
Consider an example from the anti-corruption world. Imagine an energy business that wants to expand into international markets, and management decides it has a high tolerance for corruption risk. That means you, the compliance team, pay less attention to anti-corruption compliance measures. Personally, I think that’s ill-advised, but companies make their own judgments.
But remember, we largely know the velocity of corruption violations and the damage they can cause. Enforcement happens over a period of years, and while it will cost you money, it won’t destroy your daily operations. So, defining a tolerance for corruption risk is a (relatively) straightforward exercise.
The risks of rogue AI are utterly different. Some might happen over a period of months, if the model malfunctions and feeds you bad analysis. Other risks might unfold in minutes if an AI agent decides to disable some critical system.
Risk tolerance might also depend on your industry sector. If you’re a small marketing analytics company, rogue AI might cause a privacy breach with your clients – unwelcome, but not necessarily the end of the world. At a large hospital system, however, rogue AI could kill patients, a disaster of the highest order.
The truth is that right now, the risks of rogue AI aren’t even fully known, never mind quantifiable. They unfold across too many types and at too many speeds. Companies will struggle to define what their risk tolerance should be.
One short-term solution is to strengthen your organization’s governance of AI adoption. That means employees define clear use-cases for AI, so that compliance, IT security, and other risk teams can evaluate the proper controls for that use-case. The risks of rogue AI would be one point of discussion.
Then comes the hard part: the company must assign accountability for the AI to specific people.
That’s a regulatory obligation ( Article 14 of the EU AI Act, for example) as much as it is common sense. Your organization must identify the roles and individuals who would be accountable for AI systems that malfunction. That’s what “human in the loop” means.
So, which humans? The IT department that installed the AI? The head of the business function using the AI system? The third-party vendor who provides an AI system? Who gets to decide that AI has gone rogue, anyway? The IT audit team? The low-level employee who says, “The AI is wrong again and this app stinks”?
We have no consensus for assigning accountability yet. Which underlines all the more the importance of moving deliberately, clarifying your risks and redefining roles and accountability as necessary. Rush forward at your peril.
Technical capabilities will matter
Assuming your organization solves the risk appetite, governance and accountability questions, it needs numerous technical capabilities, too. For example:
- The IT department would need processes to monitor model drift, anomaly detection (say, AI agents suddenly behaving strangely), or prompt-injection attacks meant to make AI systems go down a bad path
- If your business is in manufacturing or similar industrial sectors, you’d need access to sensor data (possibly many types of it) to see whether AI is changing the operational technology you use in some incorrect way
- If you rely on vendor technology, your third-party risk controls would need to extend to any AI systems they use as well. That might include some of the technical measures we mentioned above, as well as process-level controls around contracting, audits and service-level agreements
All businesses would need “red line” limits for what AI systems absolutely cannot do, and kill switches to make sure you can stop the activity. You’d need audit logs and disaster recovery plans, and those plans would need to include methods for reverting to human activity while you decipher why your AI went wrong and what to do next.
Most important: Human trust
It’s quite possible – likely, even – that many companies will encounter rogue AI scenarios they never contemplated. Your continuity plans, table top exercises, and emergency procedures might not fit the malfunction at hand.
That brings us to one final point: Your organization’s ability to plan in the midst of a crisis is more important than any specific plan you might already have.
So really, the most important element for success is human trust. You, other risk oversight functions, senior management, and the board will all need a shared vision for risk tolerance, risk management goals, and trust in each other that you’re working toward a common goal of keeping the business on the right path.
AI with Boundaries: Balancing Innovation and Compliance
Your teams are already using AI. This webinar gives you a practical framework to harness its potential – while managing the risks.
