Adversarial AI threatens our financial services. We need a response.

In the interconnected financial services world, artificial intelligence (AI) isn’t just a tool for efficiency: it’s at the heart of innovation and growth. Yet, while AI enhances fraud detection, decision-making, and customer experience, it also creates new vulnerabilities.

Adversarial AI, the manipulation of AI systems to behave in unintended or harmful ways, is an emerging hidden threat – and one that the banking system is not ready for. To stay ahead of this fast-evolving problem, we need to update the cybersecurity regulatory frameworks in the financial sector, adding new protections against adversarial AI.

Adversarial AI

It’s no secret the world’s multi-national financial services (FS) organisations are increasingly using AI in various ways. We hear lots of hype about AI – some well-intended; some driven more by the marketing department than the tech itself. But either way, large FS institutions are demonstrating the ability to innovative at the pace of a FinTech start-up.

The Bank of England and the Financial Conduct Authority recently found that 75% of FS companies are already using AI, and another 10% plan to start using it in the next three years.

But they are still not up to speed when it comes to new security risks this will create. Chief among them is ‘adversarial AI’.

Adversarial AI is not just about breaking into systems – what you would call the traditional ‘hack’. (Of course, this is a problem too. But most existing IT security best practice should cover that). Adversarial AI is something more subtle: manipulating a company’s AI algorithms, or the data that feeds it, in order to influence its outputs.

This is a growing problem for any organisation that relies on AI outputs. In the world financial services – banks, insurance companies, fintechs – it is potentially catastrophic. By my reckoning, there are no fewer than five different threats for FS from adversarial AI.

AI hallucinations

One of the most interesting challenges of large language models (LLMs) is their ability to ‘hallucinate’. This is when they create outputs that look and sound credible but are factually incorrect. In finance, this isn’t just inconvenient; it’s dangerous for our economy.

Let me give an example. Imagine an AI-driven market analysis tool which advises on high-stakes investments. When prompted to assess a specific company, the AI confidently generates optimistic earning projections. But if those outputs are based on hallucinations that were intentionally created by a bad actor, those projections could contradict verified regulatory filings. This misstep could lead to substantial losses and damaged trust. We know malicious actors are actively looking to create or exploit hallucinations, as the cyber-security firm Vulcan recently found.

Data poisoning

AI is only as good as the data it learns from. Poison the data, and you poison the outputs too. Again, let’s take an example. Attackers could target publicly accessible or poorly monitored datasets, injecting carefully designed adversarial samples that skew model behaviour over time.  

Although there haven’t been many examples of it yet, there are clues from other sectors. Recently, artists who think their work is being used to train models without their permission have used a data poisoning tool called ‘nightshade’ to try to weaken AI image generating models.

It's not hard to imagine how this could look with FS. They increasingly rely on large AI models to detect fraud – having trained these models with millions of examples of both legitimate and fraudulent transactions. A malicious actor, hoping to bypass these systems, might seek to inject poisoned data during the model’s retraining cycle, convincing it that fraudulent transactions are legitimate.

With no provenance tracking built into the model, or real time verification of training datasets, the AI enabled fraud detection service.

Model theft

AI models are high value intellectual property: crown jewels to protect from theft. However, a technique known as ‘model extraction’ has been used to essentially reverse engineer AI models. This is where an attacker repeatedly queries a model – and uses the outputs it receives to re-create its own version.  By chaining these model extraction techniques, adversaries can replicate proprietary models, which could start to strip businesses of their commercial competitive edge.

It’s not uncommon to see API architecture vulnerabilities exploited in complex technical supply chains. Exposing an insecure API of an advanced trading algorithm by systematically querying the system, attackers could begin to reverse-engineer its behaviour, replicating its strategic insight to mirror the architecture.

Ethical bias

Ethical bias in AI is nothing new. I wrote recently about the ethical use of autonomous AI agents in defence and national security, which has many parallels in this sector. AI outputs are driven by the data they are fed with – and that can quite easily lead to unfair, unethical, or even immoral outputs. There are already examples AI generating unfair policing and housing benefit outcomes. (I’d recommend the book Weapons of Maths Destruction if you want to know more).

In the financial services world, AI outputs could lead to unfair outcomes in approvals, credit scoring, or fraud detection. Imagine a credit scoring AI agent multi-tasking with other AI agents in the flow of customer service, consistently flagged applicants from a specific demographic group as high-risk, despite identical financial profiles to other applicants.

AI supply chain attack

Not all FS institutions – even the big ones – have the resources and expertise to design their own in-house AI models. That’s why the modern AI infrastructure ecosystem depends heavily on third-party libraries, and what’s called ‘pre-trained’ models. But these can be hijacked, as bad actors replace legitimate packages with malicious versions, or insert dormant code set to remotely activate under specific conditions.

This type of attack is becoming more common: Kaspersky uncovered a year-long attack involving masked python interfaces for ChatGPT and Claude AI, which looked like real chatbots – but were secretly carrying malware that then infected company devices. 

Improving adversarial AI cyber attack resilience

Adversarial AI isn’t a future threat, it’s here now. As financial institutions push further into AI-driven innovation, understanding and mitigating these risks will become ever more important.

We already know there are things that work – and some of them are part of good overall IT security. For example, I would suggest anyone dealing with AI security should build a multi-disciplinary ‘AI Red Team’, to constantly test your own vulnerabilities to adversarial AI threats. Similar rules apply to all Red Team exercises: break the silo mentality, be vendor agnostic, and create a multi-disciplinary team, including technical security testers , data scientists, engineers, and IT architects.

There are also a few AI-specific approaches we know can work. For example, skilled synthetic adversarial threat modelling – essentially mimicking an attacker, to help work out your own weaknesses – can help to improve resilience, by exposing models to this type of adversarial AI attack. And greater emphasis will be needed to verify data quality which go into training the models.

Since 2014, Critical National Infrastructure Banking Supervision and Evaluation Testing (also known as CBEST) has been an important way the UK financial regulators assess the cyber resilience of FS organisations, and help keep the eco-system robust. I won’t go into all the details but CBEST, on the whole, has been pretty effective.

But it’s time to modernise CBEST to reflect this new type of threat. In 2025, the CBEST implementation guide should be updated, to include adversarial AI Red Teaming under what’s known as ‘Important Business Services’. One key element of CBEST at the moment is penetration testing, where the regulators “mimic the actions of cyber attackers, intent on compromising an organisation’s important business services and disrupting the technology assets, people and processes supporting those services”. Vulnerabilities can then be identified, fixed, and learning shared. At the moment, that doesn’t include adversarial AI attacks – and it should.

This would also mean adding additional service providers with AI Machine Learning expertise to the current CBEST Service Provider terms of reference. CBEST certified individuals should also have adversarial Red Teaming as part of their skillset.

An updated CBEST can create a model for the whole sector to use – which really matters because this is a systemic threat that transcends any single firm.  I fully expect – and support – the continued widespread adoption of AI in our financial services. But for that to work, we our adoption of best practice AI safety must evolve at the same rate.

Find out how our cyber security training programmes can help protect your organisation from AI threats