School of Mathematics

Des Higham has been awarded an EPSRC New Horizons grant entitled Mathematics of Adversarial Attacks

Des Higham has been awarded an EPSRC New Horizons grant entitled Mathematics of Adversarial Attacks. The grant is built on two observations:

  1. Empirical experiments have shown that even the most sophisticated and highly-regarded artificial intelligence (AI) tools can be fooled by carefully constructed examples. For example, given a picture of a dog, we can change the picture in a way that is imperceptible to the human eye but makes the AI system change its mind and categorize the picture as a chicken. Such *adversarial attacks* can be shockingly successful, and they clearly have implications for safety, security and ethics.
  2. Although many mathematical scientists are contributing to the exciting and fast-moving body of research in AI and deep learning, the main theoretical focus so far has been on approximation power (can we build systems that satisfy a desired list of properties?) and optimization (what is the best way to fine-tune the network details?).

There is an urgent, unmet need for actionable understanding around adversarial attacks: are they inevitable, are they identifiable, and are they generalizable to other forms of attack?

This motivates the themes of the grant: Inevitability, Identifiability, and Escalation.

Here are three examples of the types of questions that we will address: A) Is it inevitable that any AI system will be susceptible to adversarial attack (in which case we should assign resources to identifying attacks rather than attempting to eliminate them)? B) Typical modern AI hardware is fast but has low accuracy (e.g., each computation may carry only 3 digits); can such imprecision be exploited by new forms of adversarial attack? C) How secure are AI systems to malicious interventions that, rather than attacking the input data, make covert alterations to the parameters in the system?

We will, for the first time, develop and extend highly relevant ideas from the field of mathematics (numerical analysis and approximation theory) to produce concepts and tools that allow us to appreciate fundamental limitations of AI technology, and identify when these limitations are being exposed; thereby contributing to issues of security, interpretability and accountability.

Overall, the ideas emerging from this project will transform our understanding of AI systems by using currently overlooked techniques from computational mathematics. Furthermore, by showing that there are challenges at the heart of AI that can be tackled by computational and applied mathematicians, we plan to transform the scale and quality of research interaction at this important mathematics-computer science interface.