UTOPIA (Understanding Threats and Optimizing mitigation, and enabling Privacy-by-design in AI) is a project funded by the Research Council of Norway. The project investigates how privacy and security threats affect AI systems, how such threats can be systematically evaluated, and how privacy-enhancing technologies can be designed and combined to support trustworthy AI. The project brings together expertise in AI, cybersecurity, privacy, and health-related applications to develop methods, tools, and evidence for more secure and privacy-preserving AI systems.

Background

Artificial intelligence is increasingly used in areas such as healthcare, public services, and data-driven decision support. At the same time, AI systems can be vulnerable to privacy leakage, data extraction, poisoning, backdoors, and other attacks that may undermine trust, safety, and regulatory compliance. In high-risk settings, these challenges are particularly important because failures may affect both individuals and institutions. UTOPIA addresses this need by studying how AI systems become vulnerable, how such vulnerabilities can be measured, and how they can be mitigated in practice.

Main objectives

UTOPIA has three main research objectives:

1. Understand the conditions under which AI systems are vulnerable to privacy and security attacks across different training paradigms, model types, and data modalities.
2. Develop and improve methods and metrics for evaluating the robustness of AI systems against such attacks.
3. Study privacy-enhancing technologies (PETs) such as anonymization, synthetic data generation, differential privacy, and federated learning, with the goal of identifying good trade-offs between privacy, security, model utility, fairness, and efficiency.

Research approach

UTOPIA combines theoretical analysis, experimental design, and empirical evaluation. The project studies realistic attack scenarios and develops formal threat models, structured evaluation metrics, and reproducible robustness-evaluation pipelines. These methods are then used to assess how different PETs perform under different conditions and how they can be combined in privacy-sensitive AI development.

A central ambition of the project is to move beyond isolated case-by-case studies and instead provide more systematic and comparable ways of evaluating privacy and security risks in AI. This includes benchmarking across different model architectures, training settings, data characteristics, and attacker capabilities.

Work packages

WP1 – Project Management and Dissemination

WP1 ensures project coordination, data management, communication, dissemination, and exploitation of results. This includes the data management plan, dissemination strategy, internal coordination, and public visibility of project outcomes.

WP2 – Threat Modeling and Robustness Evaluation of AI Systems

WP2 develops formal threat models and attack scenarios for privacy and security attacks on AI systems. It also assesses and improves metrics for AI robustness and builds structured, reproducible evaluation pipelines that can be reused across later work in the project. This work includes attack classes such as membership inference, data extraction or reconstruction, and model or data poisoning.

WP3 – Optimizing Privacy-Enhancing Technologies for AI Systems

WP3 studies how privacy-enhancing technologies can be designed to achieve good trade-offs between privacy, security, utility, fairness, and efficiency. This includes work on anonymization, synthetic data, and secure and efficient federated learning.

WP4 – Empirical Evaluation of Integrated Privacy-Enhancing Techniques

WP4 evaluates how multiple PETs can be combined in realistic AI workflows. The aim is to understand how integrated privacy-by-design approaches perform under realistic adversarial scenarios and to translate the results into guidelines and recommendations.