Artificial Intelligence, evaluation and new forms of responsibility: contributions from the EU-ALMPO project in the annual AIV Congress

On 10-12 September 2026, the University of Pisa hosted the XXVIII Congress of the Italian Evaluation Association (AIV), entitled “The Augmented Evaluator: New Postures, Tools and Intelligences for Contemporary Evaluation”. The conference brought together researchers, evaluators and practitioners to discuss how profound technological and organisational transformations are reshaping evaluation and, with it, the role of those who produce and interpret evaluative knowledge. 

The starting point of the conference was a question that is becoming increasingly relevant across evaluation practice: what does it mean to be an evaluator in an environment in which technologies can increasingly support the production, processing and interpretation of evidence? The implications extend beyond the technical dimension. They concern the epistemological foundations of evaluation, the responsibilities attached to evaluative judgement, the relationship between human expertise and artificial intelligence (AI), and the risk that evaluators may be asked to take on an ever broader range of functions. 

Building on the experience and reflections developed within the EU-ALMPO project, IRS delivered two contributions, addressing different aspects of the relationship between social research and AI.

 

When AI enters evaluative judgement 

The first contribution was part of the panel “Artificial intelligence and the transformation of evidence regimes in evaluation: from data construction to responsibility for judgement”, coordinated by researchers from the National Institute for Public Policy Analysis (INAPP). The panel explored how AI is changing the ways in which data and evidence are produced and interpreted, as well as the roles and responsibilities involved in making evaluative judgements. IRS researchers Maria Juliana Charry Camargo and Siro Ciarimboli took part with the presentation “AI, evidence and legitimacy in policy processes: the contribution of the EU-ALMPO framework”. 

The intervention provided an opportunity to introduce the work carried out so far within the EU-ALMPO project, with particular attention to the development of its analytical framework. The framework was designed to support the analysis and design of ALMPs in relation to skills mismatch, and can also provide a basis for supporting their evaluation. In this perspective, the development of AI-based tools cannot be considered separately from the contexts in which they are expected to be used. 

A central issue is the relationship between the AI tools developed in the project and their potential users. The servuction model adopted in the framework highlights the importance of considering not only the technological tool itself, but the wider environment in which knowledge is produced and used. This means starting from contextualised personas and use cases to consider how users actually interact with the tool, what information they need and what decisions they are expected to support. The objective is therefore not simply to develop increasingly sophisticated AI systems, but to establish a meaningful connection between the tool, the evidence it provides and the practices in which that evidence is ultimately used. 

This perspective also raises broader questions for evaluation theory and ethics. AI-mediated information environments challenge traditional understandings of evaluation by introducing new forms of interaction between technological systems and knowledge structures. AI does not replace evaluative expertise, but it changes the way that expertise is exercised. Evaluators and policy-makers are increasingly required to manage the use of AI, assess the relevance of its outputs and interpret them in relation to the context in which they are produced. Knowledge therefore becomes, to an increasing extent, the result of an interaction between human judgement, technological systems, available information and institutional conditions. 

This also raises questions of autonomy, accountability and legitimacy. Responsibility for an AI-supported decision cannot necessarily be attributed to a single actor. Instead, it concerns the combination of actors and conditions that contribute to the decision-making process. From this perspective, reliable AI depends not only on technological performance, but also on the quality of the information environment, the conditions of interaction, the institutional setting and the role attributed to users. 

The servuction model offers one way of looking at this broader configuration. The credibility of an AI-supported evaluative process depends on the quality of the operational environment in which it takes place, as well as on the possibility of preserving human judgement and questioning the outputs produced by technological systems. 

 

Making bias visible: gender and AI 

A further intervention took place within the thematic group on equal opportunities, where IRS senior researcher Flavia Pesce took part as speaker in the Sparking Discussion “Working on AI ethics to ensure evaluation that supports equal opportunities”, focusing on gender stereotypes and bias in AI and their implications for evaluation. 

The discussion highlighted how AI can reproduce and potentially reinforce inequalities already present in society. This concerns not only the data used to train systems, but also who participates in designing and governing them, which perspectives are represented and which experiences may remain invisible. As discussed during the session, bias can emerge before data and algorithms themselves, through the composition of the teams developing the technology and the assumptions that inform its design. 

Data represent another important dimension. Historical datasets reflect existing social conditions and inequalities and can therefore reproduce patterns from the past when used to inform future decisions. Bias can also be less immediately visible. In evaluation, for example, an AI-supported analysis may identify an average effect accurately while failing to reveal relevant differences between groups. A technically sophisticated analysis may consequently provide limited insight into questions of equal opportunities if important differences are not made visible. 

This places some familiar evaluative questions at the centre of discussions on responsible AI: who is represented in the data, who is missing, which assumptions are being made, and which differences may remain hidden? From this perspective, evaluation can contribute not only by examining the effects of AI systems after implementation, but also by informing their development, helping identify relevant groups, anticipate risks and define evidence for monitoring impacts over time. 

The discussion also considered the growing role of impact assessment in AI governance. The UNESCO “Recommendation on the Ethics of Artificial Intelligence” sets out principles including fairness and non-discrimination, inclusion, gender equality, transparency, accountability and human oversight, while its Ethical Impact Assessment translates them into questions about data, development teams, transparency, stakeholder involvement and potential impacts. This makes ethics something that can be examined through criteria, questions and evidence. The EU “AI Act” similarly introduces a Fundamental Rights Impact Assessment for certain high-risk AI systems, including the identification of affected groups and specific risks. More recently, the Council of Europe’s “Recommendation on Equality” and Artificial Intelligence calls for equality and non-discrimination to be addressed throughout the entire lifecycle of AI systems. 

 

Bringing project experience into the wider debate 

The AIV Congress therefore provided an opportunity to bring insights from EU-ALMPO into a broader discussion on AI, evaluation, knowledge production and the relationship between technological tools and their users. At the same time, the exchange offered an opportunity to revisit some of the questions emerging from the project, particularly concerning the changing role of AI in professional practice and the ethical implications of increasingly AI-mediated processes.