A documents annotation IT tool to aid experts in manually annotating documents with comprehensive metadata

Deliverable D3.1 presents the EU-ALMPO Documents Annotation Tool, an AI-assisted tool developed within WP3 to support experts in annotating documents on Active Labour Market Policies with structured metadata.

Building on the project’s analytical and data collection work, the tool helps transform a broad collection of studies, policy reports and evaluation material into a validated, machine-interpretable knowledge base. By combining Large Language Model-based annotation with expert human validation, it makes policy evidence easier to search, analyse and reuse, while keeping experts in control of the final results.

The annotated knowledge base will support the EU-ALMPO Observatory and future AI-driven services, including the Interactive ALMP Design Wizard.

Why it matters

This deliverable matters because reliable AI-supported policy tools depend on reliable, structured evidence.

Active Labour Market Policy knowledge is currently scattered across academic studies, policy reports, evaluations and institutional documents, often using different terminology and formats. D3.1 addresses this problem by turning complex, unstructured documents into a validated knowledge base that can be searched, compared and reused.

By combining AI-assisted annotation with expert human validation, the tool improves the efficiency of evidence processing without removing expert judgement from the process. This is essential for building trustworthy AI services within the EU-ALMPO Observatory, including future tools such as the Interactive ALMP Design Wizard.

In practice, D3.1 helps ensure that future recommendations and insights generated by the platform are grounded in traceable, policy-relevant and expert-validated evidence.

Key information

Objectives

Transform the heterogeneous ALMP document corpus into a structured, reusable and machine-interpretable knowledge base.
Design and apply a multi-dimensional annotation taxonomy capturing policy instruments, target groups, objectives, outcomes and evaluation evidence
Develop an AI-assisted annotation tool that combines automatic (LLM-based) annotation with expert human validation, ensuring accuracy, traceability and transparency.
Produce a validated, structured dataset able to feed the AI-driven, RAG-based components of the EU-ALMPO platform, in particular the future ALMP Design Wizard.

Target audience

Policymakers involved in the design, adaptation and evaluation of Active Labour Market Policies.
Public Employment Services and labour market institutions.
Researchers and policy analysts working on ALMPs, skills matching and labour market evidence.
Labour market practitioners involved in policy implementation and service delivery.
EU-ALMPO consortium partners contributing to the project’s knowledge base and Observatory.
Technical teams developing the EU-ALMPO digital platform, AI-driven tools and future ALMP Design Wizard.

End users

Curators: domain experts, including researchers and policy analysts, who review, correct and validate AI-generated annotations.
Lead Curators: senior experts who oversee the annotation process and ensure the quality, consistency and methodological reliability of the annotated dataset.
EU-ALMPO consortium experts responsible for transforming policy documents into structured, validated knowledge.
Researchers and analysts who will use the annotated knowledge base to support evidence-based ALMP analysis.
Technical partners who will integrate the validated dataset into the EU-ALMPO Observatory and future AI-driven tools.

Keywords →

Policy
ALMPs | Policy evidence | Expert validation| Labour market knowledge base

Analysis
Document annotation | Taxonomy | Metadata| Structured knowledge | Evidence-based analysis

AI
AI-assisted annotation | Large Language Models |Retrieval-Augmented Generation (RAG) | Human-in-the-loop AI

D3.1 presents the design, implementation and validation of the Documents Annotation Tool, developed by the University of the Peloponnese within WP3 of EU-ALMPO. The tool supports experts in transforming a large and heterogeneous corpus of documents on Active Labour Market Policies (ALMPs), including academic studies, policy evaluations, institutional reports, Public Employment Services documentation and international organisation material, into a structured, machine-interpretable knowledge base.

Building on the analytical work carried out in WP1 and WP2, the deliverable introduces an AI-assisted annotation approach that combines automatic annotation through Large Language Models with systematic human validation, following a human-in-the-loop model. This allows automation to speed up the annotation process, while experts retain full control over the final, validated metadata. Documents are divided into semantic chunks and annotated through a purpose-built taxonomy covering policy instruments, target groups, objectives, outcomes, evaluation methods and contextual conditions. Each annotation is traceable to a verbatim quote from the source text, supporting transparency and expert verification.

The resulting annotated knowledge base is a foundational building block for the EU-ALMPO Observatory. It enables structured retrieval of policy-relevant evidence and supports the future AI-driven components of the platform, most notably the Interactive ALMP Design Wizard, which will use Retrieval-Augmented Generation (RAG) to ground its recommendations in validated evidence.

D3.1 is closely connected to the project’s Analytical Framework, the document corpora contributed by IRS and GDAŃSK TECH, and the wider work carried out in WP1 and WP2. It also provides the evidence infrastructure needed for the future development of the ALMP Design Wizard and the Observatory’s analytical and AI-driven services.