- Published on
Human Oversight in Agentic AI - Preserving Fundamental Rights on the Path to Human-Level AI
- Authors
- Name
- Diego Carpintero
This is the first part of this research work, focusing on the regulatory framework.
Abstract
Agentic AI systems are now capable of acting recursively across extended time horizons, invoking external tools, and coordinating through networks of sub-agents. As their capabilities grow and become more ubiquitous, the mechanisms designed to keep humans meaningfully in control face structural pressures that current frameworks were not built to address. This work reviews the EU regulatory framework for human oversight, identifies the specific ways in which oversight erodes in practice (forthcoming), and proposes a set of implementation patterns for preserving fundamental rights on the path to human-level AI (forthcoming).
Regulatory Framework
When an AI system denies someone a loan, filters out a job application, or influences whether an asylum claim is credible, the human implications on fundamental rights are profound. If individuals cannot understand why a system reached that conclusion, they are in practice denied any meaningful ability to challenge it. Where those responsible for reviewing an AI output lack the capacity (or authority) to question and override it, oversight becomes a formality rather than a safeguard. Ultimately, who bears responsibility when a system's assessment is wrong?
EU law addresses these risks through the AI Act (Regulation (EU) 2024/1689), the General Data Protection Regulation (GDPR), and the Charter of Fundamental Rights of the European Union (the Charter).
The AI Act identifies in Article 6(1) and in Annex III specific categories of AI systems classified as high-risk, given the potential severity of harm they may cause. Non-exhaustive examples within the context of fundamental rights include:
| Area | Examples | Possible Links to the Charter |
|---|---|---|
| Education and Vocational Training | AI university tool that ranks applicants or recommends accept/reject decisions | Art. 14 (Right to Education), Art. 21 (Non-Discrimination), Art. 23 (Equality) |
| Employment, Workers' Management, and Self-Employment | AI system for scoring or filtering out job candidates | Art. 15 (Right to engage in Work), Art. 16 (Freedom to conduct a Business), Art. 21 (Non-Discrimination), Art. 23 (Equality) |
| Essential Private Services | AI system that influences decisions on whether an individual qualifies for a loan | Art. 21 (Non-Discrimination), Art. 23 (Equality) |
| Essential Public Benefits | AI system for supporting the assessment of social applications based on individuals' data | Art. 8 (Protection of Personal Data), Art. 34 (Social Security), Art. 35 (Health Care) |
| Law enforcement | AI model used to assess the likelihood of reoffending or to profile individuals in criminal investigations | Art. 47 (Right to an Effective Remedy and to a Fair Trial), Art. 48 (Presumption of Innocence and Right of Defence) |
| Migration, asylum and border management | AI tool to assess asylum claims or evaluate risk at border crossings | Art. 18 (Right to Asylum) |
| Administration of justice and democratic processes | AI system to assist in researching and interpreting facts and the law, and in applying the law to concrete facts | Art. 41 (Right to Good Administration), Art. 47 (Right to an Effective Remedy and to a Fair Trial) |
As set out in Article 6(3) of the AI Act, certain AI systems that would fall under the high-risk AI use cases defined in Annex III of the AI Act may be subject to exclusion if they do not pose a significant risk of harm to people's health, safety or fundamental rights, including by not materially influencing the outcome of decision making. Some experts have raised concerns in this regard as such an assessment is to be carried out by providers and certain deployers (Articles 9 and 27), which may lead in practice to misclassifications or circumventions.
The AI Act distinguishes the obligations between providers and deployers. Providers are defined in Article 3(3) as those who develop or commission an AI system and place it on the market under their own name or trademark. Deployers are limited in Article 3(4) as those who use AI systems under their own authority (i.e. excluding purely personal, non-professional use). For instance, a company that builds and markets a credit-scoring model would be a provider, whereas a bank that integrates that AI model into its loan approval process would be a deployer. Similarly, a technology vendor that develops an AI system to support the assessment of asylum claims would be a provider, and the deployer would be the immigration authority that applies it to process applications.
Understanding Decisions
One of the most direct impacts of AI on fundamental rights concerns the ability of people to understand the reasons for which a decision is made about them. That is, where an AI system contributes or shapes a decision, the affected person must be able to find out how, and why.
In this regard, Article 14 AI Act requires that high-risk AI systems be designed so that operators can duly monitor their operations, understand their capabilities and limitations, and correctly interpret their outputs.
Article 86 AI Act reinforces this principle from the individual's perspective. It grants affected persons a targeted right (subject to the provision’s scope and conditions) to obtain from deployers a clear and meaningful explanation of the AI system's role in the procedure and the main elements of the decision reached.
These provisions align with binding EU primary law, specifically Articles 41 and 47 of the Charter. Article 41 (right to good administration) guarantees the duty to give reasons for decisions made by EU institutions, bodies, offices, and agencies. For national authorities acting within the scope of EU law, an equivalent duty to state reasons arises from the general EU law principle of good administration. Article 47 (right to an effective remedy) defines the right to challenge a decision before a court or tribunal.
In practice, these fundamental rights would be undermined by algorithmic opacity in the absence of meaningful oversight. A decision-maker who cannot correctly interpret the AI system's outputs cannot state the underlying grounds for a decision, leaving affected individuals without knowning the factual basis on which to articulate a substantiated legal challenge.
Duty to Maintain Human Authority
Understanding an AI output is not sufficient on its own. Article 14 AI Act further establishes that natural persons to whom human oversight is assigned, shall remain aware of the tendency to over-rely on AI outputs (automation bias), and maintain both the practical capacity and authority to override, intervene in, or halt the system.
Article 22 GDPR gives individuals the right not to be subject to a decision based solely on automated processing, including profiling, where the decision produces legal effects or similarly significantly affects them. That right is subject to the exceptions in Article 22(2) and the safeguards in Article 22(3). Notably, this standard applies regardless of whether a system is classified as high-risk under the AI Act.
Within this context, the Guidelines on Automated Individual Decision-Making and Profiling provided by the Article 29 Working Party (subsequently endorsed by the European Data Protection Board) stated that human involvement must be meaningful. In other words, nominal human involvement would still fall within the scope of solely automated decision-making if a human operator lacks the practical ability (and discretion) to deviate from an automated output.
The Court of Justice of the European Union adopted a complementary, functional approach to Article 22 GDPR in the SCHUFA judgment, holding that automated outputs (e.g. scoring) playing a determining role in downstream decisions may remain within the scope of Article 22 regardless of where the formal decision sits.
Therefore, an entity cannot circumvent the safeguards required under Article 22 GDPR by a rubber-stamping process or by delegating the formal decision to another entity (while retaining the influence over the outcome), as this would fail the standard of genuine human oversight and risk legitimising unreliable or biased outputs without effective scrutiny.
Oversight-by-Design and Accountability
Human oversight cannot be bolted on after an AI system has been deployed. Article 14(3) AI Act requires that oversight measures be architecturally embedded before a system is placed on the market or put into service. This reflects the broader logic of the AI Act: risks must be addressed by design, not managed as an ad hoc and unstructured measure.
Said design and implementation responsibility is shared between providers and deployers (Article 14(3)), albeit asymmetrically: providers bear the primary obligation to build oversight into the system, or to specify what deployers must implement (Article 13(3)(d).
Furthermore, under Article 26 AI Act deployers are required to assign human oversight responsibilities to natural persons with the necessary competence, training, and authority. This is not a mere organisational constraint; it requires that those responsible for oversight are genuinely equipped and have the organisational support to exercise it.
Fundamental Rights Impact Assessments
Certain deployers are further required, under Article 27 AI Act, to conduct, on their own, a documented Fundamental Rights Impact Assessment (FRIA) before putting a high-risk AI system into service. This assessment shall be adapted as the system changes, and it needs to cover: the categories of people and groups likely to be affected by its use, specific risks of harm, a description of the implementation of human oversight measures, and the intended mitigation measures to be taken in case the risks materialise.
This obligation applies to (i) bodies governed by public law, (ii) private entities providing public services, and (iii) deployers operating in certain regulated sectors.
The Digital Omnibus on AI allows FRIAs to incorporate or cross-refer to relevant parts of a Data Protection Impact Assessment (DPIA) conducted under Article 35 GDPR, reducing duplicative documentation where the two assessments cover overlapping ground.
Proportionality Principle
Article 14(3) AI Act recognises that a single oversight model cannot fit all contexts. Compliance obligations must be commensurate with the specific risks, level of autonomy, and context of use of the system at hand.
In practice, this regulatory flexibility aims to accommodate diverse governance architectures across the risk spectrum, such as: (i) human-in-the-loop, which requires pre-authorisation or a human trigger before the AI executes any action, (ii) human-on-the-loop, which provides real-time supervisory control, and (iii) ex-post auditing, which facilitates outcome evaluation, drift detection, and systemic accountability after deployment.
Notes
This section reflects the AI Act as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI, which entered into force on 27 July 2026), the General Data Protection Regulation (GDPR), and the Charter of Fundamental Rights, as applicable. The Digital Omnibus defers application of the high-risk obligations under Chapter III, including the human oversight requirements of Article 14 and the deployer duties under Articles 26 and 27, to 2 December 2027 for standalone Annex III systems and 2 August 2028 for high-risk systems embedded in regulated products under Annex I. By way of transitional provision under Article 111, high-risk AI systems intended for use by public authorities may have until 2 August 2030 to comply.
References
[1] Yao, et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arxiv:2210.03629
[2] Snell, et al. 2024. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arxiv:2408.03314
[3] European Parliament and Council of the EU. 2024. Regulation (EU) 2024/1689 — Artificial Intelligence Act. Reg. (EU) 2024/1689.
[4] European Parliament and Council of the EU. 2024. AI Act, Annex III — High-Risk AI Systems Referred to in Article 6(2). Annex III.
[5] European Parliament and Council of the EU. 2016. Regulation (EU) 2016/679 — General Data Protection Regulation. Reg. (EU) 2016/679.
[6] European Union. 2012. Charter of Fundamental Rights of the European Union (2012/C 326/02). OJ C 326.
[7] Article 29 Data Protection Working Party. 2018. Guidelines on Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679 (WP251rev.01), adopted 6 February 2018, endorsed by the EDPB, Endorsement 1/2018, 25 May 2018. EDPB, pages 20-21
[8] Court of Justice of the EU (First Chamber). 2023. OQ v Land Hessen (SCHUFA Holding — Scoring), Case C-634/21, ECLI:EU:C:2023:957, 7 December 2023.
[9] FRA, Assessing High-Risk Artificial Intelligence - Fundamental Rights Risks. Publications Office of the European Union, Luxembourg, 2025 FRA:AI.2025
[10] European Parliament and Council of the EU. 2026. Regulation (EU) 2026/1744 - Digital Omnibus on AI (amending Regulation (EU) 2024/1689), adopted 8 July 2026, OJ L, 24 July 2026, in force 27 July 2026. Reg. (EU) 2026/1744.