Published on

Human Oversight in Agentic AI - Preserving Fundamental Rights on the Path to Human-Level AI

Authors
  • avatar
    Name
    Diego Carpintero
    Twitter

This is the first part of this research work, focusing on the regulatory framework.

Abstract

Agentic AI systems are now capable of acting recursively across extended time horizons, invoking external tools, and coordinating through networks of sub-agents. As their capabilities grow and become more ubiquitous, the mechanisms designed to keep humans meaningfully in control face structural pressures that current frameworks were not built to address. This paper reviews the EU regulatory framework for human oversight, identifies the specific ways in which oversight erodes in practice, and proposes a set of implementation patterns for preserving fundamental rights on the path to human-level AI.

Regulatory Framework

When an AI system denies someone a loan, filters out a job application, or influences whether an asylum claim is credible, the human implications are concrete. If individuals cannot understand why a system reached that conclusion, they are in practice denied any meaningful ability to challenge it. Where those responsible for reviewing an AI output lack the capacity (or authority) to question and override it, oversight becomes a formality rather than a safeguard. Ultimately, who bears responsibility when a decision is wrong?

EU law addresses these risks through the AI Act (Regulation (EU) 2024/1689), the General Data Protection Regulation (GDPR), and the Charter of Fundamental Rights of the European Union (the Charter).

The AI Act identifies in Annex III specific categories of AI systems classified as high-risk, given the potential severity of harm they may cause. Non-exhaustive examples within the context of fundamental rights include:

  • In private services and employment: AI systems used for scoring, including decisions on whether an individual qualifies for a loan or a social benefit; and those used in recruitment and employment (right to non-discrimination);
  • In law enforcement: systems used to assess the likelihood of reoffending or to profile individuals in criminal investigations (Articles 47 and 48 of the Charter);
  • In migration and border management: automated tools used to assess asylum claims or evaluate risk at border crossings (Article 18 of the Charter); and
  • In administration and justice: systems to assist in researching, interpreting facts and the law, and in applying the law to a set of facts (Article 41 of the Charter).

Understanding Decisions

One of the most direct impacts of AI on fundamental rights concerns the ability of people to understand the reasons for which a decision is made about them. That is, where an AI system contributes or shapes a decision, the affected person must be able to find out how, and why.

In this regard, Article 14 AI Act requires that high-risk AI systems be designed so that operators can duly monitor their operations, understand their capabilities and limitations, and correctly interpret their outputs.

Article 86 AI Act reinforces this principle from the individual's perspective. It grants affected persons a targeted right (subject to the provision’s scope and conditions) to obtain from deployers a clear and meaningful explanation of the AI system's role in the procedure and the main elements of the decision reached .

These provisions align with binding EU primary law, specifically Articles 41 and 47 of the Charter. Article 41 (right to good administration) guarantees the duty to give reasons for decisions made by EU institutions, bodies, offices, and agencies. For national authorities acting within the scope of EU law, an equivalent duty to state reasons arises from the general EU law principle of good administration. Article 47 (right to an effective remedy) defines the right to challenge a decision before a court or tribunal.

In practice, these fundamental rights would be undermined by algorithmic opacity in the absence of meaningful oversight. A decision-maker who cannot correctly interpret AI system's outputs cannot state the underlying grounds for a decision, leaving affected individuals without knowning the factual basis on which to articulate a substantiated legal challenge.

Duty to Maintain Human Authority

Understanding an AI output is not sufficient on its own. Article 14 AI Act further establishes that natural persons to whom human oversight is assigned, shall remain aware of the tendency to over-rely on AI outputs (automation bias), and maintain both the practical capacity and authority to override, intervene in, or halt the system.

Article 22 GDPR gives individuals the right not to be subject to a decision based solely on automated processing, including profiling, where the decision produces legal effects or similarly significantly affects them. That right is subject to the exceptions in Article 22(2) and the safeguards in Article 22(3). Notably, this standard applies regardless of whether a system is classified as high-risk under the AI Act.

Within this context, the Guidelines on Automated Individual Decision-Making and Profiling provided by the Article 29 Working Party (subsequently endorsed by the European Data Protection Board) stated that human involvement must be meaningful. In other words, nominal human involvement still falls within the scope of solely automated decision-making if a human operator lacks the practical ability (and discretion) to deviate from the automated output.

The Court of Justice of the European Union adopted a complementary, functional approach to Article 22 GDPR in the SCHUFA judgment, holding that automated outputs (e.g. scoring) playing a determining role in downstream decisions may remain within the scope of Article 22 regardless of where the formal decision sits.

Therefore, an entity cannot circumvent the safeguards required under Article 22 GDPR by a rubber-stamping process or by delegating the formal decision to another entity (while retaining the influence over the outcome), as this would fail the standard of genuine human oversight and risk legitimising unreliable or biased outputs without effective scrutiny.

Oversight-by-Design and Accountability

Human oversight cannot be bolted on after an AI system has been deployed. Article 14(3) AI Act requires that oversight measures be architecturally embedded before a system is placed on the market or put into service. This reflects the broader logic of the AI Act: risks must be addressed by design, not managed as an ad hoc measure.

The design and implementation responsibility is shared between providers and deployers (Article 14(3)), albeit asymmetrically: providers bear the primary obligation to build oversight into the system, or to specify what deployers must implement.

Furthermore, under Article 26 AI Act deployers are required to assign human oversight responsibilities to natural persons with the necessary competence, training, and authority. This is not a mere organisational constraint; it requires that those responsible for oversight are genuinely equipped and have the organisational support to exercise it.

Fundamental Rights Impact Assessments

Certain deployers are further required, under Article 27 AI Act, to conduct a documented Fundamental Rights Impact Assessment (FRIA) before putting a high-risk AI system into service. This obligation applies to (i) bodies governed by public law, (ii) private entities providing public services, and (iii) deployers operating in certain regulated sectors.

The Digital Omnibus on AI allows FRIAs to incorporate or cross-refer to relevant parts of a Data Protection Impact Assessment (DPIA) conducted under Article 35 GDPR, reducing duplicative documentation where the two assessments cover overlapping ground.

Proportionality Principle

Article 14(3) AI Act recognises that a single oversight model cannot fit all contexts. Compliance obligations must be commensurate with the specific risks, level of autonomy, and context of use of the system at hand.

In practice, this regulatory flexibility aims to accommodate diverse governance architectures across the risk spectrum, such as: (i) human-in-the-loop, which requires pre-authorisation or a human trigger before the AI executes any action, (ii) human-on-the-loop, which provides real-time supervisory control, and (iii) ex-post auditing, which facilitates outcome evaluation, drift detection, and systemic accountability after deployment.

Notes

This section reflects the AI Act as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI, which entered into force on 27 July 2026), the General Data Protection Regulation (GDPR), and the Charter of Fundamental Rights, as applicable. The Digital Omnibus defers application of the high-risk obligations under Chapter III, including the human oversight requirements of Article 14 and the deployer duties under Articles 26 and 27, to 2 December 2027 for standalone Annex III systems and 2 August 2028 for high-risk systems embedded in regulated products under Annex I. By way of transitional provision under Article 111, high-risk AI systems intended for use by public authorities may have until 2 August 2030 to comply.

References

  • [1] Yao, et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arxiv:2210.03629

  • [2] Snell, et al. 2024. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arxiv:2408.03314

  • [3] European Parliament and Council of the EU. 2024. Regulation (EU) 2024/1689 — Artificial Intelligence Act. Reg. (EU) 2024/1689.

  • [4] European Parliament and Council of the EU. 2024. AI Act, Annex III — High-Risk AI Systems Referred to in Article 6(2). Annex III.

  • [5] European Parliament and Council of the EU. 2016. Regulation (EU) 2016/679 — General Data Protection Regulation. Reg. (EU) 2016/679.

  • [6] European Union. 2012. Charter of Fundamental Rights of the European Union (2012/C 326/02). OJ C 326.

  • [7] Article 29 Data Protection Working Party. 2018. Guidelines on Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679 (WP251rev.01), adopted 6 February 2018, endorsed by the EDPB, Endorsement 1/2018, 25 May 2018. EDPB, pages 20-21

  • [8] Court of Justice of the EU (First Chamber). 2023. OQ v Land Hessen (SCHUFA Holding — Scoring), Case C-634/21, ECLI:EU:C:2023:957, 7 December 2023.

  • [9] European Parliament and Council of the EU. 2026. Regulation (EU) 2026/1744 - Digital Omnibus on AI (amending Regulation (EU) 2024/1689), adopted 8 July 2026, OJ L, 24 July 2026, in force 27 July 2026. Reg. (EU) 2026/1744.

View on GitHub