Trustworthy AI: Transparency, Fairness and Human Oversight

A pipeline that applies a rule identically to every change proves consistency, not soundness. This chapter turns from enforcement to substance: what a fair system, an intelligible output and an effective human intervention require. It takes the seven requirements of the European Commission's High-Level Expert Group as its frame, works through transparency and explainability, examines fairness metrics and the trade-offs between them, and describes the human oversight patterns that Article 14 of Regulation (EU) 2024/1689 (the AI Act) expects, closing with a fairness configuration, an oversight design, and a fundamental rights impact assessment that belong in version control. The underlying conviction is the one that runs through Architecture as Code: a requirement that cannot be expressed as an artefact in the repository will not survive contact with a delivery schedule (66, Nordqvist, 2025).

The seven HLEG requirements for trustworthy AI mapped to codified architectural mechanisms that feed the evidence store

Figure 10.1: Each of the seven requirements for trustworthy AI maps to at least one codified architectural mechanism, and every mechanism leaves evidence in the evidence store (the table below).

Requirement Codified mechanism
Human agency and oversight Override and stop paths (Article 14)
Technical robustness and safety Evaluation suite and guardrails
Privacy and data governance Data contracts and DPIA records
Transparency Model cards, disclosure and content marking
Diversity and fairness Fairness evaluation configuration
Societal and environmental wellbeing Impact assessment and carbon accounting
Accountability Evidence store and audit trail

Table for Figure 10.1: Requirements and the mechanisms that codify them.

The seven requirements as design input

The Ethics Guidelines for Trustworthy AI, published by the High-Level Expert Group on Artificial Intelligence in 2019, state that trustworthy AI is lawful, ethical, and robust, and set out seven requirements: human agency and oversight; technical robustness and safety; privacy and data governance; transparency; diversity, non-discrimination and fairness; societal and environmental well-being; and accountability (12, AI HLEG, 2019). The Assessment List for Trustworthy AI (ALTAI) of 2020 turns each requirement into questions a team can answer about a specific system (13, AI HLEG, 2020). The guidelines are not law, but they shaped the AI Act. UNESCO's Recommendation on the Ethics of Artificial Intelligence of 2021 sets out comparable values (29, UNESCO, 2021), and its Ethical Impact Assessment, designed for governments procuring or deploying AI, for companies developing AI systems and for researchers assessing them, can be run alongside ALTAI (31, UNESCO, n.d.).

The architectural task is to attach a mechanism and an evidence source to each requirement, as Figure 10.1 does. Human agency becomes an override path, a stop path, and a role; robustness becomes an evaluation suite and runtime guardrails; privacy becomes data contracts and data protection impact assessment (DPIA) records; transparency becomes model cards, disclosure, and content marking; fairness becomes a fairness configuration with thresholds; well-being becomes the impact assessment of Chapter 8 and an energy budget; and accountability becomes the evidence store of Chapter 11. This chapter concentrates on the three that most often lack a mechanism in practice: transparency, fairness, and oversight.

One framing must come first. Selbst et al. identify five traps that technical teams fall into when building fairness into a system, among them the framing trap of drawing the boundary around the software alone and the formalism trap of believing a mathematical definition captures a contested social concept (56, Selbst et al., 2019). Their sociotechnical framing is also the architect's: the system under design includes the caseworker, the appeal process, the training programme, and the incentives, not only the model.

Transparency and explainability

Transparency has three audiences, and conflating them produces documentation nobody can use. The deployer needs to know what the system does, where it fails, and how to operate it, which is why Article 13 requires providers of high-risk systems to supply instructions for use covering capabilities, limitations, accuracy, and oversight measures (1, European Union, 2024). The affected person needs to know that a system was involved and why the outcome was what it was: Article 86 gives a right to a clear and meaningful explanation of the system's role, and Regulation (EU) 2016/679 (the GDPR) adds information duties and safeguards around automated decision-making in Articles 13 to 15 and 22 (2, European Union, 2016). The oversight staff need enough insight into each output to exercise judgement rather than rubber-stamp it.

Model-level explainability serves the second and third audiences. Molnar distinguishes intrinsically interpretable models, such as linear models and shallow decision trees, from post-hoc methods such as SHAP, LIME, partial dependence, and counterfactual explanations that state what would have had to differ for the outcome to change (73, Molnar, 2022). The architectural choice is between a less accurate interpretable model and a more accurate opaque one whose post-hoc explanations must themselves be evaluated for faithfulness. For decisions about individuals in Annex III domains, we lean towards interpretable models or towards confining the opaque model to decision support. Counterfactual explanations are particularly valuable in public administration because they map onto the duty to give reasons: "the application would have been approved had the declared income been below this threshold" is an explanation a citizen can act on.

Large language models (LLMs) complicate this picture, because an LLM's explanation of its own output is generated text, not a faithful account of the computation. Bender et al. warned that fluent language invites readers to attribute understanding and intent the system does not possess (50, Bender et al., 2021). System-level transparency is the honest alternative: record and show which retrieved sources entered the context, which tools were called, which model version answered, and which verification steps applied.

Worked example: system-level transparency record

Illustrative example. With every answer from Nordvik Region's retrieval-augmented generation (RAG) staff assistant, the gateway writes the record below to the log and renders it in a "How this answer was produced" panel.

{
  "response_id": "resp-2026-10-30-7f3a91",
  "system": "ai-system/staff-assistant",
  "model": {"id": "vendor-llm", "version": "2026-06-15", "hosting": "eu-north"},
  "prompt_bundle": "prompts/staff-assistant@4c1d2e0",
  "retrieved_sources": [
    {"doc": "guidance/housing-allowance-2026.pdf", "version": "3", "section": "4.2"},
    {"doc": "guidance/appeals-handbook.pdf", "version": "7", "section": "2.1"}
  ],
  "sources_cited_in_answer": ["guidance/housing-allowance-2026.pdf#4.2"],
  "tools_called": [],
  "guardrails": {"personal_data_redaction": "2 identifiers pseudonymised", "grounding_check": "passed"},
  "disclosure_shown": true,
  "generated_at": "2026-10-30T10:41:07Z"
}

Every field is a fact the platform knows. The distinction between retrieved_sources and sources_cited_in_answer shows the caseworker what the system read but did not use, which is often where a missing caveat hides, and the same record is the event Chapter 11 retains under the logging obligations.

Article 50 adds disclosure obligations regardless of risk tier. Providers must ensure that people know when they interact with an AI system unless it is obvious, and that synthetic audio, image, video, and text are marked as artificially generated in machine-readable form; deployers must disclose deepfakes and AI-generated text published to inform the public unless it has undergone human editorial review (1, European Union, 2024). Disclosure therefore belongs in the channel layer as a standard component, and content marking in the gateway at the point of generation, so that no application can emit unmarked synthetic content.

Worked example: transparency notice to affected persons

Deployers of Annex III high-risk systems that make or assist decisions about natural persons must inform those persons under Article 26(11) (1, European Union, 2024), and the GDPR information duties in Articles 13 and 14 apply alongside (2, European Union, 2016). The notice must be written for a resident, not for a regulator.

Illustrative example. The notice below accompanies every acknowledgement of an application to Nordvik Region's financial assistance service and is published on its website.

# How we use a computer system to sort applications

## What the system does
The system looks at the information in your application, such as your household,
your income and whether you have told us about an urgent need. It suggests how
urgent your application is, so that people in the most difficult situations are
helped first.

## What the system does not do
It does not decide whether you get assistance or how much. A caseworker reads
your application and makes that decision, and can change the suggested order.

## What information is used
Only your application and our own case records. We do not use your sex, age or
country of birth to sort applications. We use them only, separately and
securely, to check that the system treats groups fairly.

## Your rights
- You can ask what role the system played in handling your application.
- You can ask for your application to be handled without the system's
  suggestion. A caseworker will then set its priority by hand.
- You can see and correct your personal data, and complain to our Data
  Protection Officer or to the data protection authority.
- You can appeal any decision, as explained in your decision letter.

The notice leads with what the system does not do, because residents ask that first. It offers handling without the system's suggestion as a real option, which the workflow must support, and it explains the use of protected attributes honestly, which follows from the fairness design below.

Fairness: metrics and their trade-offs

Fairness is the requirement most prone to the formalism trap, because its precise mathematical definitions disagree with each other. Barocas, Hardt, and Narayanan organise the main group-fairness criteria into three families: independence, under which the prediction is statistically independent of the protected attribute (demographic parity); separation, under which error rates are equal across groups (equalised odds); and sufficiency, under which the prediction means the same for every group (calibration). They show that these criteria cannot all hold when base rates differ, so every system embodies a choice between them (72, Barocas et al., 2023).

The choice must therefore be made explicitly, by people with the authority and domain knowledge to make it, and recorded in the system's AI Decision Record, where a later reviewer, auditor or court can find who chose and why. A fraud detection system where false accusations carry serious harm may prioritise equal false positive rates; a screening tool for a scarce service may be held to calibration so that a score of 0.7 means the same for everyone. Measuring fairness also requires data about protected attributes, some of which, such as racial or ethnic origin, are special categories under Article 9 of the GDPR. Article 4a of the AI Act, which the 2026 Digital Omnibus amendment inserted in place of the original Article 10(5), permits providers of high-risk systems to process such data for bias detection and correction where strictly necessary and under safeguards, including purpose limitation, access restriction, and deletion once the bias has been addressed (27, European Union, 2026). Join protected attributes to predictions only in an isolated evaluation environment, never in the production feature set, and log the join.

The configuration below is read by the evaluation service at the promotion gate from Chapter 9 and again by the production monitoring job.

fairness_evaluation:
  system: ai-system/benefits-eligibility-support
  decision_record: ADR-AI-2026-012
  protected_attributes:
    - {name: sex, source: evaluation-only-join}
    - {name: age_band, source: evaluation-only-join}
    - {name: country_of_birth_region, source: evaluation-only-join}
  reference_group: overall_population
  metrics:
    - name: false_negative_rate_gap
      family: separation
      threshold: 0.03
      rationale: "A wrongly rejected applicant loses a benefit; equal miss rates are the priority"
    - name: calibration_gap
      family: sufficiency
      threshold: 0.05
      rationale: "Caseworkers read the score as a probability and must be able to trust it across groups"
  intersections: [[sex, age_band]]
  minimum_group_size: 200
  on_failure: block_promotion
  monitoring: {cadence: weekly, on_breach: open_register_entry}

The minimum_group_size guards against conclusions from tiny subgroups, and the intersections list forces evaluation across combinations, because a system can be fair on each attribute separately and unfair at their intersection. Because the rationale sits beside each threshold, the answer to why a three-point gap was deemed acceptable is in the artefact, not in someone's memory.

Human oversight and Article 14

Human oversight most often exists on paper and not in practice. Three patterns describe the design space. In human-in-the-loop, a person approves each output before it takes effect, the only pattern suitable for consequential decisions about individuals. In human-on-the-loop, the system acts and a person monitors with the ability to intervene and halt, which suits high-volume, lower-stakes operations. In human-in-command, a person decides whether and where the system is used at all, which is what the governance gates of Chapter 9 implement.

Article 14 requires that the persons assigned to oversight of a high-risk system can understand its capacities and limitations, monitor its operation, remain aware of automation bias, interpret its output correctly, disregard, override, or reverse it, and interrupt the system through a stop function that brings it to a halt in a safe state (1, European Union, 2024). Article 26(2) places on deployers the duty to staff that oversight, a role Chapter 14 describes; this chapter concerns what the design must make possible.

Each clause is a design requirement: understanding requires a short operational summary in the interface rather than a forty-page document. Awareness of automation bias requires measurement: an override rate near zero on a system with a known error rate is evidence that oversight has become ceremonial. Correct interpretation requires that uncertainty is displayed, not hidden behind a single score. The ability to reverse requires a workflow that allows a different outcome without penalty, which is an organisational question as much as a software one. And the stop function requires an interruption path the architecture honours, which for an agent means halting tool execution mid-sequence.

Shneiderman argues that high automation and high human control are independent dimensions rather than opposite ends of one axis, and that trustworthy systems combine both (74, Shneiderman, 2022). In practice, an overseer who must process three hundred recommendations an hour with a single accept button has no control regardless of what the policy says. Queue design, time per case, the default when the overseer does nothing, and the ease of recording an override reason are therefore architectural decisions. A version-controlled oversight checklist, evaluated by the deployment gate, ties each Article 14 clause to evidence such as an override-rate metric or a stop-switch test; the design specification below records the decisions behind that evidence.

Worked example: human-oversight design specification

Illustrative example. Nordvik Region's benefits-triage system ranks applications for financial assistance; senior caseworkers oversee the ranking and caseworkers decide every application. The values are fictional defaults that each organisation must set from its own workload data and risk appetite.

oversight_design:
  system: ai-system/benefits-triage
  version: "1.2"
  pattern: human_in_the_loop
  oversight_role: senior-caseworker
  workload: {max_rankings_reviewed_per_hour: 40, min_review_seconds_per_case: 45}
  overseer_view:
    show: [urgency_score, score_band_uncertainty, top_3_contributing_factors,
           counterfactual_to_next_band, data_completeness_flag, model_version]
  actions:
    reorder: {requires_reason: true, reason_codes: [urgent_need_not_captured, data_error, other_free_text]}
    opt_out_case: {trigger: applicant_request_or_caseworker_judgement, effect: manual_priority_by_caseworker}
  defaults:
    on_no_action_within_hours: 24
    default_action: escalate_to_team_lead
    never_auto_accept: true
  automation_bias_controls:
    blind_sample_rate: 0.05
    min_override_rate_alert: 0.02
    max_override_rate_alert: 0.30
  stop_function:
    who_can_trigger: [senior-caseworker, head-of-social-services, ai-platform-on-call]
    effect: queue_reverts_to_date_order
    safe_state_reached_within_seconds: 60
    tested: quarterly
  competence: {training_module: OVR-TRIAGE-101, refresher_months: 12}
  decision_record: AIDR-0011

Several values are deliberate. never_auto_accept means that an unattended ranking escalates rather than silently taking effect, so inattention cannot become automation. The blind_sample_rate sends five per cent of cases to the overseer without the score, yielding a baseline against which agreement with the system can be measured; agreement far above it signals automation bias. Both a lower and an upper override-rate alert exist, because ceremonial oversight and a failing model both show up in that number. And the stop function reverts to date order, the safe state the service ran on before the system existed, as Article 14(4)(e) asks (1, European Union, 2024).

Across the usage modes

Article 14 binds high-risk systems, which in practice arise in Modes 4 and 5. In the other modes oversight rests on the individual who reads or acts on the output, so it becomes a matter of AI literacy under Article 4 and of interface defaults (1, European Union, 2024).

Mode Who oversees Oversight mechanism Typical failure
Mode 0 Ambient AI The reader, often unaware AI was involved Checking against the primary source Over-reliance on unrequested summaries
Mode 1 Conversational assistants The user, before copying text out Review before reuse; literacy training Fluent text accepted unchecked
Mode 2 Embedded workplace assistants The user, before sending Draft-and-confirm; no automatic sending Actions with the user's permissions unreviewed
Mode 3 Agentic work tools The developer or operator Per-command approval, sandbox, review Approval fatigue, blanket allow rules
Mode 4 Built AI applications A designated overseer where high-risk Oversight design above Ceremonial review, automation bias
Mode 5 Autonomous production agents Humans on the loop and in command Approval policies, sampling, stop function Volume outruns review; untested stop path

Mode 0 deserves more attention than it receives. An automatic meeting summary filed in a case record becomes a factual statement without anyone having decided to use AI, and fluent text invites trust it has not earned (50, Bender et al., 2021). Treat ambient AI output as an unverified secondary source until someone has checked it against the primary one, and say so in the acceptable-use policy of Chapter 9.

Accessibility and inclusion

Non-discrimination extends beyond the model's statistics to the people who can use the system at all. Article 16 of the AI Act requires providers of high-risk systems to comply with the accessibility requirements of Union law, which means the Web Accessibility Directive and the European Accessibility Act (1, European Union, 2024). For a citizen-facing chatbot, the Article 50 disclosure must be perceivable by screen-reader users, the alternative channel must be real rather than a dead end, and the system must perform acceptably in its users' languages, including the official minority languages of Nordic countries. Evaluate performance per language as a fairness dimension.

A public-sector worked example

The fundamental rights impact assessment (FRIA) is where these mechanisms are tied to the people they protect. Article 27(1) lists its content: the deployer's processes, the period and frequency of use, the persons and groups likely to be affected, the specific risks of harm, the human oversight measures, and the measures if risks materialise, including complaint arrangements; Article 27(3) requires notification of the results to the market surveillance authority, and Article 27(4), as amended by Regulation (EU) 2026/1744, allows the assessment to cross-refer to the relevant sections of an existing DPIA or include relevant parts of it (1, European Union, 2024) (27, European Union, 2026). Keep it in the repository with stable section identifiers, so that the register, the inventory entry, and the gates can cite individual findings.

Worked example: fundamental rights impact assessment outline

Illustrative example. The outline below is Nordvik Region's FRIA for its benefits-triage system, abridged in its findings but complete in its structure.

# Fundamental rights impact assessment: ai-system/benefits-triage
FRIA-2026-002 | Version 1.0 | Deployer: Nordvik Region | Completed: 2026-06-20
Complements: DPIA-2026-007 (GDPR Article 35) | Provider: TriageSoft (fictional)

## 1. Process (27(1)(a)) and 2. Period and frequency (27(1)(b))
The system scores urgency nightly; senior caseworkers review the ranked queue
each morning; caseworkers decide each application. Continuous for 24 months.

## 3. Persons and groups affected (27(1)(c))
All applicants; incomplete digital records (18%); paper applicants (9%);
households with children (41%); limited proficiency in the official
language (22%); persons with protected identity (<1%).

## 4. Specific risks of harm (27(1)(d)) and 6. Measures (27(1)(f))
| ID | Risk (Charter right)                          | Measure                      | Trigger and response              |
|----|-----------------------------------------------|------------------------------|-----------------------------------|
| R1 | Lower urgency for incomplete records (Art. 21, 34) | Fairness gate: FNR gap <= 0.03 | Weekly breach: suspend ranking for group |
| R2 | Urgent need in an attachment missed (Art. 1, 34)  | "Urgent attachment" flag     | Missed case in audit: case review |
| R3 | Caseworkers defer to the ranking (Art. 41)        | Blind sampling; override alerts | Override < 2% for 4 weeks: calibration |
| R4 | Applicants cannot contest prioritisation (Art. 41, 47) | Notice; explanation on request | Complaint: explanation in 10 days |
| R6 | Children's needs weighed only via parent (Art. 24) | Household composition as factor | Case review: Governance Forum |

## 5. Human oversight (27(1)(e))
Oversight design v1.2: never auto-accept, 5% blind sample, stop reverts to date
order. Oversight role holders trained (OVR-TRIAGE-101); 6 of 6 complete.

## 7. Notification (27(3))
Notified to the market surveillance authority on 2026-07-02.

Three features are worth copying: each risk has an identifier that the register entry, the fairness configuration, and the oversight design can cite, so the FRIA does not sit apart from the controls; each risk names a trigger as well as a measure, which turns Article 27(1)(f) into a monitoring rule; and the groups come from the service's own data, including groups such as paper applicants that a purely statistical fairness analysis would not have named.

Nordvik Region and Hansa Mutual

For Nordvik Region trustworthiness is a public-law duty before it is a design value. The transparency notice, the oversight specification (AIDR-0011) and FRIA-2026-002 all answer to residents' rights: the duty to give reasons, the right to be heard, and the Article 86 right to a clear and meaningful explanation from the deployer (1, European Union, 2024). The notice is written in plain terms and published, as the Region's openness tradition expects.

Hansa Mutual meets the same requirements across seven markets and several languages. The Article 50 notice for its claims correspondence assistant must be produced in each EU market's language, and national consumer information duties differ, so the group renders a notice per market and language from one source record. The United Kingdom and Swiss entities are outside Article 50, but the group applies the disclosure there as policy. Its fairness questions also differ in kind: its underwriting-support model must be tested against the insurance-specific expectations the European insurance supervisor has set out for AI governance (185, EIOPA, 2025). Nordvik owes explanations to residents; Hansa owes them to policyholders under differing national consumer rules.

Summary

Trustworthy AI is a design problem before it is an ethics problem. This chapter attached a mechanism and an evidence source to the High-Level Expert Group's requirements, within a sociotechnical boundary that includes the people around the model.

For LLMs the honest transparency is about the system, not the model's account of itself. Fairness requires an explicit, recorded choice between incompatible criteria, lawful handling of protected attributes under Article 4a, and a configuration that runs at the promotion gate and in production. Human oversight under Article 14 decomposes into an interruption path, an override path, visible uncertainty, measured automation bias, and an interface designed so that control is real.

These mechanisms address the harm a system can do when it behaves as designed. The harm an adversary can cause by making it behave otherwise, through prompt injection, poisoned data or a manipulated agent (38, OWASP, 2025), belongs to security architecture; for governance, the threat model is one more versioned artefact reviewed at the same gates. Chapter 11 closes Part III by assembling the evidence these mechanisms produce into a continuous assurance chain that serves conformity assessment, supervisory inspection and internal audit alike.

Sources

  1. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. (1, European Union, 2024)
  2. European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union. (2, European Union, 2016)
  3. High-Level Expert Group on Artificial Intelligence (2019). Ethics Guidelines for Trustworthy AI. European Commission. (12, AI HLEG, 2019)
  4. High-Level Expert Group on Artificial Intelligence (2020). Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment. European Commission. (13, AI HLEG, 2020)
  5. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. (50, Bender et al., 2021)
  6. Selbst, A. D., boyd, d., Friedler, S. A., Venkatasubramanian, S. & Vertesi, J. (2019). Fairness and Abstraction in Sociotechnical Systems. Proceedings of the Conference on Fairness, Accountability, and Transparency. (56, Selbst et al., 2019)
  7. Nordqvist, G. (2025). Architecture as Code. Companion volume to this book, published at aac.geon.se. (66, Nordqvist, 2025)
  8. Barocas, S., Hardt, M. & Narayanan, A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press. (72, Barocas et al., 2023)
  9. Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 2nd edition. (73, Molnar, 2022)
  10. Shneiderman, B. (2022). Human-Centered AI. Oxford University Press. (74, Shneiderman, 2022)
  11. UNESCO (2021). Recommendation on the Ethics of Artificial Intelligence. UNESCO. (29, UNESCO, 2021)
  12. UNESCO. Ethical Impact Assessment: A Tool of the Recommendation on the Ethics of Artificial Intelligence. UNESCO Global AI Ethics and Governance Observatory. (31, UNESCO, n.d.)
  13. European Insurance and Occupational Pensions Authority (2025). Opinion on Artificial Intelligence governance and risk management. EIOPA, 6 August 2025. (185, EIOPA, 2025)
  14. OWASP Foundation. OWASP Top 10 for Large Language Model Applications, 2025 edition. OWASP GenAI Security Project. (38, OWASP, 2025)
  15. European Parliament and Council of the European Union (2026). Regulation (EU) 2026/1744 (Digital Omnibus on AI). Official Journal of the European Union. (27, European Union, 2026)