Case Studies
Chapter 17 closed Part IV with the constraints that public authorities face. This chapter opens Part V, which turns from what should be done to what has been observed, with eight case studies. The first five are anonymised composites: each combines several organisations and engagements, details have been altered, and none describes a real, named organisation, although the pattern is real. The last three are real, documented European cases the reader can check: a Dutch judgment on government risk scoring, UK government evaluations of an embedded workplace assistant, and Italian enforcement against a conversational assistant that crossed borders through the GDPR's one-stop-shop.
Figure 18.1: The case study map, showing which recurring patterns each of the five composite cases exercised most heavily.
How to read the cases
The outcomes are deliberately modest: each organisation moved one or two systems to a state in which the evidence was in the repository and the controls were in the pipeline. Every case is read through three lenses: the usage mode of Chapter 2 (usage mode matrix), the organisation's role under Regulation (EU) 2024/1689, the AI Act, and the controls that worked, failed, or were missing. The documented cases predate the AI Act's obligations or fall outside its reach, so reading them through it asks what it would require, not whether it was breached.
Case 1: A Nordic social insurance authority introduces decision support
Context
A national authority handles well over a million sickness and parental benefit claims a year, under administrative procedure law that permits automated decisions in defined circumstances and requires reasons for every decision.
Challenge
Handling times had drawn criticism from the parliamentary ombudsman, and a data science unit proposed to route claims with a model predicting which would need more documentation. The legal function saw a step towards automated decision-making without a legal basis, the data protection officer asked for an impact assessment, and the case officers' union asked what the model knew about their members' work.
Approach
The team made decision support a structural boundary. The model produced a priority score and a list of likely missing documents, never a proposed decision; the decision record carried no reference to the score; and the audit log recorded whether the officer had opened the evidence. The authority classified the system as high-risk under Annex III of the AI Act, as it supported evaluation of eligibility for public assistance benefits, and accepted a fundamental rights impact assessment under Article 27, registration under Article 49(3), and the deployer obligations of Article 26 (1, European Union, 2024). Because no decision was based solely on automated processing, the system stayed outside Article 22 of the GDPR, as the AI Decision Record documented (2, European Union, 2016). The pipeline refused to promote a model version with an outdated register entry and published a register projection on the Dutch model (23, Dutch Ministry of the Interior, n.d.), and the union received aggregate override data from the first week.
Outcome
Median handling time for the affected claims fell by about a quarter in the first year, and the override rate settled near eight per cent, within the range defined as healthy. Explanation remained unresolved: officers could explain their decisions but not the score, so claimants were told it was a scheduling aid, and whether that satisfies Article 86 if the system's role grows is recorded as open (1, European Union, 2024).
Case 2: A pan-European bank builds a model risk platform under DORA
Context
A bank operating in a dozen Member States had several hundred models built on different tooling over fifteen years. Regulation (EU) 2022/2554, DORA, had made digital operational resilience a board-level duty, with governance under Article 5, an ICT risk management framework under Article 6, a register of information on ICT third-party arrangements under Article 28, and major incident reporting under Article 19 (3, European Union, 2022).
Challenge
The bank could not list its production models, owners and third-party components within a week, nor say for which high-risk credit models it was provider and for which deployer under the AI Act (1, European Union, 2024). The supervisor asked for the AI inventory and received a spreadsheet that three units disputed.
Approach
The bank built a model registry, an evaluation service, and an evidence store of signed promotion manifests, but the decisive move was organisational: a model not in the registry was not in production, units had a year to register, and inference endpoints could load only approved artefacts. Each entry carried a machine learning bill of materials (ML-BOM) in the CycloneDX format (43, OWASP, n.d.), which fed the DORA register of information, and a field recording the bank's AI Act role. Incident classification against the DORA thresholds was automated, so that reportability could be assessed within hours.
Outcome
Eighteen months later the registry held every production model, and the number had fallen by a fifth as units found models nobody used. The supervisor's next request was answered with a generated report anchored to a commit, and two models reclassified as high-risk had their provider obligations planned rather than discovered. Three business-critical models were forced into an uncomfortable exception process, and the bank judged the discomfort to be the point.
Case 3: A university hospital group deploys clinical AI within the European Health Data Space
Context
A group of university hospitals serving some two million people wanted AI for imaging triage, while researchers wanted to train models on its data. Regulation (EU) 2025/327, the European Health Data Space (EHDS), strengthens patients' control over primary use of their health data and provides for secondary use through data permits issued by health data access bodies and secure processing environments (11, European Union, 2025).
Challenge
Both ambitions shared infrastructure under different legal positions. A CE-marked triage tool bought from a vendor made the hospital a deployer under Article 26, and the AI Act treats AI safety components of medical devices as high-risk (1, European Union, 2024), whereas training on patient data was secondary use under the EHDS (11, European Union, 2025). The architecture did not separate the two, and the data protection officer had stopped approving projects.
Approach
The team split the estate along the primary and secondary use boundary. The clinical path ran vendor devices with local validation before go-live and a named radiologist role able to suspend the tool. The research path became a secure processing environment in which data entered only under a permit and left only as aggregated results or checked models, and the pipeline refused training jobs whose data contract did not match their registered purpose.
Outcome
The tool went live in two hospitals; local sensitivity was slightly below the vendor's figure, which was reported to the vendor. The research environment issued its first permits within six months. Improving the tool with local data, however, would make the hospital a provider and move the work into the secondary use regime, a line the group has so far declined to cross.
Case 4: A manufacturing group adopts agents for maintenance
Context
An industrial group with plants across Europe and a mature predictive maintenance programme wanted an agent to read maintenance histories, query spare-parts inventory, draft work orders, and schedule technicians through defined tool interfaces.
Challenge
In its first live week the agent created duplicate work orders, ordered parts already in stock after retrying a timed-out call with different parameters, and scheduled a technician who lacked the required certification. The German works council pointed out that allocating work to named technicians could fall within the AI Act's high-risk employment categories and was subject to co-determination, and inference costs had tripled in a week.
Approach
The group applied bounded autonomy through a version-controlled permission manifest. Parts orders and work orders above a cost threshold required a supervisor's approval through an interface showing the agent's trace, and task allocation to individuals was removed: the agent proposed a skill profile, and a supervisor assigned the person. That took the system out of the employment-related high-risk category and satisfied the works council, whose agreement was recorded in the AI Decision Record and enforced in the manifest.
# agents/maintenance-planner/permissions.yaml
agent: maintenance-planner
version: 2.3.0
budget:
max_cost_eur_per_day: 250
on_exceed: suspend_and_alert
tools:
- name: create_work_order
constraints:
assignee: forbidden # supervisor assigns people
requires_approval_above_eur: 500
- name: order_parts
requires_approval: always
- name: schedule_technician
allowed: false
idempotency:
work_order_key: [asset_id, fault_code, detected_at]
retry_policy: no_retry_on_timeout
The idempotency key addressed the duplicate orders, and the budget block added a hard stop.
Outcome
Unplanned downtime fell, mostly thanks to the underlying predictive models; the agent drafted work orders within minutes rather than at the next shift handover. Supervisor approvals ran at about forty a day per plant, the price of control, and the budget guardrail triggered twice in six months after model updates changed the agent's verbosity, and both suspensions were judged correct.
Case 5: A municipality runs a citizen chatbot
Context
A city of around half a million residents bought a multilingual conversational service about municipal services from a vendor, using retrieval-augmented generation, which grounds a model's answers in documents retrieved at query time (63, Lewis et al., 2020), over its own content.
Challenge
The pilot gave a confident, incorrect answer on eligibility for a housing supplement and invented a deadline for a school placement appeal. Residents did not always realise they were talking to a system, and the council asked whether the service should continue.
Approach
The city kept the service and changed its architecture: because Article 50(1) of the AI Act requires providers to design systems that interact directly with natural persons so that those persons are informed that they are interacting with an AI system, the disclosure opened every conversation in every language (1, European Union, 2024). A version-controlled scope allowed answers on what services exist and how to apply, but not eligibility assessments, which were handed over to a case officer with the transcript. A golden set of several hundred questions became the regression suite for every content update and vendor model change, retrieval was restricted to a curated index so that a wrong answer could be traced to a wrong page, and the city published a register entry on the Dutch model (23, Dutch Ministry of the Interior, n.d.).
Outcome
Within a year the service handled a substantial share of first-line enquiries, handover requests ran at a few per cent of conversations, and the regression suite caught two vendor model updates that would have reintroduced eligibility answers, and the council has since asked for a register entry for every automated system in the city.
Documented European cases
Case 6: SyRI, a government risk-scoring system struck down in court
What happened
System Risk Indication (SyRI) was a Dutch legal instrument for detecting benefits, allowance and tax fraud by linking data held by different public bodies; pseudonymised records were checked automatically against a risk model and decrypted only when flagged. On 5 February 2020 the District Court of The Hague held that the legislation did not strike the fair balance Article 8(2) of the European Convention on Human Rights requires between combating fraud and respect for private life (202, District Court of The Hague, 2020). The risk model and its indicators were not public, and SyRI's application was insufficiently transparent and verifiable, which weighed heavily because such a model risks discriminatory effects. The court declared the legislation unlawful and without binding effect.
Through the book's lenses
SyRI was a built system, closest to Mode 4. Under the AI Act the ministry would be provider and deployer of a high-risk system under Annex III point 5(a), with the duties of Articles 26, 27 and 49 that Case 1 accepted (1, European Union, 2024). Pseudonymisation existed; what failed was verifiability, so proportionality could not be shown.
Case 7: Embedded workplace assistants in UK central government
What happened
From 30 September to 31 December 2024 the UK Government Digital Service gave 20,000 civil servants a Microsoft 365 Copilot licence; participants reported saving an average of 26 minutes a day, calculated from self-reported ranges, and the report warned that searching with each user's own access could expose files users should not have been able to open (203, UK Government Digital Service, 2025). The Department for Business and Trade evaluated its 1,000-licence share with diaries, control-group interviews and observed tasks (204, Department for Business and Trade, 2025): written tasks saved time, scheduling and image generation took longer, there was no evidence that time savings had improved productivity, and quality assurance of outputs was inconsistent.
Through the book's lenses
This is Mode 2: the organisation configures access and features, not the model. An EU public body would be a deployer with the Article 4 literacy obligation and no high-risk classification for general drafting (1, European Union, 2024). What worked was a time-boxed trial with control groups; what failed was permission hygiene and output assurance left to individuals. Because the two reports measured different things, they also show why Chapter 16 asks for falsifiable benefit hypotheses.
Case 8: A conversational assistant, a national authority and the one-stop-shop
What happened
In March 2023 the Italian data protection authority, the Garante, temporarily banned ChatGPT in Italy, and within weeks OpenAI added a privacy notice, a training opt-out and age verification; in February 2024 OpenAI established an Irish entity, making the Irish Data Protection Commission its lead supervisory authority under the one-stop-shop (206, Christakis, 2026). The Garante's Decision No. 755 of 2 November 2024, announced after the European Data Protection Board's Opinion 28/2024 on AI models (19, EDPB, 2024), found, among other breaches, training on users' data without a prior legal basis, imposed a EUR 15 million fine and passed continuing matters to the Irish authority (205, Garante, 2024). On 18 March 2026 the Court of Rome upheld the appeal against the decision (judgment No. 4153/2026), its reasoning not yet published, and the Garante temporarily removed the decision from its website (205, Garante, 2024).
Through the book's lenses
For an organisation whose staff use ChatGPT this is Mode 1: OpenAI is the provider, and the organisation is a deployer under the AI Act and a controller under the GDPR (2, European Union, 2016) for what staff put into prompts. The controls the authority demanded sat with the provider, so the deployer's levers were the service tier and contract. Because a service can be withdrawn from one market at short notice, dependent processes need a fallback.
Patterns across the cases
The composite cases converge on the same patterns. Each drew a structural boundary around what the AI system may do, such as decision support rather than decision, and each made its register the single source of truth enforced by the pipeline, turned a regulatory obligation into a pipeline gate, gave those who could stop the programme an early seat, and recorded what remained unresolved. The documented cases confirm this from the other side: SyRI lacked the verifiability Cases 1 and 5 built in, and the assistants of Cases 7 and 8 left the deployer only permissions, contract and fallback as levers. Architecture as Code (Chapter 24) drew the same lesson: constraints are design inputs, not obstacles (66, Nordqvist, 2025).
Across the usage modes
In Mode 4 the organisation owns the decisive control, whereas in Modes 1 and 2 most controls belong to the provider. No case covers Mode 0 or Mode 3; Chapter 19 examines agents in an evaluation environment.
| Case | Usage mode | Role under the AI Act | Control that mattered most |
|---|---|---|---|
| 1 Social insurance authority (composite) | Mode 4 | Deployer of a high-risk system | Register-gated deployment and FRIA |
| 2 Pan-European bank (composite) | Mode 4 | Provider or deployer, model by model | Registry as the condition of production |
| 3 Hospital group (composite) | Mode 4, vendor medical device | Deployer; provider if it retrained | Local validation; secure processing environment |
| 4 Manufacturing group (composite) | Mode 5 | Provider and deployer of its own agent | Permission manifest and budget stop |
| 5 Municipality (composite) | Mode 4, procured | Deployer | Declared scope and regression suite |
| 6 SyRI, Netherlands (real) | Closest to Mode 4 | Would be provider and deployer, high-risk | Missing: verifiability |
| 7 UK central government (real) | Mode 2 | Outside the AI Act; deployer in the EU | Control groups; web grounding off |
| 8 ChatGPT in Italy (real) | Mode 1 | Deployer; OpenAI is provider | Deployer's tier, contract and fallback |
Nordvik Region and Hansa Mutual
Nordvik Region resembles the authority of Case 1, whose unresolved question about explaining the score is the one the Region's reasons log and transparency record in Chapter 17 try to answer, and the municipality of Case 5, whose regression suite is what a buyer of hosted models most needs. SyRI is its closest warning, which is why the claims ranker has a public register entry (NR-ALG-2026-003) and FRIA-2026-002 is kept where an outside reviewer can read it. The UK evaluations shape its Mode 2 pilot: tidy shared-drive permissions first, and measure benefits against a comparison group.
Hansa Mutual resembles the bank of Case 2, whose registry rule its shared platform needs across seven jurisdictions, and its IT-operations agent (AIDR-0027) and claims-triage agent need the approval design of Case 4. The ChatGPT case shows that a national ruling can change what staff in one market may use, so Mode 1 assistants run only through enterprise tiers under group contracts. Because usage dashboards and diary studies observe employees' work, its Mode 2 evaluation also needs works-council co-determination in Germany, works-council involvement in the Netherlands and union consultation in Sweden before measurement starts.
Summary
The composite cases show the book's disciplines under realistic constraints: decision support kept outside GDPR Article 22, a registry made the condition of production, primary and secondary health data use separated, an agent bounded by a permission manifest, and a chatbot given a declared scope and regression suite. The documented cases test them against the public record: SyRI struck down a model nobody outside could inspect, the UK evaluations show that a reported time saving is not a measured productivity gain, and the Italian enforcement shows that a conversational assistant's controls sit with the provider.
Chapter 19 sets these patterns against the OpenAI–Hugging Face sandbox escape of July 2026, in which an agent collective defeated correlated controls, and Chapter 20 catalogues the anti-patterns that appear when organisations do the opposite.
Sources
- European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
- European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union, 4 May 2016. (2, European Union, 2016)
- Ministry of the Interior and Kingdom Relations (Netherlands). The Algorithm Register of the Dutch government. (23, Dutch Ministry of the Interior, n.d.)
- European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 27 December 2022. (3, European Union, 2022)
- European Parliament and Council of the European Union. Regulation (EU) 2025/327 on the European Health Data Space. Official Journal of the European Union, 5 March 2025. (11, European Union, 2025)
- District Court of The Hague (2020). Judgment of 5 February 2020, NJCM et al. v. the State of the Netherlands (SyRI), case C/09/550982 / HA ZA 18-388, ECLI:NL:RBDHA:2020:1878 (English translation of ECLI:NL:RBDHA:2020:865). Rechtspraak.nl. (202, District Court of The Hague, 2020)
- UK Government Digital Service (2025). Microsoft 365 Copilot Experiment: Cross-Government Findings Report. GOV.UK, 2 June 2025. (203, UK Government Digital Service, 2025)
- Department for Business and Trade (2025). Evaluation of the M365 Copilot pilot in the Department for Business and Trade (corporate report "Microsoft 365 Copilot pilot: DBT evaluation report"). GOV.UK, 28 August 2025. (204, Department for Business and Trade, 2025)
- Christakis, T. (2026). Generative AI and GDPR Enforcement in Europe: A Lot of Noise, One Fine, Zero Survivors. Cross-Border Data Forum, 19 March 2026. (206, Christakis, 2026)
- European Data Protection Board. Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models. Adopted 17 December 2024. (19, EDPB, 2024)
- Garante per la protezione dei dati personali (2024). ChatGPT, the Italian Data Protection Authority closes the preliminary investigation. OpenAI will have to carry out a six-month information campaign and pay a fine of EUR 15 million (press release of 20 December 2024 on Decision No. 755 of 2 November 2024, with the Garante's note on its removal following judgment No. 4153/2026 of the Court of Rome, published 18 March 2026). Garante, docweb 10085432. (205, Garante, 2024)
- Nordqvist, G. (2025). Architecture as Code. Companion volume to this book, published at aac.geon.se. (66, Nordqvist, 2025)
- OWASP Foundation. CycloneDX Machine Learning Bill of Materials (AI/ML-BOM). (43, OWASP, n.d.)
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S. & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). (63, Lewis et al., 2020)