When is an AI system high-risk?
Appendix: Applying the Article 6(3)(a) and (d) criteria to illustrative examples
s. 52–62pp. 52–62 · PDF 56–66
Application to illustrative examples#s. 52
To demonstrate the practical application and test the robustness of the proposed criteria, this section applies the framework to illustrative cases drawn from Annex III sectors. These examples are not intended as empirical claims about industry practice but as analytical exercises designed to concretise and stress-test the criteria.
Document translation for asylum assessment#s. 52
Kvalifiserer for (d) bare når saksbehandleren har praktisk og reell tilgang til kompetanse i kildespråket; ellers stryker den på kriterium 5. Område 7 (migrasjon, asyl og grensekontroll)
Qualifies for (d) only where the case officer has practical, non-illusory access to source-language proficiency; otherwise fails Criterion 5. Area 7 (migration, asylum and border control)
Case An AI system is provided to translate asylum applicants’ written statements to the language of the receiving state. The translations serve as input to the eligibility assessment performed by the asylum case officer.
| Criterion | Assessment |
|---|---|
| 0Is the intended use within an Annex III area? | Asylum application examination is part of Annex III area 7. |
| 1Does the task involve profiling of natural persons? | The system processes personal data but does not evaluate personal aspects; translation is a linguistic operation, not an assessment of the person. |
| 2Is the task specific, defined, and bounded? | The task is specific and bounded: input is a document in the applicant’s native language; output is the same document in the target language. |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | Translation involves semantic processing (engagement with meaning) but does not generate evaluative conclusions about the applicant’s credibility, the merits of their claim, or any other aspect relevant to the following assessment. It is not merely procedural, but it is purely preparatory. |
| 4Does the task output bear on the substance of an assessment or decision? | The output (a translated document) makes information accessible but does not alter its substantive content; the same information is available in both languages. At first glance the counterfactual test is met: minor differences in translation approach would not change the underlying facts. On the other hand, important linguistic nuances might, as they say, be “lost in translation”. |
| 5Is the task output transparent and verifiable? | This is where the analysis requires contextual sensitivity. If the case officer has access to the native language proficiency – either personally or through a qualified interpreter – the translation is verifiable and the criterion is met if it can be demonstrated that such access is not illusory. If, however, the case officer must rely exclusively on the AI translation without any means of verification, the output becomes functionally determinative of the informational basis for the assessment. In the latter scenario, the criterion is not satisfied. |
| 6Is the task output inherently resistant to uncritical reliance? | A translated document presents factual content without evaluative framing and is inherently resistant to automation bias, provided the assessor is aware that the text is AI-translated, and has mandate and access to native language proficiency in cases of doubt. |
Conclusion The system may qualify for the Article 6(3)(d) carve-out, but only where the conditions of deployment ensure practical verifiability of the translation. The same AI system may qualify in one deployment context and not in another – a result consistent with the task-centred and context-dependent nature of the Article 6(3) assessment.
Selecting job interview candidates: a multi-stage recruitment workflow#s. 53
Job applicant data extraction#s. 53
Oppfyller alle seks kriterier og kvalifiserer etter både (a) og (d). Område 4(a) (arbeidsliv – rekruttering)
Satisfies all six criteria; qualifies under both (a) and (d). Area 4(a) (employment – recruitment)
Case The AI system extracts structured data fields from CVs and cover letters: names, contact details, educational qualifications, dates of employment, certifications, and language proficiencies.
| Criterion | Assessment |
|---|---|
| 0Is the intended use within an Annex III area? | Recruitment is part of Annex III area (4)(a). |
| 1Does the task involve profiling of natural persons? | The system processes personal data but does not evaluate personal aspects; it performs data extraction and conversion, not assessment of the person. The profiling threshold is not engaged. |
| 2Is the task specific, defined, and bounded? | The task is specific (extract specified data), bounded (input: CVs and cover letters; output: structured data fields), and narrow. |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | The operation is strictly syntactic: it maps unstructured text to structured data fields using formal parsing rules. It precedes a following, functionally separate assessment. It is both procedural and preparatory. |
| 4Does the task output bear on the substance of an assessment or decision? | The output (a structured set of data) does not bear on the substance of the hiring decision; the same substantive information is available to the assessor in the original CV. |
| 5Is the task output transparent and verifiable? | The output is transparent (the assessor understands that data has been extracted) and verifiable (the assessor can check each data point against the source documents). |
| 6Is the task output inherently resistant to uncritical reliance? | A structured set of data does not invite uncritical reliance; it presents factual data without suggesting evaluative conclusions. |
Conclusion The system satisfies all six criteria and qualifies for both Article 6(3) carve-out (a) and (d).
Score candidates against clear, rules-based criteria#s. 54
Transparent, regelbasert poengsetting er likevel profilering; unntakene er utilgjengelige. Kan falle utenfor KI-forordningen om den gjennomføres uten egenskapene til et KI-system (fortalepunkt 12). Område 4(a)
Transparent, rule-based scoring is still profiling; carve-outs unavailable. May fall outside the AI Act if implemented without AI-system features (Recital (12)). Area 4(a)
Case The AI system assigns numerical scores to candidates based on transparent, predefined criteria: for example, awarding points for each year of relevant work experience, for specific educational degrees, and for listed language proficiencies. The scoring formula is crafted by natural persons, documented, fixed, and deterministic.
| Criterion | Assessment |
|---|---|
| 1Does the task involve profiling of natural persons? | Despite the mechanical nature of the scoring, the system uses personal data to evaluate aspects of the natural person – their qualifications and experience – in relation to the job requirements. This constitutes profiling within GDPR Article 4(4). The assessment terminates here. |
Conclusion The Article 6(3) carve-outs do not apply. The fact that the scoring formula is transparent and rule-based does not change its characterisation as profiling in the context of Article 6(3): The relevant question under the profiling override is not whether the evaluation is sophisticated or opaque, but whether it constitutes an automated evaluation of personal aspects – which it plainly does. It might seem counterintuitive that such clear, rules-based processing should not be covered by a derogation. This use case could in fact be covered by another AI Act “safety valve”, that of the definition of “AI system” in itself: Recital (12) specifies that the notion of “AI system” should not cover systems “that are based on the rules defined solely by natural persons to automatically execute operations”. In other words, such scoring tasks can be automated outside the scope of the AI Act by assigning them to separate systems without machine learning components or other features covered by the “AI system” definition.
Score candidates against ambiguously defined criteria#s. 55
Profilering; ville også strøket på kriterium 3, 4, 5 og 6. Område 4(a)
Profiling; would also fail Criteria 3, 4, 5 and 6. Area 4(a)
Case The AI system evaluates candidates against soft criteria such as “leadership potential,” “cultural fit,” or “communication skills,” producing a qualitative or numerical assessment.
The analysis is identical to the previous sub-task in its engagement of the profiling override, but the additional ambiguity of the criteria compounds the concerns at every subsequent level. Even setting aside the profiling threshold, the task fails at Criterion 3 (the operation is semantic, requiring evaluative judgment about the person), Criterion 4 (the output bears directly on the substance of the hiring decision), Criterion 5 (the output is opaque – the basis for a “cultural fit” score cannot meaningfully be verified by the human assessor), and Criterion 6 (an evaluative score on ambiguous criteria is maximally susceptible to automation bias). Likewise, as such soft criteria are all but impossible to capture in formal rules designed solely by natural persons, the above-mentioned “AI system”-definition escape is closed.
Create a final ranking of candidates#s. 56
Profilering; ville også strøket på kriterium 3, 4 og 6. Område 4(a)
Profiling; would also fail Criteria 3, 4 and 6. Area 4(a)
Case The AI system produces a ranked list of candidates for the recruiter’s consideration.
This is the composite output of the preceding scoring stages. Whether the ranking is based on rule-based scores, ambiguous criteria, or a combination, the system evaluates personal aspects of natural persons and produces an output that directly structures the hiring decision. The profiling override applies. Even if the profiling override were not applicable, the system would fail at Criterion 3 (the ranking involves semantic evaluation of candidates’ suitability), Criterion 4 (the ranking bears on the substance of the hiring decision by directing the recruiter’s attention), and Criterion 6 (a relevance ranking is precisely the kind of output that invites uncritical reliance).
Observations from applying the criteria to the recruitment workflow#s. 56
The recruitment workflow example demonstrates a characteristic gradient: the earliest stages (data extraction) are genuinely procedural or preparatory and may qualify for a carve-out, while each successive stage introduces evaluative elements that move the system further from the procedural or preparatory domain and into the territory of substantive assessment. The example also illustrates the importance of the profiling override as a decisive threshold: the moment an AI system begins evaluating the suitability, qualifications, or personal characteristics of candidates – even through transparent, rule-based criteria – it constitutes profiling and is categorically high-risk.
Writing a medical journal: a multi-stage clinical documentation workflow#s. 56
Transcribe audio recordings to text#s. 56
Kvalifiserer for (d) forutsatt at det opprinnelige lydopptaket fortsatt er tilgjengelig for kontroll. Område 5(a) og 5(c)
Qualifies for (d) provided the original audio remains accessible for verification. Areas 5(a) and 5(c)
Case An AI system converts the audio recording of a medical consultation into a written transcript.
| Criterion | Assessment |
|---|---|
| 0Is the intended use within an Annex III area? | Clinical documentation may contribute to the evaluation of the eligibility of natural persons for healthcare services – Annex III area (5)(a) – and in risk assessment in life and health insurance – Annex III area (5)(c). AI systems generating clinical documentation might also be products covered by Article 6(1) and Annex I A(11) – such systems are out of scope for the Article 6(3) derogations. |
| 1Does the task involve profiling of natural persons? | The system processes personal data (the patient’s voice, name, and health-related statements) but does not evaluate personal aspects; it converts speech to text without assessing the person’s medical condition or any other personal characteristic. |
| 2Is the task specific, defined, and bounded? | The task is specific and bounded (input: consultation audio recording; output: text transcription of the recording). |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | The operation involves semantic processing – converting spoken language to written form requires engagement with meaning – but does not generate evaluative conclusions about the patient’s health or treatment. |
| 4Does the task output bear on the substance of an assessment or decision? | The transcript makes information accessible in a different modality without altering its content; the same information is available in the original audio recording. |
| 5Is the task output transparent and verifiable? | The output is transparent and verifiable: the physician can listen to the original recording and check the transcript. This verification capacity is critical; if the audio was not stored, or deleted before verification, the criterion would require reassessment. |
| 6Is the task output inherently resistant to uncritical reliance? | A verbatim transcript does not invite uncritical reliance in the way that a summary or analysis would. |
Conclusion The Article 6(3)(d) carve-out applies, provided that the original audio remains accessible for verification. This is analogous to the document translation example above.
Classify speakers (doctor, patient(s))#s. 58
Mer semantisk enn transkripsjon, men fortsatt forberedende; kvalifiserer for (d) hvis lydopptaket er tilgjengelig. Område 5(a) og 5(c)
More semantic than transcription but still preparatory; qualifies for (d) if audio remains accessible. Areas 5(a) and 5(c)
Case The AI system identifies and labels which portions of the transcript were spoken by the doctor and which by the patient(s).
| Criterion | Assessment |
|---|---|
| 1Does the task involve profiling of natural persons? | The system processes personal data (voice characteristics) but does not evaluate personal aspects of the patient in any meaningful sense; speaker identification is a formal classification based on acoustic or contextual features, not an assessment of the person’s health, behaviour, or characteristics. |
| 2Is the task specific, defined, and bounded? | The task is specific and bounded (input: consultation audio recording; output: text transcription of the recording with speaker labels). |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | The operation involves a significant degree of semantic processing (audio to text, distinguishing speakers by vocal characteristics) but the classification is formal (who spoke) rather than evaluative (what the speech means for the patient’s condition). |
| 4Does the task output bear on the substance of an assessment or decision? | Speaker labels do not alter the substantive content of the transcript; they merely make it more accessible for a later assessment. |
| 5Is the task output transparent and verifiable? | The output is verifiable – the physician, who was present at the consultation, can readily confirm speaker attributions. |
| 6Is the task output inherently resistant to uncritical reliance? | Speaker labels do not invite uncritical reliance in the medical assessment. |
Conclusion While the degree of semantic processing is larger than for the transcription subtask, the Article 6(3)(d) carve-out still applies, again provided that the original audio remains accessible for verification.
Classify message content (symptoms, diagnoses, medical advice, prescription references)#s. 59
Er ikke prosessuell; overordnede merkelapper (symptom/diagnose/legemiddel) kan forsvares som forberedende; detaljerte merkelapper («symptom på X») ville utløst profilering og strøket. Område 5(a) og 5(c)
Not procedural; high-level labels (symptom/diagnosis/medication) are defensibly preparatory; detailed labels (“symptom of X”) would trigger profiling and fail. Areas 5(a) and 5(c)
Case The AI system assigns semantic labels to segments of the transcript, categorising statements as descriptions of symptoms, diagnostic conclusions, therapeutic advice, medication references, and so on.
| Criterion | Assessment |
|---|---|
| 1Does the task involve profiling of natural persons? | The system uses the patient’s personal data to classify statements about their medical condition. Categorising a patient’s statement as a “symptom” involves some evaluation of the medical significance of what the patient has said – depending on the level of detail this could approach the threshold of assessing a person’s “health” within the meaning of GDPR Article 4(4). A binary “symptom: True/False” classification should pass below the threshold, for more detailed “symptom of X” classifications the profiling override will soon apply. |
| 2Is the task specific, defined, and bounded? | The task is specific and bounded (input: consultation audio recording; output: text transcription of the recording labelled by speaker and statement type). |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | Classifying whether a physician’s statement constitutes a “diagnosis” versus “medical advice” requires significant semantic understanding of medical content – a judgment about the evaluative significance of information. Still, the task is at its core preparatory in nature, making no evaluation of the significance or correctness of the labelled statements. |
| 4Does the task output bear on the substance of an assessment or decision? | Like speaker labels, statement classifications primarily make the journal transcript more accessible for a following assessment. The theoretical possibility of scenarios where such classification structures the journal in a way that shapes how subsequent readers (including the consulting physician themselves) interpret the consultation must be balanced against effet utile. |
| 5Is the task output transparent and verifiable? | The output is verifiable – the physician, who was present at the consultation, can confirm or reject the statement classifications. |
| 6Is the task output inherently resistant to uncritical reliance? | Again, like speaker labels, high level statement classification (“symptom”, “diagnosis”, “medication”) does not replace medical assessment of the statement’s content. |
Conclusion The task is clearly not procedural and challenges the boundaries of the Article 6(3)(d) “preparatory” derogation. Still, given sufficiently high-level classification categories (“symptom”, “diagnosis”, “medication”) and responsible users, an assessment of no significant risk of harm is defensible. On the other hand, a system performing more detailed classification (“symptom of X”, “diagnosis of illness Y”, “recommendation of medication Z”) would fail multiple criteria, including the initial “profiling” filter.
Remove non-relevant content (e.g. interruptions, off-topic discussion)#s. 60
Å fjerne innhold er en redaksjonell handling som påvirker realiteten i journalen; stryker på kriterium 4. Område 5(a) og 5(c)
Removing content is an editorial act bearing on the substance of the record; fails Criterion 4. Areas 5(a) and 5(c)
Case The AI system identifies and removes portions of the transcript deemed irrelevant to the medical record – social pleasantries, interruptions, digressions about unrelated topics.
| Criterion | Assessment |
|---|---|
| 1Does the task involve profiling of natural persons? | The profiling filter may or may not be engaged depending on how the system determines “non-relevance”: if the system merely predicts the degree of a statement’s general “medical relevance”, making no evaluation of the medical content of the statement, it should avoid the profiling threshold. |
| 2Is the task specific, defined, and bounded? | The task is specific and bounded (input: consultation audio recording; output: text transcription of the parts of the recording classified as medically relevant). |
| 3Is the task syntactic or semantic? If semantic, is it preparatory to an assessment? | Predicting medical relevance is a non-procedural task requiring significant semantic understanding of medical content. Furthermore, non-relevance classification includes an evaluation of the significance of statements that challenges the upper bounds of what can be defended as purely preparatory. |
| 4Does the task output bear on the substance of an assessment or decision? | The output is not a neutral data transformation; it is an editorial act that determines what information the physician and subsequent readers will and will not see in the final transcript. If the system incorrectly classifies a patient’s mention of a symptom as “off-topic” and removes it, that information is lost from the medical record – a consequence that bears directly on the substance of subsequent assessments. The counterfactual test of Criterion 4 is met: a time-pressed physician’s assessment could differ depending on what the system chose to include or exclude, what the physician themselves remembers from the consultation, and how much time they feel they can afford to spend comparing the edited transcription to the raw transcription. This terminates the case assessment. |
Conclusion Article 6(3) carve-outs do not apply. Selective removal of content from a draft medical record is an editorial function that bears on the substance of the medical documentation and, by extension, of the assessments based on it.
Score confidence of classifications to prioritise human review#s. 61
Konfidensscorene styrer legens oppmerksomhet og innbyr til å overlate deler av kontrollen til systemet; stryker på kriterium 6. Område 5(a) og 5(c)
Confidence scores direct the physician’s attention and invite delegation of review; fails Criterion 6. Areas 5(a) and 5(c)
Case The AI system assigns a confidence score to each of its classifications (from sub-tasks above) and uses these scores to flag portions of the journal for priority human review.
This sub-task has a dual character. On one hand, the confidence scoring is a meta-operation on the system’s own outputs – not a direct evaluation of the patient, or the medical content of transcribed statements. On the other hand, the practical effect of the confidence scoring is to direct the physician’s attention: high-confidence classifications may receive cursory review, while low-confidence classifications are flagged for closer scrutiny. This attention-directing function engages Criterion 6: the confidence score creates a cognitive shortcut that incentivises the physician to defer to the system’s own assessment of which parts of the journal require careful review. The system is telling the human overseer where to look – and, implicitly, where not to look. This is precisely the kind of cognitive pre-structuring that the Article 14 safeguards are designed to counteract.
Conclusion Article 6(3) carve-outs do not apply. The confidence scoring directs the human assessor’s attention in a manner that bears on the quality and thoroughness of the medical review.
Observations from applying the criteria to the clinical documentation workflow#s. 62
The speaker-labelling and statement-classification subtasks illustrate that an AI system may perform semantically rich operations and still qualify under Article 6(3)(d), provided that the task remains genuinely preparatory and does not steer the subsequent medical assessment. By contrast, the confidence-scoring subtask shows the boundary of that conclusion. A system that labels speakers or statement types without confidence scores may remain within the carve-out, while a system returning the same labels with confidence scores may fall outside it, because the scores direct the physician’s attention and invite reliance on the system’s own assessment of reliability. The result is counterintuitive: providing the physician more information shifts the system into a higher risk category. In my view, de lege lata the distinction is defensible. Article 6(3) is concerned not with the volume of information an AI system produces, but with whether the form of that information meaningfully shifts cognitive effort from the human assessor to the system. A bare classification leaves the assessor in the position of judging whether to trust each label on its own evidentiary footing; a confidence-scored classification invites the assessor to partially delegate that judgment to the system. The Article 14 design requirements are calibrated to the latter case, and so is the high-risk regime as a whole. The de lege ferenda question of whether providers should be incentivised differently, for example, by allowing a graduated regime that admits transparent confidence reporting under stricter human-oversight conditions, is taken up in Section 5.4.