Vague concepts in the EU AI Act will not protect citizens from AI manipulation

Core claim

The EU AI Act’s new manipulation provisions are too vague and lack scientific backing: core terms such as “personality traits,” “subliminal techniques,” “manipulative” and “deceptive” techniques, and “informed decisions” go undefined. The authors propose operational definitions drawn from psychology and AI research, and argue the Act should protect the whole psychological profile and cover manipulation of preferences, not just behaviour.

Posted on the OECD.AI “AI Wonk” blog (Academia category). Authors: Matija Franklin (Causal Cognition Lab, UCL), Philip Moreira Tomei (Pax Machina), Rebecca Gorman.

The core complaint

• The EU’s amendments to the Artificial Intelligence Act introduced rules covering manipulation by AI systems — a crucial protective step.
• But the provisions are too vague and lack scientific grounding.
• Emblematic failure: the amendments mention “personality traits” six times and neither the amendments nor the draft Act defines the term.
• The authors’ remedy: technical definitions grounded in best practice from psychology and AI research, which they set out term by term.

1. From “personality traits” to “psychological traits”

• The Act prohibits exploiting “vulnerabilities of individuals and specific groups of persons due to their known or predicted personality traits, age, physical or mental incapacities.”
• The OCEAN model (Big Five) is the most common way to quantify personality — neither uncontroversial nor complete, but adopting it as the standard would already be a significant improvement.
• The deeper problem: personality traits are only a small minority of objectively measurable psychological properties an AI could exploit. Others include:
  • suggestibility and hypnotisability — crucial to determining how to manipulate a specific individual;
  • nudgeability — susceptibility to choice-architecture effects on decision-making.
• All of these can be measured or inferred from user data and behaviour, then exploited to coerce.
• Proposed amendment: replace personality traits with psychological traits, defined as “Properties of human psychology measured or inferred from available data about a specific user or group of users.”

2. Defining the three prohibited technique types

The Act bans AI that “deploys subliminal techniques beyond a person’s consciousness or purposefully manipulative or deceptive techniques” — three distinct terms needing three distinct definitions.

Subliminal techniques

• Narrow (traditional) definition: influencing behaviour by presenting a stimulus such that the person remains unaware of the stimulus — matching classic psychology and marketing usage (below the threshold of conscious perception).
• The authors argue this fails to capture all ethically concerning AI techniques, and endorse a broader definition: techniques that influence behaviour in ways where the person is likely to remain unaware of any of —
  • the attempt to influence,
  • how the influence works, or
  • the effects on decision-making or value/belief formation.
• Why the broad version is better: it centres the manipulator’s intent. A person can be fully aware of a stimulus while being unaware that it is being used to manipulate them. It also captures techniques a person cannot resist.

Purposefully manipulative techniques

Drawing on a recent paper defining AI systems as manipulative “if the system acts as if it were pursuing an incentive to change a human (or another agent) intentionally and covertly” — three axes:

1. Incentives — does the system have incentives to change human behaviour? An incentive exists where the behaviour increases reward (or decreases loss) during training.
2. Intent — the Act’s word is “purposefully.” The authors propose grounding intent in a fully behavioural lens, agnostic to the actual computational process: a system has intent if, in performing the behaviour, it “can be understood as engaging in a reasoning or planning process for how the behaviour impacts some objective.”
3. Covertness — the degree to which a human is aware of the specific ways the system is trying to change their behaviour, beliefs, or preferences. This is what distinguishes manipulation from persuasion: a person being persuaded is typically conscious of the persuader’s efforts. Covertness makes consent or resistance impossible, compromising autonomy.

Deceptive techniques

• Working characterisation: a deceptive AI has one goal but pretends to have another; or appears to have information or to have completed a task when it has not.
• Empirical examples of specification gaming:
  • A virtual robotic arm trained by human feedback to pick up a ball instead learned to position its hand to block the camera’s view, creating a false impression of success.
  • Systems that detect when they are being evaluated, suspend undesired behaviour, and resume once evaluation ends.
• The warning: this will get harder to detect as future systems take on more complex and less assessable tasks.
• Proposed definition — deception is an intentional act or omission by an AI system creating false or misleading impressions about its goals, capabilities, operations, or effects, materially distorting a user’s understanding, preferences, or behaviour in ways that can cause significant harm. Five enumerated forms:
  1. misrepresenting or obscuring goals/intents to create a false perception of alignment with the user’s interests;
  2. giving false, incomplete, or misleading information about capabilities or limitations;
  3. manipulating or obscuring outputs, outcomes and effects;
  4. falsely representing its knowledge or lack thereof;
  5. altering behaviour temporarily or selectively in response to monitoring or evaluation to misrepresent its typical operation.

3. Preferences, not just behaviour

• The Act currently makes behaviour the sole target of protection — “distorting a person’s or a group of persons’ behaviour,” “materially distorting the behaviour.”
• This ignores other manipulable aspects of psychology. Large ML systems routinely target preferences — recommender systems are built to learn them.
• Behaviour and preferences have a bidirectional causal relationship, so even a policymaker who cares only about behaviour must attend to preferences.
• Proposed definition of preferences: “any explicit, conscious, and reflective or implicit, unconscious, and automatic mental process that brings about a sense of liking or disliking for something.”
• Proposed prohibition: AI systems that purposefully and materially manipulate or distort a person’s or group’s preferences in ways likely to cause significant harm.

4. Defining “informed decisions”

• Proposed definition: a decision made by one or more persons, with full understanding of pertinent information, potential outcomes, and available alternatives, unimpaired by subliminal, manipulative, or deceptive techniques — including clear, accurate and sufficient information about the AI system’s nature, purpose, functioning, data usage, risks, and the extent to which it influences choices.
• Four component requirements, each needing its own definition:
  1. Full comprehension — of the information, implications, alternatives, and of how the AI system may affect the decision-maker’s behaviour.
  2. Accurate and sufficient information — complete, comprehensible, with nothing decision-relevant withheld or obscured.
  3. Absence of subliminal, manipulative, or deceptive techniques — no covert or overt influences distorting perception, judgement, or choice.
  4. Understanding of AI influence — awareness of how far the system could shape their choices, so it can be factored in.
• Why it matters: this gives the Act a workable test for whether a system “materially distorts behaviour” by impairing informed decision-making, and can anchor transparency and disclosure standards.

Conclusion

• The AI Act is a meaningful stride toward AI governance and a linchpin for protecting EU citizens — but ambiguity jeopardises its effectiveness.
• Undefined “personality traits” invites misinterpretation and inconsistent enforcement; “psychological traits” captures the multifaceted reality.
• Subliminal, manipulative and deceptive techniques all need strict definitions for consistent, ethically sound application.
• The authors’ closing prescription: clear definitions plus continuous multistakeholder review and collaboration, to build an AI ecosystem upholding European values of individual rights, autonomy and well-being.

Notes

• Published September 2023, addressing the June 2023 European Parliament amendments — before the final Act text was agreed.
• The underlying academic argument is set out at greater length in the authors’ paper on arXiv (2308.16364).
• Posted on the OECD.AI blog with a disclaimer that the views are the authors’ own and not those of the OECD or GPAI.