What does Adversarial Attack mean?

An adversarial attack is a targeted attack on an AI model in which input data is modified to cause the model to produce an incorrect output. Attackers use this manipulation to flip classifications, bypass content filters, or gain unauthorized access. Unlike data poisoning, the trained model itself remains untouched; only the input provided at runtime is altered.

The best-known method is the Fast Gradient Sign Method. It calculates the gradient of the loss function for the fully trained model and shifts each pixel by a small amount in the direction of the error. To the human eye, the image appears unchanged. Iterative methods, such as Projected Gradient Descent, repeat this step hundreds of times to identify smaller perturbations. In the case of language models, the attack targets the text—for instance, through hidden instructions embedded in an uploaded document.

An attack surface exists wherever a model processes third-party inputs and immediately derives a decision from them. Vulnerable systems include image recognition used for access control, spam and fraud filters, and language models that pre-sort submitted documents. An attacker does not need direct access to the system; the file or image they are already permitted to submit is sufficient.

Protecting a model against adversarial attacks begins with separating content from instructions: treating inputs as data rather than commands removes the basis for the attack. Adversarial training using manipulated examples hardens image models against known patterns. Logging inputs and outputs allows for post-incident analysis by revealing which input triggered a specific output.

The EU AI Act lists robustness against manipulation as a requirement for high-risk systems. It mandates adherence to the verifiable state of the art, which includes defense against known attack methods, documentation of measures taken, and a reporting mechanism for incidents. This standard evolves with the emergence of new techniques.