Adversarial Machine Learning

Signs it might be this

• The company or independent researchers confirm that a system was fed misleading inputs.
• The same oddly wrong result appears repeatedly and only for inputs with something unusual in common.
• A cluster of bad outcomes coincides with a change in the data the system was learning from.
• The wrong answers consistently favor one outside party.
• The organization running the system has publicly acknowledged and described the problem.

Signs it is probably something else

• The error is occasional and varies at random, which is typical of ordinary model limits.
• The system was recently updated or retrained and results shifted for known reasons.
• The only evidence is that the answer seems unfair or surprising.
• The tool was used outside what it was built for.
• A person can review and correct the result when asked.
• A person reviews the automated result and corrects it without fuss.
• The error is rare, random, and fixed after you report it.
• The organization explains how the tool is tested and monitored.

Questions to ask yourself

• Is there a human who can review this decision if I ask?
• Has the organization or an independent researcher explained what went wrong?
• Is this a one-off error, or a repeated, consistent one?
• Am I treating the system’s confidence as proof of correctness?
• What other evidence do I have, apart from the system’s answer?

What it is

Adversarial machine learning studies how inputs built on purpose can push AI systems into mistakes. It also studies how to defend against that.

For ordinary people the point is simple. A filter, a fraud check or a screening tool can be fooled, and it will still return a confident answer.

Strange or wrong outputs are far more often ordinary limitations than attacks. Spotting a wrong answer is not the same as finding an attack.

Why it matters

The systems people are told to rely on, such as filters, fraud checks and screening tools, return confident answers. The person affected often has no way to see why.

If such a system has been misled, the error can land on someone who did nothing.

How it shows up

• An attempt to make an AI system misread, misjudge or misclassify what it is given.
• A concern for any tool that makes automated decisions, such as spam filters, fraud flags, content moderation and screening software.
• A field of research as well as a threat, since defenders study it to make systems safer.
• Usually invisible to the person on the receiving end, who sees only the system’s answer.
• Distinct from ordinary AI error, which happens without anyone trying to cause it.

What you can do

Pick what fits. You do not owe anyone a confrontation.

A check to run before you share or act

“This result came from an automated system, and a confident answer is not the same as a correct one. Can a person review it, and what else do I have that supports or contradicts it?”

Ask about it gently

Ask the organization in plain words how a decision was reached and whether a person reviewed it. Request the answer in writing.

Set a clear limit

Decline to treat an automated result as final for anything important. Ask for a human decision-maker to look at your case.

Step back

If a tool keeps giving you doubtful answers, stop relying on it for important matters and use a second source or a person.

Get support

Take an unexplained automated decision to a consumer protection office, an advocacy group or a legal aid service. Bring copies of what you were told.

At work, school or with services

If you work with automated tools, log odd results with dates and examples. Report them to whoever maintains the system, and keep a human check on high-stakes outputs.

Using this label fairly

• Blaming every strange AI output on an attack can excuse real design flaws. It can also hide the ordinary biases and limits that cause most errors.
• Accusing a specific person or company of tampering with a model, with no evidence, is a serious claim and can do real harm.

How it differs from look-alikes

Strategic Leaking releases real information at a chosen moment. Adversarial machine learning misleads an AI tool so that its judgments go wrong.

Strategic Deception is a broad campaign to mislead an adversary’s decisions through many channels. PSYOP Message Framing is crafted to move human audiences.

An adversarial input fools a machine’s reading of its input.