In 2020, a Dutch court halted an AI system for violating human rights. The Dutch government had deployed SyRI (System Risk Indication) to detect welfare and tax fraud by correlating massive cross-agency datasets covering housing, taxes, and benefits. SyRI flagged individuals based on probabilistic correlations rather than individual evidence, and certain population groups faced a higher risk of being tagged as fraudulent based on demographic patterns, leading to invasive audits, frozen benefits, and severe financial distress for people who had committed no wrongdoing (read more here).
SyRI is one example of a wider problem: businesses tend to think of AI risk as technical risk (model failure, system downtime, data breaches), when ethical risk lives inside the algorithms themselves. There are six distinct types of ethical concern that can arise from the way AI algorithms work. Understanding them is essential for any business that wants to use AI responsibly.
1. Inconclusive evidence: when AI mistakes correlation for certainty
A pattern across a population might be wrong about any single individual.
SyRI, the Dutch government’s welfare-fraud detection algorithm, is a clear case of this risk: it drew conclusions from statistical correlation across a population rather than from evidence about any one individual.
AI systems work with probability, not proof. A flagged pattern shows what tends to co-occur across a population. It does not show what actually happened to any one person.
A model that identifies a pattern across a population may be accurate at the group level while getting any single individual wrong. Treating that pattern as certain, without acknowledging its limits, turns a statistical shortcut into a real harm.
2. Inscrutable evidence: the problem with unexplainable AI
Sometimes outputs cannot easily be explained.
Many AI models, particularly deep learning systems, produce outputs that cannot be easily explained. Even the people who built the system often cannot trace how the data going in connects to the conclusion coming out.
This opacity becomes a problem when consequential decisions rest on outputs nobody can interrogate. In credit scoring, recruitment, or medical triage, people affected by algorithmic decisions have a legitimate interest in understanding how those decisions were reached.
3. Misguided evidence: biased data, biased outputs
Garbage in, garbage out…
The outputs of an AI system are only as good as the data it was trained on. Biased, incomplete, outdated, or tampered data produces biased, unreliable outputs. This is the GIGO principle: Garbage In, Garbage Out.
Bias gets frozen into the system at the data stage and then propagates through every prediction the model makes. When humans label the training data, their own biases get embedded in the model along with it.
4. Unfair outcomes: when accurate models still discriminate
Technical accuracy could be relying on the wrong proxy.
A technically accurate model can still produce discriminatory outcomes. A decision’s effect on a particular group can make it unfair on its own, regardless of whether the underlying data or reasoning held up.
Algorithmic systems can also latch onto proxies for protected characteristics, using postcode as a stand-in for race, for example, and discriminate on that basis without ever touching the protected attribute directly.
5. Transformative effects: how AI shapes behaviour, not just reflects it
AI systems shape the world.
AI systems shape the world at least as much as they reflect it. Recommender systems influence what people read, buy, and believe. Filtering algorithms create echo chambers. Personalisation narrows the range of options presented to individuals.
These effects can erode individual autonomy in ways that are subtle and hard to detect. When an algorithm nudges behaviour to serve third-party interests over the individual’s own, it crosses an ethical line.
6. Traceability: who is responsible when AI causes harm
Pinning down responsibility can be difficult.
When an AI system causes harm, pinning down responsibility is often extremely difficult. A gap opens between the designer’s intentions and the algorithm’s real-world behaviour, and blame can end up scattered across multiple human and algorithmic actors at once.
Good traceability, through logging, documentation, and audit mechanisms, is what makes accountability possible. Without it, harm goes unaddressed and trust erodes.
The business implication
These six risks are active in many AI systems currently deployed in business contexts. Recognising them is the first step to addressing them, through better data governance, more rigorous testing, greater transparency, and governance structures that keep humans meaningfully in control.
Illustrations: Hand-drawn by Aletta Simpson