Bias in AI arises when a system systematically disadvantages a particular group of people. The cause is usually the composition of the training data reflecting past practice, a lack of representation of certain groups, or inappropriately chosen input features that indirectly stand in for protected characteristics – for example place of residence or the length of a career break. The consequence can be unlawful discrimination with sanctions and reputational damage. It can be limited by testing outputs on individual groups both before deployment and during operation, by checking the composition of the data, and by human review of adverse decisions. It is essential that responsibility lies with the company that uses the system, not the model supplier, so testing cannot be shifted onto them. It is worthwhile documenting the results of testing, because they are precisely the evidence that the company took appropriate measures. Without a record, a defence before the supervisory authority is difficult to mount.
See also: AI in Recruitment, Fundamental rights impact assessment, Training data management.
