Testing an AI system before deployment verifies whether it solves the task reliably enough to be trusted. It is essential to test on your own real data, not on sample data, and to set an acceptance criterion expressed as a number in advance – for example, the required proportion of correct outputs or the maximum number of errors of a certain type. The test set should also include edge cases and atypical cases, because it is precisely these that matter in practice, as well as cases where no correct answer exists, in order to verify whether the system admits uncertainty instead of producing a made-up answer. A comparison with a human's result on the same sample is useful. Document the test results – they form part of the technical documentation and a basis for later comparison.
See also: Technical documentation of an AI system, AI system monitoring, Testing before investment.
