Data collection monitoring is tracking whether an automated task is actually running and whether it delivers the expected results. The most dangerous situation is not an obvious failure, but a silent error: the source website changes its structure, the tool keeps running, but stores empty or incorrect values. The reports then look fine, even though they are built on nonsense. It is worth tracking, therefore, not only successful completion of the task, but also the number of records processed compared with the usual value, the share of empty fields, and a basic sanity check, for example whether prices fall within the expected range. In the event of a deviation, the system should send an alert to the responsible person. Operations should also include a record of when the data last went through an update.
See also: Anomalies and Alerts, ETL and Data Flow, Validation of collected data.
