Data quality

Содержание и основные понятия

01
The data checks first.
Type
The date, number, text and category shall not be mixed in one column.
Pass
The empty cell isn't zero.
Units
The column's signature holds one: ruby, kg, min, %.
02
The check before the formula
1.
Save the source page with a separate version.
2.
Check the headlines, duplicates, and the range of data.
3.
Lock down the pass rule.
4.
Only then count the average, the sum per condition, or the figure.
Important
Don't correct the initial observations to confirm the desired conclusion.
03
Data quality protocol
Complete
Count the empty values for each field; the decision to "remove" or "replace" should depend on the meaning of the sign, not on the convenience.
Coherence
Bring the dates, categories and units to the same format. "1.5 kg" and "1,500 g" cannot be compared as starting lines.
Plausibility
Set the tolerance range and study the emissions. A rare value can be a mistake, or perhaps the most important observation.
Create a clean-up log: reference value, rule, new value and reason; then conversion can be repeated and checked and the raw data remain unchanged.