Metadata to Manage

Metadata can be divided into three types: business metadata, technical metadata, and operational metadata.

Specific examples of each are as follows:

• Business metadata

  • Table owner
  • Column used for table partitioning
  • Column data type
  • Whether column data contains duplicates (whether IDs are duplicated)

• Technical metadata

  • Data extraction conditions
  • Source tables or files
  • Table location
  • Scheduled update time for the table

• Operational metadata

  • Table status
    • Whether the table generation process completed successfully
  • Table last updated datetime
  • Metadata last updated datetime

Data Quality Tests

• Zero control
  • The total company sales minus the sum of sales from each business unit equals 0
• Ratio control
  • Whether each business unit's sales ratio falls within a certain range
    • Example: Is Business Unit A's monthly sales amount within 95%–105% of the previous year's figure?
• Range check
  • Whether a business unit's sales amount falls within a certain numerical range
    • Example: Business Unit A's quarterly sales amount is between 1.5 billion and 1.7 billion

Reference

Introduction to Data Analysis Infrastructure for Engineers