Overview

Notes on key points from "Introduction to Data Analytics Platforms for Engineers."

What Is a Data Analytics Platform?

• A system built specifically for data analysis

  • That is, it differs from a general data platform. According to the following book, a data platform is not limited to data analysis alone — it also assumes use by business applications.

  "Getting Started with Data Engineering on Google Cloud"

• A combination of three components: data lake, data warehouse, and data mart

  • Data lake
    • Structured & unstructured data
    • Stores raw data as-is
  • Data warehouse
    • Structured data
    • Data in an organized and managed state
  • Data mart
    • Structured data
    • There is no clear boundary between a data warehouse and a data mart
    • Built for specific purposes, such as dashboards

History of Data Analytics Platforms

• Single-node → Multi-node → Cloud
• Single-node
  • Parallel processing at the thread level per CPU
  • Multiple CPUs share the processing workload
• Multi-node
  • Products such as Apache Hadoop/Spark
  • Multiple nodes share the processing workload

People Involved in Data Analytics Platforms

• Data Engineer
  • Collect and integrate data
  • Maintain data analytics platforms and data pipelines
  • Build and manage distributed systems
  • Optimize data pipelines through data ingestion and ETL
  • Manage storage where data is stored
  • Provide user access
• Data Scientist
  • Machine learning, deep learning
  • Python, Java
• Data Analyst
  • Deep knowledge of business processes
  • SQL, BI tools
  • Extract value from data

Self-Service Model

• Traditional model
  • Data engineers also handle creating data marts and dashboards
• Self-service model
  • Data engineers focus on creating new pipelines, resolving system issues, and optimizing costs
  • Data analysts create data marts; users create dashboards

Key Principles to Keep in Mind for Data Engineering

• Provide diverse interfaces
• Collect metadata (make it easy to understand where data is located)
• Visualize data quality
• Improve performance (leverage partitioning, file formats, and compression)
• Minimize costs
• Build fault-tolerant systems
• Enable fast recovery

Layers in the Data World

• Collecting layer: collect data (batch, streaming)
• Processing layer: process data
• Storage layer: store data
• Access layer: use data

Processing Layer

ETL and Data Wrangling

• Data wrangling (also called data preparation)
  • Data structuring: unstructured → structured
  • Data cleansing: remove duplicates, format, normalize
  • Data enrichment
• Difference between ETL and data wrangling
  • Both transform data
  • "Data wrangling is used to discover value in data; the rules found through data wrangling are then defined as workflows in ETL"

Encryption

• Transparent encryption: encrypt on write, decrypt on read
• Explicit encryption: makes original data unreadable on both read and write
• De-identification: makes it harder to identify individuals by replacing or substituting data values
  • Cohort pattern
    • Swap data values
  • Subtract pattern
    • Transform values using arithmetic operations (e.g., subtraction, division)

Data Quality / Metadata Calculation

• Tools exist, but at this stage, many teams implement data quality and metadata collection themselves using libraries such as Deequ for Apache Spark

Storage Layer

Managing with 5 zones makes lifecycle configuration and access control easier:

• Raw zone: store collected data as-is (data lake)
• Gold zone: data warehouse, data mart
• Staging zone: store data in an immutable state. If original data is modified, it cannot be restored. Use staging zone data to repair the gold zone.
• Quarantine zone: isolated zone for sensitive data. Restrict access permissions.
• Temporary zone: load any desired data on an ad-hoc basis. Set up automatic deletion.

Zone Management with Tags

• Physical zone separation
  • For example, separate storage per zone
  • Moving data between zones incurs time and cost
• Tag-based zone separation
  • Change zones by simply updating a tag
  • No physical data movement; zone switches instantly
  • For example, access control can be configured per tag to simplify access management

Access Layer

• GUI
• BI tools
• Direct storage access
• API access
• Message queues

SSOT (Single Source of Truth)

• Physical SSOT: physically consolidate data into one location
• Logical SSoT: present data as if it were consolidated in one location

Backup

• Full backup: back up all data
• Partial backup: back up some data
• Versioning: higher cost but enables efficient backup and restore

Access Control

Control at the following granularity:

• Zone
• Database
• Table
• Column

Technology Stack

Collecting Layer

• Batch
  • Embulk
  • Dump
• Streaming
  • Apache Kafka
  • Amazon Kinesis
  • Google Cloud Pub/Sub

Processing Layer

• Batch
  • Apache Spark
  • Apache Hive
• Streaming
  • Spark Streaming

Workflow Engine

• Digdag
• Apache Airflow
• Rundeck

Storage Layer

Formats

• Parquet

Compression Formats

• Snappy
• gz
• bz2

Splittable means a single file can be split across multiple nodes for processing.

◆ Splittable format and compression format combinations

	Parquet	Avro	CSV/JSON
Snappy	Y	Y	-
gz	Y	Y	N
bz2	-	-	Y
No compression	Y	Y	N

Storage Considerations

• Small files: too many files that are too small
• Data skewness: uneven file sizes across files — one file is enormous while another is relatively small

Access Layer

• BI tools
  • Redash
  • Metabase
  • Power BI
  • Tableau
  • Looker
• Notebooks
  • Jupyter Notebook
• API
  • API Gateway

Access Control

• chmod/chown
• IAM

Metadata Management

• Business metadata: table definitions, domain knowledge, etc.
• Technical metadata: data quality, profiles, etc.
• Operational metadata: operation history, etc.

Business Metadata

• Database name
• Table name
• Column name
• Data type
• Partition
• Table status
• Table owner

Technical Metadata

• Table extraction conditions: how data was ingested
• Lineage, provenance
• Table format (Parquet, Avro)
• Table location
• ETL completion time
• Scheduled table generation time

Lineage and Provenance

• Lineage: how data is connected / data flow
• Provenance: the origin or birth of data

Operational Metadata

• Table status
  • In-Service: today's workflow completed
  • Error: an error occurred in today's workflow
  • Investigating
• Metadata last updated timestamp
• Median file size
• Who is accessing the data

Data Profiling

• Cardinality
  • How spread out the data values are
  • Gender has low cardinality; IP address has high cardinality
  • Not particularly important to focus on in data analytics platforms
• Selectivity
  • How unique column values are; higher means more unique; maximum is 1
  • All 5 records have different values → selectivity = 1 = 5/5
  • Out of 5 records, 2 have the same value, so the column has 4 distinct values → 0.8 = 4/5
• Density (NULL density)
  • Density of NULL values
  • High NULL density may lead to the decision to exclude a column from queries
• Consistency
  • Whether data representation is consistent
  • Example: mixing UTC and JST timestamps
• Referential integrity
  • Whether ID formats and logical column names are consistent
• Completeness
  • What proportion of records have values in all columns
• Data type
• Range: whether values fall within a specific range
• Format: digit count for postal codes, etc.
• Value occurrence frequency
• Data redundancy
  • Minimum value is 1; lower is better; 1 is ideal
  • Higher values mean copies exist in many places
  • Lower values mean data is consolidated — i.e., SSoT is achieved
• Validity level
• Access frequency
  • How often a table is accessed
  • Used to detect unnecessary data
• Others
  • Sum, max, min, average, standard deviation

Data Catalog

What data exists. The catalog records information such as:

• Data title
• Format (CSV, JSON)
• Data owner