Parquet

Parquet is a columnar binary storage format, an Apache project used for OLAP analytics, compressing by column for efficient queries.

Purpose

To provide columnar storage and encoding so analytical queries read only the columns they need, greatly reducing I/O.

Mechanism

A file is split into row groups, and each row group stores column data in column chunks. Within a column, data is compressed with RLE/dictionary/delta encoding. The footer stores the schema and statistics (min/max). Reads push down predicates per column to filter.

Use cases

Spark/Hive analytical queries
Data lakes (Iceberg, Delta)
BI report data sources
S3/Azure big data storage

Examples

Input
```
table data
```
Output
```
A columnar compressed file
```
Stored by column

Common errors

Too many small files inflates metadata overhead: Consolidate into larger row groups

Related terms

Avro is a row-oriented binary serialization format from the Hadoop ecosystem; its schema is embedded or external, and it is used for big data storage and Kafka.

CSV (comma-separated values) is a tabular data format that separates fields with commas; RFC 4180 defines the standard, and it is widely used for data import and export.

JSON (JavaScript Object Notation) is a lightweight data interchange format based on JavaScript object literals, using key-value pairs and arrays.

References