Avro

Avro is a row-oriented binary serialization format from the Hadoop ecosystem; its schema is embedded or external, and it is used for big data storage and Kafka.

Purpose

To provide compact, schema-aware serialization for big data and stream processing.

Mechanism

When data is encoded, the schema can be embedded (written into the header) or external (a registry). There are no field tags (the schema locates fields), so it is more compact than Protobuf. It supports schema evolution (field addition and removal with default values). Row-oriented storage suits OLAP.

Use cases

Kafka message serialization
Hadoop file format
Hive table storage
Confluent Schema Registry

Examples

Input
```
schema + data
```
Output
```
Compact binary bytes
```
Schema-aware encoding

Common errors

Schema mismatch causes decoding errors: Manage schema versions with a Schema Registry

Related terms

Protocol Buffers

Protocol Buffers is Google's language-neutral binary serialization format; it requires a .proto schema definition and is more compact and type-safe than JSON/XML.

Parquet is a columnar binary storage format, an Apache project used for OLAP analytics, compressing by column for efficient queries.

JSON (JavaScript Object Notation) is a lightweight data interchange format based on JavaScript object literals, using key-value pairs and arrays.

References