Apache Avro Specification
Avro 是数据 serialization 系统具有 schemas, used 在 Hadoop 和 Kafka 为 compact 二进制 storage.
Schema-Based Serialization
Avro requires a JSON-defined schema describing the data structure. The schema is stored with the data for self-describing files, or exchanged separately for compact wire format. Data is encoded in a compact binary format using the schema for type information. Schema evolution is supported with rules for adding, removing, and renaming fields with defaults. Avro supports complex types: records, enums, arrays, maps, and unions.
- schema
- JSON-defined structure
- evolution
- Add fields with defaults
- unions
- Nullable and polymorphic
stored or externalbackward compatible["null","string"]Ecosystem Usage
Avro is the primary serialization format for Kafka with Confluent Schema Registry. Hadoop uses Avro for file storage with built-in compression and splitting. The format supports both binary and JSON encodings. Avro RPC uses schemas for type-safe remote calls. The schema registry pattern enables centralized schema management and compatibility enforcement. Avro is a strong choice for big data pipelines.
- Kafka
- Primary format
- Hadoop
- File storage
- RPC
- Schema-based calls
Schema Registrysplittable, compressedtype-safe