Avro Compact Encoding
Avro compact encoding is Avro's binary serialization scheme, using type tags plus compact value encoding, much smaller than JSON Avro.
Purpose
To provide efficient Avro serialization in Kafka and Hadoop, with the schema not transmitted per record (pre-registered).
Mechanism
A record is encoded in field order: int/long use zigzag plus varint, string is a varint length plus UTF-8, bytes is a varint length plus raw bytes, a union uses the union index plus the value, and arrays/maps use block counts plus elements.
Use cases
- Kafka Avro messages
- Hadoop data files
- Efficient RPC