Decode protobuf by hand so every step is an exact byte you can assert against: a varint 150 is 0x96 0x01, a tag for field 1 length-delimited is 0x0A, zigzag maps -1 to 1 and -2 to 3, a fixed32 is four little-endian bytes, a nested message recurses, a packed repeated list unpacks from one length-delimited chunk, an absent field takes its proto3 default, and an unknown field number is preserved as raw wire bytes. Every lesson is one concrete spec with real bytes in and structured values out.
Over 25 lessons you build a working Protocol Buffers codec from the wire format up: a library that decodes raw protobuf bytes into structured fields and encodes them back, byte for byte. There is no generated code and no .proto compiler anywhere in the project. You read and write the bytes yourself, so every step is a concrete value you can assert against and the same code works in any language.
You start with base-128 varints, the variable-length integers at the heart of the format, decoding and encoding them and pinning the truncated and over-long error cases. Then come tags and the four wire types, the scalar family (varint-encoded integers, zigzag for signed values, little-endian fixed32 and fixed64, and length-delimited strings and bytes), nested messages that recurse, and repeated and packed fields. The final chapter adds a tiny descriptor so you can decode into named fields with proto3 default values, preserve unknown fields as raw wire bytes, resolve last-one-wins for duplicated scalars, and round-trip a real message back to equivalent bytes.
This is a teaching-grade codec focused squarely on the proto3 binary wire format, and it is honest about its edges: it never generates code from a .proto file (you supply a tiny descriptor by hand), it covers the four current wire types but not the deprecated start-group and end-group types, and it does not implement maps, oneof, services, or proto2-only semantics as distinct concepts, since those all reduce to the wire-format primitives you build here. The result is a real decoder and encoder for the format that Google, gRPC, and countless systems move data with every day.
Every protobuf message is a flat run of bytes read strictly left to right, so the first thing we need is a cursor that hands out the next byte and remembers where it is. Today you build that reader, the spine the whole decoder threads through.
Build a reader over a byte slice that returns the next byte and advances its position.
A protobuf message has no framing, no delimiters, and no length at the front - it is just a sequence of bytes that you interpret in order. Because of that, almost every operation in this project is “read the next byte (or few), decide what it means, advance.” A small cursor that owns the buffer and a position is the one abstraction every later lesson leans on.
Today is deliberately tiny: wrap a byte slice, hand back bytes one at a time, and
keep track of how far you have read. The pos field is the whole point - it is
what lets tomorrow’s varint reader consume a variable number of bytes and leave
the cursor sitting exactly on the next field. Getting AtEnd right matters too:
the top-level decoder will loop “until the reader is at the end” to know when a
message is fully consumed.
// the whole decoder reads through one of thesetype Reader struct {buf []bytepos int}func NewReader(b []byte) *Reader { return &Reader{buf: b} }func (r *Reader) ReadByte() byte { b := r.buf[r.pos]; r.pos++; return b }func (r *Reader) AtEnd() bool { return r.pos >= len(r.buf) }
The codec correctly decodes and encodes every proto3 scalar type, nested messages, and packed and unpacked repeated fields, with proto3 defaults and unknown-field preservation and a byte-for-byte round-trip, but it trusts a hand-built descriptor rather than validating wire types against it, and it has no maps, oneof, groups, or .proto-file parsing.
The canonical specification of the binary wire format: base-128 varints, the field tag, the four wire types, zigzag for signed integers, length-delimited values, packed repeated fields, and the exact worked examples (150 is 0x96 0x01) this project pins.
The .proto schema language that names field numbers and types: scalar types, defaults, enums, repeated and packed, oneof, and maps. The descriptor you hand-build in the final chapter is a tiny slice of what this guide formalizes.
A focused deep dive on the variable-length integer that underlies everything in protobuf: why seven payload bits and a continuation bit, little-endian group order, and how the encoding trades bytes for range.
The why behind the format: a compact, schema-driven, forward and backward compatible serialization, and how unknown-field preservation and defaults make old and new code interoperate. Useful context for the schema-aware chapter.
A worthwhile comparison: MessagePack is a schema-less, self-describing binary format where each value carries a type byte, whereas protobuf is schema-driven and encodes only field numbers and wire types. Reading both clarifies what a descriptor buys you and what it costs.