Projects/Build an HTTP Client

Build an HTTP Client

Model the transport as a plain byte stream and everything becomes checkable offline: a request is exact bytes ending CRLFCRLF, a response is those bytes read back. Each lesson pins one rule with concrete values - the default port for a scheme, a chunk that ends in 0 CRLF CRLF, a header value that keeps its colons, whether a 303 turns your POST into a GET. The finished core drives a real socket in the capstone.

41 lessonsMedium~20 min / lessonHTTP/1.1Message parsingChunked encoding
The project

What you'll build over the next 41 lessons

Over 41 lessons you build a working HTTP/1.1 client from first principles, one rule of the wire format at a time. You start by parsing a URL into scheme, host, port, path, query, and fragment - with default ports and percent-encoding - then serialize a request to the exact raw bytes an HTTP server expects: a request line, a Host header, a sorted header block, and a body framed by Content-Length, all ending in the blank line that terminates a message. Then you turn around and parse a response back from a byte stream: the status line, the headers (case-insensitive, colons and folding and all), and the body - by Content-Length, by chunked transfer-encoding, or by reading to the end of the stream.

With the message core solid you handle the things that make a client real: reading several responses off one keep-alive connection, following 301/302/303/307/308 redirects (and knowing which ones turn a POST into a GET), decoding and re-emitting cookies through a jar, and encoding Basic auth and application/x-www-form-urlencoded bodies. The whole transport is modeled as an abstract byte stream, so every lesson is verified offline against in-memory buffers with exact expected bytes.

This is a teaching-grade client built around the real HTTP/1.1 message grammar. The capstone finalize wraps the unchanged request/response core in a real TCP socket and performs a live GET, printing the status, headers, and body. It is honest about its limits: it speaks HTTP/1.1 in the clear with no TLS (so no https transport), does not do gzip or other content-codings, and its redirect and cookie handling follow the common rules rather than every corner of the specifications.

build-an-http-client / lesson-01.md
Lesson 01 · Parsing the URL

The scheme

Every request starts from a URL, and the very first thing a client reads off it is the scheme - http or https - which decides how to talk to the far end. Today you build the smallest possible URL parser, one that pulls just the scheme, so the type every later lesson thickens exists from day one.

The goal

Parse a URL string into a value whose scheme field holds the lowercased scheme.

Start here - the target
TO DO
Scenario: Reading the scheme off a URL
Giventhe URL string "http://example.com"
Whenit is parsed
Thenthe parsed URL has scheme "http"
Andparsing "HTTPS://example.com" gives scheme "https" (schemes are lowercased)
Background

A URL is the address a client aims at, and its first component is the scheme: the http or https before the ://. The scheme is the single most important thing about a URL because it decides everything downstream - which default port to use, whether the connection is encrypted, how the request is framed. So the parser earns its keep by reading the scheme first.

Schemes are case-insensitive, so HTTP, Http, and http all mean the same thing; the convention is to store them lowercased so later comparisons are simple equality checks. Keep the type small today - a struct with one filled field and room for the rest. Everything else about the URL arrives one lesson at a time on top of this.

Make it work
// the whole client will grow around this type
type URL struct {
Scheme string
// Host, Port, Path, Query, Fragment arrive over the next lessons
}
// the scheme is everything before the first "://", lowercased
func Parse(raw string) (*URL, error) {
// split on "://", lowercase the left half
}
CheckpointDONE
You can construct a URL value and read its scheme, lowercased. Commit and stop here.
Scope & extensions

Where this project stops - and where to go next

The finished client correctly speaks plaintext HTTP/1.1 over a real TCP socket end to end - GET, headers, chunked and length framing, keep-alive, redirects, and cookies - but it deliberately stops short of a production client: no TLS, no compression, and several simplified subsystems.

Extend it next
  • Add TLS/HTTPS support (dial 443 with a TLS handshake for https URLs) - the single biggest gap, since almost every real site requires it today
  • Support gzip/deflate content-coding by sending Accept-Encoding and decompressing a Content-Encoding: gzip response body
  • Implement full relative-reference URL resolution (dot-segments, paths relative to the current directory) instead of only absolute-URL and absolute-path Location values
  • Build a real cookie jar with per-cookie Domain, Path, and Expires tracking instead of the current name-only, send-everywhere jar
  • Add connection pooling so requests and redirect hops reuse one keep-alive connection per host instead of dialing a fresh TCP socket each time
  • Expose configurable timeouts and a redirect limit so a slow or looping server cannot stall the client
Recommended reading

Books & references that go deeper

  • The current definition of methods, status codes, header fields, redirects, and content semantics - the "what" of HTTP, independent of the wire version. The authority for the redirect and header rules in this project.

  • The HTTP/1.1 message format itself: the request line, the header block, message framing by Content-Length and chunked transfer-encoding, and connection management. This is the byte grammar the whole client is built to.

  • The URI grammar and percent-encoding rules behind chapter one - scheme, authority, path, query, fragment, and which characters are reserved.

  • A readable, example-driven companion to the RFCs: headers, methods, status codes, cookies, and redirects with worked examples. The friendliest place to look up a header while you build.

  • How Set-Cookie and Cookie actually work - parsing cookie attributes and the storage model behind the cookie jar in chapter six.

  • The base64 alphabet and padding rules you implement for Basic authentication in chapter five.