Chunk extensions. Most people know HTTP/1.1 can return a "chunked" response body: it breaks the body up into chunks, so that we can send a response whose length we don't know in advance, but also it allows us to keep the connection open after we're done. What most people don't know is that chunks can carry key-value metadata. The spec technically requires an implementation to at least parse them, though I think it is permitted to ignore them. I've never seen anything ever use this, and I hope that never changes. They're gone in HTTP/2. (So, also, if you thought HTTP/2 was backwards compatible: not technically!)
The "Authorization" header: like "Referer", this header is misspelled. (It should be spelled, "Authentication".) Same applies to "401 Unauthorized", which really ought to be "401 Unauthenticated". ("Unauthorized" is "403 Forbidden", or sometimes "404 Not Found".)
Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets".
Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications. Any proxy or reverse proxy are allowed to strip them. Indeed it can be argued that a conformant proxy is required to strip them, due to MUST ignore unknown extensions value requirement. (I suspect most do not strip them, and there is an argument to be made that blindly passing them through if not changing encoding could be considered ignoring them, but I'm not certain that is actually a conforming interpretation).
Plus surely there are many crusty middleboxes that will break if anybody tried to use that feature. Remember all the hoops websockets had to jump through to have much of a chance working for most people because of those? Many break badly if anything they were not programmed to handle tries to pass through.
> Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications.
Oof, I hadn't mentally connected those dots, but you're completely right. (As Transfer-Encoding is hop-by-hop, not end-to-end…)
Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them.
Lots of servers I've encountered with my browser stealth actually violate the spec and send "lengths" that do not match the sent payload lengths afterwards. Some reverse proxies also mess up the last chunk, so they're violating the spec there, too, and send a chunk with a negative length...and the spec doesn't even define how to handle this. I've also seen servers send random lengths in between, but without a payload that follows.
I would also like to add range requests (206 partial content) here. In practice, it's totally unpredictable how a server behaves when requesting multiple content ranges. Some reply with no range at all, even with correct headers. Some reply with more or less ranges than requested. Some even reply with out of boundary ranges that are larger than the content length header of the same response because they seem to use a faulty regexp on the server side.
"Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them."
Always wondered if developers found that easy.
I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads.
With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.
> send a chunk with a negative length...and the spec doesn't even define how to handle this
Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)
Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets".
You are supposed to treat all of them as "opaque octets"... or something like this might happen:
You can't. At some point, you have to actually make use of the headers, and some of those uses require decoding to a string. There is some wiggle room here, such as doing things like,
header_as_raw_bytes == b"chunked"
which I would argue is still decoding the header: your language of choice had to encode that string into bytes in some encoding in the first place, so even though you're comparing the encoded forms, there's still a character encoding at work.
But, some of the headers are case-insensitive. E.g., Content-Type, Accept, Expect, etc.
That golang bug is precisely not treating the non-characters (the "opaque octets", as defined by the standard, that is, the octets that form obs-text) as if they were characters. You won't hit that bug in the safe subset, presuming you're implementing other parts of the standard correctly. (Which is… a huge assumption, given HTTP's complexity, but that's sort of the point here.)
What you pass in the "Authorization" header is an user identity, which is established through authentication. And the server uses this identity to decide if you are authorized.
I've been searching for a while for a good way to know whether a client has disconnected in the middle of a long-running HTTP request. (We do heavyweight SQL queries of indeterminate length in response to those requests, and we'd like to be able to cancel them, rather than wasting DB-server CPU cycles calculating reports nobody's going to consume.)
You can't actually know whether the outgoing side of a TCP socket is closed, unless you write something to it. But it's hard to come up with something to write to an HTTP/1.1-over-TCP socket before you respond with anything, that would be a valid NOP according to all the protocol layers in play. (TCP keepalives would be perfect for this... if routers didn't silently drop them.)
But I guess sending an HTTP 102 every second or two could be used for exactly this: prodding the socket with something that middleboxes will be sure to pass back to the client.
If so, that's awesome! ...and also something I wish could be handled for me automatically by web frameworks, because getting that working sounds kind of ridiculous :)
Actually using HTTP 1xx will trigger hideous bugs, and worse those bugs will be timing-dependent. There is exactly one (non-websocket, which isn't really HTTP just pretends to be) codepath used in the wild: preapproval for a large file upload.
This problem is one reason why success/error should NOT be the first thing to send. It should be a trailer.
(HTTP/HTML tendency to substitute the response body for a human-visible error would require another mechanism to "reset" the response body.)
As of right now, while server authors will continue to need to support it, it is unlikely that it is a well tested code path, and it will likely break in weird ways even trying to use it.
So pre-approval for a large file upload is not even valid anymore.
> TCP keepalives would be perfect for this... if routers didn't silently drop them.
There are lots of middleboxes that don't pass along empty TCP packets. TCP keepalive is in a similar situation to IPsec: great for an Intranet, or for two public-Internet static peers with a clear layer-3 path between them; but everything falls apart in B2C scenarios.
Plus, to add to this problem: HTTP has gateways (proxies et al.) Doing TCP keepalive on the server end, only tells you whether the last gateway in the chain before the server is still connected to the server, rather than whether the client is still connected to the server.
Unless you can get every gateway in the chain to "propagate" keepalive (i.e. to push keepalive down to its client connection, iff the server pushes keepalive down onto it), silent undetected TCP disconnections will still happen—and even worse, you'll have false confidence that they aren't happening, as all your sockets will look like they're actively alive.
For what I'm doing, the client end isn't likely to have any gateways, so TCP keepalives "would be" workable for my use-case if not for the middlebox thing. But in full generality, TCP keepalives aren't workable, because there's always those corporate L7 caching proxies + outbound WAFs messing things up, even when L4 middleboxes aren't.
Keep your TCP keepalives for running connection-oriented stream protocols within your VPC. For HTTP on the open web, they're pretty unsuited. You need L7 keepalives. (If you've ever wondered, this is why websockets have their own L7 keepalives, a.k.a. "ping and pong" frames.)
> and trying to read
An HTTP client connection can legally half-close (i.e. close the output end) when it's done sending its last request; and this will result in a read(2) on the server's socket returning EOF. But this doesn't mean that the client's input end is closed! You have to do a write(2) to the server's socket to detect that.
And, since empty TCP packets aren't guaranteed to make the trip, that means you need to write a nonzero number of bytes of ...something. Without that actually messing up the state-machine of your L7 protocol.
You can do this with TCP keepalives [0]. Under Linux, a process can enable the SO_KEEPALIVE option on sockets to request that the Linux kernel send TCP keepalives periodically. A kernel option determines how frequently the kernel sends TCP keepalive packets. It is a single option that applies to all sockets of all processes. One can also configure the OS to enable SO_KEEPALIVE by default on all sockets of all processes.
Golang's GRPC library implements keepalives at the GRPC protocol level [1]. It provides a `Context` value [2] that code can use to detect peer disconnect and cancel expensive operations.
Golang's HTTP server API does not provide any way to detect peer disconnect before sending the final response [3].
Rust cannot set SO_KEEPALIVE [4]. One could possibly implement keepalives by writing zero-length chunks to the socket.
Java's Netty server library can set SO_KEEPALIVE [5]. One can then code a request handler that periodically checks if the socket is connected [6] and cancels expensive operations. Unfortunately, there is standard tooling to do this.
EDIT: You did mention TCP keepalives. I was not aware that some routers drop them. Can you link to any data on the prevalence of tcp-keepalive dropping for various kinds of client connections: home router, corporate wifi, mobile carrier-grade-NAT?
I can't offer any data myself, but I can suggest that chasing up the reason that more "modern" runtimes like Go's and Rust's don't bother to expose SO_KEEPALIVE support — i.e. the discussions that ensued when someone proposed adding this support, as they certainly did at some point — would be a good place to find that data cited.
I can point out the obvious "analytical evidence", though: note how all the platform APIs that did expose SO_KEEPALIVE are from the 90s at the latest — i.e. before the proliferation of L4 middleboxes. And note how modern protocols like Websockets, gRPC, and even HTTP/2 (https://webconcepts.info/concepts/http2-frame-type/0x6) always do their own L7 keepalives, rather than relying on TCP keepalives — even when there's no technical obstacle to relying on TCP keepalives.
> Golang's HTTP server API does not provide any way to detect peer disconnect before sending the final response [3].
Iirc, the context mechanism can be used to detect
the client's disconnection or cancellation of the request
in some cases. From [1]:
> For incoming server requests, the context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.
Another thing to note about custom headers is that when used in an XHR (eg: X-Requested-With), they will force a preflight request (with the OPTIONS method). If your webserver isn't configured to handle OPTIONS and return the correct CORS headers, that will effectively break clients.
Yep, you've got to be careful with browser HTTP requests! Conveniently on this very same site I built a CORS tool that knows all those rules and can tell you how they work for every case: https://httptoolkit.tech/will-it-cors/
yeah, those techniques predate CORS, but even back then, you'd typically add your anti-csrf token to the payload rather than the header. CSRF is application level logic rather than protocol level.
I’m guessing based on the username OP is the original author, caught a typo that could trip a novice up if they’re reading :
This becomes useful though if you send a request including a Except: 100-continue header. That header tells the server you expect a 100 response, and you're not going to send the full request body until you receive it.
I’m guessing that should be Expect?
Overall interesting article, thanks for writing it!
Another one is that it's technically valid to have a request target of '*' for the HTTP OPTIONS request type. It's supposed to return general information about the whole server. You can try it out with e.g. `curl -XOPTIONS http://google.com --request-target '*'`
Nginx gives you a 400 Bad Request response, Apache does nothing, and other servers vary in whether they return a non-error code.
You're not making a mistake, they just don't give an interesting response to this type of request. The only server I know of that actually uses it for something is Icecast (a music streaming server).
You can include the same header multiple time in a HTTP message, and this is equivalent to having one such header with a comma-separated list of values.
Then there's WWW-Authenticate (the one telling you to re-try with credentials). It has a comma-separated list of parameters.
The combination of those two leads to brokenness, like how recently an API thing would not get Firefox to ask for username and password, because it happened to have put "Bearer" before "Basic" in the list.
This article [1] is a really great read on some of the pitfalls you encounter due to the way duplicate headers are parsed in different browsers (skip to "Let's talk about HTTP headers" if you want to jump right into the code).
And some headers have their own exceptions to this.
The Set-Cookie header (sent by the server) should always be sent as multiple headers, not comma separated as user agents may follow Netscape's original spec.
On the other hand in HTTP/1.1 the Cookie header should always be sent as a single header, not multiple. In HTTP/2, they may be sent as separate headers to improve compression. :)
Fun Fact: When http first came out and was hidden in Academia, there were no headers. Then the need for metadata was realized and second-system syndrome took over to create a bunch of crazy headers. My favorite odd-ball header is "Charge-To:"
Readit News
Chunk extensions. Most people know HTTP/1.1 can return a "chunked" response body: it breaks the body up into chunks, so that we can send a response whose length we don't know in advance, but also it allows us to keep the connection open after we're done. What most people don't know is that chunks can carry key-value metadata. The spec technically requires an implementation to at least parse them, though I think it is permitted to ignore them. I've never seen anything ever use this, and I hope that never changes. They're gone in HTTP/2. (So, also, if you thought HTTP/2 was backwards compatible: not technically!)
The "Authorization" header: like "Referer", this header is misspelled. (It should be spelled, "Authentication".) Same applies to "401 Unauthorized", which really ought to be "401 Unauthenticated". ("Unauthorized" is "403 Forbidden", or sometimes "404 Not Found".)
Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets".
Plus surely there are many crusty middleboxes that will break if anybody tried to use that feature. Remember all the hoops websockets had to jump through to have much of a chance working for most people because of those? Many break badly if anything they were not programmed to handle tries to pass through.
Oof, I hadn't mentally connected those dots, but you're completely right. (As Transfer-Encoding is hop-by-hop, not end-to-end…)
Lots of servers I've encountered with my browser stealth actually violate the spec and send "lengths" that do not match the sent payload lengths afterwards. Some reverse proxies also mess up the last chunk, so they're violating the spec there, too, and send a chunk with a negative length...and the spec doesn't even define how to handle this. I've also seen servers send random lengths in between, but without a payload that follows.
I would also like to add range requests (206 partial content) here. In practice, it's totally unpredictable how a server behaves when requesting multiple content ranges. Some reply with no range at all, even with correct headers. Some reply with more or less ranges than requested. Some even reply with out of boundary ranges that are larger than the content length header of the same response because they seem to use a faulty regexp on the server side.
It's a total shitshow.
Always wondered if developers found that easy.
I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads.
With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.
Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)
You are supposed to treat all of them as "opaque octets"... or something like this might happen:
https://news.ycombinator.com/item?id=25857729
But, some of the headers are case-insensitive. E.g., Content-Type, Accept, Expect, etc.
That golang bug is precisely not treating the non-characters (the "opaque octets", as defined by the standard, that is, the octets that form obs-text) as if they were characters. You won't hit that bug in the safe subset, presuming you're implementing other parts of the standard correctly. (Which is… a huge assumption, given HTTP's complexity, but that's sort of the point here.)
Deleted Comment
What you pass in the "Authorization" header is an user identity, which is established through authentication. And the server uses this identity to decide if you are authorized.
Deleted Comment
Deleted Comment
You can't actually know whether the outgoing side of a TCP socket is closed, unless you write something to it. But it's hard to come up with something to write to an HTTP/1.1-over-TCP socket before you respond with anything, that would be a valid NOP according to all the protocol layers in play. (TCP keepalives would be perfect for this... if routers didn't silently drop them.)
But I guess sending an HTTP 102 every second or two could be used for exactly this: prodding the socket with something that middleboxes will be sure to pass back to the client.
If so, that's awesome! ...and also something I wish could be handled for me automatically by web frameworks, because getting that working sounds kind of ridiculous :)
This problem is one reason why success/error should NOT be the first thing to send. It should be a trailer.
(HTTP/HTML tendency to substitute the response body for a human-visible error would require another mechanism to "reset" the response body.)
There is no current browser, or client that by default will send an Expect: 100-Continue.
cURL removed it because it was too often broken. See https://curl.se/mail/lib-2017-07/0013.html
As of right now, while server authors will continue to need to support it, it is unlikely that it is a well tested code path, and it will likely break in weird ways even trying to use it.
So pre-approval for a large file upload is not even valid anymore.
The TLS 1.3 spec states "Zero-length fragments of Application Data MAY be sent, as they are potentially useful as a traffic analysis countermeasure."
I guess that tls libraries wouldn't expose an api to do that which is problematic for this approach.
Deleted Comment
Wouldn't setting appropriate net.ipv4.tcp_keepalive_* and trying to read work?
Like I said:
> TCP keepalives would be perfect for this... if routers didn't silently drop them.
There are lots of middleboxes that don't pass along empty TCP packets. TCP keepalive is in a similar situation to IPsec: great for an Intranet, or for two public-Internet static peers with a clear layer-3 path between them; but everything falls apart in B2C scenarios.
Plus, to add to this problem: HTTP has gateways (proxies et al.) Doing TCP keepalive on the server end, only tells you whether the last gateway in the chain before the server is still connected to the server, rather than whether the client is still connected to the server.
Unless you can get every gateway in the chain to "propagate" keepalive (i.e. to push keepalive down to its client connection, iff the server pushes keepalive down onto it), silent undetected TCP disconnections will still happen—and even worse, you'll have false confidence that they aren't happening, as all your sockets will look like they're actively alive.
For what I'm doing, the client end isn't likely to have any gateways, so TCP keepalives "would be" workable for my use-case if not for the middlebox thing. But in full generality, TCP keepalives aren't workable, because there's always those corporate L7 caching proxies + outbound WAFs messing things up, even when L4 middleboxes aren't.
Keep your TCP keepalives for running connection-oriented stream protocols within your VPC. For HTTP on the open web, they're pretty unsuited. You need L7 keepalives. (If you've ever wondered, this is why websockets have their own L7 keepalives, a.k.a. "ping and pong" frames.)
> and trying to read
An HTTP client connection can legally half-close (i.e. close the output end) when it's done sending its last request; and this will result in a read(2) on the server's socket returning EOF. But this doesn't mean that the client's input end is closed! You have to do a write(2) to the server's socket to detect that.
And, since empty TCP packets aren't guaranteed to make the trip, that means you need to write a nonzero number of bytes of ...something. Without that actually messing up the state-machine of your L7 protocol.
Golang's GRPC library implements keepalives at the GRPC protocol level [1]. It provides a `Context` value [2] that code can use to detect peer disconnect and cancel expensive operations.
Golang's HTTP server API does not provide any way to detect peer disconnect before sending the final response [3].
Rust cannot set SO_KEEPALIVE [4]. One could possibly implement keepalives by writing zero-length chunks to the socket.
Java's Netty server library can set SO_KEEPALIVE [5]. One can then code a request handler that periodically checks if the socket is connected [6] and cancels expensive operations. Unfortunately, there is standard tooling to do this.
EDIT: You did mention TCP keepalives. I was not aware that some routers drop them. Can you link to any data on the prevalence of tcp-keepalive dropping for various kinds of client connections: home router, corporate wifi, mobile carrier-grade-NAT?
[0] https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
[1] https://pkg.go.dev/google.golang.org/grpc/keepalive
[2] https://pkg.go.dev/google.golang.org/grpc#ServerStream
[3] https://pkg.go.dev/net/http#HandlerFunc
[4] https://github.com/rust-lang/rust/issues/69774
[5] https://netty.io/4.1/api/io/netty/channel/ChannelOption.html...
[6] https://netty.io/4.1/api/io/netty/channel/Channel.html#isOpe...
I can point out the obvious "analytical evidence", though: note how all the platform APIs that did expose SO_KEEPALIVE are from the 90s at the latest — i.e. before the proliferation of L4 middleboxes. And note how modern protocols like Websockets, gRPC, and even HTTP/2 (https://webconcepts.info/concepts/http2-frame-type/0x6) always do their own L7 keepalives, rather than relying on TCP keepalives — even when there's no technical obstacle to relying on TCP keepalives.
Iirc, the context mechanism can be used to detect the client's disconnection or cancellation of the request in some cases. From [1]:
> For incoming server requests, the context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.
[1]: https://golang.org/pkg/net/http/#Request.Context
Deleted Comment
Best to just never use custom headers.
I've written more about this here: https://developer.akamai.com/blog/2015/08/17/solving-options...
That's why they're so great. use a custom header and never worry about CSRF issues.
Use custom header and be sure that if request comes from the browser it was made by legitimate code from your origin.
This becomes useful though if you send a request including a Except: 100-continue header. That header tells the server you expect a 100 response, and you're not going to send the full request body until you receive it.
I’m guessing that should be Expect?
Overall interesting article, thanks for writing it!
Nginx gives you a 400 Bad Request response, Apache does nothing, and other servers vary in whether they return a non-error code.
https://curl.se/mail/lib-2016-08/0167.html
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
You can include the same header multiple time in a HTTP message, and this is equivalent to having one such header with a comma-separated list of values.
Then there's WWW-Authenticate (the one telling you to re-try with credentials). It has a comma-separated list of parameters.
The combination of those two leads to brokenness, like how recently an API thing would not get Firefox to ask for username and password, because it happened to have put "Bearer" before "Basic" in the list.
https://tools.ietf.org/html/rfc7235#section-4.1
[1]: https://fasterthanli.me/articles/aiming-for-correctness-with...
The Set-Cookie header (sent by the server) should always be sent as multiple headers, not comma separated as user agents may follow Netscape's original spec.
On the other hand in HTTP/1.1 the Cookie header should always be sent as a single header, not multiple. In HTTP/2, they may be sent as separate headers to improve compression. :)https://www.w3.org/Protocols/HTTP/HTRQ_Headers.html
This is not a custom X- header but an official header. Also Email: and some other odd headers were standardized at that time.
That's why you spelled 'spelled': spelt :D