"I would absolutely love to discover the original code review for this and why this was chosen as a default. If the PRs from 2011 are any indication, it was probably to get unit tests to pass faster."
This was a pretty interesting read.
https://withinboredom.info/blog/2022/12/29/golang-is-evil-on-shitty-networks/
@nyquildotorg Here you go, from the code's author: https://news.ycombinator.com/item?id=34181846
And John Nagle commenting on the tradeoffs and problems: https://news.ycombinator.com/item?id=34180239
And the rest of the comments go, in a roundabout way, into wondering why git-lfs is making write syscalls 50 bytes at a time in the first place, regardless of what the network stack does beyond that, because that's going to suck regardless of what the network stack does beyond that.
@danderson @nyquildotorg Gotta admit, I'm team TCP_NODELAY. Nagle has surprised people for generations, and "But I want to take 20k syscalls to transfer 1MB and have it look efficient" doesn't make me very sympathetic!
@sgf @danderson @nyquildotorg Nagle gets used to "fixup" a bunch of problems (eg silly window syndrome etc).
In general, there are two types of flow: elephants (bandwidth heavy) and mice (latency sensitive). You want nagle for the first class (keep overheads as low as possible for maximum throughput), and not for the second (keep latency low as possible).
@isomer @sgf @danderson @nyquildotorg
I would expect that elephants will buffer (and when they temporarily become mice they will either reshuffle things so that the bufferedwriter is out of the picture or they will simply keep flushing the writer at appropriate times). If that's the case then by disabling Nagle's algorithm we're wasting at most one packet each time the buffer is emptied (pessimistically we will emit one one-byte packet then). So, Nagle should be superfluous if we buffer with a buffer that's much larger than a packet or that's chosen to be a multiple of a packet's size. Am I missing some reason Nagle is useful?
@robryk @sgf @danderson @nyquildotorg that's all true, but you can't always keep your buffer full. Eg when reading data from disk. Esp for long fast networks.
Connections can often flip between mice and elephants. It's common to say "do you want this data?" then wait for a reply, then send the entire data. The first part is latency sensitive, the second part is bandwidth heavy.
@isomer @sgf @danderson @nyquildotorg
If the buffer is not full, the buffered writer will not write until it gets full. People who deal with buffered writers are used to flushing them at appropriate times (and there are all those funny affordances like stdio's "flush out when someone's reading in").
@robryk @isomer @danderson @nyquildotorg I've now got the dev party of my brain going "You could argue that Nagle is just a defense against badly-written programs that can't buffer properly", and the SRE part of my brain going "Yes! And we need defenses against badly written programs!".
Can we rename TCP_NODELAY to TCP_TRUSTME?
@sgf @robryk @danderson @nyquildotorg if the network is slow then the end users (like the upstream author) eventually notice and fix the problem.
If the network is inefficient, then generally the people in the middle of the network notice and cannot even figure out what is creating inefficient data (it should be encrypted right?) to complain.
@isomer @sgf @danderson @nyquildotorg
I loathe doing something worse, so that we move the pain to a better place. I don't know whether it's reasonable to loathe that.
That said, if we want to use pain to cause things to be fixed and don't care much about debuggability of that pain, shouldn't we rather only use Nagle (a) when we notice many small packets with no reads inbetween on a socket without TCP_NDELAY equivalent set (b) warn about that happening in e.g. kernel logs (rate limited)?
@isomer @sgf @danderson @nyquildotorg
That's a very weird way of doing that IMO. Once you learn that there's data to be read from the socket, but the protocol handler doesn't want more data now:
- you can't read it, because it's a remotely-triggered out-of-memory DoS,
- you need to unset the "wake on read available" flag in the subsequent polls so they don't immediately return and keep track of "this socket has some data available".
So why set the "wait on read available" flag until the protocol handler is in a state in which it's waiting for data?
@robryk @sgf @danderson @nyquildotorg with poll() style event loops, there are always reads in between.
Logging to syslog isn't a particularly useful way to get people to notice problems.