Foo: Hello_______world\r\n
Here the whitespace has to be preserved, as per the field-content production.So RFC2616 allowed:
Foo: Hello_______\r\n
____world\r\n
to be reduced to: Foo: Hello world\r\n
Incidentally RFC7230 (it's successor) says something subtly different:"A server that receives an obs-fold in a request message that is not within a message/http container MUST either reject the message by sending a 400 (Bad Request), preferably with a representation explaining that obsolete line folding is unacceptable, or replace each received obs-fold with one or more SP octets prior to interpreting the field value or forwarding the message downstream."
So now it's been loosened to one or more SP octets... useful, except there's no mention of TAB octets, which can follow a CRLF as part of an obs-fold... so you still can't just remove the CRLF and preserve all the whitespace... preserving tabs would be illegal. So the new rules don't help streaming either. Joyents parser did this regardless (not sure if it still does).
You'll also notice it says the obs-fold can be replaced with a sequence of SP characters... according to the grammar that's the CRLF and the following whitespace, not any whitespace preceding the CRLF. You'd think that would be helpful because it means any whitespace before the CRLF can always be streamed as part of the field value, right? Except...
Foo: Hello_______\r\n
____\r\n
would still simplify to: Foo: Hello_______[one or more SP octets]\r\n
and then what? Well, presumably you then have to trim off the trailing whitespace to produce a value of just "Hello"... so all the whitespace you buffered before you reached the CRLF (in case you reached another field-vchar like the 'w' in 'world') has to be discarded.However, the easy option is to default to the slow code path on edge cases which considering it's probably a rare enough it's not important to make fast as long it's bug free. IMO, optimizations are always a balancing act between minimizing the computer's effort and minimizing the coders effort while trying to maintain long term readability. But, you can always keep track of more than one buffer so the option is there.
Foo: Hello World\r\n Bar: smeg