News for hackers. Stuff that matters.
Re:Golang's network stack attempts to parse HTTP headers as ... by panpanna on Thursday January 21, 2021 @06:50AM (#25857938)
Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8.

Now, what are the security implications of this?

(beside the chunk thingy)

Re:Golang's network stack attempts to parse HTTP headers as ... by _-___________-_ on Thursday January 21, 2021 @06:58AM (#25857992)
> Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8.

That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken.

> Now, what are the security implications of this? (beside the chunk thingy)

Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context.

Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!

Re:Golang's network stack attempts to parse HTTP headers as ... by yokaze on Thursday January 21, 2021 @07:34AM (#25858204)
> That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken.

Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1

Re:Golang's network stack attempts to parse HTTP headers as ... by vaduz on Thursday January 21, 2021 @09:28AM (#25859125)
> Actually, they are explicitly specified as ASCII

They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2.

https://tools.ietf.org/html/rfc7230#section-3.2.4

Re:Golang's network stack attempts to parse HTTP headers as ... by Thaxll on Thursday January 21, 2021 @08:34AM (#25858615)
Re:Golang's network stack attempts to parse HTTP headers as ... by panpanna on Thursday January 21, 2021 @07:40AM (#25858242)
> That's a terrible reason.

Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang.

It leads to all kind of issues, not only security problems.

Re:Golang's network stack attempts to parse HTTP headers as ... by kerng on Thursday January 21, 2021 @08:06AM (#25858408)
But that's a Golang issue then, not http
Re:Golang's network stack attempts to parse HTTP headers as ... by panpanna on Thursday January 21, 2021 @08:47AM (#25858709)
I know, I am just trying to explain how this could happen :)
Re:Golang's network stack attempts to parse HTTP headers as ... by _-___________-_ on Thursday January 21, 2021 @10:46PM (#25868085)
Yes, but this simply means that header values are not (Go) strings, they're byte arrays. Many languages represent them this way, with a (fallible) function to get the string value iff the byte array contains valid UTF-8.

The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.

Re:Golang's network stack attempts to parse HTTP headers as ... by 0xjnml on Friday January 22, 2021 @05:59PM (#25877373)
1) Go string type is specified as a sequence of bytes. Full stop. Any encoding, binary blob, string does not care. Particular stdlib string functions _do_ care, expect UTF-8 encoding and document the fact. Other stdlib string function work with the sequence as a sequence.

3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals.

2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.

Re:Golang's network stack attempts to parse HTTP headers as ... by albinowax_ on Thursday January 21, 2021 @07:06AM (#25858035)
I think it's worth spelling out that 'chunk thingy' can lead to complete site takeover. For example, using this behaviour I was able to get persistent control of PayPal's login page.

That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015

Re:Golang's network stack attempts to parse HTTP headers as ... by panpanna on Thursday January 21, 2021 @07:38AM (#25858232)
Do you have any pointers to the PayPal hack?
Re:Golang's network stack attempts to parse HTTP headers as ... by albinowax_ on Thursday January 21, 2021 @07:47AM (#25858279)
Yep, the original reports are here: https://hackerone.com/reports/488147 https://hackerone.com/reports/510152

And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request...

Re:Golang's network stack attempts to parse HTTP headers as ... by panpanna on Thursday January 21, 2021 @08:02AM (#25858378)
This is awesome, thanks!
Re:Golang's network stack attempts to parse HTTP headers as ... by Uberphallus on Thursday January 21, 2021 @09:51AM (#25859372)
Wow, excellent work!
Re:Golang's network stack attempts to parse HTTP headers as ... by superjan on Thursday January 21, 2021 @07:15AM (#25858092)
It could get hairy when here is also another parser that has different assumptions. They don’t see the same URL. Psychic paper [0] comes to mind.

[0] https://daringfireball.net/linked/2020/05/02/psychic-paper

Re:Golang's network stack attempts to parse HTTP headers as ... by kerng on Thursday January 21, 2021 @08:05AM (#25858398)
Burp employee made like 70k dollars in bug bounty some time back with smuggling attacks on various sites
Re:Golang's network stack attempts to parse HTTP headers as ... by jlouis on Thursday January 21, 2021 @07:15AM (#25858094)
The string type is utf8.

You would think an old http header is a string. It's not. It looks like a string though :)

Re:Golang's network stack attempts to parse HTTP headers as ... by papaf on Thursday January 21, 2021 @08:01AM (#25858373)
The string type is utf8.

Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types

However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.

Re:Golang's network stack attempts to parse HTTP headers as ... by user-the-name on Thursday January 21, 2021 @09:41AM (#25859264)
The real problem is that Go does not actually have a string type. It only has byte strings which can accidentally also be text strings encoded in utf-8.

This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software.

Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.

Re:Golang's network stack attempts to parse HTTP headers as ... by naniwaduni on Thursday January 21, 2021 @11:04AM (#25860299)
The fact is that there is no correct "text string" type, because there are no valid operations on so-called text strings in general.
Re:Golang's network stack attempts to parse HTTP headers as ... by user-the-name on Friday January 22, 2021 @04:11AM (#25869863)
That... makes no sense? Of course there are a lot of valid operations on text strings?
Re:Golang's network stack attempts to parse HTTP headers as ... by Thaxll on Thursday January 21, 2021 @08:32AM (#25858596)
In Go a string is an immutable byte slice not UTF-8.
Re:Golang's network stack attempts to parse HTTP headers as ... by jrimbault on Thursday January 21, 2021 @08:35AM (#25858626)
I'll also play along the role of the pedant :

> In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8.

Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?

Re:Golang's network stack attempts to parse HTTP headers as ... by papaf on Thursday January 21, 2021 @08:52AM (#25858741)
Its a common mistake to assume strings are UTF-8 but it is a mistake.

This is not a defence of Go but more of a warning to Go programmers.

Re:Golang's network stack attempts to parse HTTP headers as ... by adonovan on Thursday January 21, 2021 @12:14PM (#25861291)
Exactly. It's a convention, not a law of physics. If a string's UTF-8 validity is important to the correctness of your function, check it. It's not expensive.