News for hackers. Stuff that matters.
Re:The 7-bit Internet by jperras on Wednesday July 10, 2013 @04:55PM (#6022742)
For some reason, I feel as though the people in this thread against text-based protocols have never used telnet to inspect or debug an HTTP request/response. Or used telnet to help develop a client for something like memcached.

The ease of discoverability of a protocol should not be underrated.

Re:The 7-bit Internet by josteink on Wednesday July 10, 2013 @05:00PM (#6022779)
Yeah. For some things seeing truly is believing.

I guess if you've never had that moment where it all clicked for you, just by looking at traffic, the significance of human-readable protocols is easy to underestimate.

But hey, Google says that with binary-compressed headers of sorts, they can fit all their tracking cookies in one TCP-packet and that means you wont feel inconvenienced by the tracking payload they are throwing at you everywhere you go on the internet.

You know what? I want that huge bag of tracking cookies to hurt. If you violate my privacy, that should come at a cost.

Google: You can take your SPDY and keep nice, old HTTP alone.

Re:The 7-bit Internet by flipperypokery on Wednesday July 10, 2013 @05:39PM (#6023026)
I feel as though the people in this thread against (well-designed) binary protocols have never tried implementing a binary protocol.

Length-prefixed fields and direct encoding of binary data go a long way to simplifying implementing a protocol.

The ease of implementation of a simple binary protocol should not be underrated.

Re:The 7-bit Internet by sergiosgc on Wednesday July 10, 2013 @06:59PM (#6023373)
It's not just the binary. It's that HTTP 2 is hugely more complex. Excessive complexity is a general engineering smell. It can't be all that difficult to implement a protocol for applying a few verbs to URIs.

Complex + non-discoverable = bad engineering

Re:The 7-bit Internet by pbsdp on Wednesday July 10, 2013 @08:05PM (#6023664)
HTTP is already painfully complex to implement correctly. It's also a terribly wasteful and poor engineered protocol for anything other than fetching documents.
Re:The 7-bit Internet by jimktrains2 on Wednesday July 10, 2013 @10:52PM (#6024214)
> It's also a terribly wasteful and poor engineered protocol for anything other than fetching documents.

I wonder if that was the original purpose of the protocol;)

Re:The 7-bit Internet by sergiosgc on Thursday July 11, 2013 @09:36AM (#6025898)
Those who do not know History are doomed to repeat it. Back in the day of the OSI stack, the telecom guys were huge fans of binary protocols. The arguments were exactly those described here on HN: they're easier to implement; with the right tooling, they're just as readable; they're more efficient.

The fact is, on the application layer, text based protocols wiped the floor with binary protocols. The answer is in the little details, which everyone knows is where the devil spends its time. They are not easier to implement, because they're more difficult to debug (and because text parsers are, let's be honest, a solved problem); they are not as readable, because debug tooling is never perfectly implemented and never ever present in every system; they are more efficient, but at this layer the efficiency does not pay off (i.e. the size of the HTTP header leading that 5MB image does not really matter).

One of the comments here pulled a comparison I did not remember any more: SS7 vs SIP. Go and have a look at both protocols. They are both mature, so you don't really need to wait for the fantastic debug tooling to appear. Study both ecosystems and then form your opinion on binary vs text application protocols.

Re:The 7-bit Internet by flipperypokery on Thursday July 11, 2013 @11:47AM (#6026785)
Those who don't understand history are doomed to misinterpret its lessons.

Do you remember the OSI binary protocols as designed back in the day?

ASN.1 as far as the eye could see. OIDs. MIBs. Loosely defined extensibility. Terrible, complex protocol designs.

This had nothing to do with the protocols being binary or text; the protocols themselves were painfully complex, making it difficult and frustrating to write a working implementation, much less a complete and interoperating implementations.

> ... at this layer the efficiency does not pay off (i.e. the size of the HTTP header leading that 5MB image does not really matter).

"I don't care about RTT time, the lack of bidirectional communication, and the inability to inline binary data in the protocol stream" ... said no mobile/wireless/desktop developer, ever.

> One of the comments here pulled a comparison I did not remember any more: SS7 vs SIP.

I've implemented SIP. It was a massive hassle because it was text-based and annoying and difficult to parse. The available SIP libraries tend to be buggy and incomplete, and people actually pay for workable SIP stacks. I'd rather not look at SIP ever again.

That isn't to say SS7 is better, it doesn't appear to be. As far as I can tell, that's because it was badly designed, not because of a flaw inherent in using binary encodings.

Re:The 7-bit Internet by lstamour on Thursday July 11, 2013 @01:20PM (#6027467)
> "I don't care about RTT time, the lack of bidirectional communication, and the inability to inline binary data in the protocol stream" ... said no mobile/wireless/desktop developer, ever.

+1

http://www.igvita.com/slides/2013/breaking-1s-mobile-barrier... if you want to read about this from a mobile perspective.

Re:The 7-bit Internet by jimktrains2 on Wednesday July 10, 2013 @06:59PM (#6023372)
Easy to implement in C, perhaps, but not Python, PERL, and the ilk. Most importantly, it's not as easy to do it with a shell and nc to build a simple server.
Re:The 7-bit Internet by lucian1900 on Thursday July 11, 2013 @05:47AM (#6025201)
Implementing binary protocols in Python is quite easy. I'll take length-prefixed and encoding-tagged binaries over "text" any day.
Re:The 7-bit Internet by itcmcgrath on Wednesday July 10, 2013 @06:41PM (#6023271)
So you are saying you are using a tool (telnet) to view/interact with the current protocol and this is good because it is easy to read and use.

Now we build a different tool, let's call it http2net. With HTTP being so prevalent, perhaps it isn't unreasonable to think it becomes part of most OS distributions.

So now you are using a tool (http2net) to view/interact with the new protocol and this is good because it is easy to read and use.

In summary, is binary vs text really the big problem here?

Re:The 7-bit Internet by xymostech on Wednesday July 10, 2013 @07:41PM (#6023573)
I think the reason I like telnet is because I control what is being sent down the line. It's not like telnet has any contextual knowledge of what I'm sending, so it can't do any translations on it; I get to choose exactly how the bytes get sent to the server. This means that telnet can be used for every text-based protocol out there (so long as it accepts \r\n line breaks), whereas your proposed http2net tool would be specifically for turning commands into bytes to be sent to HTTP 2.0 servers, and that response.
Re:The 7-bit Internet by vidarh on Thursday July 11, 2013 @06:17AM (#6025259)
> so it can't do any translations on it;

You'd think so, but while it's true we can mostly pretend telnet gives you a raw TCP connection, telnet intercepts some character sequences. It just doesn't interfere much with plain text.

My problem with the hypothetical "http2net" tool is that to this day I still have to deal with systems that doesn't have basic tools like tcpdump, curl or wget. But telnet, nc or some other way of getting a semi-raw tcp connection is pretty much always available.

So I have every expectation that it'd take 30+ years before this "http2net" tool would be available everywhere I'd want it.

Re:The 7-bit Internet by teddyh on Thursday July 11, 2013 @07:33AM (#6025440)
No, Telnet does not, in fact, intercept any character sequences unless when connecting to the telnet port (port 23).
Re:The 7-bit Internet by vidarh on Thursday July 11, 2013 @08:47AM (#6025682)
This is not true on the client I have on my OS X box at work, nor on my Debian and Ubuntu machines at home, nor, I believe, with pretty much any other telnet client I've used in the last twenty or so years. I just tested several to verify whether I'd somehow missed something so basic.

Regardless of the issue of negotiated options, most telnet clients goes into a command mode if you enter an escape character, defaulting to ctrl+], unless you explicitly change it or tell it not to with a command line switch. This happens for me regardless of port.

tcpdump and strace also confirms what I thought I knew, namely that it also does LF => CR+LF conversion also when connecting to other ports.

EDIT: Here's a simple demonstration of how it is decidedly not 8-bit clean:

    echo -e "test\035help" | telnet www.google.com 80
Not only will this get you the help text for telnet rather than send the unmodified byte stream to Google's unsuspecting web server, but with default options not a single byte will generally get sent over this connection on Linux at least, as the client starts out line buffered and gets put into command mode without getting a line feed.

Confirm with tcpdump, or this way if you have strace installed:

    echo -e "test\035help" | strace telnet www.google.com 80 2>&1 | grep sendto 
(the sendto calls you will get are DNS lookups)
Re:The 7-bit Internet by teddyh on Friday July 12, 2013 @03:20PM (#6034564)
Right, the interactive escape character. That's easily fixed with "telnet -E". That is,

    echo -e "test\035help" | telnet -E www.google.com 80
will not give any help text output.

(I was, in fact referring to telnet option negotiation which will not happen if not using the telnet port. Since the escape character for interactive use is so easily remedied, it did not occur to me to consider it a problem.)

Re:The 7-bit Internet by drawkbox on Wednesday July 10, 2013 @09:47PM (#6024024)
I agree. Much of that attitude is also pompous.

Engineering should be about taking complex things and making them simple and easy (the best engineers do this), on the other hand you have engineers that try to add complexity because they understand something and over engineer it to obscure it on some premature optimization or cool factor. Taking something simple and making it more complex is the epitome of bad engineering.

Cool is useful and useful is simple or at least simple parts. Keep entry to technology simple, just like good games are simple to start, deeper in it gets more difficult. The door should be easy to enter even though the labyrinth might be immense.

Re:The 7-bit Internet by pbsdp on Thursday July 11, 2013 @12:04PM (#6026913)
Engineering is about making complex things simple and easy for your users. Sometimes that means increasing complexity at one level of the stack to increase performance and functionality at a higher level of the stack.
Re:The 7-bit Internet by drawkbox on Thursday July 11, 2013 @03:38PM (#6028616)
Many times in engineering the users are other engineers. Complexity only where necessary. The OSI layers beneath this layer are much more complex for those reasons. But in the end they are still simple parts to an overall complex computing and delivery system. Each engineer, if they are good, simplify things even for other engineers, this leads to less hurdles on the next platform innovations.
Re:The 7-bit Internet by pbsdp on Thursday July 11, 2013 @03:44PM (#6028653)
HTTP improvements are necessary. If this raises the bar past what some developers can manage, that's unfortunate, but necessary -- the current system actually prevents what better developers want and need to do.
Re:The 7-bit Internet by MichaelGG on Wednesday July 10, 2013 @08:44PM (#6023817)
I've written a couple parsers and stacks for SIP, a HTTP-derived protocol.

The text-based aspect isn't bad in and of itself. It's the crazy moronic rules and pointlessly flexible syntax that makes it bad. It's the fact that "text-based" is often taken to mean "should allow humans to be flexible in writing it", instead of "uses ASCII". Oh, and UTF8 if you're lucky. If you dare want a non-ASCII value that needs to go into a header, you're toast.

HTTP: Comments inside headers, line folding, context-sensitive header value parsing, and on and on. There is zero legitimate need for these things, yet RFC writers cannot seem to help themselves and design the most complicated syntax as possible.

Text-based formats frequently trick people into thinking they're right because it looks easy on the surface to get something to work. Shit, most HTTP clients/servers aren't actually fully compliant.

It opens more possibility for security holes due to potentially ambiguous parsing (where one implementation misses an edge case, but another doesn't). Protocols are not programming languages. They don't need flexible syntax.

There's also crap left over from a time when people read/write these protocols by hand. Idiocy like the IETF preferred date formats "Sun, 06 Nov 1994". Really? Including "Sun", the English 3-letter abbreviation, in a protocol? How is that useful?

HTTP is being used more and more for high-performance work. Take a look at a high-perf text-based parser, and you see all sorts of ugly hacks. Like doing bitwise comparisons on word-sized integers to determine the request method. (Both my personal code and nginx ended up with similar solutions, so it's safe to say it's a common approach for perf.)

Having a proper binary format that can be quickly, unambiguously, and safely parsed is a huge boon for interoperability and performance. The slight detriment for analyzing raw bytes (which, when encrypted makes it all moot) is not worth it. For a popular protocol like HTTP, you're gonna have plenty of tools to properly parse and analyze.

IP, UDP and TCP aren't text based, and I have no problems regularly analyzing them, nor do they seem to have adoption issues. But you can bet your ass if UDP specified port numbers as a flexible text field, you'd find all sorts of fun bugs and implementation issues.

Edit: After skimming the HTTP 2.0 spec, implementing this as a text-based protocol sounds like a nightmare with no benefit. It's not like you're going to write multiple streams out by hand or something (and use some terrible multipart-mime approach, or another "fun" ASCII-delimiting-binary thing).

Re:The 7-bit Internet by vidarh on Thursday July 11, 2013 @05:28AM (#6025161)
IMAP is the protocol I love to hate with a burning passion... I get that it is enormously flexible. But it is also a total, utter nightmare to work with, for not very good reasons.

Though I don't really agree with you that you need a binary format for interop and performance. You'd get the same benefits if people just constrained the text protocols. E.g. Your performance "hack" for HTTP request methods is only a hack because method names can be variable length. We could constrain method names to 4 bytes and make the whole thing cleaner while still getting the text protocol benefits of being able to manually inspect request/responses. Similarly, picking a sane date format like iso-8601 and restricting it further to a specific set of options would do wonders while still being readable/writable for humans.

I do agree with you that once a protocol gets as widespread as HTTP most of the benefit is lost as we get better tools to use anyway, though. Especially with debug tools built into most browsers these days.

The issue is not that non-text protocols are impossible to analyse, but that each new non-text protocol requires new tools to do so. For IP, UDP and TCP we've had decades to create good tools. If a common binary serialisation format was agreed and most protocols stuck to that, it would be much less of a big deal to ditch text-only protocols and rely on client side tools to allow reading or manipulating them as text.

Re:The 7-bit Internet by msds on Wednesday July 10, 2013 @09:11PM (#6023914)
> Protocols are not programming languages. They don't need flexible syntax.

Or, as the langsec crowd claims, protocols don't need flexible syntax, lest they actually become programming languages.

Re:The 7-bit Internet by archivator on Thursday July 11, 2013 @06:37AM (#6025312)
Relevant talk: Meredith Patterson on Turing-complete protocols and weird machines - https://www.youtube.com/watch?v=3kEfedtQVOY
Re:The 7-bit Internet by penguindev on Thursday July 11, 2013 @10:01AM (#6026028)
I disagree with your last line, if you know what http chunked encoding is, it is a 'simple ascii delimited binary thing' - it just doesn't have a 'stream id' in it, but there could be. Of course, people always forget to check the chunk length is <0 which makes for a constant stream of security holes over the years, but you would have the exact same problem in binary.

I think it's sad that HTTP/2.0 is primarily just trying to multiplex TCP and the rest seems like pure micro-optimizations (the one exception being server push; I haven't read a lot on that yet). It seems like the wrong layer in which to attack the problem, and is basically a huge 'meh'.

Re:The 7-bit Internet by DougWebb on Wednesday July 10, 2013 @10:53PM (#6024215)
What, exactly, prevents a binary protocol definition from being overly flexible and complex? I've seen quite a few like that, and I've seen quite a few simple and easy to parse text protocols.
Re:The 7-bit Internet by ajanuary on Thursday July 11, 2013 @06:48AM (#6025336)
Attitude. One of the big advantages that seems to be touted for text protocols is easy to use manually by humans, and with that can come a desire to make it forgiving of human error.

You can have simple text protocols and complex binary protocols, but it's more tempting to make text protocols more complex.

(In this comment, simple/complex is in terms of syntax only)

Re:The 7-bit Internet by rjsw on Wednesday July 10, 2013 @05:24PM (#6022940)
Writing new protocol modules for tcpdump isn't difficult.
Re:The 7-bit Internet by mzs on Wednesday July 10, 2013 @06:49PM (#6023319)
After that you can see what? You also have to use something like sed, xxd, and nc at least to make a request unless you are really good with bitfields and a hex editor. You want to keep a connection open or use what comes back for he next request? Maybe you can quickly cobble together a crude client in expect and tcl, perl, or python.
Re:The 7-bit Internet by iso-8859-1 on Wednesday July 10, 2013 @07:02PM (#6023384)
A lot more difficult than using netcat though.
Re:The 7-bit Internet by lmm on Wednesday July 10, 2013 @07:26PM (#6023506)
It's incredibly rude to just dismiss everyone who disagrees with you like that.
Re:The 7-bit Internet by trentmb on Wednesday July 10, 2013 @09:05PM (#6023896)
What do you expect from a group that thinks trans-compiling C++ to JavaScript is a good idea...