1960s-style arbitrary field size limitations: the wave of the future! No doubt any day now we'll reorganize the internet around shipping punched-card images around, too. We could call the project the "Because It's Time Network", or BITNET.
> Of course, most people would argue that you don't save enough time or bandwidth or cpu cycles to make a difference, but it would make for easier-to-write parsers.
Most people would argue that, yes; famously, Kernighan and Plauger did argue that in The Elements of Programming Style in 1974. By "easier-to-write parsers", you mean easier than dict(((key.lower(), value.lstrip()) for key, value in (line.split(':', 1) for line in header.split('\r\n'))))? Because I think that's going to be a pretty tough bar to fit under. (Yeah, I know you need another couple of lines of code if you're going to handle indented continuation lines, but you can get rid of those without returning your protocol design to the Summer of Love. You could also get rid of the .lower() and the .lstrip() while you're at it.)
> Add to this that every header could have a 32-bit identifier (4 bytes)
Padding out sub-byte-sized values to fill out fixed-width fields: the Intelligent Man's Approach to Saving Bandwidth! Or you could just use one-letter names in an ASCII protocol.
> [Fixed-length fields] solves crazy problems like header injection and request integrity checking.
Clearly we've never had parsing bugs in binary protocols full of fixed-width fields, now have we? Surely not bugs that produced security holes? Except maybe TCP, IP, and DNS. And X.400, and X.500, and X.509, and some of those were the fault of ASN.1 BER and DER, which are hardly fixed-width formats. And surely silently truncating a value to put it into a fixed-width field would never change its semantics, right?
> as to a general question "why use binary when we can compress stuff", what is smaller: a string of bits, or a compressed string of digits?
Well, let's see. How many files do I have here?
$ find | wc
5946 17074 330147
Each one has a ctime, an mtime, an atime, and an inode number. The ctime is generally going to be the same as the mtime in this case, so we'll leave it out. I think they're technically 64-bit values in the current inode structure, but let's count them as 32 bits instead, since none of my files are from after 2038. The inode number is also 32 bits. So if we take these three 32-bit values per file, we have 5946×12 bytes, or 71352 bytes. And if we print them out as digits and compress them? $ find -printf %.10A@%.10T@%i | gzip -9c | wc -c
46278
But wait, that string of digits isn't parsable; it's just digits. So let's add delimiters. $ find -printf %.10A@\ %.10T@\ %i\\n | head -3
1373396230 1373338685 2359306
1372967497 1372967489 5246218
1365458166 1365458157 5248264
$ find -printf %.10A@\ %.10T@\ %i\\n | gzip -9c | wc -c
47506
So a compressed string of digits is a lot smaller in this case. But you could argue that that's just because my data is highly redundant, since most of the timestamps are going to be within the current few years, which is true. But then, most data is highly redundant. How bad can it get, in the worst case of representing uniformly distributed random 32-bit values as compressed strings of digits, with spaces between? It adds about 44% overhead: $ dd </dev/urandom bs=1024 count=1 |
od -w1024 -l | tr -s ' ' ' ' |
gzip -9c | wc -c
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000622635 s, 1.6 MB/s
1460
> Little inefficiencies like this don't go away if you just hack onto the same old protocol forever.But you know what? They do go away if you tunnel over a deflate-compressed VPN, or if you increase the available bandwidth by a few percent. Maybe instead of trying to take us back to bug-prone 1960s designs you should be working on that.
2. Trying to fit an HTTP field parser into one line is not the way to win a programming argument.
3. One ASCII character names? 255 (edit: 37 alphanumeric) possible headers, of which only a few might correlate to what you'd expect? You're trolling me now.
4. Yes, software bugs happen! It's crazy I know. But let's go ahead and assume that the security holes that still plague applications today due to design flaws are not the same as a couple off-by-one bugs decades ago.
5. By virtue of the algorithm, compression works better the more you have of the same thing. You won't have 70KB of headers to compress at once; more like 400 bytes. The compression of individual header groups each time will not benefit from the previous data's compression, as TLS or SPDY might do. The eventual overhead would not only be larger than a bitstream but take more CPU to decode.
6. Not only are they inefficient, they add complication to the parsing of the protocol, which is one more thing an application can mess up. Not only is it slower, it's more prone to errors. A VPN does not fix that. Perhaps you should focus on real solutions to real problems versus quoting what decade network application programming began in.
I wasn't suggesting encoding "Host: news.ycombinator.com\r\n" as "Hnews.ycombinator.com\r\n" but as "H:news.ycombinator.com\r\n". As long as you keep the colon, you can still use long names for other headers.
> Trying to fit an HTTP field parser into one line is not the way to win a programming argument.
You said parsing would be "simpler". It's going to be hard to get simpler than something that you can fit into one line.
> Perhaps you should focus on real solutions to real problems versus quoting what decade network application programming began in.
Well, that's kind of what I'm saying: let's focus on solving real problems, instead of recreating new ones that we'd already solved decades ago, in order to "solve" non-problems like HTTP header encoding.
> It's going to be hard to get simpler than something that you can fit into one line.
This is a terrible argument, as you can fit anything onto one line if you string it along enough. But here's one example of something simpler:
strncpy( frame_struct, buffer, sizeof(frame_struct) );
And i'm not proposing we merely solve the problems of HTTP. That would make too much sense; people are much more willing to put up with bullshit than do the hard work to make things work correctly. I was proposing we make things work better, simpler, and more reliable, and throw away the completely nonsense argument that using netcat once a year is worth a ridiculously impractical protocol format. But whatever, it's not like this thread will amount to a hill of beans.
So let's just convert all of HTTP into XML.
And I don't really think "H:news.ycombinator.com" is quite as illegible as your suggested 32-bit integer space — which, by the way, is small enough that you'll probably need a central registry to prevent header name conflicts — and it also occupies only two bytes instead of four for the header type. So, from my point of view, the "completely illogical" thing is to go from, "The header names currently in HTTP are too long!" to "Therefore let's replace them with 32-bit integers in a binary protocol" instead of "Therefore let's shorten the header names in HTTP", which solves the problem more thoroughly and with less collateral damage.
And what is this about "if you string it along enough"? We're talking about a parser (for RFC-822 headers without continuation lines) that fits into 110 characters, here, without the least obfuscation. Less than a Tweet. In fact, I just Tweeted it. And it worked on the first try.
> the completely nonsense argument that using netcat once a year is worth a ridiculously impractical protocol format.
You know, we did kind of try binary protocols already: the whole IPX stack, CIFS, X.everything, SNMP, TFTP, ICB, Sun RPC and thus NFS and NIS, and so on. A few survive in common use: DNS, TCP, IP, ICMP, SSL, SSH, BGP, and to some extent, SNMP. And there are lots of them working fine inside of particular companies, rather than between implementations by different vendors. But for the most part, they've been replaced with textual protocols, despite the lower efficiency and in many cases the first-mover advantage: HTTP, SMTP, and IRC, and previously FTP, Gopher, and Finger. You seem to be arguing that was an accident, or a mistake. It's not.
Why do you find HTTP impractical? There seems to be a lot of evidence to the contrary.
This was mentioned once in the IETF discussion, before someone said "but uhm, SPDY is binary, and we have data from SPDY, and yeah".
After that everyone was too busy discussing how horrible unstrict and ambigious text-formats can be, before jumping off to a 20 email discussion about which endianess should be preferred and how the clients and server should determine which one to use, or maybe clients should support both kinds endianess.
All without a hint of irony. It's like a Bizarro world IETF discussion.
First, OSI layer 6 called and it wants its old job back. It sat around connecting layers 5 and 7 peacefully since the dawn of the ARPANET, all while the HTTPbis guys were passing around messages back and forth trying to obsolete it.
Second, you can't type a binary protocol. Yet, you have to somehow make the server work without a client or vice versa (for the initial implementation). That's going to be a lot more difficult. With HTTP, you could literally hook up a teletype to the Internet, let it print incoming requests and type the response back to the user agent. I've done this occasionally on a terminal emulator for debugging purposes.
Third, the Internet is big-endian while most common processors in use today are little-endian. This is going to haunt peoples lives forever because you have to continuously convert between the two and although the conversions are orthogonal, the methods aren't idempotent (as opposed to converting a string to ASCII, or a text buffer to DOS style line endings).
You mention 32-bit identifiers as opposed to a string of digits. This is more error-prone than you think, two's complement isn't the only integer representation out there. Implementations written in C would have to deal with their underlying architecture as the standard allows for 3 different representations (so the compiler wouldn't help you out). Then there's signed and unsigned, either of which might not be available in the implementers programming language. You end up unpacking the identifier by hand, which may end up being slower than just looping through a string. ASCII is hardly an inefficient serialisation format.
Fourth, any fixed-length records are going to be useless at some point in the future. Several versions later the fixed length records are going to either point to an extra set of records tailing the 2.0 records or will simply have a (designated) backwards compatible value for consumption by older peers. With HTTP, we can add a header anywhere in the request or response except for the very top. We can even shuffle them at will without adverse effects on peers.
Fifth, it doesn't make sense to optimise a tiny fraction of the entire HTTP session. Any benefits are too small to be worthwhile and would therefore result in a net-negative to most implementers.
Sixth, you can still make improvements to HTTP without moving to a binary protocol. Not sending the same headers on every single request isn't one of them. HTTP is essentially a stateless protocol and every request could be handled by a different server. You can architect clusters of servers routing incoming requests however you please and satisfy every one of them correctly and efficiently. For starters, you can replace any underlying protocol in the stack with a more cluster-friendly protocol in transit.
Seventh, just because no one is using a given content type in HTTP (I think you were referring to multipart/related) doesn't mean the protocol used to transfer that content is bad. Heck, it's not even part of any HTTP standard.
2. Amazingly, people have made binary protocols work before, in spite of no preexisting implementation, so it's not impossible. I'm sure we will be able to meet the challenge.
3. Do not try to sell me the endianess issue. I have written multi-arch tcp/ip stacks and i'm not a CS major. Trust me, it will be okay.
4. Yes, and IPv6 address space will someday expire. But not soon. And as many fixed-length frame protocols have done in the past, you leave an "extra frame options" bit to stack more fields on. It's fine.
5. It's really not about optimization at all. It's about common sense. The computer works better when you talk to it in computer-speak, and we gain absolutely nothing by talking to it in English human-speak. The benefits are a net-positive because parsing is easier, because a computer is parsing it, not a human. There is no sane argument that can validly claim that parsing human-readable English is easier for a computer than fixed-length bitstrings. CPUs don't grok ASCII, they grok BINARY.
6. Modern designs for clusters of web applications route by session, not by individual request. You are session-oriented instead of connection-oriented, though in practice it's almost the same thing. And see previous comment on why adding onto HTTP willy-nilly is just a hack.
7. No, just the jgc's re-implementation of multipart is bad, for previously stated reasons.
In the end, it is formats that people can understand that win the day. You can't just write that off as if it has no value. It plays out in technical ways: all the CORBA implementations ended up having very poor interporability partly because they were hard to debug. Nobody could actually look at a CORBA exchange and see what was wrong with it.
HTTP is not comparable because it never really changes. It's a fixed format. And frankly the majority of developers never need to go down to that level anyway.
1. It definitely is an argument and in fact my main argument. At least explain how this would be any less valid than your "it's a hack" and (bandwagon) "wave of the future" arguments.
2. My turn to invoke "not an argument". Just because one can simply copy a struct over a socket doesn't mean it's a good idea to do so. Especially in light of the flourishing culture of diversity we have on the Internet.
3. You conveniently choose to ignore one half of the argument, but miss it entirely. The point is not that we can't overcome endianness mismatch, it's that we shouldn't have to. At least not inside layer 7.
4. Except the old records will have to remain there forever. HTTP implementations dropped the Pragma header a long time ago and today we can simply pretend it was never there.
5. When it comes to common sense, ASCII is right there. That's because a protocol on the Internet needs to interoperable with many systems. Sure, all of those systems use binary one way or another. But human operators are still going to have to program those systems and ASCII is a useful representation which enables us to do just that. Furthermore, the draft proposes to encode binary headers in base64 in order to transfer them in an HTTP/1.1 upgrade request. Now you have 3 ways of transferring HTTP headers instead of just one and we'll have to support all of them in any case. This might seem trivial to you, but it's a problem to servers and quite a huge one at that for intermediaries (proxies).
6. Amending HTTP with a new header again is much less a hack than providing a way to switch to a binary protocol and resume communications from there. Your buggy argument doesn't stand, for HTTP is not the car. It's the pavement upon which old buggies can ride along just fine until it's no longer considered safe amongst the faster carriages.
7. Yes, I'm not convinced client-provided request identifiers are the way forward myself. Though I would consider the proposal a better starting point for discussions than the current HTTP/2.0 draft because it leverages existing mechanisms better.
2. My argument isn't "just because you can", it's "you can." You seemed to be saying it would be difficult if not impossible. I was saying, no it isn't.
3. Endianess will always be an issue, forever. The only time it will go away is when every architecture picks one format. It's a really simple operation and it's part of how computers expect us to behave due to their nature and design. Hacking around it doesn't make it disappear, nor does it help anything.
4. What old records? Pragma was deprecated in 1.1 yet included anyway for god knows why. There's no reason they should do so again, but if they do, it will exist both in text and non-text versions. This is a non-issue.
5.
> a protocol on the Internet needs to interoperable with many systems
You mean like IP, ICMP, TCP and UDP? > But human operators are still going to have to program those systems
Yep. > and ASCII is a useful representation which enables us to do just that
Sure. My C code editor displays ASCII. It totally enables me to write IP, TCP and UDP code, using an ASCII display with code in ASCII. And it neatly compiles down to binary and runs a binary protocol. Amazing!!!! (seriously though, if your argument is that ASCII is just easier to "program" as a protocol, you're up shit creek; you have to write more code to handle converting ASCII to binary and back anyway. your high-level language abstractions hide this fact from you, and you think it's a convenience because you never have to learn what a constant is) > it's a problem to servers and quite a huge one at that for intermediaries
That's backwards compatibility for you. If the alternative is to simply mangle and bungle the existing format into a frankenstein into eternity, it's not going to be any better.6. Are you comparing extending HTTP/1.1 for a single feature to the backwards-compatibility support of HTTP/2.0? Because that makes no sense. The vehicle analogy is just weird at this point.
7. See, this is where the vehicle analogy works again. "leveraging existing mechanisms". In other words, let's throw one more feature on top. It never ends, because all you have to do is keep adding more lines, and modify the browser, and modify either the server or your web app, and keep going to support god knows what. At some point they'll implement an incredibly complex binary protocol and embed it in base64-encoded ASCII HTTP/1.1 headers, because "leveraging existing technologies" is thought of as a neat thing to do. It will also be insane. At some point you need to just make a better <whatever> instead of hacking and hacking and hacking onto it to make it do what you want.
Like building the great pyramid of Giza out of tinker toys. Sure, it's easier for people to use tinker toys. It's easy to understand. You don't have to do any real work. And it's also not meant for that task. At some point you need to throw out the toys and use stone.
I can even go further. ASCII is too old to use. Really, it's been antiquated by UTF-8. It is telegraphic codes for teleprinters. And ASCII itself was micro-optimized to only be 7 bits, and the 8th bit was used as a parity bit because perforated tape had space for 8. ASCII is so antiquated (1960) that nobody should be using it anymore.
Clearly we need to implement HTTP/2.0 in UTF-8 wide characters, so connections to China, Japan and India will support their native language in the protocol. (After all, what's the point of a native-language protocol if only English speakers can read it?) Also, we should include the byte order mark at the beginning of all messages so we don't have to worry about how endianess works.
No need to wait: http://tools.ietf.org/id/draft-ietf-httpbis-header-compressi...
Just look at that and weep. That whole document deals with how to represent HTTP headers. It doesn't define them, their behaviour and how they should interact. No. This multi-page document merely documents how these headers should be represented.
You know, things which up until now has been:
Lines of text with key-value-pairs delimited by a colon-sign.
Noticed how that didn't take eighteen pages and pressumed anything about current generation consumer DSL MTUs? Yeah. That's a nice, simple and good spec.Obviously this HTTP2 binary monstrosity is being done all in the holy Google-name of micro-optimizing performance.
This is terrible design and quite literally obfuscation more than anything else. I cannot believe the IETF is even considering this junk.
Edit: Link to an IETF discussion on the subject: http://www.w3.org/Search/Mail/Public/search?keywords=&hdr-1-...
2. I don't disagree it's easy to come up with a binary protocol, taking a short cut is always easier. Just like it's easier not to write a test harness with full coverage for a software project, that's entirely up to you. When a regression causes havoc down the road before you realise what's going on, well, rather you than me.
3. You're defending a regression, as of right now it's a non-issue. And are you really calling ASCII a hack around endianness issues? On what planet?
4. Records that are going to be deprecated down the road, which I think is fair to consider inevitable. All I'm saying it's been a problem in binary protocols before, so let's not do that. You don't see this as a problem at all, so I'll digress.
5. You mean like IP, ICMP, TCP and UDP?
Yes, these are built into the operating system. Once you start using e.g. netcat (who hasn't piped tar into netcat for a quick backup?), all of that becomes transparent.
No, my argument isn't that ASCII is ipso facto easier to implement. It's that it's easier to test, debug and always see exactly what's going on over the wire.
If the alternative to a text-based frankenstein format is its binary-bastard child, I'll have the former thank you.
6. No, I'm telling you to think of HTTP as a conveyor rather then a payload. There's a difference, and that's why the vehicle analogy is weird.
7. So you're proposing that every N years we create an entirely new HTTP and upgrade to that? At what point will the streaming pile of upgrade requests yield a noticeable reduction in performance?
Also, UTF-8 characters are not "wide", they're variable length but not wide as in multibyte encodings. Then you go on to suggest we use a BOM at the start of every (UTF-8, mind you) message, I'll leave it up to yourself to let that sink in. You even spelled it out.
(Caveat, I'm back-of-the-enveloping this on my phone about to go to sleep But still!)
Many of the complaints on HTTP are really complaints about MIME messaging which the entire internet is really built on (standards anyways) and has ran pretty smoothly for a very long time. Approaching improving HTTP by addendum like SPDY is a better idea. Or possibly transporting it better over streamed protocols like SCTP: http://tools.ietf.org/html/rfc6525 no need to modify the packaging/messaging format.
MIME/HTTP/HTTPS are very flexible and if you want binary can be added in and has been in multipart, EDI/HTTP/AS2 and other RFCs use this. Multipart isn't used as much because it is more problematic (used heavily in email and custom protocols) so making the whole spec that way would be bad overall. The points on the OSI layers is very key, let's not revert to binary + base64 everything just to get data across the wire. You can put anything in there, basing it in text and human readable is always a good idea. That is really what this whole layer is about. Binary type of movement pushes us back to the days of non standard blobs, problems that http messaging then content as xml then json solved by standardizing readable exchange of data. When you are exchanging data in a standard way it should be very basic to minimize problems not collude. Throwing out all of MIME just to speed up HTTP when other protocols exist for any needs that are faster (real-time, attaching files, streaming etc) is a bad idea. Also changing support from HTTP < 1.0, to 1.0 to 1.1 had many problems, unless this adds considerable benefit, changing it adds more problems.
That's a very 1970s way to develop a new protocol.
These days you specify the protocol in, say, an XML- or JSON-based file format, and then run a code generator to produce client and server libraries directly from the spec. This has the advantage that the implementation is derived directly from the specification, so there is little room for ambiguity.
Wayland is one example in the open-source world of where this is done, but I've seen the technique used in proprietary shops as well.
The point is, the ability to "type" a protocol is irrelevant to how modern distributed software gets developed. Maybe it mattered in the days when comms were at 300 baud, machines had kilobytes or perhaps megabytes of core, and the Mark I eyeball was the best way to debug machine-to-machine comms, but these days we have tools that can decipher binary wire protocols for us. Performance and adaptability are far more important now than human readability. That war has been lost.
I can't tell you how many times I've manually read HTTP. To be sure, it's insignificant compared to how many HTTP headers have passed through my computer unseen by me.
ASCII may be inefficient, but computers are really fast, people are not. I don't have any measurements, but in making a browser, HTTP header parsing/writing was never near a performance issue. Bandwidth wise it also tiny compared to that image file you'd inevitably download for every page visit.
And sometimes you don't have tools. Sometimes you don't want tools. Sometimes you want to use tools that work with text to analyse your problem.
(You other arguments might still stand, though :) )
Since header lengths are not limited, and a single TCP packet's payload is quite limited, long headers can cause very measurable latency difference. Additionally, while I agree the generation / parsing overhead is probably quite small, saving it for every HTTP request is still a boon.
I'm also curious where you are reading raw HTTP from?
For me it's primarily in two situations, reading a packet capture from WireShark, or in the browser's debugger. In both cases, the tool will end up translating the request for me.
Just because I use Charles or Wireshark doesn't mean that I only want to use those specialized tools. I have definitely been in situations where I'm doing something like running nc as a proxy and looking at raw HTTP. I wouldn't choose to throw that away and revert to the bad old days without a big win.
But now nc supports SSL natively, making it super easy. Just as it will support binary HTTP natively, making it super easy. And everyone will finally stop caring about ASCII.
I'm questioning the measurability of this, though. Smells like premature optimization. You could be right, but I'd like to at least measure it before we go about changing one of the fundamental protocols on the internet.
Last time I read raw HTTP was when writing a script to automate some stuff on a web page. I specifically did not want the browser's headers and behaviors. I had a bug which only happened from my script, and raw HTTP helped me track it down. I could have used wireshark, but I am much faster in vim for a simple task like that.
As a comparison to your scripting story, you would use Wget or Curl or LWP::UserAgent or a thousand other things to automate HTTP requests. One function call to do what you did manually. To find bugs you would use an HTTP fuzzer like Skipfish to automate the process. If you think somehow your manual process was faster, I say to you, teach a man to fish...
(I automate things in web pages for a living, and I only use tools like Firebug and LWP)
Also sometimes you need something spcific and then you have to option to code it up yourself, or change a open source library quickly and efficently.
The advantages of text based protocol remain the same. The disadvantage is lessened by faster CPUs.
If anything, the more complex the protocol, the more redundant the text becomes, because we have to write tools to parse the text and output it so we can understand it better or identify flaws in it, and work around bugs introduced by the human element of the protocol. The ability to view and interpret the protocol in a text editor is equivalent to the ability to view and interpret the protocol as output from a debugging tool or log file - except the tool can give you much more detail than the text file in a variety of ways. Text files are inferior, but they can be quicker/simpler, depending on what you're doing.
You still need a library or tool to write the protocol out, as it's complicated and needs to be structured for the machine, not a person.
Second, saying "it's ok that it's slow, we'll just buy a faster CPU" is not a good argument for anything ever. It's part of the reason it's taken so long to adopt encrypted services everywhere. Someone (Google) had to finally prove it wasn't slow so people would adopt it.
Third, the state of modern computers is that there is no difference in speed between interpreting most text protocols and binary protocols. But that has nothing to do with efficiency, or what the machine is naturally suited to doing. You have to translate from English into machine code for a computer to know what the hell another machine is talking about. Machines don't care about line-by-line, or capitalization, or indentation, spaces, or any vestige of our natural language. Strip all those things away and machines purr along happily with less bullshit to deal with, which means simpler, more efficient code. Note that I didn't say faster.
Fourth, your performance and history observation is flawed. We need a lot more performance today than we did before, as we're scaling existing technology to many, many orders of magnitude higher than anything that existed when it was invented. Yes, we have faster CPUs. We also have more users and more data, and we don't have time to sit around reading packet dumps in text editors.
This is entirely false, as anyone who ever had to debug a malfunctioning http proxy or a misbehaving IMAP can tell you. Nothing beats netcat for a quick bug isolation test. As for the need for formal parsing, again it is true for production code, entirely false for sysops transient tasks.
Compare debugging a corba server with debugging http for a whiff of the he difference.
Tools aren't omnipresent. My miryad busybox embedded devices won't ever likely have a protocol analyzer. If I'm in need of one there, I'm done with.
With a binary protocol you're entirely dependent on tools (except you want to trawl through it with a hex editor)
Seeing as we use binary protocols every day of our lives, and the tools to work with them have existed for years, and nobody has any problem with using them, let's let this argument rest.
d-(^.^)z
"The history is doomed to repeat itself".
Fifth, redundancies introduced from the beginning of time need to go away I wholeheartedly agree with this, but it doesn't automatically warrant binary encoding.
Human readability is a huge bonus in any protocol or format. Not because normally people read those protocols, but because people read ASCII and therefore they have good tools to work with ASCII.
HTTP is a layer 7 communication protocol. HTML/CSS are markup languages for designing an interface. JSON is a data interchange format. RSS is a content syndication format.
They are all wildly, vastly different. The only thing they have in common is they're all ASCII. If anything, you're making my argument for me: a communications protocol is not a format for displaying documents, it is a language for communicating machine instructions to network applications. Historically they have always been binary because it works better that way.
Your argument that "people can read ASCII, so ASCII is good" leaves out a couple points. Like, human beings do not read an HTTP statement, go into a file folder, bring out a document and present it to their computer. It's the other way around.
Really this just reflects a strange phobia people seem to have. Like your brain is tricking you into thinking you'll lose something by not looking directly "at the wire".
When you look at HTTP headers, 90% of the time you're actually looking at a pre-parsed, normalized set of fields. If you look at a raw packet dump, the whole message may not show up in one packet; you may have to reassemble it, which means parsing. If you have multiple requests in one connection, you have to find the end of the last request, which means seeking through the stream; seeing requests broken down individually means a tool already parsed them. Firebug and wireshark and other tools all take care of the automated, machine-operated work for you.
And what's left? What do you have to do with HTTP, really? Apache rules? They'd stay human-readable. Application testing? We use proxies that handle it, and APIs for client/server programming. Firewalling? Handled by tools and appliances.
Stop giving me the blanket "ASCII is great for everything" excuse and tell me one thing, one single thing, that only humans are able to do with HTTP using their eyes without a tool. But you don't have to, because that's impossible: HTTP is not for humans.
I look forward to servers having different text representation of the same binary headers in their config files.
tell me one thing, one single thing, that only humans are able to do with HTTP using their eyes
You're missing the point.
No one writes HTML manually anymore either. People generate it using tools (string processing tools in a language or templates) and read it using browsers. Heck, even Notepad++ is a tool, but a generic one.
If you want, you can generate all your HTML using DOM. But almost no one does that, because DOM tools are clumsy, while text-based tools are easy to use.
I was up until four last night doing just this. It's commonly done for templating purposes all the time, or quick hacks and placeholders.
Have you lost your mind?
If no one writes HTML manually anymore, then we have no need for it to look like English when the computer interprets it! We can compile the HTML down to bytecode and have it be interpreted much quicker by the computer, which won't have to do the job of lexing, compiling, assembling, etc. Here, two steps would be eliminated immediately, resulting in increased speed and more efficient storage and transmission: http://www.html5rocks.com/en/tutorials/internals/howbrowsers...
For that matter, if it's generated by tools, and we use programs designed to interpret and decipher and color-code it, all of that can happen without it being in English!
On top of that, you missed when I said HTTP is a communications protocol. Ever seen the movie The Matrix? Know how the sentinels would sometimes look at each other and make scuttling noises, then shovel off somewhere? They weren't speaking English ASCII. They were speaking a binary communications protocol. Know how I know? BECAUSE MACHINES AREN'T HUMANS! It would be absolutely moronic for them to speak English to each other. It would be like dogs saying the English word "bark" instead of just barking. Completely unnecessary and crazy. But that's what an ASCII communications protocol for machines is.
On top of that, there is no benefit, not one at all, to humans being able to read it when tools already exist to interpret and display it even more human-readable than its natural state. We squish and compress and strip HTML and JS already just to make it more efficient, and then undo the whole process just to read it. It's insane.
The web is made by people, not computers. Open an ubiquitous text-editor and you can start working on something right away. If you have to download a dozen different compilers and IDEs to do that, it's definitely not the same.
Face it. Your love affair with ASCII is just that: an emotion.
(As to your original question: humans haven't needed to program in binary or assembly for decades. That's what so great about computers: they do the hard work for us, so we don't need to type everything manually into a text editor. Is that such a hard pill to swallow?)
Yes, binary is more efficient, but then tell me why is JSON the most popular data interchange format on the web today?
Yes, let's base HTML 6 on Word .doc.
Also, the machines in The Matrix were hostile to humans. We'd like machines in the real world to be... not so.
debugging.
With a text based protocol, you can inspect it visually with no special tools, and munge it with general purpose tools that you already know how to use (shell script, sed, awk, perl, python, ruby, what have you) with no special support libraries or anything of the sort. Support libraries can help you with the more complex aspects of the protocols, but for basic debugging purposes, you can do it all with general purpose tools.
With a binary protocol, you need those libraries to even have a chance of being able to work with it. Now you can't use a general purpose shell pipeline to munge it; no more nc | grep or what have you. You have to have a wireshark dissector; and good luck figuring out how to grep through the results of what a wireshark dissector generates.
The main point is that the overhead of the ASCII encoding isn't the main problem with HTTP. Reading ASCII encoded CRLF delimited headers is a solved problem (and heck, you could probably switch that to just LF delimiters, since I'm sure that most processors already handle that case just fine).
The problems are things like having to repeat headers over and over again for each request in a session, enormous cookies that need to get sent with every request, and the like. But you can solve those without throwing away the easily debuggable ASCII-encoded headers; and compression really does solve most of the problem with the inefficiency of ASCII encoding (and you're going to want to use anyhow, since the HTML, CSS, and JavaScript that you're delivering is all a fairly inefficient ASCII representation too).
There isn't one, don't bother looking. Corba is the poster child for the problems with binary protocols: fragmentation, buggy implementations, incompatible extensions.
I'd rather not see HTTP follow the Corba path.
It can be, but that doesn't really make sense. The vast majority of web development is done without manually editing HTTP headers. It could change to a binary format tomorrow and - so long as our tools preserved their interfaces - we wouldn't even notice. The same cannot be said for any of the other technologies you listed.
http://php.net/manual/en/function.header.php
This is a good example where adding an object-oriented representation to every header out there would require a lot of work. Not sure if it would justify the gains.
It could change to a binary format tomorrow and - so long as our tools preserved their interfaces - we wouldn't even notice.
Until you try to use grep or something of that sort for some non-trivial analysis operation. Everything 'speaks' ASCII. Custom tools for binary format would take years to evolve to be as powerful as generic text tools.
All you need is one parsing tool that produces a textual representation of the binary protocol, and you can once again use grep and friends.
Most of the tools I've used represent the header as a hash/dictionary. I fail to see how that approach "requires a lot of work".
> Until you try to use grep or something of that sort for some non-trivial analysis operation.
You're arguing from the assumption that a binary protocol would be implemented by idiots. Custom binary tools can always emit a textual representation, at which point you can grep through it to your heart's content. This is the exact same problem that we've been solving with compilers for generations. It isn't nearly as insurmountable as you seem to believe.
HTML isn't XML based; there is an XML-based relative of HTML (XHTML) which was originally (before HTML5) viewed as a potential successor to HTML, and with HTML5 there is an available XML-based serialization of the HTML's semantics, but HTML is its own thing (prior to HTML 5, HTML was SGML-based; XML was inspired by HTML rather than serving as the basis for it.)
That's pretty annoying to see this kind of thinking. The reason why everyone codes in JS and uses HTML, CSS is because its ASCII. Its easy to understand, hack, etc. Same reason python is so popular. Even Go, is pretty simple like that. Sure its languages vs protocols, but the reasoning is exactly the same.
And in fact, the comparison works with protocols as well: SMTP, IMAP, HTTP, IRC are EXTREMELY easy to understand and code for. Binary protocols are a huge PITA to code for. The argument that you're going to use a lib or whatever tool just doesn't hold any water. You want to understand what exactly happens.
Thats how everyone learns, etc. I could write my own SMTP, IRC clients when i was 10. I could understand it. It works. No way I could understand fully the documented binary protocols. I tried, and it was just too painful and not fun at all (hey, I was 10).
I'm not certain the added performance of using a binary format and some of the other advantages are really good enough to make the world unable to understand whats going on anymore by just looking at it.
Sure purely technically speaking, it sounds like "binary is the way to go" for pure performance.
But if you think about it, making hacking around that stuff a niche thing is perhaps a much greater loss. Even the reliability of a binary protocol is VERY arguable.
In fact I'll put a last comparison. Shell pipes and ASCII. Many tried to replace them with smart binary protocols, objects, etc. Its cool. Its more powerful. More efficient. At the end of the day tho, a quick hack with regular pipes transferring ascii is just easier to understand and we all use those - not the fancy binary objects.
Of course, you can say TCP/IP is still binary, and it is true. But TCP/IP tools are built in every OS in existence now, so they do not form a real entrance barrier. Would HTTP tools be in the same position? I'm not sure - most HTTP tools right now are not standard and do not cover even HTTP/1.1 completely, what reason is there to expect they'd cover the whole 2.0 protocol properly and be widely standard and available on the level tcp/ip tools are? Which means much higher barrier of entry.
That is always my stance on things - if one computer is going to run something, write it in python, make it bleed memory, just make it work. If it is going to run on a million, you have to consider the raw power waste of inefficient programming. If it is going to run on trillions of devices for decades, your choices are few in my mind.
>>> If it is going to run on trillions of devices for decades, your choices are few in my mind.
The history suggests otherwise - majority of mass-produced software is not written with performance as an ultimate concern. You would find a lot of software written in languages like Python or Java, even though using C or assembly would probably produce better performance. But using C or assembly that software would probably never be produced because its complexity would be harder to manage.
Of course, performance does matter - even writing in Python, you have to worry about performance. But here we effectively see an argument saying "since we have a lot of software in Python, if we switch it to C we'll have massive performance gains". I think it is a wrong line of argument, if we switch to C a lot of this software wouldn't be written. (Note it's not against Python or C - I use both and they both are great in their areas :)
So I guess optimized protocol does have its uses for high-volume websites - but I am concerned its advantages would be offset by its complexity. The designation of it as HTTP/2.0 implies it is the next version of HTTP - but it's rather a rather different thing with different use case. I'd rather have it as a separate protocol for high-traffic websites.
http://www.catb.org/~esr/writings/taoup/html/textualitychapt...
Why aren't we using binary file formats anymore?
XML is more easily debugged when the machine tools don't work.
It's also more accessible, more easily created and modified, and thus, more available to a wider range of people than just web design professionals.
First, ASCII is inefficient. People don't interpret HTTP, computers do. Web servers and browsers. People only look at HTTP when they want to troubleshoot without any tools. With real tools, you can find out what's broken much quicker. And there's plenty of things you can miss without a real HTTP interpreter. Most hackers prefer to think of themselves as wizards that can spy 0's and 1's and tell you what the weather is. It doesn't make for a better protocol, though.
Second, we can already break HTTP responses up in multiple parts, using a novel idea called "multipart". It sucks and nobody has used it since HTML/JS found new ways of providing content. http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
Third, it's a hack. If you want to improve the protocol, improve the protocol, don't just hack onto it to make it do what you want. I could make a horse and buggy go 60mph, but would it be a good idea? How about just designing a better buggy that is intended to go 60mph?
Fourth, fixed-length records are the wave of the future! It solves crazy problems like header injection and request integrity checking. Moreover, it makes for simpler, more efficient parsing of requests.
Fifth, redundancies introduced from the beginning of time need to go away, like terminating every record with "\r\n", or passing the same headers on every single damn request when once should be just fine for a stream of requests. Little inefficiencies like this don't go away if you just hack onto the same old protocol forever.
Sixth, the flow control improvements can make different applications more efficient by both not having to hold state of where and when traffic is coming and improving flow across disparate network hops.
Seventh, as to a general question "why use binary when we can compress stuff", what is smaller: a string of bits, or a compressed string of digits? Add to this that every header could have a 32-bit identifier (4 bytes) and you've got more efficient compression than gzip. Of course, most people would argue that you don't save enough time or bandwidth or cpu cycles to make a difference, but it would make for easier-to-write parsers, which would make working with the protocol in general more attractive. But then you have your binary-detractor-wizard-hackers and the whole conversation becomes an infinite loop.