News for hackers. Stuff that matters.
Re:Althttpd: Simple webserver in a single C file by lmilcin on Tuesday June 08, 2021 @06:07AM (#27432587)
Binary protocols aren't (or at least don't have to be) any more difficult and frequently are even easier to implement.

Text protocols have difficult problems like escaping or detecting the end of particular field that are frequent source of mistakes.

The issue is that many (especially scripting) languages treat binary data as second class.

The only real issue is that inspecting the binary message visually is little bit more difficult. You can usually easily tell if your data structure is broken if it is in text form. What I do is I usually have some simple converter that converts the binary message to text form and this helps me inspect the message easily.

Re:Althttpd: Simple webserver in a single C file by chrismorgan on Tuesday June 08, 2021 @06:39AM (#27433081)
And capitalisation issues, and encodings… give me a binary protocol to parse any time. They’re normally comparatively well-defined, whereas text protocols are seldom properly defined and so have undefined behaviour left, right and centre, which inevitably leads to security bugs—or even if they are well-defined, they’re probably done in a way that makes your language’s string type unsuitable, and makes parsing more complicated.
Re:Althttpd: Simple webserver in a single C file by pjc50 on Tuesday June 08, 2021 @10:01AM (#27435320)
People say this, but then there's ASN.1 which has had critical security bugs in: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1

I will agree that exhaustively defining text protocols is extremely hard, starting from character set / encoding and getting worse from there.

Re:Althttpd: Simple webserver in a single C file by lmilcin on Tuesday June 08, 2021 @11:44AM (#27436657)
ASN.1 is not a binary protocol. It is a language to describe messages.

Typically you create message description which is then compiled to code that can serialize/deserialize messages in BER-TLV or PER-TLV.

I know because I wrote a complete parser/serializer for BER-TLV. It is simple protocol and any security issue is in the parser/serializer and not the protocol itself. That for simple reason that the protocol is nothing more than a format to serialize/deserialize the data.

Re:Althttpd: Simple webserver in a single C file by lmz on Tuesday June 08, 2021 @07:33PM (#27442442)
True, but they certainly don't make it too easy either with the indefinite length constructs.
Re:Althttpd: Simple webserver in a single C file by admax88q on Tuesday June 08, 2021 @04:15PM (#27440182)
I guess if you're going to pick the absolute worst example of a binary protocol.
Re:Althttpd: Simple webserver in a single C file by lmilcin on Tuesday June 08, 2021 @05:06PM (#27440794)
That is absolutely not fair.

BER-TLV is really nice protocol. I have worked with it for couple of years when I worked on an EMV application. It uses BER-TLV to communicate with the credit card but it is also very convenient format for all sorts of other uses and I would use it wherever I could. Think of it as Json but in binary form. It is not complicated and I would not even bother parsing the messages -- I could interpret hex dumps of them by sight very easily.

Re:Althttpd: Simple webserver in a single C file by the_arun on Tuesday June 08, 2021 @11:52PM (#27444031)
>>* The author disclaims copyright to this source code. In place of

>>* a legal notice, here is a blessing:

>>*

>>* May you do good and not evil.

>>* May you find forgiveness for yourself and forgive others.

>>* May you share freely, never taking more than you give.

Above all, this note at the beginning of the source code impressed me.

Re:Althttpd: Simple webserver in a single C file by scruple on Tuesday June 08, 2021 @12:59PM (#27437755)
> give me a binary protocol to parse any time

I don't disagree in principle but I've come across a handful of very poorly documentated binary protocols in my years and that is an extremely painful thing to deal with compared to text-based protocols.

Re:Althttpd: Simple webserver in a single C file by jrockway on Tuesday June 08, 2021 @02:23PM (#27438896)
I think it's pretty easy to make poor use of HTTP to the same end. Imagine you're dumping traffic between a client and a server and the exchange is:

   GET /asdfasdfasdfdsaf
   X-Asdf-Asdf: 83e7234
   
   HTTP/1.0 202
   X-83e7233: 1
   X-83f730b: 4
   
It's text, but you still have no idea what's going on.

Overall, I think it's kind of a wash. It's basically equally easy to take a documented text or binary protocol and write a working parser. Neither format solves any intrinsic problems -- malicious input has to be accounted for, you have to write fuzz tests, you have to deal with broken producers. It's a wash.

People like text because they can type it into telnet and see something work, which is kind of cool, but probably not a productive use of time. I can type HTTP messages, but use curl anyway. (SMTP was always my favorite. "HELO there / MAIL FROM foo / RCPT TO bar / email body / ." Felt like a real conversation going on. Still not sure how to send an email that consists of a single dot on its own line though.)

Re:Althttpd: Simple webserver in a single C file by knome on Tuesday June 08, 2021 @04:16PM (#27440189)
>Still not sure how to send an email that consists of a single dot on its own line though.

It seems you prefix any line of data starting with a period with an additional period, to therefore distinguish it from the end of mail period.

https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.2

Re:Althttpd: Simple webserver in a single C file by taejo on Tuesday June 08, 2021 @04:14PM (#27440169)
> Still not sure how to send an email that consists of a single dot on its own line though.

A line starting with a period is escaped by adding an extra period; the receiving side removes the first character of a line if it is a period.

Re:Althttpd: Simple webserver in a single C file by funny_falcon on Wednesday June 09, 2021 @11:09AM (#27448790)
It's the dirties hack I've ever met.

Except I did something like this once in the past (but with zero byte instead of dot).

Re:Althttpd: Simple webserver in a single C file by chrismorgan on Tuesday June 08, 2021 @01:26PM (#27438158)
Ugh, flashbacks. Now that you’ve mentioned that I feel the burning need to change my wording to add the condition “fairly described”. I have also come across very poorly documented binary protocols!
Re:Althttpd: Simple webserver in a single C file by touisteur on Tuesday June 08, 2021 @06:56AM (#27433375)
Just use recordflux for binary parser proved absent of runtime errors... ;-)
Re:Althttpd: Simple webserver in a single C file by einpoklum on Tuesday June 08, 2021 @07:23AM (#27433640)
Can you expand on that a little?
Re:Althttpd: Simple webserver in a single C file by michaelmior on Tuesday June 08, 2021 @09:24AM (#27434830)
RecordFlux[0] is a DSL written in Ada for specifying messages in a binary protocol. Code for parsing these messages is then generated automatically with a number of useful properties automatically proven including that no runtime errors will occur.

[0] https://github.com/Componolit/RecordFlux

Re:Althttpd: Simple webserver in a single C file by jchw on Tuesday June 08, 2021 @02:07PM (#27438725)
They definitely don't have to be. I've echoed these sentiments before.

However, HTTP/2 and HTTP/3 certainly are, though the reasons why they are complicated have nothing to do with choosing to use a binary based format. (They are complicated for good reason, though, and I hope that browsers and servers can continue to support HTTP/1 as the baseline till the sun burns out, just to make life easier.)

Re:Althttpd: Simple webserver in a single C file by mucholove on Tuesday June 08, 2021 @07:37AM (#27433764)
I did implement a webserver (that supports websocket-a binary protocol) and also implemented a `make_binary_string` function along the wah so as to not lose my bearings. Printf is 100% amazing and I generally leave my printf statements inside all my code because I can switch them on and off very easily for any individual function by using macros. Also clutch :)
Re:Althttpd: Simple webserver in a single C file by daitangio on Tuesday June 08, 2021 @08:33AM (#27434275)
Yes and no. Debugging a binary protocol is a bit more difficult then a text one; which is the reason HTTP is ascii based.
Re:Althttpd: Simple webserver in a single C file by zaarn on Tuesday June 08, 2021 @08:44AM (#27434387)
The reason HTTP is ASCII based isn't because it's easier to debug. It's because back then the other end was as likely to be a human as a piece of software. Because HTTP in it's early days barely had headers or even formatting, so people typed "GET /" at the server directly or used the same method to send mail directly.

Nobody does that anymore and debugging is easily solved by converting your binary protocol to a textual form.

Re:Althttpd: Simple webserver in a single C file by throw0101a on Tuesday June 08, 2021 @09:11AM (#27434676)
I question the ascertain that humans were doing "GET /" or "HELO my.fq.dn" on any regular basis. There were mail clients from the very beginning for example:

* https://en.wikipedia.org/wiki/History_of_email

Perusing document-based information was also done via clients, first with Gopher and then with the WWW: either GUIs like Mosaic, or on the CLI via (e.g.) Lynx.

Re:Althttpd: Simple webserver in a single C file by michaelmior on Tuesday June 08, 2021 @09:26AM (#27434842)
I definitely did back in the day :) But I'm only one single human.
Re:Althttpd: Simple webserver in a single C file by lmilcin on Tuesday June 08, 2021 @09:51AM (#27435189)
I did both send mails over SMTP as well as downloading files from FTP using Telnet.
Re:Althttpd: Simple webserver in a single C file by jbverschoor on Tuesday June 08, 2021 @12:13PM (#27437107)
same
Re:Althttpd: Simple webserver in a single C file by zaarn on Tuesday June 08, 2021 @09:16AM (#27434733)
I would question if this weren't the case often enough that people would demand it be compatible with their teleprompter.
Re:Althttpd: Simple webserver in a single C file by choeger on Tuesday June 08, 2021 @10:31AM (#27435673)
I don't think this is about human readability (albeit it might be to some extent during the design phase). Instead, I think this was the sensible thing to do when byte order was way more variable across a network. If you only send single bytes to begin with, you can as well use a textual format, IMO.
Re:Althttpd: Simple webserver in a single C file by squarefoot on Tuesday June 08, 2021 @11:54AM (#27436807)
It depends from the nature of the payload. If I have to send text, I'll use a text based protocol because I can debug it more easily, but if I have to send binary information, I'd rather send it using well defined structures after taking into account word sizes, alignment, endianess etc. on the involved hardware. Back in the day I had to do that with small agents on different achitectures (IBM Power and x86) with different languages (C and Object Pascal), and the correct textbook way to do that was to use xml, but that way everything had to be converted to text, interpreted and then translated back for replies. No way, hardware was slow and we aimed at speed, so we used plain C/Pascal structures and no other manipulation except for adjusting type sizes and endianess if the hardware or language required so to make all modules compatible no matter what they were written in and where they would run. I also built my own tools for testing and debugging which were nothing more than agents that sent predefined packets and echoed at screen the return values, so that any error in the data would be quickly spotted, while a xml translation of a corrupt buffer could have missed any error not comprised in the marked text, hence hiding there was a problem somewhere. A solution like that is highly debatable for sure, but I'm among the ones who want their software to crash loudly at the first problem, not try hiding it until it creates a bigger damage.
Re:Althttpd: Simple webserver in a single C file by Iwan-Zotow on Tuesday June 08, 2021 @09:57AM (#27435277)
> which is the reason HTTP is ascii based.

well, other side of HTTP was telnet in the terminal - this is why it is ASCII to begin with

Re:Althttpd: Simple webserver in a single C file by tdeck on Tuesday June 08, 2021 @01:53PM (#27438538)
This is not true. HTTP was developed for the first (graphical) web browser, WorldWideWeb. It was specifically intended to serve HTML and images, neither of which is particularly well suited for browsing with telnet.
Re:Althttpd: Simple webserver in a single C file by eloop on Wednesday June 09, 2021 @12:14AM (#27444155)
Not sure that is strictly correct. I vaguely remember a command line browser early in the piece. The inline image ability and <img> tag came a bit later in the story when Andreessen put it into the NCSA's Mosaic browser. I've always wondered if it was that single thing that really made the web take off, more so than hyperlinking which had been around for quite a while. People like pretty pictures, especially advertisers.
Re:Althttpd: Simple webserver in a single C file by tdeck on Wednesday June 09, 2021 @12:57PM (#27450257)
You're describing the line mode browser:

https://en.wikipedia.org/wiki/Line_Mode_Browser

But that didn't just spit out the HTTP response - it parsed and rendered the HTML.

Re:Althttpd: Simple webserver in a single C file by arcticbull on Friday June 11, 2021 @12:41PM (#27475533)
Yeah, humans can roughly speaking infer the schema of a text-based protocol but require one for a binary protocol.