And I'm not sure there's a good solution. The obvious approach, as you already know, is to separate headers into two groups: one group is "properly" handled, and the other group gets tossed into an "other" bucket of key/value string pairs. But it's going to be awfully tempting over time to have "proper" support for the most common non-RFC2616 headers, especially those which have widely-accepted standards of their own.
But that's a bit messy and much less than ideal, as you've also discovered. And I don't know of another way to approach the problem that still lets you have the stuff you want from the type system, and doesn't cut people off the minute they start using extension headers your library doesn't know about.
And that's without getting into all the completely stupid things it's technically legal to do in an HTTP header, and the even more stupid things that aren't legal but that people do anyway. My usual torture test of an HTTP library these days runs through things like non-ASCII content in header values (legal if properly encoded, and not everything in the wild properly encodes), looking to see how large a value a library will accept, whether it will let me throw newlines into headers, etc. etc.
I'll give you a real-world example. Mono. HttpWebRequest used to store the Content-Size value as an int32. What about files larger than 4gb? Oops...
That was a big problem for us. Thankfully Mono is open source, we were able to peruse the code and come up with some magic incantations using reflection to work around the fact that the framework libraries would not allow us to set the Content-Size header without going through the int32 property.
Now, normally you would file that as a bug, however, it (A) took years for a version of mono to be released with that fixed this and (B) could easily have been some misunderstanding in the implementation of the HTTP spec that was overlooked. You're implementing a spec right now. Can you guarantee that the your implementation is 100% correct? Can you guarantee that the spec is 100% correct? Can you guarantee that the spec is implemented 100% correct everywhere?
No, you can't.
So, don't limit me. Help me, but do not prevent me from doing what I need to in order to do my job.
https://groups.google.com/forum/#!searchin/golang-nuts/s3/go...
The short summary is that Amazon S3 does not behave like a normal web server and expects HTTP path's to be "url encoded". S3 is pretty obviously something you would want to interoperate with. Go's http client (which isn't very strict at all) causes friction here.
In the past when I have used such libraries they have been a pain - especially for loose formats like HTTP headers. Type system come with a cost, you need to carefully consider the burden you are placing on the programmer to see if the benefits out way the cost.
Remember Postel's law: "Be conservative in what you do, be liberal in what you accept from others."
Unfortunately, you will have to return URLs to your users that are invalid. You will have to parse slightly malformed dates. Your clients will expect to be able to pass through malformed values from bad endpoints and not introduce a point of failure in your library. (Do you know about the "Cneonction: close" trick, for instance? All real-world proxies do all sorts of batshit-crazy HTTP header munging.)
If your library can't accept and pass through arbitrary strings for arbitrary headers, and return them to users, and then accept them back again, then it is unlikely to be successful in any sufficiently complex and robust HTTP service.
As a (albeit poor example), I once inherited a click tracking system that uses Java Jersey. The standard response object has a .redirect(URI uri) method. Unfortunately, many "URI"s we receive from the world are not actually value, and the java URI object refuses to build URI objects from the string. Fortunately, one can simply call .setHeader(String, String) and be done with it.
When the framework treats a a valid http header as invalid the framework is probably using the wrong abstraction for http headers.
You can be both . . .
IMO, all the fields need to have a Printable and Readable trait (or whatever Rust's version of those are). If you want to do special things for certain fields to make them easier to work with, I think that's cool too. But if you force people not to treat headers as text, I think you're going to have a problem.
It seems to me that, in the worst case, the downside of your current scheme is an extra field dereference, as in `request.headers.extensions.get("crazyNewField")` instead of `request.headers.get("crazyNewField")`. I think that cost is easily outweighed by the benefits.
response["headre"] = ...
People say the same thing about null pointer exceptions, but null pointer exceptions are a huge problem in practice. Every bug is "obvious" in retrospect.
I do understand the desire to prevent these sorts of errors at the language level, but not at the cost of functionality. I don't agree with this one, but I do (for example) appreciate libraries such as the STL which (hopefully) prevent that pointer issue you brought up (and more). The STL doesn't (often) force me to work around silly limitations though.
Does this library provide for creating arbitrary headers in some other fashion? If not, why?
I love that there are people who say things like this totally seriously.
I've worked on very large, complex, mission critical systems my entire career. Medical devices mostly. If you're not catching these sorts of errors early then you aren't modeling and testing your software correctly (or at all). All of the objects in the world aren't going to prevent you from writing bugs and crappy code. OO is no panacea and we've known that for quite some time now. I don't want to use a library which is limiting because it thinks it is doing me a favor.
I don't think I'm infallible (straw man much?), but I do think that dead simple errors such as this would be caught by any amount of actual testing. That said, I wouldn't be against a better design. Why doesn't this library provide a way to add arbitrary headers? Oh, right, that would still involve using a string. Gasp, no, that shouldn't be allowed. We have to protect the poor users of our library. Yeesh.
You can't always protect a programmer from themselves while still allowing the flexibility/power needed to get shit done. I'd rather use the map than some overly architected and frustrating library. I have a problem with abstractions that get in my way.
You are suggesting that you are infallible, as the "dead simple errors" you refer to get out into production systems every single day. More than one serious security hole in rails has been due to very similar problems. You are very much implying that you are a better programmer than the entire rails team. And virtually everyone else, as examples of these kinds of bugs are everywhere.
And finally, crying straw man while presenting a straw man is poor form. If a particular library has a deficiency, that's unfortunate. But it has absolutely nothing to do with type safety. Haskell for example is more type safe, and it has two http client libraries, both of which provide safety while also allowing arbitrary headers (and methods, etc).
EDIT: Eh... looking back at my original comment I don't think I got that across at all. It reads as more of a rant against a typed system in general, which is not what I intended.
I love that there are people who say things like this totally seriously.
If people like OP are dissing a language because they don't like a particular library's API, they have very strange expectations from the language. And if you are worried about this section of users, then I would say in order to make good use of your time, it is best to ignore them.
If it's hard to do X in your language, but easy in other languages - then that's a weakness for your language, despite that 90% of X's will be outside of the language core.
And even 'optional extras' in your language can be weaknesses instead of features - if they are often used by library makers for less readable/maintainable code than the other options.
Why not add a weak checking for k-v strings? Something like during compile time:
if not k in LIST_OF_WELLKNOWNS
warn('Bad header maybe?')
I really like the spray way. You can add a custom/nonstandard header by using RawHeader(key, value), but most of the time you use e.g. Headers.ContentType(MediaTypes.`application/json`), and it just completely eliminates a class of trivial but annoying bugs.
rust-http is, at present at least, rejecting the standard practice of stringly typed headers and is using strongly typed headers.
> ... and even types for every ‘known’ HTTP header.
> The known HTTP header part was particularly ridiculous. HTTP headers are defined as simple key/value pairs. You’re supposed to be allowed to add arbitrary headers. You’re not supposed to be constrained by the framework you’re using not having any support for CORS yet.
He has taken a strong position against this strong typing; I take a similarly strong position for such strong typing.
Yes, HTTP headers are defined as simple key/value pairs. But headers are not text; they are data. XML is defined as text. Should we use regular expressions to work with it? (True, it also has the DOM defined. But HTTP headers also have a grammar defined—it just happens to include the admissibility of extension headers.)
HTTP is a serialisation format for messages; messages made up of computer-readable data, not text.
I wrote a handful of paragraphs about this subject at https://github.com/chris-morgan/rust-http/issues/1 where using a map instead of typed headers was suggested.
As I also note there, I would not attempt such a scheme in a language like Python (my primary language for some years) because its type system is not well adjusted to something like this; I am certain it would cause many problems and feel it likely it would cause more than it solved. But in a language with a solid type system capable of expressing these things, taking advantage of it should lead to a faster, more correct program.
In the end, I believe strongly typed headers are a good thing to have (it will actually make most code distinctly nicer—e.g. ``response.headers.date = Some(now())`` instead of ``response.headers.set("Date", now().to_http_header())``). The implementation of the scheme in rust-http is certainly not perfect yet; I'm not happy with it at present; I'm expecting to turn the extension headers into a trait object, and I've even considered crazy ideas like allowing users of the library to make their own header collection (request.headers and response.headers) types—that would allow custom headers to be made first-class citizens but could lead to compatibility problems.
Still, I don't want to make something that I believe should be the best HTTP library around and find that people hate to use it. What do you think about this header matter?