const response = await new Promise( (resolve, reject) => {
const req = https.request(url, {
}, res => {
let body = "";
res.on("data", data => {
body += data;
});
res.on('end', () => {
resolve(body);
});
});
req.end();
});> Undici is an HTTP client library that powers the fetch API in Node.js. It was written from scratch and does not rely on the built-in HTTP client in Node.js. It includes a number of features that make it a good choice for high-performance applications.
This seems like a pretty good reason to fork to me.
> Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python,
Or Javascript (well node), or golang (http/net is _worse_ than urllib IMO), Rust , Java (UrlRequest is the same as python's), even dotnet's HttpClient is... fine.
Honestly the thing that consistently surprises me is that requests hasn't been standardised and brought into the standard library
HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_1_1)
.followRedirects(Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(20))
.proxy(ProxySelector.of(
new InetSocketAddress("proxy.example.com", 80)
))
.authenticator(Authenticator.getDefault())
.build();
HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());
> What's the matter with this?
To me what makes this very "Java" is the arguments being passed, and all the OOP stuff that isn't providing any benefit and isn't really modeling real-world-ish objects (which IMHO is where OOP shines). .version(Version.HTTP_1_1) and .followRedirects(Redirect.NORMAL) I can sort of accept, but it requires knowing what class and value to pass, which is lookups/documentation reference. These are spread out over a bunch of classes. But we start getting so "Java" with the next ones. .connectTimeout(Duration.ofSeconds(20)) (why can't I just pass 20 or 20_000 or something? Do we really need another class and method here?) .proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80))), geez that's complex. .authenticator(Authenticator.getDefault()), why not just pass bearer token or something? Now I have to look up this Authenticator class, initialize it, figure out where it's getting the credentials, how it's inserting them, how I put the credentials in the right place, etc. The important details are hidden/obscured behind needless abstraction layers IMHO.
I think Java is a good language, but most modern Java patterns can get ludicrous with the abstractions. When I was writing lots of Java, I was constantly setting up an ncat listener to hit so I could see what it's actually writing, and then have to hunt down where a certain thing is being done and figuring out the right way to get it to behave correctly. Contrast with a typical Typescript HTTP request and you can mostly tell just from reading the snippet what the actual HTTP request is going to look like.
Unless you use a text editor without any coding capabilities, your IDE should show you which values you can pass. The alternative is to have more methods, I guess?
> why can't I just pass 20 or 20_000 or something
20 what? Milliseconds? Seconds? Minutes? While I wouldn't write the full Duration.ofSeconds(20) (you can save the "Duration."), I don't understand how one could prefer a version that makes you guess the unit.
> proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80))), geez that's complex
Yes it is, can't add anything here. There's a tradeoff between "do the simple thing" and "make all things possible", and Java chooses the second here.
> .authenticator(Authenticator.getDefault()), why not just pass bearer token or something?
Because this Authenticator is meant for prompting a user interactively. I concur that this is very confusing, but if you want a Bearer token, just set the header.
> Unless you use a text editor without any coding capabilities, your IDE should show you which values you can pass. The alternative is to have more methods, I guess?
Fair enough, as much as I don't like it, in Java world it's safe to assume everyone is using an IDE. And when your language is (essentially) dependent on an IDE, this becomes a non-issue (actually I might argue it's even a nice feature since it's very type safe).
> 20 what? Milliseconds? Seconds? Minutes? While I wouldn't write the full Duration.ofSeconds(20) (you can save the "Duration."), I don't understand how one could prefer a version that makes you guess the unit.
I would assume milliseconds and would probably have it in the method name, like timeoutMs(...) or something. I will say it's very readable, but if I was writing it I'd find it annoying. But optimizing for readability is a reasonable decision, especially since 80% of coding is reading rather than writing (on average).
If you've ever dealt with time, you'll be grateful it's a duration and not some random int.
using HttpClient client = new();
HttpResponseMessage response = await client.GetAsync("https://...");
if (response.StatusCode is HttpStatusCode.OK)
{
string s = await response.Content.ReadAsStringAsync();
// ...
}"Common IHttpClientFactory usage issues"
https://learn.microsoft.com/en-us/dotnet/core/extensions/htt...
"Guidelines for using HttpClient"
https://learn.microsoft.com/en-us/dotnet/fundamentals/networ...
And this doesn't account for all gotchas as per .NET version, than only us old timers remember to cross check.
I feel like you answered yourself. Java makes you do this by not supporting proper keyword arguments.
HttpClient client = HttpClient.newBuilder()
.followRedirects(Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(20))
.build();It does enable code that "hides" implementation very well, like the quoted examples authentication API lets you authenticate in any way you can imagine, as in literally any way imaginable.
Its incredibly flexible. Want to only be able to send the request out after you've touched a file, send of a Message through a message broker and then maybe flex by waiting for the response of that async communication and use that as a custom attribute in the payload, additionally to a dynamically negotiated header to be set according to the response of a DNS query? yeah, we can do that! and the caller doesnt have to know any of that... at least as long as it works as intended
Same with the Proxy layer, the client is _entirely_ extensible, it is what Inversion of Control enables.
It just comes with the unfortunate side-effect of forcing the dev to be extremely fluent in enterprisey patterns. I dont mind it anymore, myself. the other day ive even implemented a custom "dependency injection" inspired system for data in a very dynamic application at my dayjob. I did that so the caller wont even need to know what data he needs! it just get automatically resolved through the abstraction. But i strongly suspect if a jr develeoper which hasnt gotten used to the java ecosystem will come across it, he'll be completely out of his depth how the grander system works - even though a dev thats used to it will likely understand the system within a couple of moments.
Like everything in software, everything has advantages and disadvantages. And Java has just historically always tried to "hide complexity", which in practice however paradoxically multiplies complexity _if youre not already used to the pattern used_.
Yeah, I remember the first time I encountered a spring project (well before boot was out) and just about lost my shit with how much magic was happening.
It is productive once you know a whole lot about it though, and I already had to make that investment so might as well reap the rewards.
The stdlib may not be the best, but the fact all HTTP libs that matter are compatible with net/http is great for DX and the ecosystem at large.
It is not built for convenience. It has no methods for simply posting JSON, or marshaling a JSON response from a body automatically, no "fluent" interface, no automatic method for dealing with querystring parameters in a URL, no direct integration with any particular authentication/authorization scheme (other than Basic Authentication, which is part of the protocol). It only accepts streams for request bodys and only yields streams for response bodies, and while this is absolutely correct for a low-level library and any "request" library that mandates strings with no ability to stream in either direction is objectively wrong, it is a rather nice feature to have available when you know the request or response is going to be small. And so on and so on.
There's a lot of libraries you can grab that will fix this, if you care, everything from clones of the request library, to libraries designed explicitly to handle scraping cases, and so on. And that is in some sense also exactly why the net/http client is designed the way it is. It's designed to be in the standard library, where it can be indefinitely supported because it just reflects the protocol as directly as possible, and whatever whims of fate or fashion roll through the developer community as to the best way to make web requests may be now or in the future, those things can build on the solid foundation of net/http's Request and Response values.
Python is in fact a pretty good demonstration of the risks of trying to go too "high level" in such a client in the standard library.
Instead, official documentation seems comfortable with recommending a third party package: https://docs.python.org/3/library/urllib.request.html#module...
>The Requests package is recommended for a higher-level HTTP client interface.
Which was fine when requests were the de-facto-standard only player in town, but at some point modern problems (async, http2) required modern solutions (httpx) and thus ecosystem fragmentation began.
The h11, h2, httpcore stack is probably the closest thing to what the Python stdlib should look like to end the fragmentation but it would be a huge undertaking for the core devs.
More importantly, it would be massively breaking to remove the existing functionality (and everyone would ignore a deprecation), and confusing not to (much like it was when 2.x had both "urllib" and "urllib2").
It'd be nice to have something high level in the standard library based on urllib primitives. Offering competition to those, not so much.
Yes, and it's in the standard library (System namespace). Being Microsoft they've if anything over-featured it.
And while this article [1] says "It's been around for a while", it was only added in .NET Framework 4.5, which shows it took a while for the API to stabilise. There were other ways to make web requests before that of course, and also part of the standard library, and it's never been "difficult" to do so, but there is a history prior to HttpClient of changing ways to do requests.
For modern dotnet however it's all pretty much a solved problem, and there's only ever been HttpClient and a fairly consistent story of how to use it.
[1] https://learn.microsoft.com/en-us/dotnet/core/extensions/htt...
is 14 years not a while?
.NET’s WebRequest was available in .NET Framework 1.1 in 2003 [1].
But since then, Microsoft noticed the issues with WebRequest and came up with HttpClient in 2012. It has some issues and footguns, like those related to HttpClient lifetime, but it’s a solid library. On the other hand, the requests library for Python started in 2011 [2], but the stdlib library hasn’t seen many improvements.
[0] https://github.com/python/cpython/blob/6d7e47b8ea1b8cf82927d...
[1] https://learn.microsoft.com/en-us/dotnet/api/system.net.webr...
[2] https://github.com/psf/requests/blob/main/HISTORY.md#001-201...
it's called the STD lib for a reason...
I've often ended up reimplementing what I need because the API from the famous libraries aren't efficient. In general I'd love to send a million of requests all in the same packet and get the replies. No need to wait for the first reply to send the 2nd request and so on. They can all be on the same TCP packet but I have never met a library that lets me do that.
So for example while http3 should be more efficient and faster, since no library I've tried let me do this, I ended up using HTTP1.1 as usual and being faster as a result.
Python makes everything so easy.
I realized this the other day, and dub it Bram's Law -- Bram
Bram's Law
The easier a piece of software is to write, the worse it's implemented in practice. Why? Easy software projects can be done by almost any random person, so they are. It's possible to try to nudge your way into being the standard for an easy thing based on technical merit, but that's rather like trying to become a hollywood star based on talent and hard work. You're much better off trading it all in for a good dose of luck.
This is why HTTP is a mess while transaction engines are rock solid. Almost any programmer can do a mediocre but workable job of extending HTTP, (and boy, have they,) but most people can't write a transaction engine which even functions. The result is that very few transaction engines are written, almost all of them by very good programmers, and the few which aren't up to par tend to be really bad and hardly get used. HTTP, on the other hand, has all kinds of random people hacking on it, as a result of which Python has a 'fully http 1.1 compliant http library which raises assertion failures during normal operation.
Remember this next time you're cursing some ubiquitous but awful third party library and thinking of writing a replacement. With enough coal, even a large diamond is unlikely to be the first thing picked up. Save your efforts for more difficult problems where you can make a difference. The simple problems will continue to be dealt with incompetently. It sucks, but we'll waste a lot less time if we learn to accept this fact.
I've noticed that many languages struggle with HTTP in the standard library, even if the rest of the stdlib is great. I think it's just difficult to strike the right balance between "easy to use" and "covers every use case", with most erring (justifiably) toward the latter.
I still hear people complain about how such and such removal between "minor versions" of Python 3 (you really should be thinking of them as major versions nowadays — "Python 3 is the brand", the saying goes now), where they were warned like two years in advance about individual functions, supposedly caused a huge problem for them. It's hard for me to reconcile with the rhetoric I've heard in internal discussions; they're so worried in general about possible theoretical compatibility breaks that it seems impossible to change anything.
The notable exception is Go, which has a fantastic one. But Go is pretty notable for having an incredible standard library in general.
1. Whether or not it was "made for" webdev, people do use Rust for that.
2. Plenty of people write networked Rust that interacts with HTTP. That code requires an HTTP stack, even if it isn't web development.
3. Like all of the other examples, Rust does have an excellent third-party HTTP stack (reqwest and its underpinnings). So it's not like Rust fails to do HTTP.
(As an outsider I had the impression that Go's net/http was good, but a lot of people in this thread are complaining about it as well. So it may be 0-4 instead of 1-3).
Node.js got its production version in 2023.
Rust doesn't include an HTTP client at all.
Even for stdlib that have a client, virtually none support HTTP/3, which is used for 30% of web traffic. [1]
--
HTTP (particularly 2+) is a complex protocol, with no single correct answers for high-level and low-level needs.