News for hackers. Stuff that matters.
Re:The State of HTTP in 2022 by 1vuio0pswjnm7 on Saturday December 31, 2022 @10:24PM (#34203269)
Commenters might try to challenge the parent comment but the article's disclosures are evidence of the comment's accuracy. For example, this alleged "head-of-line blocking problem" that HTTP/2 purportedly "solves" was never a problem of HTTP outside of a specific program, the graphical web browser. The type of client that tries to pull resources from different domains for a single website. Not all programs that use HTTP need to do that.

For instance I have been using HTTP/1.1 pipelining outside the browser for close to 20 years. It has always been supported by HTTP servers and it works fast and reliably with the simple clients I have use. I still rely on HTTP/1.1 pipelining today, on a daily basis.

Thus, this "problem" that is often cited when attempting to justify the industry-sponsored "improvement" of HTTP is confined to a specific program, an "approved browser", a bloated, complex, beloved-by-developers client that is optimised for tracking and advertising, of course with JS and cookies enabled not to mention a host of other nuisances. To be clear, I am not suggesting HTTP/2 is not useful. Of course it is. I am suggesting that the HOL "problem" that HTTP/2 is cited as "solving" was never a problem outside of a certain program that happens to use HTTP: the "modern" web browser. From a user perspective, "server push" was a ridiculous idea. Glad to see it gone frome Chrome. Like "solving HOL blocking" it seems to be targeted at the "modern" web browser that serves ads. Another example may be "Early Hints". I once saw theregister.com using it and it caused problems with the proxy I use. What uses does it have other to support "modern" web browsers and online advertising. Why shold users want these "features".

HTTP/1.1 and its precursors were generally developed by academics. HTTP/2 was generally developed by corporations that profit from eyeballs, surveillance and online advertising.

When I use the web recreationally, such as reading the text of pages on sites submitted to HN, I only use HTTP to request the page, nothing else. That is, one HTTP request per site, to one domain. The DNS data for the domain has been retreived and stored in advance in the memory of a localhost-bound TLS forward proxy^1 by a different program specifically designed for retrieving DNS data in bulk. By design, any DNS lookups the browser tries to make will fail. As it happens, this configuration is generally enough to read every site submitted to HN. In fact, I cannot think of a single exception.

Normally I use a text-only browser that connects to the proxy but occasionally I test sites in a "modern" browser through the proxy. Surprisingly many, perhaps most, sites I have tested look just fine under this configuration. Making numerous requests to other domains is unnecessary in order for me to read the text on the page.

What "tech" companies and their service providers want and what www user want are not always the same things. The "new" HTTP gives the former what they want. The later are an afterhtought.

1. If one has smaller/simpler/older software, whether it is Netcat or an old version of Internet Explorer or something else, that does not keep up with the wonderful world of TLS, the proxy, which is up-to-date with the all the latest developments, can take care of that. That said, as many HN readers know, Cloudflare has a heuristic for detecting "preferred" browsers and sells this as a "protection service". IMO, it ruins the simplicity of HTTP when www users are forced to use a 135+MB program, financed by online advertising, to send HTTP requests.

Re:The State of HTTP in 2022 by mnot on Saturday December 31, 2022 @10:42PM (#34203364)
If early hints breaks your proxy, it’s likely your proxy doesn’t handle 1xx status codes correctly. Could you tell me which proxy it is (privately if you think it necessary)? I’d like to chase the bug with them.
Re:The State of HTTP in 2022 by raggi on Sunday January 01, 2023 @02:34AM (#34204411)
The big problem with pipelining in http 1.x is that a response can break the pipeline part way through and there is no way to know what the server processed. A response night for example, mid pipeline be connection:close and that’s that, did any subsequent request get processed? Who knows.
Re:The State of HTTP in 2022 by 1vuio0pswjnm7 on Sunday January 01, 2023 @06:01AM (#34205380)
HTTP/2 seems to be designed for (graphical, interactive) webpages.

The maintainer of a popular webserver has suggested HTTP/2 is slower than HTTP/1.1 for file download.

https://stackoverflow.com/questions/44019565/http-2-file-dow...

As I stated, I use HTTP/1.1 pipelining every day. I use it for a variety of information retrieval tasks, even retrieving bulk DNS data. To give an arbitrary example, sometimes I will download a website's sitemaps. This usually involves downloading a cascade of XML files. For example, there might be a main XML file called "index.xml". This file then lists hundreds more sitemap XML files, e.g., archive-2002-1.xml, archive-2002-2.xml, containing every content URL on the website beginning with some prior year all the way up to the present day. Using a real world example, index.xml contains 246 URLs. Using HTTP/1.1 pipelining I can retrieve all of them into a single file using a single TCP connection. Then I retrieve batches of the URLs contained in that file, again over a single TCP connection. Many websites allow thousands of HTTP requests HTTP/1.1-pipelined over a TCP single connection, but I usually keep the batch size at around 500-1000 max. Of course I want the responses in the same order as the requests.

The process looks something like this

    ftp -4o 1 https://[domainname]/sitemaps/index.xml
    yy030 < 1|(ka;nc0) > 2
    yy030 < 2|wc -l

    1337855
1337855 is the number of URLs for [domainname]. Content URLs, not Javascript, CSS or other garbage.

yy030 is a C program that filters URLs from standard input

ka is a shell alias that sets an environment variable that is read by the yy025 program to indicate an HTTP header, in this case the "Connection:" header set to "keep-alive" not "close" (ka- sets it back to close)

nc0 is a one line shell script

    yy025|nc -vv h1b 80|yy045
yy025 is a C program that accepts URLs, e.g., dozens to hundreds to thousands of URLs, on stdin and outputs customised HTTP

h1b is a HOSTS file entry containg the address of a localhost-bound forward TLS proxy

yy045 is a C program that removes chunked transfer encoding from standard input

To verify the download, I can look at the HTTP headers in file "2". I can also look at the log from the TLS proxy. I have it set configured to log all HTTP requests and responses.

Is this a job for HTTP/2. It does not seem like it.

This type of pipelining using only a single TCP connection is not possible using curl or libcurl. Nor is it possible using nghttp. Look around the web and one will see people opening up dozens, maybe hundreds of TCP connections and running jobs in parallel, trying to improve speed, and often getting banned. As with the comment from the Jetty maintainer, I suspect using HTTP/2 would actually be slower for this type of transfer. It is overkill.

IMHO, HTTP, i.e., in the general sense, is not just for requesting webpages and resources for webpages.

I find HTTP/1.1 to be very useful. It is certainly not just for requesting webpages full of JS, CSS, images and the like. That is only one way I might use it. Perhaps HTTP/2 is the better choice for webpages. TBH, if using a "modern" graphical browser, I would be inclined to let it use HTTP/2. Most of the time I am not using a graphical browser.

Re:The State of HTTP in 2022 by pfoof on Sunday January 01, 2023 @07:03AM (#34205661)
This comment deserves a post on it's own, so you can explain the naming scheme behind it
Re:The State of HTTP in 2022 by asdfghjhgfderty on Sunday January 01, 2023 @10:18AM (#34206927)
they are simply constructing GET request headers "by hand " based on some xml file downloaded earlier an then sendind that list of GET via `nc`. the example is just over confusing and using file named as 1, 2

I voted up because that is indeed neat tho.

Re:The State of HTTP in 2022 by 1vuio0pswjnm7 on Sunday January 01, 2023 @05:03PM (#34211494)
One of the many programmer memes is something along the lines of "naming is difficult." Yet programmers, individuals who are often obsessed with numbers, insist on trying to do it anyway. The results speak for themselves. This extends beyond programs. The so-called "tech" industry has produced some of the most absurd, non-descriptive business names in the history of the world.

I decided to try numbering the programs I write instead of naming them. I often use a prefix that can provide a hint.^1 For example, the yy prefix indicates it was created with flex and the nc in nc0 indicates it is a "wrapper script" for nc. If the program is one I use frequently, then I have no trouble remembering its number. In the event I forget a program number, I have a small text file that lists each yy program along with a short description of less than 35 chars.

1. But not always. I have some scripts that I use daily that are just a number. I also have a series of scripts that begin with "[", where the script [000 outputs a descriptive list of the scripts, [001, [002, etc. I am constantly experimenting, looking for easier, more pleasing short strings to type.

Each source file for a yy program is just a single .l file with a 3-char filename like 025.l, so searching through source code can be as simple as

     grep whatever dir/???.l
If I put descriptions in C comments at top of each .l file I can do something like

     head -5 dir/???.l 
Aesthetically, I like have a directory full of files with filenames that follow a consistent pattern and are of equal length. Look at the source code for k, ngn-k or kerf. When it comes to programming, IMO, smaller is better.