News for hackers. Stuff that matters.
Re:How to store your app's entire state in the url by dyno12345 on Monday January 09, 2023 @02:37PM (#34314915)
you could come up with a better compression scheme given that the dict keys and many of the values are probably common across everyone even if they don't repeat multiple times for a given user. a lot of flexibility would be lost though, and you would need to always update the scheme in a backwards-compatible way.
Re:How to store your app's entire state in the url by hot_gril on Monday January 09, 2023 @02:41PM (#34314957)
Could start with a smaller original payload. Use single-length keys, or better yet use protocol buffers or even ASN1.
Re:How to store your app's entire state in the url by waynesonfire on Monday January 09, 2023 @02:43PM (#34314969)
would it be feasible to include the compression dictionary along w/ the data? And, maybe just send it once on the initial request and store it on the client?
Re:How to store your app's entire state in the url by gnramires on Monday January 09, 2023 @07:05PM (#34318385)
> would it be feasible to include the compression dictionary along w/ the data

In theoretical terms (in particular Kolmogorov-complexity perspective), I believe the inclusion of a dictionary wouldn't help, it would strictly increase size. It can help (specially from a Shannon information perspective) if the compression is too computationally demanding to build the data model (which is kind of what all compressors do); but it almost certainly doesn't help in the case of something as small as an URL. But then it's possible to just pack a common compressor with the website (and just refer to it with a version code).

I think the web in general would benefit greatly from some kind of standard compression libraries. Compression tends to have a significant cold start cost to work well, if that were mitigated certainly something like an url could be far shorter.

I believe HTTP2 has header compression (RFC 7541)[1] with 'static tables', which seems to be a form of dictionary, but surprisingly to me no html compression with such shared dictionaries. There is caching libraries (unfortunately cross-domain caching seems to be deprecated[2] due to security concerns) I believe which helps in other ways, but I think a true dictionary-style compressor would bring huge bandwidth savings.

Major web entities should get together to develop those standard dictionaries (and which algorithms to use them with), so they are reasonably fair for everyone (maybe even language specific dictionaries should be used).

Security seems indeed a concern, but I think the key would be carefully preventing that a content were decoded with the incorrect dictionary (meaning the server could serve a different web page than the standard decompression) -- but overall it doesn't seem like a big issue (similar to cache, less problematic than cross-domain caching).

In the future maybe even one of those fancy neural network (or otherwise machine learning inspired) methods could be an option, specially useful for highly bandwidth constrained like rural or satellite internet (although of course performance is always a priority with mobile devices being major users).

[1] https://httpwg.org/specs/rfc7541.html#static.table.definitio...

[2] https://www.stefanjudis.com/notes/say-goodbye-to-resource-ca... Is this still accurate?

Re:How to store your app's entire state in the url by gnramires on Tuesday January 10, 2023 @08:11AM (#34324294)
> would benefit greatly from some kind of standard compression libraries

Correction: standard compression dictionaries

Re:How to store your app's entire state in the url by WorldMaker on Tuesday January 10, 2023 @04:02PM (#34331445)
> I believe HTTP2 has header compression (RFC 7541)[1] with 'static tables', which seems to be a form of dictionary, but surprisingly to me no html compression with such shared dictionaries.

To my understanding, that static table in HTTP2 is directly in context of recommending and discussing Brotli compression, which does use a table like that as a standard static dictionary in HTTP2+ scenarios including other static dictionary inclusions derived from a corpus of HTML documents.

https://en.wikipedia.org/wiki/Brotli

Re:How to store your app's entire state in the url by gnramires on Wednesday January 11, 2023 @09:00AM (#34338506)
Thanks, I didn't know Brotli! Indeed it seems to include a dictionary in its compression which seems to help significantly with small pages. I hope compression continues to improve in this way.

> To my understanding, that static table in HTTP2 is directly in context of recommending and discussing Brotli compression

It seems separate, that would be a header compression for HTTP2 header itself, while Brotli encodes content (usually html, css or js).

Re:How to store your app's entire state in the url by robryk on Wednesday January 11, 2023 @09:15AM (#34338691)
The header compression in http2 has atypical requirements places upon it: contents of different headers in a single request must not impact each other's compressed size[^], lest an attacker able to manipulate one of them can use that to guess the other. Thus, LZ77-style compression is out of the window, as well as using Huffman codes selected based on _all_ the header values. In essence, each header value (in a single request) has to be compressed independently. IIUC the dictionary you reference is used for header keys (as opposed to values).

[^] this statement is a bit of a lie for reasons of simplification

Re:How to store your app's entire state in the url by mikepurvis on Monday January 09, 2023 @02:55PM (#34315126)
Once you've already given up on human-readability, it probably doesn't much matter how long/gross the result is. Rather than spending effort and adding complexity, just provide a built in URL shortener.
Re:How to store your app's entire state in the url by hot_gril on Monday January 09, 2023 @03:13PM (#34315327)
I agree, same reason people use json in the first place despite the inefficiency, it's easier.
Re:How to store your app's entire state in the url by wtetzner on Monday January 09, 2023 @07:51PM (#34318794)
Well, there are limits to the lengths of URLs in much of the internet’s infrastructure, so better compression could still be useful.
Re:How to store your app's entire state in the url by jsvaughan on Tuesday January 10, 2023 @05:35AM (#34322836)
This is the critical point to make. Paste a few long texts into the boxes and the url goes into the hundreds of thousands of characters long, whereas this says that anything more than about 2000 chars puts you into uncertain territory:

https://stackoverflow.com/questions/417142/what-is-the-maxim...

Re:How to store your app's entire state in the url by p1necone on Monday January 09, 2023 @08:57PM (#34319343)
Isn't storing state in urls and then using a url shortener just a key/value store with extra steps?
Re:How to store your app's entire state in the url by taneq on Tuesday January 10, 2023 @02:36AM (#34321671)
But with added session id energy too!
Re:How to store your app's entire state in the url by mikepurvis on Tuesday January 10, 2023 @11:10AM (#34327035)
True, though I think the differences around shareability matter— a mapping-table entry is immutable, whereas even a non-logged-in session token is probably not.