News for hackers. Stuff that matters.
Re:BlurHash: Algorithm to generate compact representation of... by css on Thursday February 20, 2020 @09:10AM (#22375139)
Judging by the example string on the website, `LEHV6nWB2yk8pyoJadR*.7kCMdnj`[0], you cannot use these hash strings as filenames because they can have disallowed characters.

You could save a step by just sending which image to download and use the filename as the hash you use to render the result, but this algorithm requires you store the relationship between the hash and the image somewhere (unless I am missing something obvious).

[0] source code: https://github.com/woltapp/blurhash-python/blob/7469c813ea64...

Re:BlurHash: Algorithm to generate compact representation of... by lvh on Thursday February 20, 2020 @09:30AM (#22375287)
1. Using the hashes as principal indices seems unsafe in general (which includes image upload) anyway: it looks pretty simple to generate collisions. The suggested use case (save the hash in the database next to the original image or a reference to it) sounds fine.

2. Which disallowed characters? The dictionary appears to be "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz#$%*+,-.:;=?@[]^_{|}~". No slashes or null bytes, which are the common disallowed characters on server platforms. It does have characters your shell cares about, but that's only a problem if you're not quoting.

Re:BlurHash: Algorithm to generate compact representation of... by css on Thursday February 20, 2020 @09:38AM (#22375340)
`:` is disallowed on MacOS and several of those characters are disallowed on Windows. Generally I view images on a frontend, not on my servers.
Re:BlurHash: Algorithm to generate compact representation of... by lvh on Thursday February 20, 2020 @09:41AM (#22375359)
Fair! I would not expect the image be saved to a Windows path when displaying a blurred image in the example use cases (mobile apps and web pages). You're thinking Electron, perhaps?

Point of order though: I don't think that's right for MacOS, though you're certainly right for Windows. HFS+ lets you use any Unicode symbol including NUL (because the filenames are encoded Pascal-style). I don't know the details how how APFS does it, but it also appears to support any Unicode code point, though it additionally mandates UTF-8. [0]

[0] https://developer.apple.com/library/archive/documentation/Fi... grep "filenames"

Re:BlurHash: Algorithm to generate compact representation of... by css on Thursday February 20, 2020 @09:54AM (#22375438)
Interesting, I was not aware about MacOS. If I run `touch :.:` it will run fine and create the file, but Finder disallows this for whatever reason:

    The name “:.:” can’t be used.
    Try using a name with fewer characters, or with no punctuation marks.
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @10:19AM (#22375631)
Finder exchanges : and / in filenames, for compatibility with classic MacOS, which used : as a directory separator and allowed / in filenames. You can create a file named / in Finder, and it will show up as : on the file system.
Re:BlurHash: Algorithm to generate compact representation of... by _kst_ on Thursday February 20, 2020 @01:41PM (#22377434)
A file name with an embedded NUL character couldn't be passed to fopen(), or to any other function that takes a pointer to a C-style string. (Not directly relevant here.)
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @10:18AM (#22375624)
: was disallowed on classic MacOS. On modern macOS, it is allowed in the POSIX layer just fine. However, Finder and other parts of the UI will translate a : in the file name to a / on display, and vice versa.

This is because classic MacOS used : as a directory separator instead of /, and this behaviour is preserved in the UI, but obviously not in the underlying layers.

Re:BlurHash: Algorithm to generate compact representation of... by Wowfunhappy on Thursday February 20, 2020 @12:08PM (#22376538)
Is it not present in the underlying layer anymore? Applescript displays paths with : separators unless you specifically tell it you want a “POSIX path”. (This is actually really annoying.)
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @12:12PM (#22376581)
No, the underlying layer is POSIXy, and uses /. The highest UI layers do the mapping between : and /, and apparently also AppleScript.
Re:BlurHash: Algorithm to generate compact representation of... by cosmojg on Thursday February 20, 2020 @01:21PM (#22377233)
Simple solution: run a POSIX-compliant OS on the frontend. All Linux distributions, all BSD distributions, modern macOS, and Windows Subsystem for Linux all support ':' in filenames.
Re:BlurHash: Algorithm to generate compact representation of... by joosters on Thursday February 20, 2020 @09:42AM (#22375367)
':' works fine in MacOS file names for me.
Re:BlurHash: Algorithm to generate compact representation of... by tyingq on Thursday February 20, 2020 @10:15AM (#22375597)
They mention it's a custom version of base83: https://github.com/woltapp/blurhash/blob/master/Algorithm.md...

With this charset: 0123456789ABCDEFGHIJKLMNOPQ RSTUVWXYZabcdefghijklmnopqr stuvwxyz#$%*+,-.:;=@[]^_{|}~.

I imagine you could tweak that to replace problematic chars with safe substitutes.

Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @10:39AM (#22375781)
These are the safest I could find. They are perfectly fine for using in filenames.
Re:BlurHash: Algorithm to generate compact representation of... by svnpenn on Thursday February 20, 2020 @01:01PM (#22377029)
Windows "A file name can't contain any of the following characters":

    \ / : * ? " < > |

which means "*:|" would need to be removed if you want to support that use case
Re:BlurHash: Algorithm to generate compact representation of... by ygra on Thursday February 20, 2020 @12:00PM (#22376475)
That depends very much on the platform ...
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @12:13PM (#22376593)
Well, Windows is very picky, yes.
Re:BlurHash: Algorithm to generate compact representation of... by mlyle on Thursday February 20, 2020 @12:38PM (#22376821)
> you cannot use these hash strings as filenames because they can have disallowed characters.

You can't really use them as filenames because they won't be collision free, too. They're not meant to be an identifier, but instead a compact representation of the image that can be stored in a database.

Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @09:37AM (#22375338)
This is a tempting idea, but as was already pointed out, the problem is not the character set, but that similar images will encode to the same string.

Anyway, chances are you are already storing a filename in your database, so you would need a second field for the blurhash string.

Re:BlurHash: Algorithm to generate compact representation of... by mzs on Thursday February 20, 2020 @02:05PM (#22377633)

  assets/example.png/LEHV6nWB2yk8pyoJadR*.7kCMdnj
  assets/example.jpg/LEHV6nWB2yk8pyoJadR*.7kCMdnj
Re:BlurHash: Algorithm to generate compact representation of... by presumably on Thursday February 20, 2020 @04:43PM (#22378934)
s/g\//g#/

    assets/example.png#LEHV6nWB2yk8pyoJadR*.7kCMdnj
    assets/example.jpg#LEHV6nWB2yk8pyoJadR*.7kCMdnj
No need to send the extra data to the server for each request.
Re:BlurHash: Algorithm to generate compact representation of... by PabloSichert on Thursday February 20, 2020 @10:27AM (#22375686)
Hm, that's an interesting choice. I also wonder why they didn't use an encoding that has an alphabet with 2^n characters (that way you can directly map 1 charater to n bits when decoding).

Coincidentally, I've just finished some work on a project[1] that is in the same space (identifiers for images).

For the reasons you pointed out, I found Douglas Crockford's base32[2] encoding to be a good fit.

[1] https://github.com/pablosichert/ciid

[2] https://www.crockford.com/base32.html

Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @10:29AM (#22375702)
That choice is explained in the README: https://github.com/woltapp/blurhash

(83 is about as many safe characters as you can reasonably find, and it allows some nice ways of packing values together.)

Re:BlurHash: Algorithm to generate compact representation of... by PabloSichert on Thursday February 20, 2020 @10:38AM (#22375770)
Could you explain what "AC components" refers to? I couldn't figure that out just by reading your README.

I wonder how the efficiency compares to just encoding on the bit level.

Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @10:45AM (#22375826)
It is a term often used for DCT-transformed data. DCT, in this case, breaks the image down into basically an average colour of the whole image, referred to as the DC component, and a bunch of waves that make up the detail of the image, referred to as AC components.

https://github.com/woltapp/blurhash/blob/master/Algorithm.md

Re:BlurHash: Algorithm to generate compact representation of... by sampo on Thursday February 20, 2020 @11:05AM (#22376010)
> Could you explain what "AC components" refers to?

All but the first component of the Fourier transform. (The first component is the average of the data.) The term comes from electrical engineering, but Fourier transform has lots of applications also outside of electrical engineering.

Re:BlurHash: Algorithm to generate compact representation of... by bausano_michael on Thursday February 20, 2020 @09:24AM (#22375244)
I quite like that idea. Maybe you could run base64/base32 on the blurhash. That brings two potential issues I could think of. Is the size of the result less than filesystem max file name length? And what about collisions where two slightly different images have the same blurhash?
Re:BlurHash: Algorithm to generate compact representation of... by lvh on Thursday February 20, 2020 @09:33AM (#22375307)
As I mentioned upthread: none of those are disallowed on common server platforms, but ironically canonical Base64 _will actually_ break, and it'll break rarely enough that you're likely to miss it in testing! (The last character in canonical Base64, for symbol 63b10, is /, which is disallowed on (*nix) filesystems and does not occur in the native BlurHash dictionary. Unless you're suggesting encoding the encoded format, I guess.)
Re:BlurHash: Algorithm to generate compact representation of... by ReverseCold on Thursday February 20, 2020 @04:41PM (#22378918)
I accidentally made a folder (in my pinephone's / folder) the other day called '', and I was about to delete it with rm -rf .

Maybe that shouldn't be allowed, but it is :P

Re:BlurHash: Algorithm to generate compact representation of... by bootloop on Thursday February 20, 2020 @09:27AM (#22375257)
If you want to do that just use a different encoding which outputs a string safe to be used for filenames.
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @09:35AM (#22375325)
This one is picked very carefully to be as safe as reasonably possible with this number of characters. As was pointed out elsewhere, it is actually safe for filenames.