News for hackers. Stuff that matters.
Re:How Discord Stores Billions of Messages (2017) by latchkey on Tuesday August 24, 2021 @02:39PM (#28293062)
Unique images or just copied from elsewhere?
Re:How Discord Stores Billions of Messages (2017) by objektif on Tuesday August 24, 2021 @06:38PM (#28295862)
How does that matter? Do they keep track of all images on the internet?
Re:How Discord Stores Billions of Messages (2017) by jjice on Tuesday August 24, 2021 @07:26PM (#28296267)
I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image.

I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerprint images. I have a utility on my machine (czkawka maybe?) that does really good image de-duplication with what seemed like a common algorithm (based on a quick look at the source).

No idea though, just spit balling.

Re:How Discord Stores Billions of Messages (2017) by vortico on Tuesday August 24, 2021 @08:32PM (#28296741)
I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.
Re:How Discord Stores Billions of Messages (2017) by Ekaros on Wednesday August 25, 2021 @07:28AM (#28300220)
I think it might, spamming same meme images over and over is quite common in some servers. On other hand the bigger pictures might overhelm these just in size.
Re:How Discord Stores Billions of Messages (2017) by vortico on Wednesday August 25, 2021 @05:38PM (#28307412)
Yeah, that's why I assumed it wouldn't help that much. People re-upload 100kB memes all the time, but the bulk would probably be 5MB phone pictures that won't typically be re-uploaded.
Re:How Discord Stores Billions of Messages (2017) by kroltan on Tuesday August 24, 2021 @10:25PM (#28297414)
The plural of anecdote isn't data, but about 20% of the images I post on Discord come from Discord in the first place, cross-posting among different servers.
Re:How Discord Stores Billions of Messages (2017) by PeterCorless on Tuesday August 24, 2021 @07:55PM (#28296504)
Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."
Re:How Discord Stores Billions of Messages (2017) by objektif on Tuesday August 24, 2021 @08:08PM (#28296599)
Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".
Re:How Discord Stores Billions of Messages (2017) by PeterCorless on Tuesday August 24, 2021 @11:46PM (#28297841)
Oof! No, you can't. :D