Re:How Discord Stores Billions of Messages (2017)
by latchkey on Tuesday August 24, 2021 @02:39PM
(#28293062)
Unique images or just copied from elsewhere?
[ Parent
| on HN ]
Re:How Discord Stores Billions of Messages (2017)
by objektif on Tuesday August 24, 2021 @06:38PM
(#28295862)
How does that matter? Do they keep track of all images on the internet?
Re:How Discord Stores Billions of Messages (2017)
by jjice on Tuesday August 24, 2021 @07:26PM
(#28296267)
I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image.
I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerprint images. I have a utility on my machine (czkawka maybe?) that does really good image de-duplication with what seemed like a common algorithm (based on a quick look at the source).
No idea though, just spit balling.
Re:How Discord Stores Billions of Messages (2017)
by vortico on Tuesday August 24, 2021 @08:32PM
(#28296741)
I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.
Re:How Discord Stores Billions of Messages (2017)
by Ekaros on Wednesday August 25, 2021 @07:28AM
(#28300220)
I think it might, spamming same meme images over and over is quite common in some servers. On other hand the bigger pictures might overhelm these just in size.
Re:How Discord Stores Billions of Messages (2017)
by vortico on Wednesday August 25, 2021 @05:38PM
(#28307412)
Yeah, that's why I assumed it wouldn't help that much. People re-upload 100kB memes all the time, but the bulk would probably be 5MB phone pictures that won't typically be re-uploaded.
Re:How Discord Stores Billions of Messages (2017)
by kroltan on Tuesday August 24, 2021 @10:25PM
(#28297414)
The plural of anecdote isn't data, but about 20% of the images I post on Discord come from Discord in the first place, cross-posting among different servers.
Re:How Discord Stores Billions of Messages (2017)
by PeterCorless on Tuesday August 24, 2021 @07:55PM
(#28296504)
Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."
Re:How Discord Stores Billions of Messages (2017)
by objektif on Tuesday August 24, 2021 @08:08PM
(#28296599)
Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".
Re:How Discord Stores Billions of Messages (2017)
by PeterCorless on Tuesday August 24, 2021 @11:46PM
(#28297841)
Oof! No, you can't. :D