News for hackers. Stuff that matters.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @12:29PM (#15697155)
Why don't more companies resize images client-side first using <canvas> and then save the server some work by only asking it to verify the result by

- resizing to the same size

- removing metadata

This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients.

https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende...

# Edit: On Keeping Full Resolution Images

Some people mention having original highest-resolution images are important. I don't think that is true for most applications.

Most apps don't need hi-resolution history as much as current, live engagement so older photos being smaller isn't a big deal. As technology moves on you simply start allowing higher-res uploads. Youtube, facebook, and others have done this fine as the older stuff is replaced with the new/current/now() content.

In fact, even our highest resolution images are still low-quality for the future. Pick a good max size for your site (4k?) and resize everything down to that. In a year, bump it up to 6k, then 10k, etc...

Keeping costs low has it's benefits, especially for us startups. Now if you have massive collateral, then knock yourself out.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by AndrewStephens on Tuesday November 14, 2017 @12:35PM (#15697220)
A few reasons:

1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad.

2) The original image has some metadata that they want to keep (geolocation, etc).

3) They think they can do a better and more consistent job resizing than the various browsers, which is probably true.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by malux85 on Tuesday November 14, 2017 @01:34PM (#15697826)
agree on 3) most browsers just use linear interpolation when resizing images, which makes sense from a performance point of view, but looks terrible. Better to use a bi-linear or cubic resize, more computing up front, but better images, this is probably the reason they do it
Re:How Discord Resizes 150M Images Every Day with Go and C++ by amelius on Tuesday November 14, 2017 @02:55PM (#15698547)
But soon you can do any type of resizing through WASM on the browser.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by XCSme on Tuesday November 14, 2017 @04:03PM (#15699092)
You can already do it, just use a library or implement your own scaling function and don't use the built-in image resize functions.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by searealist on Tuesday November 14, 2017 @11:24PM (#15701441)
bi-linear is linear interpolation
Re:How Discord Resizes 150M Images Every Day with Go and C++ by nice_byte on Wednesday November 15, 2017 @01:44AM (#15701900)
I think they may have meant "nearest-neighbor", which isn't true for any browsers that I know of.

Regardless, there are still better filters than bilinear, i.e. Lanczos, which I'm pretty sure none of the browsers use.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by trumpeta on Tuesday November 14, 2017 @04:39PM (#15699415)
This is my understanding as well, you could kill a browser trying to do single-threaded bilinear or cubic resize on a sufficiently large image.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Poefke on Wednesday November 15, 2017 @07:03AM (#15703160)
If you resize the image in steps, with each resize at least 50% of the previous step, you can do a pretty decent approximation of cubic resize using the canvas. Doing this for a year now, we've gotten no complaints and we have designers as clients :)
Re:How Discord Resizes 150M Images Every Day with Go and C++ by SaltyBackendGuy on Tuesday November 14, 2017 @01:41PM (#15697882)
> 2) The original image has some metadata that they want to keep (geolocation, etc).

Isn't exif data something you should strip out?

Re:How Discord Resizes 150M Images Every Day with Go and C++ by indiandennis on Tuesday November 14, 2017 @01:59PM (#15698063)
Discord makes most of their revenue from selling user data so they probably want to keep as much as possible.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by RussianCow on Tuesday November 14, 2017 @02:02PM (#15698104)
Do you have a source? That's a pretty bold claim, and I haven't seen anything else to back that up.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by hintss on Tuesday November 14, 2017 @04:11PM (#15699165)
pretty sure they've directly stated that they're not selling user data
Re:How Discord Resizes 150M Images Every Day with Go and C++ by gruez on Tuesday November 14, 2017 @04:32PM (#15699346)
to be fair, you could say the same of facebook/google. they're not selling the data (giving it to third parties), but they're making money off it.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Operyl on Tuesday November 14, 2017 @05:47PM (#15699954)
Sure, they serve said data to other authorized users and clients (intended recipients). Name a business that doesn’t do that. ;)
Re:How Discord Resizes 150M Images Every Day with Go and C++ by gruez on Tuesday November 14, 2017 @07:39PM (#15700597)
point is, their statement does not preclude them from using data for marketing purposes. some people are content with that, but others take it as a sign that they are (or will) use the data to build dossiers on users.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Moru on Tuesday November 14, 2017 @03:04PM (#15698629)
Our site would have been happy with full res images from the start. As it is now we are stuck with 80x80 images that needs replacing with higher res images since the originals was not kept in any sorted order.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @03:23PM (#15698758)
So you kept the originals, but didn't organize them into a way you could use them later?
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Moru on Wednesday November 15, 2017 @09:13AM (#15703813)
Long story but from the start we kept originals organized. Then we restructured and the new people couldnt care less and threw away originals or left them named 1,2,3 and so on. All useless now.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Lutin on Tuesday November 14, 2017 @01:21PM (#15697679)
This is for proxying images that users link in chat, not for when users upload images to the service. It doesn't make sense to talk about doing this resize on the client, as the client doesn't have the image.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by TrinaryWorksToo on Tuesday November 14, 2017 @01:50PM (#15697963)
That is a great point. It could still be feasible to cache the image on the client, and have it do the resize. Although I probably wouldn't accept it as a client, especially if my internet cuts out halfway through.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @01:56PM (#15698029)
Yes, the use-case of proxying images is a different mater. I was talking about client uploads since so many companies seem determined to waste my bandwidth and time uploading without resizing first.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by drdaeman on Tuesday November 14, 2017 @02:54PM (#15698541)
Client uploads with client-provided thumbnails could lead to some pranks.

You know, a thumbnail with a cute kitten but something completely different after you click on it ;)

Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @03:24PM (#15698768)
I think you misunderstood. The client is uploading a single data blob for each image, but the blob can be smaller if the client resizes it first.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by b1naryth1ef on Tuesday November 14, 2017 @12:38PM (#15697257)
As mentioned in the post, one of our core product features is preventing your IP from being shared. Given that requirement, images shared in chat have to be proxied through our infrastructure. When doing this we save a lot of money and improve client performance by reducing image sizes.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @12:40PM (#15697275)
So why the image can't first be resized/compressed before being sent through your infrastructure...?
Re:How Discord Resizes 150M Images Every Day with Go and C++ by b1naryth1ef on Tuesday November 14, 2017 @12:43PM (#15697301)
Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by radicalbyte on Tuesday November 14, 2017 @01:27PM (#15697758)
You should seriously consider doing this for your mobile client; the worst thing about Discord is that it eats mobile data if you're uploading lots of images.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by tzahola on Tuesday November 14, 2017 @03:47PM (#15698947)
That’s exactly the reason why the iOS Mail app asks you whether you want to attach an image in small/medium/large/original size.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by ehsankia on Tuesday November 14, 2017 @10:19PM (#15701216)
Data aside, my phone takes pictures at ridiculously high resolutions. Whenever I go to send pictures with discord, it takes a good 30 seconds and half the time it'll just break.

As an aside, I wish the "share" button would share a lower resolution image instead. I don't mind storing the full quality picture, but handling a 10mb image is seriously silly.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by rockostrich on Wednesday November 15, 2017 @10:17AM (#15704314)
The other side of this is if you're on mobile data and they decide to resize the images on the client then people will complain about the app eating up battery life because it's using so much CPU to resize the images. Plus, they won't have the full size image to share with people you may be chatting with on desktop. If you want to not use so much data while uploading images then you should probably resize them yourself or just not upload images unless you're on wifi.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by mark-r on Tuesday November 14, 2017 @01:01PM (#15697487)
There's no reason this couldn't be a two-step process, resizing to something reasonable on the client then fine-tuning it on the server. I'm presuming you don't see the need to start with multi-megapixel images.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by wastedhours on Tuesday November 14, 2017 @01:21PM (#15697691)
> I'm presuming you don't see the need to start with multi-megapixel images.

Might be a fair presumption today, but might not be for the future with hiDPI screens, VR etc... for the relative storage costs, it'd be better to have the original, and then you can programmatically run from there.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by mark-r on Tuesday November 14, 2017 @01:33PM (#15697812)
Today's hiDPI screens can already display more detail than your eye can perceive. The issue wasn't about storage costs, it was about transmission costs which still matter for the foreseeable future.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by dboreham on Tuesday November 14, 2017 @05:07PM (#15699653)
Perhaps, but I think you can compress current-day phone images considerably without losing any actual image fidelity (because the sensor pitch significantly exceeds the lens resolution, and .. noise).
Re:How Discord Resizes 150M Images Every Day with Go and C++ by pmelendez on Tuesday November 14, 2017 @12:40PM (#15697274)
From the notes of your link:

"drawImage() will ignore all EXIF metadata in images, including the Orientation. This behavior is espacially troublesome on iOS devices. You should detect the Orientation yourself and use rotate() to make it right. "

If the origin of the image is the client and you got the client side resize wrong, then you might introduce artifacts when trying to fix it on the server because the data loss. Also if clients are mobile, you might like to optimize the battery of clients instead of computing time on the server.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by Matt3o12_ on Tuesday November 14, 2017 @01:05PM (#15697522)
The question now is, what drains more power, sending the image and resizing it before sending. If the user has a good WiFi or 4G connection, sending the file as it, should be quicker and more energy efficient. With a 2G or 3G connection, uploading a photograph can take significantly longer (1min on poor/average 3G, which means the antenna is working for that duration and draws a lot of battery). Converting it should not take more then a second. Furthermore, I would prefer using less data then using a tiny amount of battery.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @12:43PM (#15697296)
Even several years ago there were libraries on github that accounted for iOS defects. However, that aside, just skip the resize on iOS and send as-is. The server still has to verify the result anyway.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by jlg23 on Tuesday November 14, 2017 @04:48PM (#15699491)
First: Please don't use "Edit" for responding to responses to your comment; it make following threads much, much harder.

On Topic:

> Some people mention having original highest-resolution images are important. I don't think that is true for most applications.

It is true for every application when the next generation of displays hits the market. The question is not the long term usability of our current low-res images but just the migration to the next step. At the moment Acorn announces their new APhone and has a million handsets sold by tomorrow, you want your service to deliver at least viewable images. It's not always the app that sets the bar, sometimes it is the device.

Edit: As someone who travels rather remote places of this planet regularly I'm grateful for every app that does not put the burden on the client. My battery packs only last so long.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by antisthenes on Tuesday November 14, 2017 @02:07PM (#15698134)
The last thing I want is client-side image resizing when my browser is already choking on heavy javascript.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by timdorr on Tuesday November 14, 2017 @12:39PM (#15697260)
These images are often links from the web or posted by a bot, so they're not on a client until after they've been served.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by codedokode on Tuesday November 14, 2017 @04:43PM (#15699459)
As I understand they needed to download images from any external servers using an URL. I am not sure if that is possible even with CORS.

Also they don't save preview images and generate them when needed as I understood. So what you are suggesting requires a lot of disk space to keep thumbnails that might be never needed later.

And if you don't have millions of uploads per day then it makes no sense trying to save some seconds of CPU time by unnecessarily complicating the system. In most languages there already are libraries for resizing images.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by megy on Tuesday November 14, 2017 @06:04PM (#15700092)
I have no idea how what you are saying related to compressing images before they are sent?
Re:How Discord Resizes 150M Images Every Day with Go and C++ by jasonlotito on Tuesday November 14, 2017 @01:01PM (#15697486)
Many do resizing initially, but even when resizing, you still need to resize images for different reasons, such as thumbnails. So what you need to do is resize on client as low as you are willing to go, and then upload that. But you still need to resize for different needs. You don't want the client doing multiple resizes and uploads for that.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by CamTin on Tuesday November 14, 2017 @12:33PM (#15697201)
Because then you don't get to build out fun infrastructure like this and write it up in your company blog.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by fleitz on Tuesday November 14, 2017 @01:30PM (#15697789)
Agreed. I wish the posts contained a "it cost X developer hours to recreate thumbor or $$$ total, and we saved Y dollars per month" meaning in approximately 15 years we'll have broken even on this investment. Oh yeah and we don't even do intelligent resizing like thumbor does.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by rockostrich on Wednesday November 15, 2017 @10:29AM (#15704386)
It looks like thumbor is built on a Python stack similar to what Discord was using in their original service. What makes you say they didn't consider it or benchmark it against their previous service and make the decision that it wasn't as good as they needed?
Re:How Discord Resizes 150M Images Every Day with Go and C++ by Xeoncross on Tuesday November 14, 2017 @12:35PM (#15697225)
Sure you do! Don't you know that adding "Free" to a title increases the ROI by 40%?

"How Discord Resizes 150M Images Every Day for Free"

Re:How Discord Resizes 150M Images Every Day with Go and C++ by batmansmk on Tuesday November 14, 2017 @12:37PM (#15697243)
It may work in certain cases.

But it also means you need to know how you will display when you save it. Layout changes, screens change, how do you anticipate the future dimensions / resolution you will need out of the original?

Re:How Discord Resizes 150M Images Every Day with Go and C++ by user5994461 on Tuesday November 14, 2017 @02:22PM (#15698274)
Historically, resizing images client side doesn't work because most clients are not able to render the images, let alone resize them.

The image file formats are very very complicated, many are platform specific, some are covered by patents.

For example of a common issue, another comment mentioned the rotation parameter, it's set by many cameras but the support is inconsistent.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by sgk284 on Tuesday November 14, 2017 @12:42PM (#15697287)
You often want the original because different clients may display different sizes.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by dajohnson89 on Tuesday November 14, 2017 @10:27PM (#15701240)
isn't it a meme at this point, pushing computational work to the client side? I have a laptop or mobile device, please don't hog my limited cpu and battery life by forcing my device to resize images.
Re:How Discord Resizes 150M Images Every Day with Go and C++ by _Tev on Wednesday November 15, 2017 @07:42AM (#15703321)
Are you sure your wifi connection will not eat more power with huge upload than processor/gpu doing the scaling?
Re:How Discord Resizes 150M Images Every Day with Go and C++ by dajohnson89 on Wednesday November 15, 2017 @02:17PM (#15706870)
I dont see how that matters at all.

also -- there's also page load time. if it's an intensive calculation then the overhead of sending the results over http is still less than the browser calc time.

Re:How Discord Resizes 150M Images Every Day with Go and C++ by gsich on Tuesday November 14, 2017 @02:01PM (#15698087)
Mandatory XKCD: https://xkcd.com/1683/
Re:How Discord Resizes 150M Images Every Day with Go and C++ by greenleafjacob on Tuesday November 14, 2017 @12:48PM (#15697356)
Imgur does this.