Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by Jabbles on Thursday May 30, 2013 @12:28PM
(#5793444)
“We just couldn’t crunch those images down to a smaller size,” Zacharias remembers. “It would have taken a significant amount of Google’s entire processing power just to do that.”
What? Processing a thumbnail is insignificant compared to processing a video - in fact it's on the same magnitude as making a single frame of video... Maybe Google uses a lot of hardware encoders for YouTube?
In any case, I can't believe this is a reason. A billion images? Is nothing for Google. Google was offering a billion hours of cpu time to scientists for free in 2011. Don't tell me that a thumbnail takes an hour... http://googleblog.blogspot.com/2011/04/1-billion-computing-c...
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by cma on Thursday May 30, 2013 @04:19PM
(#5794940)
Not to mention they could just do lazy initialization; most youtube thumbnails will never ever be viewed, so you don't have to convert them until they are needed.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by dgreensp on Thursday May 30, 2013 @11:00AM
(#5792908)
I tried imgix for thumbnails in a web photo album. It was much slower (higher latency) and double the price per bandwidth of Amazon Cloudfront. In general, then, resizing all the images in advance and serving them from a traditional CDN offers a better experience at a better price. Only if that's impossible is imgix a good choice.
Being a premium CDN seems a bit like being a premium credit card processor -- as customers grow, they have more and more incentive to switch to a commodity alternative.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by dgreensp on Thursday May 30, 2013 @11:23AM
(#5793046)
To be more constructive:
- Be competitive on price and performance for the basic CDN use case
- Amazon's strategy is to pass on their own costs with a very small margin. If you are serving each Zillow thumbnail by resizing it once in a while on a GPU and caching it in RAM, your costs should be tiny. Then you are just over-charging for bandwidth.
Biggest take-away is think hard about the pricing model. Maybe have big tiers with flat prices and pass on the bandwidth at cost.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by qhoxie on Thursday May 30, 2013 @01:12PM
(#5793751)
I think that "double the price per bandwidth" figure is way off – where did you get that? imgix pricing looks to be about a 30% premium over the US CloudFront pricing, and much less than that internationally (imgix doesn't appear to distinguish).
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by jtanner on Thursday May 30, 2013 @12:05PM
(#5793288)
I host large images between 500k and 700k. I store the original image along with 4 different resized versions. The resized images on average account for 35% of disk space.
I store about 300GBs of images. The resized versions take up about 100GBs of the 300.
If I were to then to add retina versions (2x) of each thumbnail, I might be running at 70%+ disk space being used up by thumbnails.
imgix is very interesting, as it would cut my storage costs, decrease my user image upload times ( because I wouldn't have to resize after uploading ). Also I would be free to create 2x retina images, or even new thumbnail sizes as desired.
Only things to justify, are the lock-in with using imgix ( re-writing my site to use it ) and the monthly cost.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by cunac on Thursday May 30, 2013 @02:24PM
(#5794209)
how 300GB is storage problem? I would understand if you say 300TB , but with 2TB HDD < 100$ and 0.5TB SSD < 400$ I can't really see that a major use case. And that is one time fixed payment.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by ekurutepe on Thursday May 30, 2013 @11:10AM
(#5792970)
Isn't it a bit wasteful to resize all those images on the fly? I'd think it's much better to resize them at creation and serve different files as they're needed, assumption being here that dumbly serving a file is more power efficient/cheaper than rescaling a large image every time a smaller version is needed.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by zacman85 on Thursday May 30, 2013 @11:13AM
(#5792986)
We cache aggressively, at multiple levels, and manage the dependencies between caches. In the overwhelming amount of cases, we are serving straight out of SSD caches. The on-the-fly work only happens if the image does not already exist.
Resizing or optimizing images, in isolation, is not a hard problem. I've solved subsets of this problem a handful of times before, and I could do it again, but I gladly pay for imgix. The time and headaches saved are easily worth it. The next time I have to rely on ImageMagick will be too soon.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by 300bps on Thursday May 30, 2013 @04:08PM
(#5794882)
>Is this really such a hard problem to solve?
I must be misunderstanding because resizing an image, rotating it or many other basic operations consists of about 5 lines of .NET code and very little processing power. I imagine it's not any more difficult in other frameworks. I couldn't conceive of offloading that to an external service under just about any use case.
It seems like they plan to offer video editing via SaaS as well which is more valuable, but I would think the transfer time and bandwidth would kill any speed improvement over a massively parallel SaaS video editor as opposed to using a local open source library.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by benjamincburns on Thursday May 30, 2013 @04:46PM
(#5795120)
> 5 lines of .NET code
Yep.
> very little processing power
Not really. Those 5 lines are hiding a pretty big chunk of smarts. Simple scale/rotate transformations are an O(n) operation. That's not so bad. However image encoding/decoding is a bit more expensive.
JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform. Good JPEG encoders will also do some expensive analysis to maximize precision while minimizing size during quantization (edit: though most just use a set of standard quantization matrices). And then there's I/O. It seems fast when you're only processing one image at a time, but to do this as a real-time service for a number of high-traffic websites is very nontrivial.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by astrange on Friday May 31, 2013 @03:30AM
(#5797347)
> JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform.
Scuse me?
JPEG's frequency transform is a set of fixed linear transforms and is always 8x8, so this step is just O(n) of the area of the image. Also, it's not really "a DCT", just any approximation that will look good when run through the approximation of the opposite transform at the other side.
The compression steps after are the most difficult, since they have poor ILP and can involve division. But JPEG is really simple, it's not a burden these days.
One interesting part of it is minimizing the memory use of your application, by using the smallest possible ring buffer for pixels rather than keeping the whole image in memory at once.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by jakejake on Thursday May 30, 2013 @01:23PM
(#5793828)
I thought the same thing. I think this would be hard to do at scale and they offer quite a lot more features than most would need (sepia tone filter). I couldn't personally see paying for re service, but obviously it must be useful to others.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by dminor on Thursday May 30, 2013 @11:46AM
(#5793189)
> when someone visits the site, imgix will deliver the appropriate photos and, in need be, reformat or resize them. If someone visit your site with an Apple iPad, the service can instantly reformat an image for the tablet’s “Retina” display.
I didn't see any documentation about this on their site. Can they deliver, via a single URL, differently sized images for mobile vs desktop? This is one of the biggest headaches when trying to do responsive design.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by e98cuenc on Thursday May 30, 2013 @12:49PM
(#5793589)
We have a service similar to imgix (thumbr.it). The way we serve retina images is that you add @2x to the name of the images with JS, and when we see the image name ends with @2x we serve double the size you requested (see http://thumbr.it/api/documentation/#retina).
I guess the imgix guys do something similar.
Serving different images with exactly the same URL is tricky because it will confuse the cache.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by kaffeinecoma on Thursday May 30, 2013 @02:19PM
(#5794183)
Stupid question: are you just doubling every pixel of a non-retina source for the @2x images? Does that really provide any extra clarity on a Retina display (beyond setting the proper size of the image?) Or are you actually starting with an @2x image and halving it for non-2x default?
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by e98cuenc on Thursday May 30, 2013 @02:30PM
(#5794263)
The retina modifier is only useful when you request a smaller thumbnail than the original.
So your original may be 1000x1000px, you use a 128x128 thumbnail on your regular (non-retina) site, and you serve 256x256 thumbnail to your retina users.
If your original image is small then there is nothing to do.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by mhmtmhmt on Friday May 31, 2013 @04:20AM
(#5797453)
VIPS values are very good and you should notice it says "All timings are for a 5,000 by 5,000 pixel 8-bit RGB image in uncompressed tiled TIFF format, 128 by 128 pixel tiles". I haven't used VIPS in production. It seems like it's better for big files. I heard VIPS is not that performing with small files.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by chiph on Thursday May 30, 2013 @10:17AM
(#5792602)
“We would just walk into one Apple store after another and buy them out”
Apple has good terms for business purchases. Spend more than $2k and get 18 months interest-free financing. No need to drive around to all the stores (unless you just want the airline points on your credit card)
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by ams6110 on Thursday May 30, 2013 @10:36AM
(#5792756)
Is the graphics card in a Mac Mini really that good? Seems there would be cheaper ways to buy commodity hardware and put the savings towards a better graphics card.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by adlpz on Thursday May 30, 2013 @10:58AM
(#5792895)
I know I'm parallel-commenting here the guy himself, but seriously, I believe this is just what happens when you either have too much money, are too lazy, or both.
I am pretty sure they could have come up with a basic MoBo+GPU+CPU+RAM combo to buy from a large distributor, assemble them in series in no-time and get a vastly cheaper and more powerful farm.
Walking around buying mac minis a dozen a time to build a data center is just silly in my opinion, whatever the reasoning.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by zacman85 on Thursday May 30, 2013 @11:08AM
(#5792959)
You are not wrong. And we were very well aware of this. However, we found the reality of the problem being solved is much more about network latency, routing, and caching than actual graphics processing throughput. Our pain points have been very different than what you are prescribing as a better course of action.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by zacman85 on Thursday May 30, 2013 @10:50AM
(#5792851)
They do a fair enough job for what we needed. Really, it was more about what we could get quickly, with a certain expectation of consistency, just to get us moving forward.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by chiph on Thursday May 30, 2013 @11:52AM
(#5793222)
I'd have to do the math, but the Mini is very power efficient¹, so perhaps it was just enough graphics capability for the watts it consumes (11 watts idle, 85 watts max, if I recall correctly)
¹A lot of data centers these days bill you based on the power + bandwidth you consume, not the # of U's you use in a rack.
Re:Ex-YouTube Man Builds Graphics Card for Entire Internet
by mseebach on Thursday May 30, 2013 @04:20PM
(#5794955)
Heh. This was my (unsuccessful) pitch for Startup Weekend Copenhagen three years ago. My own fault for not just building it, but always nice to have an idea validated.
What? Processing a thumbnail is insignificant compared to processing a video - in fact it's on the same magnitude as making a single frame of video... Maybe Google uses a lot of hardware encoders for YouTube?
In any case, I can't believe this is a reason. A billion images? Is nothing for Google. Google was offering a billion hours of cpu time to scientists for free in 2011. Don't tell me that a thumbnail takes an hour... http://googleblog.blogspot.com/2011/04/1-billion-computing-c...
Google App Engine now offers the Image Service, but perhaps that was launched after this. https://developers.google.com/appengine/docs/java/images/ove...
Being a premium CDN seems a bit like being a premium credit card processor -- as customers grow, they have more and more incentive to switch to a commodity alternative.
- Be competitive on price and performance for the basic CDN use case
- Amazon's strategy is to pass on their own costs with a very small margin. If you are serving each Zillow thumbnail by resizing it once in a while on a GPU and caching it in RAM, your costs should be tiny. Then you are just over-charging for bandwidth.
Biggest take-away is think hard about the pricing model. Maybe have big tiers with flat prices and pass on the bandwidth at cost.
I store about 300GBs of images. The resized versions take up about 100GBs of the 300.
If I were to then to add retina versions (2x) of each thumbnail, I might be running at 70%+ disk space being used up by thumbnails.
imgix is very interesting, as it would cut my storage costs, decrease my user image upload times ( because I wouldn't have to resize after uploading ). Also I would be free to create 2x retina images, or even new thumbnail sizes as desired.
Only things to justify, are the lock-in with using imgix ( re-writing my site to use it ) and the monthly cost.
Nice marketing though.
http://littleutils.sourceforge.net/
Also what does this has to do with being "graphics card for the entire internet"?
Resizing or optimizing images, in isolation, is not a hard problem. I've solved subsets of this problem a handful of times before, and I could do it again, but I gladly pay for imgix. The time and headaches saved are easily worth it. The next time I have to rely on ImageMagick will be too soon.
I must be misunderstanding because resizing an image, rotating it or many other basic operations consists of about 5 lines of .NET code and very little processing power. I imagine it's not any more difficult in other frameworks. I couldn't conceive of offloading that to an external service under just about any use case.
It seems like they plan to offer video editing via SaaS as well which is more valuable, but I would think the transfer time and bandwidth would kill any speed improvement over a massively parallel SaaS video editor as opposed to using a local open source library.
Yep.
> very little processing power
Not really. Those 5 lines are hiding a pretty big chunk of smarts. Simple scale/rotate transformations are an O(n) operation. That's not so bad. However image encoding/decoding is a bit more expensive.
JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform. Good JPEG encoders will also do some expensive analysis to maximize precision while minimizing size during quantization (edit: though most just use a set of standard quantization matrices). And then there's I/O. It seems fast when you're only processing one image at a time, but to do this as a real-time service for a number of high-traffic websites is very nontrivial.
Scuse me?
JPEG's frequency transform is a set of fixed linear transforms and is always 8x8, so this step is just O(n) of the area of the image. Also, it's not really "a DCT", just any approximation that will look good when run through the approximation of the opposite transform at the other side.
See http://multimedia.cx/eggs/dct-pr/.
The compression steps after are the most difficult, since they have poor ILP and can involve division. But JPEG is really simple, it's not a burden these days.
One interesting part of it is minimizing the memory use of your application, by using the smallest possible ring buffer for pixels rather than keeping the whole image in memory at once.
I didn't see any documentation about this on their site. Can they deliver, via a single URL, differently sized images for mobile vs desktop? This is one of the biggest headaches when trying to do responsive design.
I guess the imgix guys do something similar.
Serving different images with exactly the same URL is tricky because it will confuse the cache.
So your original may be 1000x1000px, you use a 128x128 thumbnail on your regular (non-retina) site, and you serve 256x256 thumbnail to your retina users.
If your original image is small then there is nothing to do.
Apple has good terms for business purchases. Spend more than $2k and get 18 months interest-free financing. No need to drive around to all the stores (unless you just want the airline points on your credit card)
I am pretty sure they could have come up with a basic MoBo+GPU+CPU+RAM combo to buy from a large distributor, assemble them in series in no-time and get a vastly cheaper and more powerful farm.
Walking around buying mac minis a dozen a time to build a data center is just silly in my opinion, whatever the reasoning.
Its rated around 100 megahashes. A 6990 gets you 800MH.
¹A lot of data centers these days bill you based on the power + bandwidth you consume, not the # of U's you use in a rack.
But well, Ex-YouTube? They have way more money to throw at this. I'll have to put that idea aside.