News for hackers. Stuff that matters.
Re:How 500px serves up 500TB of high res photos by callumjones on Friday May 15, 2015 @05:41PM (#9553855)
Interesting that they upload to S3 first, only to pull it off soon after and process it.

I imagine this reduces complexity and lets them increase their throughput of uploads but I wonder if 500px toyed around with the idea of first uploading it to local (temp) storage, having the workers process it and then finally persist it. But I guess the cons are a) some point it needs to go to S3 in original form b) increases data loss

Re:How 500px serves up 500TB of high res photos by Renat on Friday May 15, 2015 @09:04PM (#9554572)
I'm the CTO of 500px. Decoupling uploaders and converters simplifies the overall design and makes it more robust. Converters are a part of the legacy design that we are planning to retire. In the past, we would generate multiple sizes and crops for every image so it can be efficiently displaced on the site and/or mobile apps.

Today, we rely on the new Resizer service that can resize and crop images on the fly. It's an interesting piece of technology that we will be writing about on our blog: developers.500px.com

Re:How 500px serves up 500TB of high res photos by ranrub on Saturday May 16, 2015 @03:00AM (#9555280)
Thanks for sharing this and being on HN to answer - very interesting architecture! How do you deal with customers from remote locations (Asia, Oceania) complaining about slow upload speeds?
Re:How 500px serves up 500TB of high res photos by bliti on Friday May 15, 2015 @09:20PM (#9554614)
I will keep posted to your blog. Thanks for sharing the link. Always looking to learn about memory heavy operations. Currently am the lead dev on software that revolves around user generated content (images and audio). Its an ever ending road of (fun) challenges.
Re:How 500px serves up 500TB of high res photos by renaudg on Saturday May 16, 2015 @12:09AM (#9555031)
Is there a specific reason why you didn't use an off the shelf open source solution like Thumbor ( http://thumbor.org/ ) ?
Re:How 500px serves up 500TB of high res photos by Renat on Saturday May 16, 2015 @09:19AM (#9555991)
performance is the main reason. Additionally we needed a solution to modify images on the fly (watermarking and attribution). That's why we decided to build our own service using go
Re:How 500px serves up 500TB of high res photos by frik on Saturday May 16, 2015 @02:27AM (#9555224)
I assume you still generate a small thumbnail for every picture, right?
Re:How 500px serves up 500TB of high res photos by discardorama on Friday May 15, 2015 @11:57PM (#9555014)
I am surprised that it is faster to resize the image, than to pass the (resized) bits right through. Is the outbound network a bottleneck?
Re:How 500px serves up 500TB of high res photos by kalleboo on Saturday May 16, 2015 @01:18AM (#9555136)
It could be that there are a lot of images/versions that simply never get seen, or seen rarely enough that it's inefficient to store them. And they're being cached by the CDN anyway, so popular images will still only be generated once per edge.
Re:How 500px serves up 500TB of high res photos by Renat on Saturday May 16, 2015 @09:06AM (#9555944)
bingo! Extra storage for all the resized images that never get seen, lack of flexibility to introduce new sizes, plus requirements for watermarking and attribution on images are the main reasons why we are moving away from pre-conversion to dynamic image resizing
Re:How 500px serves up 500TB of high res photos by yeukhon on Friday May 15, 2015 @06:52PM (#9554168)
To me this is a very common practice. At least in my experience a lot of AWS processing are done with S3 throughout the pipeline, whether you run EMR or run simple workflow. You can also create API/temp credential per user in your app allows the user to upload things to "storage" (but in this case S3) and process from there.
Re:How 500px serves up 500TB of high res photos by ranrub on Friday May 15, 2015 @07:48PM (#9554361)
And with Lambda based on S3 events, you can run stuff automatically on that as well. However, if you want your uploaders to get feedback on their uploads immediately and/or synchronously, you need to upload to app servers.
Re:How 500px serves up 500TB of high res photos by ranrub on Friday May 15, 2015 @06:46PM (#9554141)
It also means that you need to scale your upload servers (IO & long requests), this is more difficult than trusting S3 to scale for you. However having your S3 open to uploads has it's downsides. We at Cloudinary do uploads to our own (autoscaled) instances and only then persist.
Re:How 500px serves up 500TB of high res photos by e12e on Friday May 15, 2015 @08:59PM (#9554553)
My initial thought was: wow, this looks like a great validation of the idea behind Joyent Manta[1] storage+process. I wonder if it would've been a good fit for 500px, and how it would compare in terms of price/performance? Granted, they now have a working system on s3, so maybe for the next up-and-comming competitor? ;-)

Great write-up, btw. And thanks for the heads-up about vips/nip2 -- I wasn't aware of those.

[1] https://www.joyent.com/object-storage

Re:How 500px serves up 500TB of high res photos by dataminer on Friday May 15, 2015 @05:31PM (#9553801)
Thumbs up for VIPS, used it 4-5 years ago, for building a system which processed high resolution images. Its fast, lean and an amazing piece of software.
Re:How 500px serves up 500TB of high res photos by jeffreyrogers on Friday May 22, 2015 @11:00AM (#9588691)
I'm replying here to let Melraidin (who posted a sibling comment) know that their account is dead, for reasons I can't understand since their comments seem helpful.
Re:How 500px serves up 500TB of high res photos by fotios on Tuesday May 19, 2015 @11:33AM (#9570870)
Totally agree, I've used it for some projects as well. The only thing I wasn't able to grok (which still kills me to this day), was that I couldn't get it to handle EPS files correctly. If anybody has done that with VIPS/IM, I would love to hear about it.
Re:How 500px serves up 500TB of high res photos by Renat on Friday May 15, 2015 @09:05PM (#9554577)
Performace is the reason why we switched from ImageMagick
Re:How 500px serves up 500TB of high res photos by quasse on Friday May 15, 2015 @05:36PM (#9553832)
Are they somehow intercepting middle click on their site so that links do not open in a new tab? Control-click works fine for me, but middle click would just replace the article, which was inconvenient. (latest Chrome on Ubuntu)
Re:How 500px serves up 500TB of high res photos by leftnode on Friday May 15, 2015 @05:50PM (#9553888)
I'm getting the same thing and it's incredibly frustrating on an otherwise interesting article. I had to Right Click > "Open In New Tab" for links I wanted to read later.
Re:How 500px serves up 500TB of high res photos by distantsounds on Friday May 15, 2015 @06:50PM (#9554156)
Same here, Chrome on W7.
Re:How 500px serves up 500TB of high res photos by aroch on Friday May 15, 2015 @05:41PM (#9553854)
Middle-click seems to be working fine here (FF41, OSX)
Re:How 500px serves up 500TB of high res photos by nols on Friday May 15, 2015 @11:41PM (#9554975)
Same for me (Chrome on Mint)
Re:How 500px serves up 500TB of high res photos by Retr0spectrum on Friday May 15, 2015 @07:20PM (#9554268)
Works for me on Firefox 38 on Arch Linux.
Re:How 500px serves up 500TB of high res photos by happytrails on Friday May 15, 2015 @08:28PM (#9554470)
All the services! An Engineer's paradise.
Re:How 500px serves up 500TB of high res photos by cconcepts on Saturday May 16, 2015 @02:48AM (#9555251)
I was getting excited about using 500px for the specific reason that it seems to load much faster than flickr. Then I loaded it up in a public space to get some inspiration by browsing other user's work - interspersed were various images of women in various states of undress. I quickly exited the site and never went back for fear of someone glancing over my shoulder and assuming I'm doing something other than getting artistic inspiration.

Not arguing that the images aren't art but I personally don't want them popping up on my screen when I'm just trying to get inspiration for landscape photography.

Re:How 500px serves up 500TB of high res photos by dagw on Saturday May 16, 2015 @02:59AM (#9555273)
500px at least used to have a feature to autofilter out nude photos.
Re:How 500px serves up 500TB of high res photos by Renat on Saturday May 16, 2015 @09:22AM (#9555995)
we still have "Display Adult Content" flag that can be disabled on https://500px.com/settings
Re:How 500px serves up 500TB of high res photos by cconcepts on Sunday May 17, 2015 @07:33AM (#9559628)
Cool, thanks for that. It was a knee-jerk reaction to bail on you folks the way I did but I thought perhaps I wasn't the only one.
Re:How 500px serves up 500TB of high res photos by ranrub on Friday May 15, 2015 @04:05PM (#9553307)
Popular photo sharing company opens the kimono on their photo processing stack.