Re:AWS Case Study: Parse (YC S11)
by jedberg on Thursday March 07, 2013 @01:25AM
(#5336431)
I'm really humbled to see the architecture that we came up with at reddit with help from Heroku and JustinTV, which I then taught to Amazon and AWS, is still the one AWS is highlighting today (and presumably teaching their customers). It's an amazing feeling to see your work live on beyond your involvement.
* They flew me up to Seattle to teach Amazon retail how to move to AWS back when retail had just started that process. Amusingly, the guy mainly responsible for that move now works for Pinterest (a company that "does cloud right").
Re:AWS Case Study: Parse (YC S11)
by jedberg on Thursday March 07, 2013 @06:22PM
(#5341076)
I was surprised when they asked, that's for sure.
It was a little odd because I felt like there is no way I could teach these people anything they didn't already know, but I got really good questions and feedback so at the end I didn't feel nearly as weird about it.
Re:AWS Case Study: Parse (YC S11)
by gfodor on Wednesday March 06, 2013 @08:35PM
(#5335375)
I realize the diagram is likely heavily over-simplified, but what's the reason for the nginx layer? It doesn't look like you guys are serving assets through those boxes, so it seems like you have an unnecessary extra hop from the ELB to the app servers. My guess is this is either nginx caching, a holdover from the pre-aws architecture or a measure to keep the system decoupled from ELB?
Re:AWS Case Study: Parse (YC S11)
by tcwc on Wednesday March 06, 2013 @09:14PM
(#5335544)
I can't speak for Parse, but I've come up with something similar in the past. Nginx/HAproxy as a combo is far more flexible than the ELB alone, you might want to use it for rate limiting, better load balancing algorithms, better logging, tweaking headers, handling errors, or controlling buffer sizes for example.
Re:AWS Case Study: Parse (YC S11)
by pjscott on Thursday March 07, 2013 @01:59AM
(#5336511)
Or for preventing DoS attacks from slow client connections. Unicorn is not designed to be exposed to the outside world; it needs a reverse proxy that does request buffering, like nginx (which the Unicorn docs recommend).
Re:AWS Case Study: Parse (YC S11)
by lolz on Thursday March 07, 2013 @12:45AM
(#5336327)
Can you tell us more about RAID10 comment?
Are you doing RAID0 now instead for disk volumes?
What would happen if there's another EBS outage that would take out a lot of your MongoDBs. Would you just recreate them from the very latest S3 backups?
Re:AWS Case Study: Parse (YC S11)
by manuelflara on Wednesday March 06, 2013 @09:47PM
(#5335705)
I can't speak for Parse, but I use both of these technologies in one site. Memcached is, well, for any kind data that you want to cache: results of SQL queries, blocks of the site (HTML), etc. Redis, on the other hand, it's a final data storage on its own (like MySQL, because it's also persistent), but it's faster than MySQL for certain things. For example, for things like "List of profiles I've visited / have visited me recently on Facebook". This seems like a very simple MySQL table and query, but once it becomes big it starts to get slow. On the other hand, you can have one key per user on Redis, like "<id_user>_visit_history" which is a sorted set of <time, id_user>. You could do the same with Memcached (although it doesn't support the same data types), but then it wouldn't be persistent.
Re:AWS Case Study: Parse (YC S11)
by trustfundbaby on Wednesday March 06, 2013 @10:34PM
(#5335906)
Right ... thats my point. redis is as fast if not faster than memcached for caching exactly the same things you talked about (results of sql queries, blocks of the site etc) so why not just use redis for everything and eliminate memcached.
Re:AWS Case Study: Parse (YC S11)
by spimmy on Wednesday March 06, 2013 @09:47PM
(#5335707)
We use redis because we use resque extensively for the push notifications architecture. Memcache is superior to redis for simple caching in front of data sources.
Re:AWS Case Study: Parse (YC S11)
by AznHisoka on Wednesday March 06, 2013 @10:20PM
(#5335842)
Any reason why you haven't tried using Sidekiq, which is much more efficient in terms of using memory resources? Or is Resque just good enough for now?
Re:AWS Case Study: Parse (YC S11)
by spimmy on Wednesday March 06, 2013 @10:31PM
(#5335892)
Hm, well Sidekiq wasn't out when we started developing. :) Memory has never been a constraint for us with resque though. The only problems we've had with it have been things like improper signal handling, pretty easy to deal with.
* They flew me up to Seattle to teach Amazon retail how to move to AWS back when retail had just started that process. Amusingly, the guy mainly responsible for that move now works for Pinterest (a company that "does cloud right").
It was a little odd because I felt like there is no way I could teach these people anything they didn't already know, but I got really good questions and feedback so at the end I didn't feel nearly as weird about it.
probably to load balance between unicorn instances in the particular amazon EC2 instance
Are you doing RAID0 now instead for disk volumes?
What would happen if there's another EBS outage that would take out a lot of your MongoDBs. Would you just recreate them from the very latest S3 backups?