News for hackers. Stuff that matters.
Re:Ask HN: Which cloud do you use, and why? by shortstuffsushi on Thursday May 21, 2015 @11:52AM (#9583511)
Couldn't you potentially push that aspect elsewhere, into a data storage solution like S3 or something (not sure of the cost efficiency, not a huge cloud guy myself). Then you'd just be utilizing the DO server for processing. It would have the added benefit of allowing you to scale your crawling horizontally if you wanted, as all your servers could have access to the same set of data.
Re:Ask HN: Which cloud do you use, and why? by 3pt14159 on Thursday May 21, 2015 @02:43PM (#9584612)
Yeah I could, and I currently do - but I don't like to. I want the data on the same network for lots of reasons and I also don't like maintaining two sets of deployment systems. I'd prefer to just use Cloud66 + Digital Ocean and focus on writing code, not setting up and maintaining servers.
Re:Ask HN: Which cloud do you use, and why? by shortstuffsushi on Friday May 22, 2015 @02:48PM (#9590203)
Oh yeah, I totally understand that, was just offering an alternative as a non-ideal workaround. I have the same set up for a service I offer, primarily because (afaik) it's hard to beat S3 pricing.
Re:Ask HN: Which cloud do you use, and why? by icebraining on Thursday May 21, 2015 @12:02PM (#9583587)
The traffic between datacenters would be significantly slower and more costly, though. Besides, S3 is not a replacement for a real drive.
Re:Ask HN: Which cloud do you use, and why? by shortstuffsushi on Thursday May 21, 2015 @12:11PM (#9583646)
It would be slower than on disk for sure, so if that's a concern, it wouldn't be a good choice. If the poster is just storing off results, pushing it to S3, or even potentially a document database could be a logical choice. They said they're just using it for occasionally re-parsing at a later date, so immediacy doesn't seem like a huge concern. Just a thought, at any rate.