Re:Removing your files from S3 can cost thousands of dollars
by jbellis on Monday May 10, 2010 @08:55AM
(#1334233)
nbio isn't a magic wand; context switching is virtually free in a language without a GIL
http://paultyma.blogspot.com/2008/03/writing-java-multithrea...
Re:Removing your files from S3 can cost thousands of dollars
by jvaleski on Monday May 10, 2010 @03:21PM
(#1335639)
the solution I used leveraged non-blocking I/O in a roundabout way. 150 separate procs (quick and dirty) per instance. no threads. I let the OS manage the I/O in that regard. My thinking was that if I had "a lot" of procs trying I/O... I'd get the same effect. again... quick/dirty.
It's not a trivial task, but reasonably exprienced Python (or maybe even C) programmer would do it easily. (I'd say in a couple of days) And I can confirm that even on EC2 small instances 700 HTTP req's a sec is achievable (not sure about S3 API usage limiting).