You might also be happy to learn that some of the talent behind Picloud is now part of the Docker team itself :)
There are tons of companies that are working on Docker based solutions to my problems. In six months, or a year, there will be plenty of great ways for me to use Docker. The only problem is that PiCloud shuts down on February 25, so I'm in a bit of a hurry. Today, I'm in a gap, and there isn't a great replacement. Specifically I'd need:
* Per second pricing (per minute would be Ok). This makes parallelism "free"
* Large instances with big memory and CPU
* Ideally on AWS (My data is on s3, and it would be good to avoid bandwidth charges).
Nothing seems to fill these requirements yet. Any of you Docker startups out there want to help me out? Any pointers would be great if I'm missing something.
The docker gap is one part of it. For me I only really need one environment though (I have a customised one I run on picloud) so that's not the major concern. The bit that wasn't as obvious to solve is moving the python library dependencies from the client to the server. I looked into libraries to handle this but I couldn't find anything obvious. All the other processing libraries use a model where you need to have the code you want to run up on the remote machine before you start.
Do you expect the open source pickup by Multyvac to be a bust?
the team has decided to join Dropbox...
We’ll be bringing our API-building expertise to the Dropbox Platform, ...
Even though the team is moving on ...
I also hope that Multyvac will be a success. That said, I hope they'll start communicating soon. I can't really judge if they'll be able to be a success... but I hope so.
I've been looking around for a good replacement, and there is a gap. PiCloud really had three components. The first was an ability to run code with arbitrary dependencies with no more than a second start up time. PiCloud called them environments. This is really the same problem Docker is addressing. Both were using LXC and AUFS to solve the problem. Unfortunately, Docker-as-a-service is very early days. People like orchardup.com are getting there, but so far only have hourly pricing. There is enginedock.com which has per minute pricing, but is it super early on. I'm in a "Docker gap", where these services are really not at the same level that PiCloud was.
The second was the nice Python API, RESTful interface, and the command line interface for triggering off jobs, querying when the finish, and getting the results. That part will presumably be open sourced, but it is really all tied in to the hosting portion.
Then the third level is to have a bunch of machine you can load balance across, to allow per second pricing on AWS. They had such lovely high memory, high CPU machines to run on. The split up some of the biggest EC2 cluster instances, and you could run each process on your 1/8 share. The nearest current replacement is iron.io's IronWorker. However, they only have 320MB per job, which doesn't work for my application. Also Heroku workers are billed per second, but they only have up to 1GB of memory. I'm hoping that this (new?) entity Multyvac can serve that role successfully. Hopefully, they'll tell us more detail on this soon.
Anyway, best of luck to the PiCloud team. You were great. Especially Aaron who did much of the tech support. Thanks.