> Any AppEngine developer could tell you not to use it like this

I'd really wish for some AppEngine developer to tell me how's the proper way of designing entity groups. The only advice given is to not exceed a single user's data, otherwise it's too big.

Really, I want to see an example of a request handler that creates data in multiple entities and takes care to not create duplicates.

> It would be very interesting to see what the results would be with some more realistic testing.

IMHO, my experience with AppEngine's datastore has been pretty bad ... the performance varies greatly based on factors you can't control ... i.e. processing something that does well on slow Fridays, may throw the 30 secs timeout error on Mondays.

Also, I don't know how people deploy real-world apps on it ... a non-relational datastore requires custom indexes (i.e. precalculated queries done by building specialized entities out of your quasi-normalized data).

You can't do that easily because of the 30 secs limit that also applies to cron or queue jobs. I tried a workaround by having a queue task that processes 5 rows at a time and then reschedules itself. The stuff was so god-damn awful to debug, and the results so unpredictable (30 secs timeout for 5 rows / concurrency issues) that I felt like crying.

But yeah, it's fine for hosting a blog engine on top of it.

Another issue we're running into now (as a result of having a growing number of users) is that we're noticing that downtime on the App Engine is shockingly bad, and getting worse.

Over the last few weeks, we've seen periods of downtime pretty much daily, and serious downtime (i.e. more than a few minutes) monthly. It's grossly under-reported in the App Engine status console, which shows a euphemistic warning (which in practice might mean that all server requests will return errors.)

Our software is used in the classroom during lectures, and we've had several customers experience system failure due to App Engine downtime. On top of that we've had more than one customer demo go wrong for the same reason.

It's pretty infuriating actually, because 1. you can't do anything other than just sit there and wait it out, and 2. there is no one to talk to at Google. Finally, the problems are only getting worse.

We were going to do a slow migration out of the App Engine over the course of a few months (because we've got a ton of code to rewrite.) At this point we're doing an emergency 2 week migration with all devs pulled from their projects for the task.

I honestly have no idea why these issues aren't getting more mainstream attention, given how many people are using the App Engine. The only thing I can come up with is that there must be tons and tons of tiny non-critical projects, and very few serious startups betting their business on the App Engine.

I was about to start a big project on App Engine, so your comment is worrying.

Anybody else having reliability problems with GAE or is this just an isolated case?

If you're interested in more gory details feel free to email me at mike at tophatmonocle dot com.

For what it's worth, we're an angel funded startup with some extremely bright hackers and we were signing praises about the App Engine until about 3 months in, when we started hitting it's various brick walls.

Oh, and take a look at: http://code.google.com/status/appengine

Click through the various items with green checkmarks to see the actual data that GAE considers "no significant issues." Including fun stuff like 80% datastore error rates.

There might be some problems but it's great to know that Google system administrators (instead of me) are taking care of it. When I hosted my application on EC2 twice I almost lost all my files due to hardware problems. I had to spend all night preparing a new server, not funny. There are no servers to fail in App Engine, I love that!

>I'd really wish for some AppEngine developer to tell me how's the proper way of designing entity groups.

here are some helpful links:

http://code.google.com/appengine/articles/datastore/overview...

http://code.google.com/appengine/articles/datastore/overview...

http://code.google.com/appengine/articles/scaling/contention...

You can split it up any way you want to, so long as you don't require more than 1-2 transaction per second in each entity group. On the other hand, lots of tiny entities doesn't perform well either. So it's a pretty severe design constraint.

To get decent performance for batch jobs, I use the remote API with bulk gets and bulk puts and no transactions. But that's useful only for inserts or when you've taken the application offline.

And a useful video:

http://code.google.com/events/io/2009/sessions/OfflineProces...

Well, maybe you're right, but I'll say the following:

1. Unless I'm missing something here the entity group thing is a red herring. Only if the authors of the paper explicitly setup all entities in the datastore as having the same root entity, then this would be an issue. However, since there isn't anything to suggest that they did this, all datastore entities are their own groups (and hence don't lock up the entire db,) so transactions on them would not have had any speed penalty. Please someone correct me if I'm wrong here.

2. Unfortunately their results are consistent with my experience over the last ~8 months running a startup on the App Engine (and now switching over to EC2 over the next few weeks.)