Re:Google App Engine is now much faster, after latest mainte...
by mmastrac on Saturday November 06, 2010 @10:15PM
(#1878410)
Our app (http://gri.pe) feels a lot snappier after this latest bit of datastore maintenance. We've been using as much memcache as possible to avoid datastore access because prior to this maintenance, datastore access was spiky and even the simplest of queries had a small chance of timing out.
I can say for sure that some of our datastore-heavy cron jobs were taking > 200ms, but are around 70ms now.
It's a shame that the stats for > 12hrs ago seem to have been wiped. I bet we'd be seeing a marked improvement on our ms/request stat.
Re:Google App Engine is now much faster, after latest mainte...
by bad_user on Sunday November 07, 2010 @02:09AM
(#1878910)
Unrelated question: how are you doing those cron jobs.
I tried using the task queue, but it kept timing out on me, even when I broke my tasks into smaller ones (couldn't figure out a threshold as the behavior of the datastore was totally unpredictable).
Re:Google App Engine is now much faster, after latest mainte...
by mshafrir on Sunday November 07, 2010 @08:19AM
(#1879270)
The key is to combine cron jobs with Tasks. Tasks are am easy to use and powerful feature of Google Appengine that let you deal with the standard limitations. For example, rather than doing some operation on 1000 entities in one request, you spin off 1000 tasks, each handling the operation on one entity. Read up about them at http://code.google.com/appengine/docs/python/taskqueue/
Re:Google App Engine is now much faster, after latest mainte...
by mmastrac on Sunday November 07, 2010 @11:38AM
(#1879666)
We have to break our cron jobs down into single datastore queries for them to succeed near 100% of the time. If you want more complex logic, use a cron job to trigger tasks that process smaller bits of your operation.
Even our cron job to compute the top users on our site (a simple query + ordering) uses one second of CPU time between the application and the datastore. It's just something you get used to in AppEngine.
Re:Google App Engine is now much faster, after latest mainte...
by mmastrac on Sunday November 07, 2010 @11:39AM
(#1879670)
Sorry, I meant the internal application statistics. You can visit your dashboard to view "milliseconds per request", but the data doesn't exist from before this maintenance period.
One interesting aside: our nightly datastore backup (using the remote API) no longer causes 500 errors! Hooray!
Re:Google App Engine is now much faster, after latest mainte...
by pjscott on Sunday November 07, 2010 @06:50PM
(#1880764)
It would have been sexier on a semi-log-y scale. That said, as someone who's always been annoyed at the high latency of App Engine's datastore, this is huge if they can deliver this level of latency with low variability.
Re:Google App Engine is now much faster, after latest mainte...
by k7d on Sunday November 07, 2010 @04:19AM
(#1879052)
Good news but I really hope Google figures out how to solve DeadlineExceededError. They are popping up from time to time for my app (http://www.knowcase.com) and I don't think it's datastore related.
Re:Google App Engine is now much faster, after latest mainte...
by wkornewald on Sunday November 07, 2010 @02:34AM
(#1878955)
It's still too early to tell. This would not be the first time that speed improves after a maintenance and then it slowly gets worse every week until it looks almost like before the maintenance.
Re:Google App Engine is now much faster, after latest mainte...
by benjaminfox on Sunday November 07, 2010 @03:24AM
(#1879012)
Is it possible that the degradation in performance is due to more developers building more apps and driving more traffic to App Engine as time goes on?
Re:Google App Engine is now much faster, after latest mainte...
by wkornewald on Sunday November 07, 2010 @12:08PM
(#1879743)
The performance degrades too quickly for this. In two or three months we might be on our old performance level, again. Well, I seriously hope that we'll stay fast this time. Only time will tell.
Re:Google App Engine is now much faster, after latest mainte...
by tropin on Saturday November 06, 2010 @09:26PM
(#1878318)
This site, probably NSFW, makes use of the datastore (for the thumbnails) and it's now much faster, it's a pity we can't compare speeds with the old latency.
Re:Google App Engine is now much faster, after latest mainte...
by forgotusername on Saturday November 06, 2010 @10:33PM
(#1878455)
From the maintenances mailing list:
The maintenance work has been completed. The datastore was in a read-only period between 12:02-12:45 for a total of 43 minutes. Some users reported two periods of 500 errors unrelated to datastore writes, but these issues should no longer be occurring.
Re:Google App Engine is now much faster, after latest mainte...
by cageface on Sunday November 07, 2010 @02:32AM
(#1878952)
The sense that the app store has been a neglected, second-tier priority is deep enough now that it will take them a while to regain significant developer trust.
AWS is very close to a de-facto standard now.
Edit: I mean app engine, the hosted web service. Not the app store. Thanks.
App Gallery was just a GAE application show-case center, and most likely was causing confusion to Google Apps customers too.
I am still a developer who trusts GAE (my app was on the appgallery and is also listed on marketplace) and believe that they will continue to compete for developer mind-share, as they proved yet again with this performance update.
Let go of the fear, uncertainty & doubt and hop on to try an app. You will like it too.
Re:Google App Engine is now much faster, after latest mainte...
by c1sc0 on Sunday November 07, 2010 @04:32AM
(#1879061)
Can't repeat this enough: the main reason I develop on appengine is that it removes server admin headaches completely from the equation. Ok, you get other headaches in return, but most of the time it's stuff I can deal with.
I can say for sure that some of our datastore-heavy cron jobs were taking > 200ms, but are around 70ms now.
It's a shame that the stats for > 12hrs ago seem to have been wiped. I bet we'd be seeing a marked improvement on our ms/request stat.
I tried using the task queue, but it kept timing out on me, even when I broke my tasks into smaller ones (couldn't figure out a threshold as the behavior of the datastore was totally unpredictable).
You can also explore the new Mapper API at http://blog.notdot.net/2010/05/Exploring-the-new-mapper-API
AppEngine requires careful thought of application design at this point. Everything is an HTTP request and subject to the same restrictions.
I've heard rumours that they'll be launching longer-lived tasks at some point in the future, but I have no idea when that will happen.
Even our cron job to compute the top users on our site (a simple query + ordering) uses one second of CPU time between the application and the datastore. It's just something you get used to in AppEngine.
Just click the left-facing arrow next to the date. Here are the stats for 11/5: http://code.google.com/status/appengine/detail/datastore/201...
and here are the stats for 11/4: http://code.google.com/status/appengine/detail/datastore/201...
One interesting aside: our nightly datastore backup (using the remote API) no longer causes 500 errors! Hooray!
EDIT: I neglected to see the other datastore figures.
It definitely makes me reconsider using App Engine for new projects.
Disclaimer: I have no connection to Appbrain aside from meeting the developers once.
Kudos to the team, and I, too, hope that the latency will remain this low forever and ever…
http://wallres.saiswa.com/?tags=girls&pagina=23
(By the way, the above site was my gift for my son on his fourth birthday)
The maintenance work has been completed. The datastore was in a read-only period between 12:02-12:45 for a total of 43 minutes. Some users reported two periods of 500 errors unrelated to datastore writes, but these issues should no longer be occurring.
On Nov 6, 12:01 pm, App Engine Downtime Notify <[email protected]>
AWS is very close to a de-facto standard now.
Edit: I mean app engine, the hosted web service. Not the app store. Thanks.
I guess you are confusing the Google App Engine's App Gallery : http://appgallery.appspot.com/, which has been removed for some reasons and the Google Apps Marketplace : http://www.google.com/enterprise/marketplace, which is the actual App Store, where you could pay and install apps.
App Gallery was just a GAE application show-case center, and most likely was causing confusion to Google Apps customers too.
I am still a developer who trusts GAE (my app was on the appgallery and is also listed on marketplace) and believe that they will continue to compete for developer mind-share, as they proved yet again with this performance update.
Let go of the fear, uncertainty & doubt and hop on to try an app. You will like it too.
http://appgallery.appspot.com/results?topapps=true
Also, I guess Google folks would post about the performance boost later.