1) It's not fair at all to say that Google "continue to ignore the two critical issues of uptime and data store latency". The HR datastore is specifically designed to address the concerns about variable latency of datastore operations, and to prevent both planned and unplanned downtime.

2) In my experience, high CPU cost for datastore operations is generally tied to things like having large numbers of indexes, or doing queries that don't have the right indexes. Each index written involves a bigtable write, if you are doing hundreds of these per entity it can become very expensive. That said, in general it isn't easy enough to know when you are doing something that will be expensive.

3) The bulkloader is definitely painful to use if you aren't familiar with it. In particular, casting schemaless data to a format such as CSV is hard because of the problem that rows can have unexpected keys. An improved "Bulk Datastore Import and Export tool" is on the 6-month roadmap for the product.

Can you talk about how big some of the larger customer apps are and how much traffic/uers they serve?

It would be easier for ppl in a position like mine to make decisions about AppEngine if there were big examples/case studies to point to in the same way Amazon, Heroku, Rackspace etc. do

I'd love to hear more about tweaks to existing indexes that can alleviate these sorts of problems.

Six month roadmap?

The 's~' prefix as an internal detail isn't going to go away, but we're trying to smooth out any of the situations where you have to see/deal with it.

I somewhat disagree with the author about scalability. There is a very narrow sweet spot of apps for which App Engine is a quite natural solution. If you're in the sweet spot, you'll probably scale well without too much up-front engineering investment.

The sweet spot _is_ narrow, though. For example, as the OP states: geo data doesn't really belong on GAE -- you can do a limited set of bounding box/proximity queries with the third party geomodel library, but wow, it's expensive and dog slow!

I do agree with the author about both the status dashboard (it only sometimes reflects my current experience with the system) and the surprising variation it data store latency. Latency has been much improved of late with 1.4 and beyond.

http://www.youtube.com/watch?v=ofhEyDBpngM

we still haven't seen any of these improvements. in fact, they took it off the roadmap without any explanation.

1) Extremely simple or very flattened data-model 2) Few writes, TONs of reads (I peak at 150 requests/second every day) 3) The occasional DeadlineExceededError causes me little or no headache. For some people this would be frustrating.

Also AE is awesome as a CDN. The latency is very tolerable for static assets.

Isn't the point of GAE is that it scales (as long as you don't do stupid queries)?

If we have to put everything in memcached, what's the point of using GAE?

I also don't quite understand the push of using memcached for almost everything (especially for young startups). How do you handle data integrity? I'm guessing most data models of young startups are fairly simple and only contain at most 10 models with almost no relationship? Otherwise data integrity is painful.

That doesn't mean you will have a fast site, though.

On GAE the datastore is pretty slow (it's a lot better now, but it used to be terrible), so a common pattern it to use a read-though cache to improve performance.

I wrote a thing that touched on this a couple of months back: http://nicklothian.com/blog/2010/11/23/a-pragmatic-approach-...

I heard stories where some people that use Spring MVC would have their request fails because the call stack is too deep.

The worst offender is the data access libraries. JPA is bad, JDP slightly better, but something like Objectify works really well.

Typically, Spring isn't the problem (I guess it could be if it is doing a lot of classpath scanning stuff or something though)

The point in question is why are you doing this on a new application? The sort of caching you describe is something you put in place after your site starts seeing a lot of traffic, and running queries for every request start to slow things down to the point where optimizing them doesn't help.

If you're designing your app well and running it on a solid stack, that should be something you need to worry about in year 3, after you're being TechCrunched on a regular basis. In the Rails/Django/AppEngine world, it seems to be the case that you need to resort to that level of caching just to see regular, day to day, 100 request/second traffic.

It raises red flags to see that.

  from google.appengine.api import memcache
  from google.appengine.ext import db

  class User(db.Model):
    email = db.StringProperty()
    password = db.StringProperty()
    sessionKey = db.StringProperty()

    def userKey(self):
      return "User.session="+self.sessionKey

    def put(self):
      super(User, self).put()
      memcache.delete(self.userKey())
      memcache.add(self.userKey(), self, 600)

    def getCurrent(self, request):
      user = memcache.get(self.userKey())
      if user is None:
        user = User.all().filter('sessionKey = ', request.header.cookie.get('user'))
        memcache.add(self.userKey(), user, 600)
      return user

The thing is, if you turn it on right away you never get a chance to see and fix a bunch of low hanging fruit optimizations that can really help you out. You don't know about them until you start to run up against scaling issues despite all your caching. Once you're there, it's not any fun to try to go back, find and fix those original bottlenecks, so you don't have much option but to start throwing hardware at it.

If, on the other hand you hold off and handle your first few scaling episodes by making your queries and code faster, only adding caching after you're not seeing returns from that other stuff anymore, you'll be able to go a lot farther before you have to start adding hardware.

  @Cached
  public class MyEntity {
      @Id Long id;
      ...
  }

That box is running about 20 sites, some big, some small, usually no more than one of them getting TechCrunched/Redditted at a time.

Here's a writeup of one particularly heavy day in the life of that box:

http://expatsoftware.com/articles/2008/03/6-million-hits-day...

Should Python programmers write the unofficial guide to migrating off of heroku?

Clearly, not every web app is a good candidate for GAE.

I have found objectify-appengine to be nicer to work with than the official Java data store APIs and I think it helps minimize loading request times.

I never used it, but it sure looks interesting if your application doesn't fit GAE anymore or you want to make specific infrastructural adjustments. They support multiple different database/http/.. servers too.

Those are defensible positions in the first 10 minute impression of app engine, but building a simple app (and testing it with GAEUnit) should put both to rest immediately.

Yes, the Datastore is unreliable, but a newer, more reliable version has been released.

App engine is still my preferred platform of choice -- just waiting for ssl, naked domains, and per-entity-group selection of which datastore service level to use.

One thing I worry about is the reported downtime. I'm not sure whether or not that will affect a static site.

(knowing of course that we're more or less tied to Google's infrastructure and the GAE way of doing things)

If you know what you're building and you want to scale it using the sort of tools AE provides -- i.e. the datastore, taskqueue, etc. all fits your app well -- it's quite good. And quite cheap; it's not really fair to compare per-resource pricing to AWS because on AE you'll only pay for what you use instead of paying for idle time on a server instance. But it definitely constrains what you can do, and also locks you to a single hosting provider. If you're still exploring what you want to do, that level of design and vendor lock-in can be a pretty severe liability.

In the real world however, requirements change: * You come up with a new idea that requires a certain library. Chances are, the library won't work on GAE out of the box. * You find out that you need to change your schema. It is pretty hard to update to the new schema while keeping everything in sync.

Finally, you pay the Google cost. When Google implements a new feature, they spend enormous time making sure that it scales well. They need to do so since they could be looking at millions of users on day 1. Most of us however, are looking to build something as cheap as we can, not knowing whether anyone is going to bother to look at it. However, you have to do the same performance optimizations that Google has to do so that your app scales. Chances are, it will be wasted effort - unless your objective is to just learn. I find it funny that GAE goes completely against the rule that "Premature Optimization is the root of all evil". Yes, you should think about your application's scalability. But your bigger problem should be about finding traction, and being able to react fast, not optimize for millions of views.

At a high load (50+ requests/sec), we saw database timeouts tens of times per minute, and often more (every request in a 10-60 sec period would fail).

That's not a terribly high load by any modern standard, you can easily service that on one or two fairly commodity machines these days.

Assuming the point of GAE is scalability, this issue didn't make any sense to me. One would assume that Google would simply load balance requests out to replicated instances of your app and that those apps would in turn access Google's vaunted super scalable data services. Thousands of requests a second shouldn't even phase it.

I've noticed that response times can be highly variable, but so far (we're pre-launch) I haven't noticed anything just simply not getting returned or the kinds of slowness he's describing. Without more detail, something else must be going on here.