ORM's aren't a problem at all as long as you have the ability to override problematic queries with named queries, etc.
ORM's can provide very real advantages when it comes to caching, development time, etc., as long as you review what the ORM is doing and notice when it's doing it wrong.
I've just spent 2 years architecting a high transaction global video game system using an ORM, and it worked well. In our case, the ORM provided acceptable SQL for about 85% of the queries, and we overrode the rest.
The ability to quickly and easily allow the developers to write their own SQL, to be reviewed later by a DBA, was a life saver. Combine that with our stress and load testing, it was easy to see where the hot spots were and deal with them effectively.
The problem comes from people who rely on the ORM to do everything for them without truly understanding how it works.
ORM's, like anything, are a tool, and there is a time and place for them.
In both cases, we had a large number of smart developers who we empowered with the use of an ORM; they understood the domain model, and they didn't have to worry about waiting for a "DB type" to write stored procedures, or develop a data model, etc. As a matter of fact, in both cases, I was the only DBA on the project, and it was a predominately part-time role. We'd meet, ensure we were all on the same page with the object/data model, and then they'd go and build it. The developers were able to immediately build and run and test and integrate something that was functional and operational, when they needed it. This was HUGE, and something that most people don't properly appreciate. Timelines were already insane enough as it was, the last thing we needed to do was artificially constrain ourselves by waiting for other (db) devs before work could go on. Especially when requirements had the potential to change from one day to the next.
In both situations, we took advantage of very, very sophisticated testing procedures that would happen nightly, both functional and stress/load, and it pointed us at the bottlenecks of each nightly build that would require tuning and investigation. We intentionally set up our testing to be able to monitor and test the effectiveness of the ORM, and to point it out when it didn't work efficiently. The devs would do the majority of the heavy lifting with the initial data model, and the results would be tested, reviewed, and then modified if required. The performance modifications were not a lot of effort to fix, either. Usually it was a very slight data model change, or using a named query to take advantage of a database-specific features. And CLOBS. Every database seems to handle them differently, so we had to hack some solutions.
Having done large scale database development for almost 25 years, using the classic stored procedure approach and the ORM approach, I'll say again that ORM's are a great solution for certain projects with the right staff, and aren't a crutch or some lazy choice if used properly.
Separating your data access code out of the application logic also allows you to change it much more easily as data conditions change, including on the fly, without an application deployment. That's often extremely useful.
MySpace scale may be at an extreme end of the spectrum, but we had formidable hardware to throw at it too (although x86, so nothing TOO crazy). So I think the ratio of hardware to scale at other sites is comparable, and so I think the same lessons apply. I have no experience working with oracle, but would you say that a 7 node oracle cluster is some pretty serious hardware? I myself really don't know, but it is a question I have :).
EDIT: I'm not discounting your experience, i just want to point out that i've experienced conditions where I think the orm approach would have broken down. If others have had different experiences, the more data points the better, but i think the scale/complexity/cost(hw) ratios play into the debate as well.
EDIT #2: Oh and I forgot to mention that the automated test suite you had is an incredible asset, and no doubt made it easier to discover problems early and deal with them effectively. But you do have to invest resources in creating one, and something like that is no small cost at a start up.
I was just pointing out that ORM's are indeed quite effective in online systems that are more complex than a blogging site.
If you're going to say "no, don't use it", based on a development situation that is very much an outlier (MySpace), and use that experience to discount it for any but trivial use, then I'm not sure what to say.
They can and do offer real-world advantages with minimal downside if you treat them like any other tool, and not use them blindly, in reasonably complex and large systems, as I've tried to demonstrate.
As to your environment, the data requirements were quite different than ours. Our systems were more like online banking systems; very much an even split of fast writes and reads, transactionally bound to third party systems (in-game payment, in-game "real time" use of consumables, etc), real-time analytics for fraud detection, etc. We were very much high IO, and our caching opportunitites were few and far between.
And in our environment, we HAD to have sophisticated testing. I ensured that the stress and load testing was done so that we could directly simulate the load of our expected user base, with realistic profiles, in order to better engineer our databases and disk IO. It also allowed us to measure the impacts of feature additions, etc. If it failed in Production, it made the news, and we had millions of gamer-freaks bitching everywhere.
In my case, the middle-tier was not an issue... we enabled minimal caching on a per-box basis, and other than that, they were stateless, and we could add/remove them at will; the application WAS the database.
And you can still abstract various parts of the database while using an ORM. We did write a few special stored procedures, and used some forced query plans, views, etc., to tweak the performance.
And yes, Oracle can scale out quite well. Cache Fusion, high speed and low latency interconnects, and shared block access provides incredible scaling without having to do anything special in the middle tier.
Because people have been running massive systems on Oracle, DB2, Sybase, TeraData etc for years now.
I assume (among this crowd, anyway) that scaling out is more desirable vs. scaling up because the majority of the hardware costs are variable, whereas scaling up requires a step function of large cash investments that startups often can't afford.
Do those massive systems on Oracle etc scale out, or simply scale up with expensive hardware?
Both. If you still need ACID guarantees and want hundreds of thousands (or even millions, if necessary) of TPM, you will need to pay the piper.
No you're not. Look at ActiveRecord, it lets you drop to any level of SQL optimization you need. In ActiveRecord 3 with ARel queries are composable, allowing lazy loading and the breaking of queries into appropriate locations according to your code architecture.
I can't speak to other ORMs, maybe they really are as bad as your opinion would indicate, but I suspect what you're really complaining about people who don't know how SQL works being enabled to write horrible data persistence code by ORMs with a pretty facade. That's a legitimate problem, but the fact that a tool can be abused is not an argument against the tool itself. We'll never build anything great if we are driven primarily by what the ignorant will do with it, after all, every single person on the planet is ignorant of most things, our tool development should be driven by what they enable experts to do.
Here's what you're missing, and why ActiveRecord works: if you halfway know what you're doing with a RDBMS, 95-99% of the time the query generation just saves you writing a lot of boilerplate SQL. It's true that sometimes you have to drop to a lower level to hand-craft a query, but ActiveRecord in no way prevents that. Again, I don't know what kind of ORM hell you have been put through, but I assure you that an ORM does not need to be this horrendous performance killing black box that you think it is.
I understand where you're coming from and what you describe may be workable in a smaller company with 7-14 devs where everyone knows what they are doing and understands well what happens under the hood. I think it's less likely to work at a company with 50+ devs though where you inevitably start trusting people less, or just at a company where you don't trust everyone. I've worked at both types. There is also the question of the complexity of your data and the way you need to query it. Right now we do essentially a ton of graph queries that we optimize highly in sql (ends up working much faster than any graph database since the schema and the queries are optimized for the exact data we are working on). Some of the functions that I write for this would not be implementable in an orm. I suppose that could be the case where you drop down into raw sql, but that happens to be a fair chunk of our code.
Maybe you can make it work better than I'm expecting, but if you were starting from scratch would you really want to go down that path anyway, all things considered? My original argument was that you are better off choosing a different way. I suppose that point of view will be difficult to change for me :).
I get the impression that you don't really understand the complexity and capabilities of a modern, robust ORM, or how it can be used.
And you're working on a data warehouse, which isn't usually a viable candidate for ORM in the first place.
And i'm not working at a data warehouse.. why do you think that?
users = User.where(:age => 10) # no rows fetched
# Once you access something on users, then the query happens users.each do |user| # something end
It's not really that "magic", it's useful.
ActiveRecord uses joins (mostly). If you are using a "non-standard" table schema, you can tell ActiveRecord what column to use on joins.
The disadvantage is that due to some organizational dysfunction at MSFT there's still no really satisfactory ORM infrastructure surrounding the query engines.
(as for your "misguided engineering idea in itself" claim, I don't really see how it's fundamentally different from writing SQL in the first place to be translated by the database into query execution plans, vs. writing the query execution plans directly).
As for checking for type safety, I think frameworks that do sql-to-object mapping (with type safety), and also handle cache for you, are a very useful thing. Making raw calls on database connections is definitely too far "in the other direction" :).
I don't have much experience with ORMs or mapping frameworks other than LINQ-based ones, but it seems like it would be pretty difficult to typecheck queries expressed as SQL strings, at least dynamic ones, at compile time. Do the frameworks you mention typecheck the actual query itself at compile time, or do they just check at runtime that the data returned from the query matches what you want?
Then explain the massive success of Rails. Quite simply, you are wrong.
I don't think they're necessarily misguided. DataMapper made efforts to circumvent the N+1 problem, in most cases probably pretty effectively.
Partial Updates are also pretty easy. Slamming every field into every INSERT/UPDATE is obviously a bad idea.
I think the missing sauce for ORMs is funding. Getting the basics together takes time and money, and it's hard to pull off in your free time.
On the other hand, having written many an ORM, I think there's still plenty of room to advance the state of the art. One of the biggest untapped (AFAIK) opportunities is using Statistics for query tuning. It's the life-blood of databases, but statistics are noticeably lacking in ORMs. Even simple counters could allow you to tune lazy-loads, JOINs, pre-fetches, etc on-the-fly.
I would go so far as to say that sql writing ORMs are a deeply misguided engineering idea in and of itself, not just badly implemented in its current incarnations. You can't possibly write data access logic entirely in your front end and expect some system to magically create and query a data store for you in the best or even close to the best way.
I think the real reason people use ORMs is because they don't have someone at the company that can actually competently operate a sql database, and at any company of a decent size traffic-wise that's simply a fatal mistake. Unless you are going 100% nosql, at which point this discussion is irrelevant.