On its own, the code representation is not an improvement. Where it shines is when you want to compose result sets.

For example, in ActiveRecord (the Ruby library that this looks very much influenced by), given:

  posts = Post.where(user_id: 1)

...you can now do things like:

  recent_posts = posts.order(created_at: :desc).limit(100)

or

  tagged = recent_posts.where('tag in ?', ['hacker', 'news'])

and then you can extract data:

  tagged.group_by('tag')

or do mutate it:

  tagged.update_attributes({author_id: 2})

or

  tagged.destroy_all

...And so on. By encapsulating a result set as something that can be augmented with new query clauses (where, limit, select) and so on, you can incrementally build queries, pass them to other functions, store them as member variables so that they be reused as "scopes" across multiple calls, and so on.

If the query were just a string, this sort of thing becomes awkward, verbose, brittle, and generally not type-safe.

The problem with this approach is that you are forced to work in a subset of SQL due to the type system mismatch of SQL and the host language.

A "proper" solution to this problem requires language changes (as Microsoft did with C# 3.0).

While it's true that solving this at the language level would be even better, it's not required. All you need is an extensible AST data model.

For example, ActiveRecord doesn't have support for window functions, but its underlying AST (a library called ARel) allows you to extend it, which at one point I did, allowing me to write something like this:

  things = Arel::Table.new('things')
  things.project(Arel::Nodes::NamedFunction.new(:sum, [things.count]).over('my_window')).
    window('my_window').
      order(things[:name]).
      range(Arel::Nodes::Preceding.new)

which generates something like:

  select sum(count(*)) over my_window
  from things
  window my_window as (order by name rows unbounded preceding)

(It's been a while since I looked at this particular code, which is no longer in production use, so it might not be correct.)

In other words, I could continue to express whatever I wanted of SQL, at the language level — no type mismatches.

The SQL builder code isn't pretty, to be sure, but it's something you typically encapsulate into a lower-level module rather than writing in application code. And as it's highly analogous to the SQL code it generates, it's not like it's that much more complicated. (ARel isn't a shining example of how to do it right, either. I expect Diesel implements the AST aspect more cleanly.)

Edit: I see what you're saying about typed record types. I was thinking more about the grammar, not the shape of the data.

Yes I should have been more clear that I was talking about a static typing context. In a dynamic language the problem is much easier, as you are basically only limited by your languages metaprogramming facilities.

But isn't the same true for a statically typed language with sufficiently powerful macros? I don't know if Rust falls into that category, having never used it, but I don't think it makes a difference whether the metaprogramming happens during compile time or runtime.

Yeah, if they're talking about what I think they're talking about, yes, something like LINQ could be implemented with macros (compiler plugin variant, I think) in Rust.

> basically only limited by your languages metaprogramming facilities

I'd like to point out that strong static-typing does not preclude powerful metaprogramming! Nim's macro system is extremely powerful because of it's great interplay between it and the type system.

http://nim-lang.org/docs/manual.html#macros

You should take a closer look at what Diesel is doing with it's query builder. We actually do attempt to support the full power of every supported backend (of course there's still holes which need to be filled), with proper type guarantees which map to the semantics of Postgres. The main difference being that Diesel can check your queries for correctness at compile time.

The problem is that joins and projections give rise to new record types. The only way to support this in a type safe and natural way is if you have anonymous records in the host language, which Rust doesn't. On top of this there is some other metaprogramming stuff that is necessary in order to facilitate the DSL embedding.

Look at what Microsoft did with LINQ. You can't pull that off in Rust no matter how much you want to.

This can be accomplished with procedural macros.

This can be accomplished without procedural macros, too. ;)

Nevertheless, the goal here is to build an ORM for Rust. Why waste time describing what you think it can't do?

SQLAlchemy is a Python ORM that matches SQL almost 1:1.

The main reasons I like this approach are:

* it makes way easier (and safer) to reuse query parts compared to string composition/interpolation

* the ORM takes care of differences between the backends, so it is easier if I have to port the application to another RDBMS.

That's normally done in SQL with JOIN, or in some cases, temporary tables.

The ORM model of building a query dynamically over multiple steps in multiple contexts gives you a lot more flexibility and more resilient code. You can share code that builds queries for multiple tables that share some, but not all, fields or relation patterns much more easily. And you can do it all without making multiple calls to the database.

See Sandi Metz' recent piece, The Wrong Abstraction [1]. I prefer a SQL template approach such as Yesql [2]. The trouble with an ORM like ActiveRecord is a lack of control of when and how the query is performed.

1. http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstractio...

2. https://github.com/krisajenkins/yesql

That's a problem with ActiveRecord and not with ORMs in general. Plenty of ORMs are explicit about how and when queries are performed. Check out SQLAlchemy for an example.

In that very specific example, you are right -- the benefits are pretty minimal (mostly just the deserialization). However, you're actually able to represent the relationship between those two models however you want, but still deal with relationships between types, and not what the exact join clause is.

Additionally, SQL is extremely hard to compose. In my applications, I very rarely have some specific one off query that I want to run, I have multiple predicates that I want to be able to mix and match easily. String munging is fragile, while this actually lets me build reasonable domain abstractions.

To your final point "Unless this has some huge performance gain over rust-postgres"

Diesel performs 33% faster than rust-postgres, and will be competitive with if not faster than ideomatic C.

> Additionally, SQL is extremely hard to compose.

I've seen almost everyone who doesn't use SQL directly say this but I always feel it's the exact opposite. SQL is easy to write once you do it enough. I've gotten to the point where I compose zero SQL in any application's code and, instead, do it all inside of stored procedures (of which they do not generate dynamic SQL). Then my code just calls stored procedures, my database users for the web application never have more access than executing a specific subset of stored procedures, and if there is a bug found in a stored procedure I can deploy them separately from the web application.

I'm guessing it's just different philosophies but almost every time I've used something that abstracts an existing interface to another, completely different interface, I always end up running into major issues.

"Compose" here doesn't mean "to write". It means to build up a database query in multiple steps, gradually adding (or removing) restrictions or relations dynamically in response to input. So one method might add a certain restriction that will get translated to a part of the where clause. Another might change which fields are being selected for or joined on or add transformations to the field. Meanwhile, the developer doesn't have to worry about special cases, or whether you need an AND or an OR, or how to go about wrapping a field in the SELECT clause after the fact. These methods might be in totally different contexts in the underlying code, and it would not be possible to make that style of query composition possible without abstracting away the SQL syntax. And that's exactly what an ORM does for you.

Oh I know what compose means. Having written a ton of stored procedures for large scale web applications I've never found the need to dynamically build such queries. Instead I focus on what I need a specific query to do, provide the options to a stored procedure or function (depending on the RDMS) and that's it. Much like unix with simple programs that do very niche things I look at procedures much the same way.

Don't get me wrong I see the simplicity in it from the ORM side I've just never had a good experience with ORM tools. Having worked directly against many RDMS and key value stores at least for me I don't see much value in them.

But anything that helps get a product out the door more power to ya, in my opinion.

What do you do when you want e.g. user controlled sorting and filtering?

I just implemented this recently in my job, but it was even tougher than that; I was joining dynamically created tables, and the number of joins depended on criteria. I ended up building my own AST to represent filters, and compiling to SQL. Every extra table join hurts performance - typically filters are implented as joins against a filtered sorted limited subquery returning ids, so fewer different table joins means less data to pull in when evaluating filters even when all the data is needed for the results page.

Point being, I had to compose complex dynamic queries and SQL was not a particularly pleasant target language to target directly. Abstractions (like asts) help with composition.

You can use a CASE-statement for each possible filter and apply it only if the filter value(s) are non-trivial.

    -- Parameter :filter
    --
    AND CASE WHEN :filter IS NOT NULL THEN
        table.column = :filter
    ELSE TRUE END

This has zero cost in the case where :filter is unset.

The filters are predicates, not values.

> What do you do when you want e.g. user controlled sorting and filtering?

Sorting is easy. The stored procedure just accepts a sorting flag, add it into your query and your done. Filtering though, I could think of a dozen types of things you may want to filter on and I still may not hit your use case so it's hard to say how to do it based on what data is stored in the database, how it's stored and which RDMS you're using.

I've primarily done this type of work in MS SQL where I've built some pretty complex queries but I was always able to replace dynamically generated SQL with SQL that could generate a cacheable query plan.

Not saying there won't be edge cases and I don't know how your filtering worked so I don't have any type of satisfying answer for you. I would just be surprised if it wasn't doable somehow (I've seen some crazy shit in SQL queries :D)

The job I have in mind was basically implementing a subset of Excel's sorting and filtering capabilities into a paginated table view on a SPA web page.

Sorting has a couple of wrinkles. When Excel sorts rows, it changes the in-memory order; its sort is also stable. That means if you sort by column C, then B, then A, you're effectively doing "order by a, b, c". Seems simple enough. But that's not the only wrinkle in this problem.

Not all sortable columns are in the same table. If you do something naive like "select x.* , y.* from x join y [...] where [...] order by x.a, y.b limit 10 offset 20000", you're doing too much work - you're sorting a wide result set. It's normally better to do your pagination in a subquery and join against it, like this:

    select x.*, y.* from x join y [...] join (
      select x.id from x join y [...] where [...]
      order by x.a, y.b limit 10 offset 20000
    ) f on f.id = x.id
    order by x.a, y.b

Depending on the body of the where clause, you want to join the minimum set of tables to include just enough data to evaluate the page of ids - every join has a cost.

The body of the where clause is another story, it's pretty much entirely arbitrary. The user may choose to filter on any column, just like Excel autofilter. Creating good indexes up front isn't feasible - you don't know how many columns to include in any one index, they take time to create and they're not free in disk storage either. And the filtered columns span multiple tables; the final result set may contain several hundred columns across all these tables.

So what I did is take the criteria - the filter clause, which might look something like this:

    (and (or (eq (field 'x.f1') "foo")
             (eq (field 'y.f2') "bar"))
         (between (field 'm.timestamp') "date1" "date2"))

And turn it into a where clause. I analyze the filter tree to determine the minimum set of joins required to access all the fields needed by the filter and sorting criteria. I can then put that into a subquery to join against.

There are more wrinkles. To give a good experience, we need to populate a dropdown of potential filter values, like Excel autofilter. Potential filter values need to be filtered themselves - we only display filter values for that are potentially visible given the current filter. They also need to be searchable, because there may be millions of them. And they may include joins, like when you want to filter by tags or some other 1:N style relation, like row:tags. There are a bunch of different shapes to the different queries, but there are common composable underlyers to them.

All the above runs on mostly unindexed tables - we're relying on pure query compute performance within the designed data limits (less than 3 million rows). Sub-millisecond query response times aren't the goal here; we're supporting the user doing reasonably complex data manipulation operations with worst case performance measured in single-digit seconds.

(This is all in MySQL, because reasons that include business model constraints.)

So you don't even build dynamic queries in the stored procedures themselves (e.g. with string concatenations and EXECUTE)?

That's where I usually get fed up with stored procedures, as pseudo-ADA ain't a nice language for list and string manipulation. I'd rather do that in a language better suited, and that quickly leads to query builders…

Stored procedures are no more composeable than SQL strings are. Now it's certainly easy to say that I haven't used SQL directly. But I did spend years working that way (though of course this statement shouldn't give you much confidence by itself).

Ultimately an ORM can help you to build more maintainable and resilient (to changing market conditions) software than direct SQL. Diesel in particular can catch mistakes that SQL strings cannot.

I do understand why a lot of people have reservations about ORMs. I share a lot of them. I've maintained Active Record for about 2 years now, and learned a lot from doing so (note: This does not mean that I like AR or want this to be anything like it).

TL;DR: Your points aren't wrong. Diesel is different. Maybe give it a shot. ;)

> Stored procedures are no more composeable than SQL strings are[...]Ultimately an ORM can help you to build more maintainable and resilient (to changing market conditions) software than direct SQL.

I couldn't disagree more. Creating the separation between SQL DB and web application allows you to maintain, deploy, fix and upgrade things independently and it's easy (maybe not quite as easy as an ORM would make it but not really harder either; you're going to have to know how SQL works anyway if you want to effectively optimize and secure your data).

Not only that but you can provide far better security controls because you can limit certain users access to certain procedures, never direct access to tables and various other things and if you don't do any dynamic SQL in the procedures itself you not only never have to worry about SQL injection but unless there is a critical vulnerability in the SQL system itself it's going to be impossible to conduct an injection attack.

Yeah you can separate your code to make it easier to deploy separately to get a similar benefit and yeah your code can prevent SQL injections too but it's so much more work, more possible points of failure and the differences between databases can really bite you in the ass when you do everything generically. SQL is fun :)

> TL;DR: Your points aren't wrong. Diesel is different. Maybe give it a shot. ;)

Yeah I'm not saying NEVER use ORMs; they're great for creating things quickly that don't need to scale. But after my experience with them I'm highly skeptical using them outside of that type of use case. Of course you can scale ORMs but I've never seen it done well.

I think we're violently agreeing in a lot of ways. :)

I'm curious what you mean by scale. Are you referring to performance, or code size? If you're referring to performance, you'll probably find it interesting that we out-perform or are competitive with ideomatic C.

By scale I mean performance though I'm not sure the application's language matters as much as the SQL that's being generated and run against the database. I've just never had good luck with ORMs generating the best, most optimized SQL for the query I'm trying to run and I've known a few people who used an ORM until they started getting heavy traffic, tested out talking directly to the RDMS, found it to be more performant and end up dropping the ORM.

Don't get me wrong ORMs can certainly be performant. I just haven't seen or know anyone who used it in a manner that scaled to thousands of concurrent users and, at the same time, didn't have a lot of pain trying to do that with the ORM.

I think this conversation shows a mismatch between two different philosophies about how to use the database. The traditional way of doing things you've described involves multiple different applications connecting to the database. Data access is restricted by locking their database users down to specific APIs exposed by stored procedures. Non-trivial logic lives in the database in order to present these APIs.

A lot of systems lately are being built in a different way, that is, only one application connects to the database. This application presents an API (ReST, protobufs, thrift) for other apps that need that data. It handles access control and business logic, using the DB as a backing store. That means you don't have to worry about dababase user access control, because you don't have more than one user.

I don't have a strong opinion about either of these, but the latter way of working is how we do things at my current job (reasonably large payments software company), and it does work quite well. One advantage I appreciate is that there isn't arbitrary business logic sitting in the database where it can be changed on the fly - I wouldn't consider that a feature.

I'll make a stronger statement - stored procedures are generally not composable in production.

At least in Postgres they act as an optimization fence. In practice this means you can use them as an abstraction layer only in very limited ways where they happen to fit into a decent execution plan.

SQL is hard to programatically compose. At the very least, you have to use a Lang/SQL.

That's a new term to me - lang/SQL?

I guess he means a procedural language for sql, like PL/Pyton, PL/V8, etc., so PL/

Oh god, ick. I've used T-SQL and Pl/SQL procedurally and hate them with the burning passion of many suns.

I think somebody needs to actually justify the need for composable SQL. With examples. It's far from obvious.

In my experience outside of extension code the need for composable SQL is very limited. Most uses cases can be satisfied by Rails 2 scopes.

Now, if composability was free - sure it's great. But it's not free.

So, on one hand:

* it would be nice to have infinitely composable SQL

On the other hand:

* it would be nice to just have the actual SQL with all the features, well understood syntax and semantics that cost decades and billions of dollars to develop

* and no wrapper-on-top-of-SQL limitations and translation bugs, and need for training

With these language-on-top-of-SQL you don't get both. In fact you get severe compromises in the second group.

User-generated data in a user-defined schema, where the user wants an Excel-autofilter-like sorting and filtering experience over rowsets that range from 10 thousand to 3 million rows, and 2 columns to 150 columns.

Columns may be sorted and filtered individually or in combination. Filters may be based on value set membership or range criteria, comparisons with fields in the data or via a join to other tables, or based on the presence of an entry in a join table (think: tags or labels, and you want to filter by presence or absence of tag).

(This is what I spent the past couple of months implementing. The simplest way of composing SQL is via union / except / intersect, but it's also the worst performing; doesn't work great past a few hundred thousand rows. And you still need to compose together the filter conditions into a where clause.)

Rails 2 scopes are literally the case for composeability...

Yes, but a very limited one.

I'd love to see a solution with tradeoffs made the other way:

  * syntax and semantics 99% like underlying SQL
  * support for 99% of underlying DB functionality
  * traditional areas where ORMs add value:
    * deserialization into convenient data structures
    * casting, quoting and escaping on the way in and out
    * using stored association metadata for concise joins in the query

Every single one of those bullet points is a primary design goal of Diesel (caveat: Our semantics generally map to PG, because they apply well to other backends and are the least nonsensical)

   Diesel performs 33% faster than rust-postgres, and will be
   competitive with if not faster than ideomatic C.

Interesting. What makes it faster than rust-postgress?

Short version: Our compile time guarantees allow us to omit a lot of runtime checks that you would otherwise have.

Long version: http://bikeshed.fm/49

Longer version: Planning on doing a really in depth write up for the website soon. I'll also be going into more detail during my talk at the SF rust meetup, which there's a link to the live stream towards the bottom of the thread

I'm interested if some of those guarantees can be used upstream in rust-postgres?

As far as I saw, Diesel is essentially shipping with its own PostgreSQL driver.

It's possible. I tried to build on rust-postgres originally, but its design basically makes it impossible to abstract over. We also have fundamentally different views on how `ToSql` should work.

I'd love to find out what made it impossible to abstract over?

Please include in your larger talk!

The short version is its refusal to box things means that you have to have the connection, the statement, and the cursor all as local variables in the same lexical scope (you almost always want to abstract away the statement and cursor), and it's use of trait objects for serialization and deserialization. I might go into it briefly in my talk, but I don't like to rag on other people's work so I probably won't say much on it.

(Sorry for the double post, I think I replied to my own comment and not yours!)

And rusqlite (or libsqlite-3) doesn't have those issues?

I don't think you're ragging on rust-postgress, although I'd probably attempt to change rust-postgres first :P

> And rusqlite (or libsqlite-3) doesn't have those issues?

rusqlite has the same issues. libsqlite doesn't since it's just C bindings. Same as pq_sys. Ironically, rusqlite makes the same design choices as rust-postgres, but it's actually wrapping a heap allocated object anyway, so it gains nothing by doing so...

> I'd probably attempt to change rust-postgres first

I did, too. It turned out to be a pretty major rewrite.

Perhaps the question should be rather:

Why is

    Post::belonging_to(usre).load(&connection)

an improvement over

    SELECT * FROM posts WHERE usre_id = 1;

?

... to which the answer is probably more obvious.

Hah I love this answer. ^_^

>> it nails the database schema into your Rust code

On the contrary, the purpose of an ORM is to let your code be a schema for your database. This is my personal use case for an ORM; I want persistence but am not overly concerned with the database. There are plenty of use cases [1] where a schema update from outside is unlikely or impossible. In these cases, there is a very low cost for (by way of an ORM) tightly coupling your code to your schema.

Also, an ORM lets somebody else worry about sanitisation, and database specific-code. It can abstract away the choice of database, another very valuable feature. I think that the example you quote is actually a poor demonstration of the value of an ORM. The other ones on the linked page are far better :)

[1] A program running on only one machine, like an Android app or a music player on Linux desktops etc

If you change your schema without changing your client code, your program is broken - you just haven't found out about it yet. The fact that changing schema becomes a compile-time error with an ORM is a feature, not a bug.

Well for one thing the "SELECT *" query will typically yield for results a runtime-typed flat list of fields/values as accessed via a generic database access api.

Whereas as an applications programmer I'd rather deal with structures that make sense in my domain of interest, and let the marshalling code from SQL results be handled for me, and with compile-time checking for correctness. When that's coupled with a database schema reverse engineering tool (not sure if this project provides one), that way whenever the database changes, compile errors will result, which errors allow me to quickly hunt down places in my code that are affected and need changing.

For one thing, I believe SELECT * is generally considered harmful (I certainly view it as such), and without wildcard you would need to update the query regardless of the method you use to construct it.

Yeah, I've been wondering if I should note somewhere in the examples that we don't actually execute `SELECT *`, we do something more reliable. I felt like listing every column would bury the lede though.

Sorry, I wasn't at all criticizing the example, I was replying to the parent poster. Your library looks awesome :)

Oh I know, I didn't think you were. Just stating a random thought I've had about the examples on the site. XD

I generally agree. Are there frameworks or db access libraries that just offer some helpers around plain ole SQL? Seems like very framework/ORM invents it's own query DSL.

In the .NET world we have Dapper (https://github.com/StackExchange/dapper-dot-net), which lets us write queries in plain old SQL. It's a very useful little library. I believe StackOverflow use it for their data access.

Example:

  var posts = connection.Query<Post>("select * from posts where user_id = @UserId", param: new { UserId = 1 });

I really like Dapper - allows you to write "raw" SQL but does the tedious bit of mapping to/from objects in a nice way.

In Java I really liked MyBatis for this, and I think fits what you are asking.

"If someone adds a new field to a row, you have to rebuild all your Rust programs that use the database."

Are you sure? Why do you say that?

Read the Diesel build process, especially the section on "migrations".[1]

[1] http://diesel.rs/guides/getting-started/

This isn't true at all. You can add as many fields as you want. You'll need to recompile to use them, of course.