News for hackers. Stuff that matters.
Re:The SQL filter clause: selective aggregates by d33 on Monday May 30, 2016 @09:40AM (#11801264)
It would be nice to see an example for this kind of query.

BTW, SQL seems like a terrible language. For example, why would it need to enforce ordering for keywords like ORDER BY, WHERE, GROUP BY? So many times I had made a mistake that boiled down to putting them in the correct order...

Re:The SQL filter clause: selective aggregates by MichaelBurge on Monday May 30, 2016 @09:45AM (#11801297)
One reason is to remind you of the SQL order-of-operations:

* The from clause retrieves records from data sources like set functions, tables, or views.

* The on/using clause joins tables

* The where clause filters out records

* The group by clause assigns records to buckets

* The having clause filters out records

* The select clause runs

* The order by sorts

* The limit clause picks records

In this query:

> select sum(1) over () from (select 1 a) x join (select 1 a) y using (a) where 1 = 1 group by true having true order by 1 limit 1;

The only out-of-place component is the select and window function(which runs after the group by clause). In a more complex query with multiple selects in scope, the order of terms helps remind you which parts are running in which order.

Note that the optimizer won't strictly enforce the order; it's just a guide to interpret what the results should be.

If you need to do something out-of-order, you can use a subselect:

> select * from (select sum(1) over () from (select 1 a) x join (select 1 a) y using (a) where 1 = 1 group by true having true limit 1 ) x order by 1;

Here, I've swapped the limit clause and the order-by clause, although you can still see the order of operations in the explain plan being separated by a 'Subquery scan'.

Re:The SQL filter clause: selective aggregates by kbenson on Monday May 30, 2016 @02:12PM (#11802390)
> The having clause filters out records

I think you mean "The having clause filters out buckets", right?

Re:The SQL filter clause: selective aggregates by jakobegger on Monday May 30, 2016 @01:07PM (#11802146)
Assume that you have a table 'orders' similar to the following:

    nr | date       | product | quantity
    ---+------------+---------+----------
     1 | 2016-05-29 | a       | 2
     2 | 2016-05-29 | a       | 7
     3 | 2016-05-29 | b       | 3
     4 | 2016-05-29 | a       | 1
     5 | 2016-05-30 | b       | 5
     6 | 2016-05-30 | a       | 2

And you want to get daily order totals for each product.

Then you do:

    SELECT
      date,
      SUM(quantity) FILTER(WHERE product='a') AS product_a,
      SUM(quantity) FILTER(WHERE product='b') AS product_b,
      SUM(quantity) AS total
    FROM orders
    GROUP BY date
    ORDER BY date
And the result is

    date       | product_a | product_b | total
    -----------+-----------+-----------+-------
    2016-05-29 | 10        | 3         | 13
    2016-05-30 | 3         | 5         | 8
Re:The SQL filter clause: selective aggregates by snissn on Monday May 30, 2016 @01:25PM (#11802230)
Thank you.
Re:The SQL filter clause: selective aggregates by return0 on Monday May 30, 2016 @09:59AM (#11801366)
> SQL seems like a terrible language.

The reason you give is too trivial to justify calling it "a terrible language"

Re:The SQL filter clause: selective aggregates by JustUhThought on Monday May 30, 2016 @10:51AM (#11801580)
E.F. Codd (author of the whitepaper that is the foundation of all RDBMSs, for which SQL was created to query) was of the opinion that the SQL was a terrible language. His reasoning for saying so are very clearly and well speelled out in several books and articles of his.
Re:The SQL filter clause: selective aggregates by mamcx on Monday May 30, 2016 @02:22PM (#11802421)
That reason is in fact a good one. The order is "wrong" from the point of view of a developer and maybe ok from the POV of the original intent of SQL: The end-user was supposed to run SQL against databases. Few ad-hoc queries, even complicated, can be managed. But using SQL all the time, and the language is clearly the wrong thing to talk to the DB (is like JS: Is what you have instead of what you want).

Also: The fact that everyone run (even if eventually fail) for a ORM instead of SQL, or the way people imagine NOSQL is "better" -is very telling is NO SQL- show this.

The relational model is so simple and powerfull, but SQL is not a good seller of the idea.

Re:The SQL filter clause: selective aggregates by bluejekyll on Monday May 30, 2016 @03:36PM (#11802717)
Is there another language agnostic option?

I suppose it would be interesting if you could use something like the llvm to target a DB's core features, but through your language of choice.

I haven't seen anything like that, if it exists. SQL is that standard API today, and ORMs provide the language specific implementation (though sometimes in ways that make it inobvious what the generated SQL will look like).

Re:The SQL filter clause: selective aggregates by pjungwir on Monday May 30, 2016 @04:12PM (#11802900)
I actually like SQL, but I'm glad people are willing to ask, "Can we do better?" I think for a working programmer, SQL is the only practical choice, but Tutorial D is an interesting attempt to improve (by C.J. Date, described at length in his books Database in Depth and The Third Manifesto), with an open source implementation called Rel. There is also a commercial database called Dataphor that uses D4, a cousin to Tutorial D. There is also a newer project called Andl that is another non-SQL relational query language.

Personally, although I'm not a researcher, I've become intrigued by ways we model change-over-time in relational databases. Snodgrass's book _Developing Time-Oriented Database Applications in SQL_ is a good entry point to all that, and more recently we have SQL2011's Temporal standard. I still have a lot to read here myself, but I am keen to see what happens in the next ten years in how the relational model can incorporate change.

Re:The SQL filter clause: selective aggregates by smt88 on Monday May 30, 2016 @06:15PM (#11803527)
SQL is a lot like JavaScript: a lot of people hate working with it, but it's the only practical choice, as you say.

It's interesting to note that transpiling to SQL is even more popular than the recent trend of transpiling to JavaScript. Every major language has a query builder (many of which end up with syntax similar to Andl). We're abstracting away from SQL and have been for a long time.

With that in mind, I wonder whether we need a new query language so desperately. SQL is so highly optimized in the major RDBMS that any new language would probably compile internally to SQL, meaning it has the same semantics (and is essentially sugar) OR it would be way behind SQL in terms of optimization.

Am I just totally off base here?

Re:The SQL filter clause: selective aggregates by pjungwir on Monday May 30, 2016 @06:37PM (#11803614)
After I wrote that comment I thought it might be worth submitting this Andl story to HN about Andl-on-Postgres: [0]. They write, "Relational expressions are converted into equivalent SQL and passed back to the Postgres SPI interface for execution," so you are right about transpiling. But the software world is more mature than it was 20 years ago, so it seems reasonable for a project to get a toehold by building on top of existing systems. That needn't limit its future growth necessarily. It sounds like Andl will work on more than just Postgres, so I can imagine them eventually building a "native" version too. Using existing RDBMSes is a pretty good way to get an MVP though. :-)

[0] https://news.ycombinator.com/item?id=11802917

Re:The SQL filter clause: selective aggregates by smt88 on Monday May 30, 2016 @07:37PM (#11803805)
> Relational expressions are converted into equivalent SQL and passed back to the Postgres SPI interface for execution

I actually don't think I like that approach.

If I transpile my own queries (meaning I use a query builder), then I can cache the generated SQL. There would be some placeholders for arguments, but the query itself doesn't change.

Andl's approach could add significant overhead in certain applications where lots of queries are being executed at once.

And then what happens to EXPLAIN support? Is it harder to debug an Andl query?

Re:The SQL filter clause: selective aggregates by MichaelGG on Monday May 30, 2016 @04:29PM (#11802997)
Wasn't QUEL a lot better?
Re:The SQL filter clause: selective aggregates by return0 on Monday May 30, 2016 @09:39PM (#11804105)
I think people use an ORM so that they can "switch data stores". Personally i never used one since SQL is very easy to write and its actually a Standard. Even when i did switch databases, having a standard guarantees there will be fewer changes needed, than if switching orms. Also, what people "imagine" is of zero concern.
Re:The SQL filter clause: selective aggregates by mamcx on Tuesday May 31, 2016 @08:40AM (#11806178)
In theory maybe. I think instead is looking for a nicer interface (BTW: I learn new languages building a ORM!).
Re:The SQL filter clause: selective aggregates by d33 on Monday May 30, 2016 @10:14AM (#11801433)
Okay, so another reasons are: that 1) it's terribly difficult to parse and 2) on large tables, messing with query planner is actually more work than just specifying the algorithm you'd like to use. Also: 3) yes, there is an SQL standard, but generally if you're writing any application that uses SQL, you're going to end up writing code for a specific database or use an ORM.
Re:The SQL filter clause: selective aggregates by ufmace on Tuesday May 31, 2016 @02:39PM (#11808920)
The only real beef I have with SQL is that some of the aspects of the ordering seem weird.

Why does SELECT put the column list at the start instead of the end? For all but the simplest queries, I want to set up my tables and joins and group bys and whatnot and then think about what columns I want from them. Seems kinda cumbersome to start with SELECT * FROM ... and then go back and fill in the columns later.

Why does UPDATE put the SET before the WHERE? Feels like an accident waiting to happen. I got into the habit long ago of writing SET x = x WHERE..., writing the correct WHERE statement, then going back and writing correct/valid sets, just to make sure I don't accidentally execute the query too soon and update the whole table.

Re:The SQL filter clause: selective aggregates by snaky on Monday May 30, 2016 @10:20PM (#11804209)
> SQL seems like a terrible language

That's because idea of SQL was the same as COBOL - kinda-natural more or less ad-hoc end-user-oriented language.

Codd ('Relational Completeness of Data Base Sublanguages') - “Clearly, the majority of users should not have to learn either the relational calculus or algebra in order to interact with data bases. However, requesting data by its properties is far more natural than devising a particular algorithm or sequence of operations for its retrieval. Thus, a calculus-oriented language provides a good target language for a more user-oriented source language.”