News for hackers. Stuff that matters.

M_Rys_Msft

Re:Introducing U-SQL, a language for big data processing on Thursday October 01, 2015 @01:06PM (#10312666)
Because Pig's overall adoption is low. 80% of Hadoop jobs last time I checked were written in Hive.

Besides Pig's extensibility story is not that good.

But I am glad you like Pig, because Pig is supported in the system as well :).

Re:Introducing U-SQL, a language for big data processing on Thursday October 01, 2015 @01:03PM (#10312656)
Having been one of the co-inventors of XQuery, I completely understand the benefit of placing the FROM in the beginning. However, there is a vast set of existing familiarity among users along SQL that we felt supporting this will help people to learn the language quicker than introducing a new syntax.

As to your join syntax: Conceptually, U-SQL is a functional lambda expression composition language. One of the major lambdas are rowset expressions. Thus in your example above a is by itself a rowset lambda, "a JOIN b ON comp" is a rowset lambda etc. What does this mean? It means that eventually, the language may allow you to just write:

a join b on a.id == b.id

instead of

SELECT * FROM a join b on a.id == b.id;

At that point the SELECT ... FROM clause becomes a projection and applier of scalar transforms and aggregators.

Today, U-SQL is not quite as flexible, but we do support this pattern already for Table-Valued Functions where you can say

TVF(42)

in addition to

SELECT * FROM TVF(42)

Good discussion though... keep it coming!

Re:Introducing U-SQL, a language for big data processing on Thursday October 01, 2015 @12:54PM (#10312582)
Actually U-SQL gives you the ability to map files to a virtual table (aka view :)).

CREATE VIEW V AS EXTRACT <insert schema here> FROM "/a/b/{*}.csv" USING Extractors.Csv();

If the data is more complex, you can write your own Extractors that for example handle the divergence in columns. Although we are planning on some features here in the future.