News for hackers. Stuff that matters.
Re:Introducing U-SQL, a language for big data processing by joshuaellinger on Wednesday September 30, 2015 @10:15AM (#10304093)
Interesting extensions.

The nice part of the syntax is that: 1. it is clear when you are going external. 2. procedural code can be written in C# -- big win there.

But it misses the two big issues in ETL that I always run into:

    1. You have a bunch files that are 95% the same but have 2-5 columns that are different.
    2. Someone sticks the wrong type of data a field.
For those issues, you'd be better off mapping a file (or a group of files) to a virtual table so you can specify the schema up front.
Re:Introducing U-SQL, a language for big data processing by M_Rys_Msft on Thursday October 01, 2015 @12:54PM (#10312582)
Actually U-SQL gives you the ability to map files to a virtual table (aka view :)).

CREATE VIEW V AS EXTRACT <insert schema here> FROM "/a/b/{*}.csv" USING Extractors.Csv();

If the data is more complex, you can write your own Extractors that for example handle the divergence in columns. Although we are planning on some features here in the future.