Re:Introducing U-SQL, a language for big data processing
by M_Rys_Msft on Thursday October 01, 2015 @12:54PM
(#10312582)
Actually U-SQL gives you the ability to map files to a virtual table (aka view :)).
CREATE VIEW V AS EXTRACT <insert schema here> FROM "/a/b/{*}.csv" USING Extractors.Csv();
If the data is more complex, you can write your own Extractors that for example handle the divergence in columns. Although we are planning on some features here in the future.
The nice part of the syntax is that: 1. it is clear when you are going external. 2. procedural code can be written in C# -- big win there.
But it misses the two big issues in ETL that I always run into:
For those issues, you'd be better off mapping a file (or a group of files) to a virtual table so you can specify the schema up front.