News for hackers. Stuff that matters.
Re:Sidehelm: a pipeline to validate, test, and pull CSV data by iforgotmypass on Thursday January 19, 2017 @02:28PM (#13438715)
Dear HN users,

Could you please link to alternative solutions?

Does AWS Glue solve the same problems? https://aws.amazon.com/glue/

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by ZenoArrow on Thursday January 19, 2017 @02:41PM (#13438832)
I could do with those suggestions too, we have a strong need for a CSV validation tool.

After a brief search, this seems promising:

https://theodi.org/blog/introducing-csvlint

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by ckorhonen on Thursday January 19, 2017 @03:03PM (#13439011)
CSVLint is useful in terms of identifying potential parsing issues, though I think the larger issue is that very few applications actually generate valid CSV files - even Excel has it's issues with certain encodings.

In most cases programmers seem to have an incorrect set of assumptions around encodings, escaping etc.

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by iamwil on Thursday January 19, 2017 @02:47PM (#13438882)
What do you do for work that you have a strong need for a CSV validation tool?
Re:Sidehelm: a pipeline to validate, test, and pull CSV data by ZenoArrow on Friday January 20, 2017 @02:34AM (#13442260)
I'm a software tester, but the company I work for gets a lot of third-party data from CSV files, which frequently do not match the requirements set out for those files. Using a CSV validation tool (ideally a web service controlled by the company I work for) would allow our clients to validate the files before they sent them to us.

I'm slowly putting together my own solution for this, but if suitable open-source software already exists I'd be happy to make use of or modify that.

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by iamwil on Tuesday January 24, 2017 @10:24PM (#13478194)
I'm guessing you'd want the validation tool to be controlled by the company you work for, because of the sensitivity of the data being processed?

What type of data/industry is it? It'd help us to know where to look for people that might need us.

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by dgudkov on Thursday January 19, 2017 @03:08PM (#13439053)
If you don't deal with terabytes of new data every day then my EasyMorph might be of help: http://easymorph.com