News for hackers. Stuff that matters.
Re:Show HN: Quilt – manage data like code by RangerScience on Friday July 14, 2017 @01:43PM (#14771669)
Also, if we want to produce our own data set for community consumption, what are your requirements, and what kind of payout could we expect?
Re:Show HN: Quilt – manage data like code by akarve on Friday July 14, 2017 @02:11PM (#14771933)
To start with, we can add stars (pay with prestige). Getting more into science fiction--but very possible science fiction--we can put data on the blockchain and let people transact. The data owner would get the lion's share of the transaction.
Re:Show HN: Quilt – manage data like code by RangerScience on Friday July 14, 2017 @02:51PM (#14772260)
Hmm. I can see some ways for this to work.

Awhile ago I started a project converting government voting records (both elections and congressional) into a database. Would that be interesting to you?

Here's an idea: You could also host a data bounty program, and/or start a grant program for the production of these data sets.

Still missing an answer to "what are you requirements"? How do you verify data quality, etc? What format(s)?

Re:Show HN: Quilt – manage data like code by akarve on Friday July 14, 2017 @07:08PM (#14773939)
We support arbitrary data formats in that Quilt falls back to a raw copy if it can't parse the file. On the columnar side (things we convert to Parquet) we support XLS, CSV, TSV, and actually anything that `pandas.read_csv` can parse. We use pandas and pyarrow for column type inference. We want to add a "data linter" that checks data against user-provided rules, and welcome such feature requests on GitHub or in our Slack Channel.
Re:Show HN: Quilt – manage data like code by kevinemoore on Friday July 14, 2017 @03:25PM (#14772471)
Currently, we support 2 "targets" a Pandas DataFrame and a file. Files can be any format. The Quilt build logic uses Pandas to read files into DataFrames so any format Pandas can read should work in Quilt to create a DataFrame node.