The ideas there are now getting realised in Frictionless Data https://frictionlessdata.io/
This an initiative providing a simple way of "packaging" data like software plus an ecosystem of tools including a package manager etc - https://frictionlessdata.io/data-packages/.
Aims to be minimal, easy to adopt etc (e.g. based on CSV). It has got significant traction with integration and adoption into Pandas, OpenRefine etc.
https://datahub.io/ itself is entirely rebuilt around Data Packages and includes a package manager tool "data".
If you're interested to talk more please come chat on http://gitter.im/datahubio/chat
https://us.wikibudgets.org/w/united-states-budget-2016
I would like to create laymen oriented central repository for all public spending data of the world.
* Hosting problems - I make my own copy of the data.
* Format problems - clearing and formatting data from different sources is a real pain. Once it is on my website, I offer CSV download or COPY/PASTE tabulated data.
* Update problems - no versioning or public API yet.
* Provenance problems - there is a link to the source of the data.
* Presentation problems - tailored to displaying budgets. Not cross-browser or full mobile support yet.
I haven't seen many other platforms offer the same kind of functionality.
e.g This one is a dataset of CCTVs across Leicester. You can easily see all of the columns, sort data around, display a chart of camera types, see their location on a map etc.
https://data.opendatasoft.com/explore/dataset/cctv-cameras%4...
Screenshots: https://postimg.cc/gallery/j5e5thv2/
A few portals powered by them:
- Paris https://opendata.paris.fr/explore/
- Mannheim https://mannheim.opendatasoft.com/page/home/
- Durham NC https://opendurham.nc.gov/pages/home/
disclaimer: I was an intern there 4 years ago
For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was.
Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!
A web interface would be nice so that I don't have to install a tool to browse the content. Going to give it a shot non-theless.
You would be doing us a huge solid, working through these use cases irl is beyond helpful.
It's one of those... I know I should just do it kind of things, even just to get it out there, but I haven't found the inertia.
Seeing things like dat, quiltdata, public data sets, etc. made me think what I wanted to do was unnecessary, but I also agree with your comment.
I think a core problem is data democracy / control / politics of data. Too much we still act silo'ed instead of benefiting from massive data sharing, for a multitude of reasons (especially but not limited to $$$).
* Hosting problems. The first link I tried was already broken.
* Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?).
* Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice.
* Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice.
* Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it.
Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-idea...
Approaches I have seen so far in the wild:
* figshare.com -- Addresses Hosting and Presentation.
* https://quiltdata.com/ -- (!) looks great. Still exploring.
* github.com -- works fine for small datasets (<1GB)
* packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance.
This seems to be a wide open problem to me.