I'm not really sure what I'm looking at. Is this something like Google Dataset search? It sounds like you're adding something else on top of this with the "harmonizing" you mentioned, but it doesn't appear to be implemented yet unless I'm missing something.
- FinServ has had harmonized data for years, in the form of horizontally integrated data solutions from Bloomberg (B-PIPE), GS (Marquee), S&P, and LSEG.
- A number of data marketplaces have popped up to facilitate easy access to datasets (free and paid) through standardized data sharing methods, including Snowflake Marketplace, AWS Data Exchange, Google Cloud Marketplace, Databricks Marketplace, and Datarade.
You'll want to explore the space and get an idea of what the current State of the Art is, so that you can understand how you can best contribute.
When looking at these kind of datasets, there's couple of very high level things I would like to know: 1)How granular the data is in terms of geographical location or time 2)What geographical areas and/or time scopes it covers 3)One-of dataset or constantly updated 4)License
Having these as filtering conditions or as summary would help in browsing the datasets.
This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:
* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)
* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?
* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?
I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/
This project is interesting, we at https://www.databoutique.com are building something similar, a curated dataset marketplace for web scraping data.
We believe that using standardization, quality controls and high density verticals, we can cut prices and time to value for web scraping data.
Some stuff to check out:
- AWS registry of open data: https://registry.opendata.aws/
- Datahub (metadata platform for presenting datasets with documentation): https://datahubproject.io/
- For a data exploration frontend, consider Datasette from HN regular SimonW: https://datasette.io/
- Nasdaq Data Link (formerly Quandl): https://data.nasdaq.com/search
- FinServ has had harmonized data for years, in the form of horizontally integrated data solutions from Bloomberg (B-PIPE), GS (Marquee), S&P, and LSEG.
- Data.gov: https://catalog.data.gov/dataset
- A number of data marketplaces have popped up to facilitate easy access to datasets (free and paid) through standardized data sharing methods, including Snowflake Marketplace, AWS Data Exchange, Google Cloud Marketplace, Databricks Marketplace, and Datarade.
You'll want to explore the space and get an idea of what the current State of the Art is, so that you can understand how you can best contribute.
Having these as filtering conditions or as summary would help in browsing the datasets.
This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:
* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)
* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?
* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?
I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/
I'm happy to answer any questions :)