kevinemoore

The search is powered by Elasticsearch. We want to support the full power of elastic’s query language, but so far haven’t found a great way to prevent overly intense (or malicious) queries and still allow arbitrary queries—love to hear from Elastic experts out there. Exact match on multiple words seems possible though.

We'll look into that for sure!

+1 for everything Aneesh said, but I also wanted to add that the public cloud offers opportunities in data sharing that academia hasn't yet provided, specifically the ability for collaborators to bring their code to the data. I posted a quote from Jed Sundwall, Global Open Data Lead at AWS in another thread. I think he really nails it when he says that the cloud "completely changes the dynamic for sharing data."

There certainly have been efforts in academia to provide shared computing resources. Cyverse (https://www.cyverse.org/about) comes to mind. At Wisconsin many researchers shared clusters using Condor. But, none to my knowledge come close to the scale, reliability and features of AWS and the other major cloud providers.

There's no doubt that local processing is a lot cheaper than the cloud for a lot of workloads.

That's a very interesting pattern--publishing "fake" (perhaps safe or anonymized) data online along with code to spur research and development then running the enhanced code locally on private (e.g., PII data) on local compute resources.

We hope Quilt packages can play a role to make that easier. The package serves as an interface and layer of abstraction between the code and the data so the same code can be run against the safe or private data.

The list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?

Thanks! That's really interesting feedback. We hadn't thought of that.

Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.

Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?

In general, a data publisher simply creates an S3 bucket in their account and sets the access control to allow public read. In the specific case of the AWS Registry of Open Data, data providers create a clean AWS account to hold the data (the account will have only S3 and won't run any compute services). Once the dataset is accepted into RODA, AWS will cover the costs (S3 storage and egress bandwidth) for the clean account.

Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc. Looking forward to checking out Open Images.

I just want to give a plug for sharing data in the public cloud and S3 in particular. Jed Sundwall (AWS Global Open Data Lead) sums it up really well: "The cloud completely changes the dynamic for sharing data. When data is shared in the cloud, researchers no longer have to worry about downloading or copying data before getting to work. Instead, they can deploy compute resources on-demand in the cloud, where a single copy of the data is made available. It is much more efficient to move algorithms to where the data is, than to move the data to where the algorithms are, and this makes it cheaper for researchers to ask more questions and experiment often." See the full whitepaper here: https://s3-us-west-2.amazonaws.com/opendata.aws/AWS_Sharing_...

Sirix sounds like a very interesting system! Is it similar in its internal structure to noms (https://github.com/attic-labs/noms) or Dolt (https://github.com/liquidata-inc/dolt)?

I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS?

I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.

Aneesh's co-founder here. I just want to add a word of thanks to Jed Sundwall and the AWS Registry of Open Data. The support of AWS makes publishing data at this scale possible. I also want to thank Jackson Brown and everyone else who worked so hard to compile, document and annotate these large and extremely valuable datasets.

Quilt co-founder here, no, Quilt doesn’t use tar or zip. Each package version has a manifest that specifies the set of items it contains. Each item in the collection is stored and transported separately. Items are identified by their hash and stored once even if used in more than one package version.

We definitely imagine saving edits as creating new versions of the same package. The most common pattern we've heard is adding more dataframes or files to a package as new results are generated. But, we can certainly imagine other transformations.

That would be awesome! We're happy to help, but we only know a little bit of R. We've been looking at Sparklyr in case it might help read Parquet into R.

No, no relation.

try: example.sales()

it's shorthand for: example.sales._data()

Once the interfaces are mostly settled, we plan to open-source the server so that other organizations can run Quilt registries. If you know of non-profit data-indexes that Quilt should work with or organizations who might be interested in running a Quilt registry, please let us know.

Thanks for describing the problem. That's really interesting. We can certainly aggregate counts of downloads and installs across versions in Quilt. I'll definitely look into providing DOIs within Quilt and see if it's something we can do.

No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.

The package metadata is stored in JSON so that should be pretty easy to access in either Ruby or Go. Tabular data is stored in Parquet by default. Do you know of any good libraries for reading and writing Parquet in Ruby and Go? Do either of those languages have a DataFrame-like class/struct?

Yes and yes. Do you use frictionless data packages? If so, what do you like and not like? We've looked at their specs and have thought about ways we could integrate. We'd love to hear your suggestions.

Currently, we support 2 "targets" a Pandas DataFrame and a file. Files can be any format. The Quilt build logic uses Pandas to read files into DataFrames so any format Pandas can read should work in Quilt to create a DataFrame node.

We'd love to include DOIs for each version of datasets. I think it's feasible for us to do that, but we haven't scoped out how hard or expensive that will be. In the meantime, if you have a way of creating DOIs from URLs, creating a package version will give you a permanent URL for a version of a dataset. If you have a recommendation for how to implement DOI creation, please let us know. Thanks!

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

Good suggestion! From what we've heard from academic users, they'll want a DOI for a specific version, e.g., data from a particular paper or journal article. Any thing else we should watch out for?

We're definitely planning to make column names and types more easily accessible and searchable. It'd be great to learn more about what information and metadata would be most useful to you. We also have an on-prem version we're rolling out with a couple of pilot customers in case that's helpful.

Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.

We've definitely been looking at that! Are you using frictionless data packages now?