The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

I used to like this style of naming software with a allusive dictionary noun, but the sheer volume of new code is driving me to wish there was a canonical clearing house for naming, with agreed conventions for the "given name" even. e.g. Quilt, which in documentation and prose reference is the most natural choice, and a real use handle with a context qualifier, especially e.g. Quilt_Data and the use of a clearing house website for reference and proposals, including short forms which may need to be tested by popularity in use. Quilt_DM might be a alias to my preference, but I think it's worth the effort to get the time and confusion out of accidental conflicts like this.

Java has the right idea. Namespace by [reverse] domain name. It's sadly a tad verbose though. There's high contention for short names which is at odds with uniqueuness.

Namespacing by reverse domain name is no panacea. Your company might change it's name, might be bought, or might buy another company.

On the open source side, how much stuff was org.sourceforge that then had to move when SourceForge turned into a scummy operation?

Microsoft's CLR guidelines are a little less verbose but still run into the same problem with corporate name being permanently unique.

> naming software with a allusive dictionary noun

Perhaps <adjective> <noun>, <adjective> <noun> <verb>, or <adjective> <noun> <verb> <adverb>. Or <adjective> <adjective> <noun>. And so on.

Frog (Design, 200M ggl hits); Happy Frog (soil, copy center, 500k hits); Happy Frog Swimming (pool, 4 hits); Happy Frog Swimming Sideways (0 hits).

Quilted Data (250 hits).

But my impression is common nouns have trademark advantages.

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

Quilt is an incredibly widely used tool. Many kernel developers use it as an alternative to git for patch management, and it's used by most distributions to manage their patches on top of upstream projects.

It's quite surprising. Quilt is a very common tool, e.g. for Debian developers: https://wiki.debian.org/UsingQuilt

You should probably rename the pip package and the CLI tool to quiltdata to avoid conflicts.

No worries, just make the name switch sooner rather than later. Since your target audience is obviously developers in general - to whom the patch tool is well known - it will most likely affect your business negatively if you don't...

How did this not come up during your market research?

Also, fantastic product! My only criticism is the poor name choice and lack of market research...

There is also a relatively well-known startup called Quid. They are doing something with data visualization. I first though this was them.

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

I think it conflicts a bit:

  $ brew install quilt >/dev/null 2>&1 && which quilt
  /usr/local/bin/quilt

I have used it extensively in automation in the past. I don't know if it's a good idea to call it "quilt" on the $PATH. Maybe "dataquilt" or "data-quilt"? "quilted"?

When choosing a name, you should check the package manager index sites of large distros for potential conflicts. For example in Debian:

https://packages.debian.org/search?suite=default&section=all...

Quilt is a patchset manager and used a lot in linux distro spaces (it was created by one of Linux's maintainers for his own use). Any global install of your tool will conflict with its existence since the utility names are the same, which you'll get will depend on your PATH setup.

> We built Quilt to bring package management to data.

No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories.

> so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data.

You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of data analysis. I would refuse to do any work on data that was pre processed without having the exact list Of what was done to the raw data.

Honestly all I see technically speaking in this project is a python program to download data frames... Woo-hoo. And for the business side, the now usual attempt to own a community of users and their data. Boring.

Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :)

We understand that cleaning/filtering/preparation are key to the analysis, and we plan to support those operations as part of package construction. The question is whether or not that work should be done repeatedly, or once for the benefit of your collaborators.

You can store any kind of data in Quilt. Not just data frames. What's on Quilt today is just the beginning of what is possible.

Do you also dislike GitHub? We've opened up the client source and the community for as much free data as people can publish. It seems naive to expect that we wouldn't charge for anything ever. The users always and forever control their own data, by the way.

I'm not criticizing the price here. It is obviously totally OK to pay for data hosting.

> "Contact us to start Business or On-premise service." I DO reject the whole concept of having a "hosted-only" software. To me it makes the whole project useless. It means I cannot have my own private or confidential data. I cannot use it in my company, etc. More importantly, the data is not mine anymore. It's yours.

I cannot find any place on your website that explains what happens to the data once it is uploaded! As far as I'm concerned, it means I am giving you my data, for free, without any restriction on your side. You can resell it, modify it, rebrand it, prevent me from accessing it. Since there is no mention of how the data is stored, I also cannot know if the data is encrypted on your side, or have the capability to read everything that is uploaded.

We need to clarify the EULA and terms on the website. The data belongs to the users and we want to keep it that way. Quilt is not "hosted-only". The whole point of the on-prem install is that customers can run Quilt on their own infrastructure (it's Dockerized, etc.). We'll roll out a more formal on-prem solution as the project evolves. We're doing one thing at a time right now :)

> "The data belongs to the users and we want to keep it that way." Does it mean you plan to include in the EULA that Quilt will not parse, read, sell or otherwise use the data that users upload?

> "The whole point of the on-prem install is that customers can run Quilt on their own infrastructure (it's Dockerized, etc.)."

Yes, and on premise is not possible at the moment, and on the website it is marketed the same a business use.

Anyway, I lost my cool a bit on previous comments, thanks for keeping yours, and good luck :)

> Package managers don't make users pay if they want their own package repositories.

Isn't that npm's business model?

This is a really impressive project. Do you think you would add data cleaning commands? How do you think you would handle datasets that are only available to academics or other restrictions?

On your front page you should have the owner/dataset name instead of just the dataset name so I wouldn't have to click through to find the owner name.

Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any "out of the box" cleaning functions you'd like? In the spirit of dplyr? We've looked at e.g. scikit feature for normalization, 1-of-n encoding, etc.

How about deduplication. Also table / dictionary lookup . And string replacement and regex replacement .

Got it. If you'd like me to ping you once we have custom build hooks: aneesh at quiltdata dot io. Have you tried Luigi or Bonobo for data cleaning?

No I haven't I think I just did it myself I'll look into that .

We've been thinking of making a Python build file to make it easier add richer transformations than the simple file->DataFrame that runs by default. We'd love suggestions and input on the types of transformations that would be most useful.

This looks pretty cool. Am I right to understand that this is providing a virtualized filesystem interface that dynamically loads the slices of data actually being accessed (but through a direct API rather than something generic like FUSE)?

Unrelated I found a typo on your blog. Search for "seriailize" on your "Manage data like source code" post.

Yes, essentially the quilt library abstracts the serialization (building packages) and deserialization (importing packages) so that user/client code doesn't need to include the mechanics of loading data (e.g., from a file system path or URL in S3).

I now wish there was a feature to make one's own comment less prominent yet still in thread, a self down vote in essence.

Because I took my first point in a direction away from the context, despite the reasons I wrote being the underlying reason.

I'm aiming at the extent to which data is susceptible to considerable performance improvements when you have the ability to align the physical store with the data structure.

Simultaneously, arguing as I did, to punch holes in the operating system to get to hardware comes with costs and the cost of losing backups is not trivial.

This is why I asked about the Weka FS in my comment, because if you were atop a system like that you would have theoretical redundancy already.

I ignored my dubious omission of this being another FS and not a hardware pass through my comment was about.

However the first point of contact in any case is going to be a requirement to partition a store for pass through access, taking my comment literally.

That's a great path to disaster and not what I meant but omitted the explanation required:

If you had commit access to your FS code, you could provide accommodation for the pass through request to vary metal, but just lose the point of the FS. I instead envisioned the FS providing you with hardware layouts that are suitable for the data you need running fast. I imagined a API to request that "on disk" format, and the FS to be able to indicate potential changes to balance between the requested layout and the performance of FS features such as cluster latency, computational efficiency of raid schemes and other features like T10 and other management of data overall. A scale between handing over the hardware and alerting to management consoles of the existence of a opaque store, and the agreement in tradeoffs the full features provide, where the application is placed in authority to size performance at a level of reasonable costs.

I'm thinking of serialisation that's a part of any larger system which provides reliability features in other components.

But the flexibility could exist to allow for flow under exceptional load, if doing so was critical to the performance of your overall system.

If you are buffering requests at a point where you have to pass the stream to the next component which is responsible for acknowledgement of the requests, and they will be able to resend if they have not received acknowledgement, then the opportunity to trade normal FS behaviour for raw speed, is possible under reported conditions known by the management instruments.

I'm assuming that the FS is not going to let you write data faster than it can function for jobs like replication, and then limits would indicate that the discarding of the FS roles are not always good value.

I haven't seen any sign of the purported advent of intelligence in file systems for the sort of thing I am interested in like this.

But certainly it must be a likelier possibility to cooperate with new FS companies, than the Windows design team and the weight their legacy brings.

I'm out of touch, but NTFS provides or provided interfaces for, e.g. sparse file layouts and there's nothing new in my central proposal. I've merely speculated how far it might be possible to go. And there's a entire field of data structure optimisation which can only be done in a worthwhile way with the FS cooperation. To just leave it at the hope of getting nice features in FSs else write your own driver, seems like a crude bipolar proposition in the present times

I hope this is not too tangential, but I have been thinking about the best ways to make use of direct access to non volatile memories, ignoring the block and driver level and ordering the layout in your code. I suspect that your project is one which could take that direction very usefully.

I would very much like to hear your reaction to this suggestion, and I would like to add if you have looked at the work at weka.io for their take on the convergence of storage. I increasingly like the idea of having the ability to use the hardware and have the management directed at the applications level where the developer is able to use intelligent measures and policies to tune their systems in discreet manner from the OS. Close cooperation will enable the collection of data to provide a valuable resource for administrators and directly benefit the pace of production development, by providing a comprehensive universal instrumentation context.

I know this is almost arguing about the reason why databases should use their own on disk format and there is long history of the tradeoff involved with that.

But the trouble I have with the current storage space is that the equity in the file systems is not flexible enough for the kind of smaller mixed deployment I come into contact with in the lower small business market, as example of this, CEPH or any FS which is a monolithic investment in which you are going to find the most restricted resource is management time. Additional FSs are difficult propositions to small shops. But the idea I'm looking for is the application layer should be responsible for storage management and performance tuning and follow best practice set by the software publisher learned from collected instrument data.

I think the epiphany of the general operating system is nigh or even last century.

Nobody is able to use large software programs in a turnkey way, making assumptions about the OS environment. I hand wave plenty saying that, but certainly I can follow up flippantly to add that a friend's experience providing contracted management to small businesses is not atypical by my experience, he joked that he loves Linux because it meant he got a clean install and nobody likely to be able to know how to mess it up.

The context where I see very little leveraging of OS capabilities, particularly in the Windows Server user world, it looks like a lot of wasted effort and license expense.

I beg forgiveness in advance of this facetious illustration, but in conversation with a small business Web developer recently, I cited the example of Plenty of Fish, and rhetorically asked if he knew it was a one man gig, on Windows and IIS? He was unaware of this, so I teased him that he would be forever in his first money rounds and hiring, if he had similarly accepted a bet on building such a dating site, if he kept on reading HN so much... My joke is off color sorry, but I wanted then as now to make the point where it has become all too accepted to automatically get started with a complete development stack and seek advantages in terms of the customization and deep power that is leveraging highly experienced professionals, and I worry about whether we all just do too much of this, and it's time to review the situation more broadly than my rotten humour alludes to, because the problem, if it is a problem, is much wider.

I agree wholeheartedly with this statement: "application layer should be responsible for storage management and performance tuning". I would take it one step further and say that storage should be virtualized in a high-performance way. HDF5 was the old way, Parquet/Avro is the new way, and something like Apache Arrow is the future. We are currently focused on efficient, cross-platform friendly ways of serializing [columnar] data and have chosen Parquet. Optimizing to the level of volatile caches, though, is probably not something we're ready to tackle. The performance gains to be had by eliminating parsing and lazily loading data (in the spirit of DMA) are absolutely huge. And good file-formats accomplish that. Moreover, the amount of time and performance lost to moving data around is staggering. https://weld-project.github.io/ and https://arrow.apache.org/ sketch the solution: 1) optimize the entire computation graph to minimize data materialization; 2) have a canonical in-memory representation that can quickly serialize results to a variety of clients.

I have not heard of weka.io but will take a look.

As to your slicing question, yes. Data is lazily loaded. With Parquet as our data store we can do even more (but haven't yet): load only the columns referenced.

I dream of being able to tell the file system to be ready for a range of requests when they are first anticipated.

In a extreme, I think about getting a NV store to load the pointers to the physical data, in cache and to not flush the cache until the context is released by the application thread. Going further into this reverie, anticipatory QoS for bandwidth is a personal desire. I'd even like to have a PCIe lane reserved, in my fascistic moments.

Now I'm regretting my wordy comments above, because the explanation I wrote wasn't needed.

But this dream I have to gain programmatic cooperation for data performance is surely not unique to me.

Maybe twenty years ago it made perfect sense to leave the task of resources sharing to the operating system and subsystems, but now not only do many people have a high performance headroom to exploit, but the knowledge and experience of writing concurrent schedulers and load balances is far more commonplace, just by virtue of the needs presented by the crunch expansion of the Internet.

The only point from my above comments that I think is under appreciated by the end users of software and worth approaching, is how much you can improve your software when you have a truly controlled environment. Talking straight through the operating system stack you can obtain uncoloured measurements from which you can build optimal performance applicable to all installations. I may dream, but I grew up with the worshipping of vendor benchmarks as my bete noir, and I have been set against waste created by supporting unrestrained variations, ever since. That makes sense for a commercial operating system, but the opportunity to work with completely homogeneous stacks is something that ought to be recognised for the value potential, by management. I'm convinced that this will be a critical commercial advantage for whoever first finds a solution that isn't a service only but customer deployable.

If you're talking about lazy loading of data, for your R implementation you might want to (if you're not already) look into creating a custom dplyr backend that only loads data when needed (similar to the dplyr SQL backends).

Another wish of mine, is the reporting of the installation conditions on every individual installation.

It's a long time past, but I can't forget my experience in small businesses, where I had to disbelieve any reports about performance problems, until I had full access, or even on site immediacy to get a idea of the circumstances.

I'm fed up with Microsoft getting the advantage of being the only one who collects metrics aggressively or at all.

The open source community should be the first to get the data out of customers from the production systems

How can we do this?

I'm just feeling a personal sense of futility from my experiences optimising code as well as installations to sometimes minimal effect, bounded by the hardware budget. I certainly learned a valuable part of my skills in that way, but I have ever since felt sceptical how much of the development effort would be better allocated, if true installation performance instruments were reporting the whole user base.

(one assumes a R user will be well equipped, but I expect it to be a broad spectrum of hardware, from students in India to multinational developers working with the latest generation.

Its often true that the most impoverished users gain the most from optimised software and the ethics of this result are impeccable. I'm simply asking for the concerted collection of instruments metrics to support optimisation efforts in open source software. Putting the authors, creators and hackers first, is something that I wish was done as part of the process of promoting FOSS and the provision of quality insights into how their work is used, I think should be the basic foundation of our responsibility and gratitude for their work. Not to mention the improvements in our work which will result. Surely this is not impossible to solve. I have often thought about a package manager taking snapshot performance characteristics and reporting to the developers by way of a public page update. But I've not even seen the idea anywhere else, and don't understand what gives...

Typo fixed. Thanks :)

Love the concept! Clear value proposition for anyone who knows both worlds (code + data). Look forward to giving it a try.

Also, if we want to produce our own data set for community consumption, what are your requirements, and what kind of payout could we expect?

To start with, we can add stars (pay with prestige). Getting more into science fiction--but very possible science fiction--we can put data on the blockchain and let people transact. The data owner would get the lion's share of the transaction.

Hmm. I can see some ways for this to work.

Awhile ago I started a project converting government voting records (both elections and congressional) into a database. Would that be interesting to you?

Here's an idea: You could also host a data bounty program, and/or start a grant program for the production of these data sets.

Still missing an answer to "what are you requirements"? How do you verify data quality, etc? What format(s)?

We support arbitrary data formats in that Quilt falls back to a raw copy if it can't parse the file. On the columnar side (things we convert to Parquet) we support XLS, CSV, TSV, and actually anything that `pandas.read_csv` can parse. We use pandas and pyarrow for column type inference. We want to add a "data linter" that checks data against user-provided rules, and welcome such feature requests on GitHub or in our Slack Channel.

Currently, we support 2 "targets" a Pandas DataFrame and a file. Files can be any format. The Quilt build logic uses Pandas to read files into DataFrames so any format Pandas can read should work in Quilt to create a DataFrame node.

Do you plan to curate the free plans? If not, how will you prevent abuse? (thinking in terms of both the service and its users)

Yes :) Happy to discuss in detail if you have specific types of abuse in mind. Our first line of defense is to get help from the community through downvoting useless content.

The main thing is how you are able to offer unlimited disk, when anyone can start uploading data? Some users might upload very large amounts of innocent data, others might simply abuse it as a sort of data backup system. More nefarious users might even try to distribute malware.

Could you tell us how this compares with Synapse (https://www.synapse.org/)?. They've been doing this for a long time and have a large presence in computational biology.

I know only a bit about Synapse. It seems like they have some valuable data. I think the biggest differences are in the culture and user experience. Quilt gets its DNA from the open/community-driven cultures of GitHub and npm. We want to make it extremely light-weight for anyone to push and share data packages. Synapse frankly seems to have a lot more features. We want to compete on simplicity.

Does Synapse offer server-side filtering? That's something we're thinking about.

> We want to compete on simplicity.

Bravo, this is an excellent strategy. I've been using NPM packages to share small datasets for a little while due to the simplicity of distribution.

Would be great if you also had the schema / file format information as well on the site.

Agreed. We're considering doing more along these lines, such as generating the Hive DDL. So you would like to browse the schema, e.g. under Contents?

Awesome! Yup would like to browse the schema.

What type of representation would be most useful for schema information? Avro?

That or just a basic visual editor is fine too. Goal is to just know what I'm getting before getting it in more detail.

are you aware of datapackages [1]? do you plan on opensourcing the backend components?

[1]: https://github.com/frictionlessdata

Frictonless is interesting but 1) doesn't handle serialization (which is essential for performance); 2) requires users to hand annotate schemas (we think schemas should be auto-generated whenever possible).

Yes and yes. Do you use frictionless data packages? If so, what do you like and not like? We've looked at their specs and have thought about ways we could integrate. We'd love to hear your suggestions.

Any way your web page could be made viewable without requiring Javascript?

I disable it in my web browsers because of security and privacy concerns, and it would be great if I could just read about your project without it.

Ah, that would be useful. In the meantime our docs are browsable without JS https://docs.quiltdata.com/. The articles linked at the bottom of the docs landing page are also legible without JS. Unfortunately the docs menu is not available without JS. I will look into fixing these issues. Email is feedback at quiltdata dot io if you have questions :)

The menu on https://docs.quiltdata.com/ isn't visible if you have JS disabled. I had to disable CSS to be able to navigate.

Is there planned support for any other languages than Python?

Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.

I'd definitely be interested in looking into this but I'm on an R team. Didn't see anything on how to contribute in another language or an API to hit. Will be there be information soon?

Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt (aneesh at quiltdata dot io). There is a sliver in our docs but it's not complete: https://docs.quiltdata.com/basics.html

The hooks for extending build targets are here: https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b7778...

Again it's a bit raw but we're here to help make R support easy.

Is this only cloud hosted? I have been looking for this exact solution, and I'm very excited to try yours, but banks and other institutions will not tolerate storing their data in my butt. It must be totally local without ever needing an outside network. Can you provide that?

We are rolling out on-prem where the customer runs the package registry on their own infrastructure. The set up process takes a bit of customization depending on your environment. If you have Docker containers it's easier. If you email me I can get you rolling: aneesh at quiltdata dot io.

This is pretty neat! Is there an API or anything, so we can write support for other languages?

There's a RESTful API that's used by the Python client, though it's not documented yet. If you're willing to go through the code, you can find /api/... calls in https://github.com/quiltdata/quilt/blob/master/quilt/tools/c... .

We're planning to write proper documentation soon.

Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.

Ruby and Go, personally.

The package metadata is stored in JSON so that should be pretty easy to access in either Ruby or Go. Tabular data is stored in Parquet by default. Do you know of any good libraries for reading and writing Parquet in Ruby and Go? Do either of those languages have a DataFrame-like class/struct?

This could be a big deal. Fantastic concept and great implementation.

Thanks. Where do you see it going in 2-5 years? Sometimes outsiders see things about the future that we haven't thought about :)

What's your privacy policy for private data?

Only the owner and designated collaborators can view it. We offer private data in the cloud (S3) and are rolling out on-premise in case you want to run Quilt on your own infrastructure, in which case you control blob storage. Does that answer your question?