Just today I was using the multilingual Conceptnet-numberbatch word vectors[1], which would not be possible without your work.
To your point though - you can use Amazon S3 as a seed for Bitorrent downloads, which might help some and reduce what you pay. See [2]
[1] https://github.com/commonsense/conceptnet-numberbatch
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...
I have some background in question answering over knowledge graphs, though, so I'm familiar with their strengths.
In my company Luminoso's work, it's important in building domain-specific models that can be used for topic detection, search, and classification. Beyond that, I use it for mostly the basic demos -- word similarity, text similarity, analogies, et cetera.
I believe based on its performance there that it should be a pure upgrade to the kind of applications that use word2vec, but I'd like to know what particular applications it's being used in besides my own.
Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon.
Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and I can help you get started.
It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.)
And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.
Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls.
> Maybe Dat plus Zenodo could do the trick eventually.
Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site.
We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`.
We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.
Awesome, congratulations! :):)
For distributed/p2p software these are just amazing times to be alive :)
100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch).
However, if we gain traction and become cash positive, data sets this size would be good to cater to. :)
Switching to PostgreSQL sped things up, at the cost of requiring a separate database process, dealing with psql's weird access control, and adding an inconvenient step of loading the data using COPY commands.
I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been getting the data -- building it requires about 100 GB of raw data or 15 GB of computed data that can be imported into PostgreSQL.
I once tried git-annex. It turned out not to be a good choice -- its tools were flaky, its usage patterns confusing, it leaves a permanent record of your mistakes in configuring data sources, and it was very hard to convince to use ordinary HTTP downloads instead of trying to get read-write access to S3 (which wouldn't work for anyone but me). Now I have weird branches and remotes in my repositories, and weird data in my S3 buckets, that I can't get rid of in case someone tries to use git-annex in a way I told them would work.
After that I just went with distributing the data with plain HTTP downloads from S3. I wish I could do better than this. The only semblance of versioning is putting the date in the URL, and also people in Asia tell me that the build fails because their downloads from us-east-1 get interrupted. Oh, and if I ever stop paying for S3, everything will break.
If I tried making data reproducible with Dat, would it be safe to promise people that they could use Dat to get the data? Even if in the future I don't like Dat anymore?
For instance, do I have to commit to hosting the data somewhere? If not, who does? Does it disappear when people lose interest, like BitTorrent?
[1] http://conceptnet.io