filiwickers

OpenReview | full stack engineer | Full Time | Amehert, MA + Distributed | REMOTE | https://codeforscience.org/jobs

OpenReview is building software for transparent scientific peer review, advanced review-paper matching systems, and knowledge bases for scientific communities. We are seeking an experienced full stack developer to join the dynamic team of software developers working across universities and nonprofits. We aim to revolutionize scientific peer review working with machine learning researchers, open access advocates, and a broad community of open source developers. Are you excited to build user-facing applications and inspired to develop new tools for research? Join us! Reviewing applications now!

OpenReview is a fiscally sponsored project of Code for Science & Society, a nonprofit supporting public interest technology. Read more about OpenReview + CS&S here: https://blog.codeforscience.org/welcome-and-contratuations-o...

To apply, please send a cover letter and resume to [email protected].

Thanks for your feedback and support! We're trying to push forward the specification work to support other implementations like https://datrs.yoshuawuyts.com/.

We created a protocol working group and spec are being updated at https://www.datprotocol.com/deps/. But definitely still some need to improve docs all around! We have an upcoming grant to support this :).

- joe hand, a dat person

It's been almost 5 years at this point somehow!

Check out the first commit: https://github.com/datproject/dat/tree/464679267049899eafa34...

Bit more on the funding history, etc. : https://blog.datproject.org/2017/09/15/dat-funding-history/

Ya, we've thought about that. Dat's storage is pretty flexible and we have a content-addressed storage library. A lot of our users do not want or need to storage data in IPFS though so it adds unnecessary complexity to do that by default.

Someone could built a storage that uses IPFS, similar to our dat-http storage [1].

[1] https://github.com/datproject/dat-http

Dat allows you to store any type of data. Yes, I think you could do what you're suggesting (though I don't follow 100%). You can use it like you would Dropbox, except only one person can write at the moment.

Under the hood, there are two implementations: hyperdrive and hypercore. Hyperdrive is a filesystem for storing any kinds of files. Hypercore can store any kind of data and is really great for streaming data (hyperdrive is built on top of it).

> If I uploaded a bunch of data that was obtained illegally there is nothing stopping me from doing that right?

The hosting provider is responsible for removing illegal content. Dat itself doesn't track any content. The datproject.org is more of a registry, not a host.

> Also, is your peer to peer network able to be attacked by nefarious users like a sybil attack? Is there a situation where I could alter or forge data?

No, only authorized people can write to each dat key (currently only the owner, but multi-writer is coming soon). All the writes are signed with the writers private key and then verified whenever content is downloaded.

We have a few approaches to the disappearing data.

First, we are working with libraries, universities, or other groups with large amounts of storage/bandwidth. They'd help provide hosting for datasets used inside their institutes or other essential datasets.

Second, we started to work on at-home data hosting with Project Svalbard[1]. This is kind of a SETI@home idea where people could donate server space at home to help backup "unhealthy" data (data that doesn't have many peers).

Finally, for "published" data (such as data on Zenodo or Dataverse), we can use those sites as a permanent HTTP peer. So if no data is available over p2p sites then you can get it directly from the published source.

As others said, decentralization is an approach but not a solution. It gives you the flexibility to centralize or distribute data as necessary without being tied to a specific service. But we still need to solve the problem!

[1] https://medium.com/@maxogden/project-svalbard-a-metadata-vau...

We'd love to work with them! We really like what they've done and we have a few partners that use Globus.

Underneath all the transfers in Globus use GridFTP. Using Dat could help distribute bandwidth and speed up transfers. It'll also add version control for free, which (I think) Globus does not have yet.

Ya, there are a few other related questions below. Resilio is BitTorrent-based. But I'm not 100% familiar with how Resilio differs from BitTorrent.

The core difference is in our approach. We're all open source and a non-profit. We're also really focused on the research data use case where BitTorrent is less easily deployed.

We hope that making an open and easy to use p2p protocol will enable other developers to build applications on top, and something like Resilio could be one.

We use TCP or UDP for all the connections. UDP is especially helpful for hole-punching as well.

The direct HTTP support is not fully supported yet. But we're excited for that because it'll allow you to use S3 or other static file servers as peers.

Dat works over any protocol, so it's just a matter of implementing it.

The other one I know of is datahub [1], which used to be frictionless data. It is from the same group that built CKAN.

[1] https://datahub.io/

Ah cool! I hadn't seen quilt before. You are spot on with the differences.

Dat is targeting similar users to Quilt. But we are also looking more broadly at libraries, labs, or other larger academic/gov't organizations managing data. There are a lot of data publishing tools in the sciences such as Zenodo. We'd love for it to be easier to download/publish data to those places. Dat is decentralized, so it really fits well in integrating other data tools.

You can use Dat to replace file transfer software like rsync, so it is a bit more general purpose.

Another difference not mentioned is that Dat really starts at the protocol level while Quilt is more software-focused. Dat protocol is a peer-to-peer protocol for syncing files, modeled off Git and BitTorrent. We built the data management software on top of the protocol.

Edit: I should mention we don't offer any hosting right now, all the data up there is temporarily cached. There is public hosting via Hashbase[1] from the Beaker team. The cool part about Dat being p2p is that it's really easy to switch hosts or use multiple hosts.

[1] https://hashbase.io/

Hi everyone, I'm one of the core contributors to Dat, @joeahand. Happy to answer any questions. It's an interesting time to see this posted because I've been working on a new datproject.org site recently =).

Dat Project started with a focus on increasing access to research & public data. To support the data tools we built a peer-to-peer protocol. People are doing some really cool stuff on top of Dat (such as Beaker Browser), we're really excited about it and want to make sure to support all the neat use cases.

We'll be launching an updated site soon to highlight more of the work around the protocol and what the community is building. Our main use case will still be data management but most of what you see on the current site will shift to a new domain.

Yes, good point! Those are all important as well. I did not intend in my comment to emphasize a binary identity for authors. Seek out books by authors from any lived experience dissimilar from yours to maximize your grasp of our world.

Identity of the author matters because a writer's work is informed by their experience in the world. As a white male, I will view & experience the world differently from Jane Jacobs. Even in a book about facts, a person's relation to those facts reflects their lived experience.

This is especially important in fiction books where you are immersed in the experience of main characters. If the authors identity mirrors yours, then the experience of the main character will not build my understanding of how other people live. At this point, you are reading only for entertainment (which is fine, but you must acknowledge and accept that).

Books, whether fiction or nonfiction, are stories. They present information in a narrative. That narrative will be determined by who writes it and how they've lived their life.

I was being facetious.

Women are better writers.

http://github.com/datproject. It is pretty modular so if you need help finding something hop in our chat: http://chat.datproject.org.

=)

Me either, and that's exactly the issue. Look at any top books list and it is mostly white males. Either men have an easier time getting published or men are inherently better writers. I believe the former is the case. By seeking out female writers and other diverse voices we can start to tip the balance.

Historically, men were the only ones that had the opportunity to write, so any books from older centuries are going to be by men. But that doesn't have to be the case now.

ps. Love Jane Jacobs! After you read that, if you are interested, you may want to check out The Power Broker to see the other side. It is one of my top nonfiction books. (warning, white male author ;)).

Ha-Joon Chang has very approachable books about capitalism and economics.

* Twenty-Three Things They Don't Tell You about Capitalism [1]

* Bad Samaritans: The Myth of Free Trade and the Secret History of Capitalism [2]

[1] https://www.amazon.com/Things-They-Dont-About-Capitalism/dp/...

[2] https://www.amazon.com/Bad-Samaritans-Secret-History-Capital...

If you struggle to get diverse voices inside your field, maybe you should read from diverse voices outside your field. I understand it is hard and I also gravitate towards comfortable reading, usually meaning from people that look like me. This is the problem. Start being accountable to yourself about it.

2 of 19 the books in the list are by female authors (both recommended by women). Looking for some good books from women? Check out these:

Nonfiction:

* Radical Acceptance, Tara Brach

* The Death and Life of Great American Cities, Jane Jacobs

* Freedom Is a Constant Struggle, Angela Davis

* Rising Strong, Brené Brown

* Cleopatra, Stacy Shiff

* The New Jim Crow, Michelle Alexander

Fiction:

* Anything by Ursula Le Guin

* Ancillary Justice, Ann Leckie

* Too Like the Lightning, Ada Palmer

* Citizen: An American Lyric, Claudia Rankine

... so many more

Ya, for Dat[1], we had a WebRTC implementation and it was unreliable compared to our other clients. We are hoping other in browser p2p options get developed, e.g. https://github.com/noffle/web-udp.

[1] https://datproject.org

Check out Dat: https://datproject.org/install.

We do NAT traversal but also connect to local peers over multicast DNS. Command line client should have a local or offline option to restrict only to local peers soon.

We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm).

Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls.

> Maybe Dat plus Zenodo could do the trick eventually.

Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site.

We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`.

We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.

Not very familiar with librevault, but on a cursory glance the design seems fairly consistent (p2p, encryption, offline support).

However, Dat has put a lot of focus into developing an ecosystem to build applications on top of the protocol. This is making awesome things like Beaker Browser[0] and peer-npm possible [1].

[0] - https://beakerbrowser.com/ [1] - https://github.com/noffle/peer-npm

Thanks for the links, here are a few more you may find interesting:

In progress paper - https://github.com/datproject/docs/blob/master/papers/dat-pa...

Dat protocol site - https://www.datprotocol.com/

Yes! Dat protocol supports syncing only the data you want, including storing all history or only the latest content. We just need to get the UI added for this in the desktop app & CLI.

You are correct that Dat does not offer anonymity. The primary use case for Dat is with scientific and public data. In Dat, a user will only connect with peers trying to share the same dataset, meaning a user will never get data they didn't ask for on their computer.

However, we are concerned and thinking about how to improve reader privacy - which is the main need for anonymity in the Dat: https://datproject.org/blog/2016-12-18-p2p-reader-privacy.

Nope! You only join the network for sites that you browse or create. You'll never connect to a site automatically.

Additionally, you have to explicitly opt-in to re-hosting another site. You'll never upload data unless you tell beaker to host a site.

It's called "implied consent". You have to agree to submit to a breathalyzer when driving in order to get your license. All states use this type of law for breathalyzer among other things.