News for hackers. Stuff that matters.
Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @03:03PM (#23394842)
Hi @Arathorn, any work on using IPFS as the media store? It seems like this would be also a really good project for Matrix and doubly so for p2p matrix...
Re:P2P Matrix by Arathorn on Tuesday June 02, 2020 @03:23PM (#23395094)
hey - for whatever reason, nobody's hooked up IPFS (as far as I know) as a filestore. It might be because you still need to pin the content somewhere, as vertex-four says, and so even in a P2P network the fact that the nodes keep coming and going means it might not help much.
Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @03:09PM (#23394895)
IPFS doesn't help - you still need a store-and-forward architecture for messaging, so you might as well use that for images too.

IPFS is not a store-and-forward architecture; a file does not get automatically stored by the system, so when the sender disconnects, the file is no longer accessible. There's "solutions" to this, like Filecoin, but nobody wants folks to be paying to upload images.

Or, in other words - the problem IPFS solves is peer discovery for a given hash, not storage. In Matrix, you already know who your peer is - it's whoever sent you the message referencing the image.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @03:22PM (#23395077)
My idea was more in that every p2p node of Matrix can also be a IPFS server, and be used to pin all of your own media + a small cache of your contacts/group media.

You wouldn't need to save all of the media your homeserver is dealing with, but you still could be reasonably sure that your node would be able to find all media you are supposed to access.

Given that the post talks about libp2p and how libp2p's main driver is IPFS, it seemed reasonable to think that if you are adding Dentrite to your browser you might embed an IPFS server as well.

Re:P2P Matrix by viraptor on Tuesday June 02, 2020 @07:13PM (#23397654)
> + a small cache of your contacts/group media

This idea always sounds interesting and dangerous to me. There's a lot of content you really don't want to be a provider for for legal and other reasons. Caching it for yourself is one thing - public distribution is another.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @09:11PM (#23398578)
Yeah, I stopped development of one project due to this: https://news.ycombinator.com/item?id=22246797
Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @03:24PM (#23395116)
But if you're doing that then those nodes could also implement the store-and-forward aspect of p2p matrix, which comes with its own non-IPFS discovery and thus solves the discovery problem.

libp2p is what IPFS is built on, not the other way round.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @03:50PM (#23395524)
I think I am failing to get my idea through. I am thinking more in the sense of having IPFS as a way to increase redundancy of nodes that might be able to serve some media.

Yes, discovery is not an issue for matrix. But p2p matrix on its own is not able to say "oh, your contact is sending you a message that contains a 10GB file, let me see if I can find other nodes that have this file already and get it from the swarm"

10GB is an exaggeration, but picture a group sharing hundreds of silly videos a day and you already can distribute the load quite a bit, no?

Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @04:08PM (#23395735)
Yes, possibly this could be an optimisation; although then you get to worry about leaking information about your group's contents to the world - anything you send, other clients in your group will ask random servers on the internet for, by a globally identifiable hash. In the best case, this just leaks my friends' meme sharing habits - in the worst case, this announces to the world "I am in a group sharing materials that my Government thinks are dangerous". If someone posts an original file to a public group and a private group, then anyone who downloads that file who isn't in the public group obviously knows that person well enough to be in a private group with them, leaking group membership metadata. And so on.

Random identifiers for each instance of sharing a file to a group would thus be needed, reducing the usefulness of this optimisation quite a bit, where e.g. using the results of the group's p2p matrix discovery mechanism as your peer list to download blocks from has much the same results.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @04:43PM (#23396072)
You are right about the problem of leaking metadata - just by asking for the file reveals to the world that you know about it.

However, this is something that can be fixed on IPFS itself, and I believe it is even in their roadmap: the idea of adding "friend peers". Your IPFS node could ask for sensitive files to this privileged list of nodes only.

So, your client can have a simple logic: if you have e2e encrypted message with IPFS-linked media, ask only the group member peers about the contents of the file. Messages that are not e2e (or marked as non-sensitive) can be asked to the global IPFS swarm.

Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @04:48PM (#23396125)
Sure, if IPFS gets that feature, maybe it's worth it. (Although maybe not - at that point IPFS is just an overweight tooling for downloading files from a list of known peers block by block.) I wouldn't hold my breath though.
Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @05:19PM (#23396437)
> at that point IPFS is just an overweight tooling...

Only if you assume that your only use case is to have private communications.

There are a lot of use cases and jurisdictions where sharing files do not have to be private and would benefit a lot from having redundant nodes doing the distribution. The tooling is already there, why not benefit from it?

Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @05:35PM (#23396611)
There are no jurisdictions where all possible information can be shared safely without hiding it.

And IMO, software should maintain privacy by default - doing something less private should involve actively opting into that. Matrix, for example, creates e2e encrypted rooms by default, and even for non-e2e rooms, it doesn't announce the contents of the rooms to unrelated servers.

If you want to publish the fact that you're downloading a file to the world, you should have to actively opt into doing so, every time - you should treat this like reporting your downloads to your Government voluntarily.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @06:41PM (#23397349)
> There are no jurisdictions where all possible information can be shared safely without hiding it.

This is not what I am saying. What I am saying is that there are plenty of people in countries doing illegal-on-paper things but are largely ignored by their Governments. No one in Brazil is worried about torrenting, even though it is not legal. No one connecting to a Swedish VPN provider worries much about getting copyright infringement letters, and so on. No one in Russia is worried about downloading music via VK, a social network with more ties to its Government than Facebook with the CIA.

> software should maintain privacy by default (...) you should treat this like reporting your downloads to your Government voluntarily.

I don't see any contradiction between what you are saying and what I am saying.

Yes, it should be secure by default. Yes, people should be aware that anything that is not sent encrypted and securely should be considered public knowledge.

So what?

Nothing of what you said negates the fact that there is a huge number of use cases where people want things to be public (Youtube? Instagram? Basically every social network?) and that they will benefit from having it that way.

You might not care about these use cases because you are too risk-averse to use them. Fine. There are plenty of other people who are totally okay with these risks. As long as they aware of the risks, why stop them?

Worse still, between a system that does only one thing "super-privately" vs something that gives them privacy when they want it and also gives them "things where I don't care if is public or not", which one do you think people will use? Look at another post on the same thread, you have people sharing multi-GB Anime over IRC, ffs.

Re:P2P Matrix by vertex-four on Tuesday June 02, 2020 @07:12PM (#23397642)
The point I'm trying to argue, poorly, is that implementing a feature where you can right-click and explicitly choose to download something a bit faster is... probably not worth the sheer amount of arguing for "integrate IPFS into Matrix" that I've seen at every turn, for the past few years. People largely seem to want IPFS because IPFS is their pet project, not for any particular edge that it'll give Matrix, and alternative solutions are dismissed out of hand.

Even before p2p Matrix was announced, people turned up in #matrix-dev every week and argued that Matrix should replace its existing, functional file sharing system with IPFS, but could never come up with a reason to put in the work.

Sharing multi-gigabyte files with groups of people in a public way is, yes, maybe something some people do, but if IPFS was uniquely suited for this and the integrated system was deficient, wouldn't people be posting IPFS links in rooms all the time right now, and be able to talk about why they're doing this? Then it'd be a lot clearer that this is a thing people actually want to do and maybe setting up a button to do it in fewer clicks would be worth it.

Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @08:23PM (#23398214)
> people turned up in #matrix-dev

Honestly, for a "normal" homeserver it does not make as much sense as for the p2p version of it. For the "actual server" synapse, I am more interested now in the multi-home-repo that I saw posted also on this thread.

> Sharing multi-gigabyte files with groups of people in a public way is, yes, maybe something some people do

Microsoft executives in the 90s: "Internet search is a silly idea. Once you find a site that you like, you just bookmark it. Who will be constantly search for the sites they already know?"

> if IPFS was uniquely suited (...) wouldn't people be posting IPFS links

The use-case of posting IPFS to share content is mostly covered by "posting magnet links", which is basically "why people just don't bookmark the sites they like?"

There is a lot more than could use cases that can come up once your matrix node can (a) manage data by its content (b) if it could know who also has it. And these use cases can be implemented in a way that do not make other use cases less secure.

You are right though that just having people asking for something without having no real background about the underlying challenges is a pain in the ass. Barely a year of working with blockchain stuff and I already roll my eyes every time I see someone on a forum doing a "Why don't you... X?" just after reading the word in some random blog. Let's see if I find time to brush up on my Go and contribute to the multi-home-repo project.

Re:P2P Matrix by vertex-four on Wednesday June 03, 2020 @04:55AM (#23401406)
I’m interested in what those use cases that you suggest exist are. IPFS has, so far, really not impressed me (nearly all uses I’ve seen so far could literally be replaced with BitTorrent without problems) - I don’t see how integrating it into Matrix fundamentally changes anything.
Re:P2P Matrix by Groxx on Tuesday June 02, 2020 @04:08PM (#23395734)
Anime-over-IRC can hit 10GB files pretty easily. I wouldn't label it an exaggeration so much as "uncommon".

I do like the idea of automatically deduplicating every shared gif tho, which would happen automatically since it's content-addressed.

Re:P2P Matrix by tulir on Tuesday June 02, 2020 @03:49PM (#23395517)
matrix-media-repo has an early implementation for IPFS support: https://github.com/turt2live/matrix-media-repo/issues/115#is...
Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @05:22PM (#23396465)
Thanks for sharing this. I am working on matrix hosting and a multi-homed media service will be very useful.
Re:P2P Matrix by karatestomp on Tuesday June 02, 2020 @05:30PM (#23396544)
I thought part of the point of how IPFS is designed is that you don't need to "implement" it, you just ask for a hash (file) in the IPFS mount and it goes and grabs it if it's not already there? Should just be a matter of handling IPFS URLs correctly. Which, I guess that's something to implement, kinda, though if you already have a handler for 'ipfs://' and Matrix passes off a click on such a URL to your OS to handle as it will (as it might to a browser for 'http://') then there really isn't anything to do
Re:P2P Matrix by rglullis on Tuesday June 02, 2020 @05:43PM (#23396704)
I am talking about the media store, the subsystem of the Matrix homeserver that stores all media that is uploaded.
Re:P2P Matrix by karatestomp on Tuesday June 02, 2020 @05:58PM (#23396865)
Ah, yeah, that's less straightforward.

checks the docs

Looks like you could point it at IPFS "MFS" mount (Mutable File System) instead of whatever directory it's using now for its normal local storage media store (which I assume exists?), and that'd do the trick. Files stored in IPFS (so mirrorable through that, or whatever it is that you've got in mind) but appearing as ordinary files with normal names, so the normal file storage media store might manage to Just Work.

[EDIT] Incidentally MFS is exactly what I wished existed back when was kinda halfway considering using IPFS for my personal files. Been added since the last time I looked in on the project. Was too easy to "lose" files in IPFS otherwise. That sort-of fixes the problem, at least, provided the file tree can also be replicated pretty easily.

Re:P2P Matrix by WhatIsDukkha on Tuesday June 02, 2020 @07:05PM (#23397579)
Have you looked at git-annex recently?

https://git-annex.branchable.com/special_remotes/ipfs/

It takes a little bit to work out why you want it and why its so much better in that unixy way many of us insist on vs git-lfs (supporting ipfs should be a clue there).

*as a warning some of the defaults in last years git-annex 7 versions were aggressively annexing files in the git archives but fixed now.