News for hackers. Stuff that matters.
Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by QasimK on Wednesday February 21, 2018 @05:05PM (#16433412)
I was thinking of something similar for applications. Just earlier today I needed a large dataset for some machine learning with Jupyter notebook in a git repo. I don't want to store the dataset in the git repo due to the size, and I don't really want to deal with "downloading the file if it isn't already downloaded into some location".

What would be awesome is something like this little bit of hypothetical Python code:

  import ipfs
  file = ipfs.file(FILE_HASH, keep=True, share=True)  # THIS LINE
  with open(file) as fin:
      # Do stuff like a normal file.
No need to distinguish between local files and cloud files! You can choose whether to keep the file permanently or temporarily, and whether to share it back to other IPFS nodes. You would probably need a system service to do this seamlessly as you suggested.

Sure, it might take a some time to download it at first - but you have to deal with that either way. On subsequent runs, it all just works. (I assume for popular enough files you don't have to worry about the availability of the file, especially if people tend to share back by default).

The big benefit being it is simpler to share datasets and actually use datasets - you can do both at the same time with one little file content identifier (FILE_HASH above).

Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by Jtsummers on Wednesday February 21, 2018 @06:05PM (#16433770)
IPFS includes a way to mount it as a filesystem. So if you know the IPNS URL for some data, or the hash, you can access it like accessing any other file. If it's already downloaded then there shouldn't be any overhead versus just opening a file, if it hasn't it'll download it. You oughtn't need any extra library or logic for it.
Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by QasimK on Wednesday February 21, 2018 @06:24PM (#16433876)
That's brilliant! I think that means it should also support reading the file in chunks, as it gets downloaded, automatically.

I wonder how purging less important files is/could be handled through this? It would be good if there was some way to say, "this file is likely only going to be used temporarily (or just once), so feel free to delete it if disk space is required". That hinting might not be necessary as IPFS could use some least-recently used algorithm, but there should be some way to at least pin files, "this is really important, don't delete this".

Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by foota on Wednesday February 21, 2018 @09:50PM (#16434810)
Iirc pinning is basically required for a file to continue existing.
Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by Jtsummers on Wednesday February 21, 2018 @09:56PM (#16434834)
To guarantee it persists, yes. But it'll remain cached if recently used on the local system until garbage collection runs.
Re:Paper review: IPFS – Content addressed, versioned, P2P fi... by voltagex_ on Wednesday February 21, 2018 @10:48PM (#16435027)
Windows has been experimenting with variations of remote-files-as-local via OneDrive for some time now. I really think it needs to be baked in to the filesystem or at least the file browser to be successful, and be absolutely transparent for the application that's requesting the file.