So I would see all my pictures list, if I decide to open the first one, it would take a few seconds to download, and then I start browsing the pictures, it would figure out I plan to look at all of them, and pre-fetch them from the cloud, so there would be on average no perceived latency.
They recently shut down their free plan.
I don't know a common search/find system that open()s or read()s files during the search by default. AFAIK Spotlight and Windows search are indexed searches. As for the indexing operations, I don't know how that is handled, they could disable indexing for remote files, or they could somehow integrate with indexing.
Based on my testing of a pre-released version of the feature (it isn't released yet), if you were to do something like `find ~/Dropbox -type f -exec md5 {} +`, it would download files.
As a user it did exactly what I expected. I was truly amazed. It was totally seamless and amazing.
Compared to the complexity of what has already been implemented, solving the problem of "I want to recursively open/read every file in my Dropbox, but I don't want it to download terabytes of data and fill by hard drive" seems fairly simple. For example there could be a setting for the maximum about of space Dropbox will use up, e.g., 40 GB, plus Dropbox could be smart enough to detect disk usage. If you `grep -R` it may download/open/read the files, once you reach 40 GB or near your disk capacity, Dropbox could start removing local copies of files that are not pinned to be local, i.e., remove the files that were downloaded because of the open()/read(), not the files you explicitly told it to keep local. I don't know how the team will choose to implement these features, but I'm confident that it will be well-thought-out and tested.
Remember, Dropbox is the company that especially monkey patched the Finder to get the sync icons (http://mjtsai.com/blog/2011/03/22/disabling-dropboxs-haxie/). They will go to great lengths for a seamless user experience, and do a ton of testing. I have no doubt that when Project Infinite is widely available it will be amazing, seamless, and have functionality many people thought wasn't possible or only dreamed existed.
... grep?
In any event, if they were to index and provide search as a service as well, I wouldn't think it's something they do quietly. It would most likely include it's own huge marketing campaign.
That seems like a lousy tradeoff.
Does not mean the files will get indexed, but there is no chance that Spotlight will trigger a unexpected terabyte download in the background.
Is that still true for sshfs ?
People used to ask us if they should rsync to us directly or sshfs mount and then rsync to the mount, and we told them not to do that since the original rsync stat would basically download all files simply to look at them / size them.
But I don't think that's the case anymore. I think sshfs (or perhaps something about FUSE underneath) is smart about that now ... isn't it ?
Are you guys allowing full access to the machine now through LXC containers or some sort of VM?
No - it is the customer, on the client side, that creates an sshfs mount representing their rsync.net account.
It works very well and it is very nice to have a plain old mount point that represents your rsync.net account - especially since you can just browse right into your historical ZFS snapshots, etc.
But in the past, people did that and they got the bright idea to rsync to that local mount point, to do their backups, and that didn't work well.
But my understanding is that nowadays it would work better - you wouldn't download every single file that rsync simply stat'd or listed ...
We still don't recommend it, though. No reason to add that complexity.
That seems wrong to me. It would violate the assumptions of software that does stat() on directory entries and not only verifies presence but also non-zero size.
So it's risking buggy behavior to gain a latency edge over other networked filesystems. I think smart prefetching while preserving correctness would be better.
$ ls
$ truncate -s 1M foo
$ ls -lhp
total 0
-rw-r--r-- 1 catwell wheel 1.0M Apr 26 18:39 foo
$ du
0 .I confess, I did not watch the video, and only briefly skimmed the announcement.
Edit: I'm asking a genuine question of real technical interest here. How can this be implemented with no latency and real file sizes immediately available for inspection, while taking up no disk space? I went back and read the announcement again, and there are no hard details I can see that I missed in my initial skim. There has to be something stored locally, right? Hell, I'm running gigabit fiber here, and I still notice latency in the CLI for anything that requires a network connection. Perhaps I misunderstood the parent?
For example if you have a directory that is all stored in the cloud you can `cd` to it without any network delay, you can do `ls -lh` and see a list with real sizes without a delay (e.g., see that an ISO is 650 MB), and you can do `du -sh` and see that all the files are taking up zero space.
If you open a file in that directory, it will open, even from command line, then do `du -sh` and see that that file is now taking up space, while all the others in the directory are not.
You can right-click to pin files and directories to be stored locally, and right-click to send them back to the cloud so they don't take up space.
This is actually very different than traditional network file systems like SMB, NFS, WebDAV, and SSHFS. With a normal network file system over the WAN you would have major latency problems trying to `cd` and `ls` the remote file system. Most of them also don't have any ability to cache files locally when offline, or the ability to manually select which files are stored locally and which are remote. AFS does have some similar capabilities.