> So you're going to implement algorithms then?

Yes, we will implement the algorithms, purely on the client side, such that we don't hold indexes to your personal data.

But I understand how that piece of text could have thrown you off, I'll think of ways to rephrase it. Thanks for pointing it out.

Actually I'm really curious how you do this. If the photos aren't stored client side, then how do you search? Do you have a thumbnail of every photo client side? Is that enough? I mean ImageNet scores are still pretty low for small/fast neural nets. And ImageNet isn't even representative of real world photos. So obviously to be successful you're going to have to continue training. So how do you do this in a privacy preserving way? Even federated learning can have some issues because images can be reconstructed from gradients.

> Do you have a thumbnail of every photo client side

In the happy path the files/thumbnails are indexed before they are uploaded. But we are designing a framework that will pull files/thumbnails for indexing if they are unindexed or indexed by older models.

> how do you do this in a privacy preserving way

Our accuracy will not match that offered by services who index your data on their servers. But there's a trade off between user experience and privacy here, and we are hopeful that ente will be a viable option for an audience who is willing to sacrifice a bit of one for a lot of the other.

As someone who has worked on systems like these let me translate:

“You stuff will be private but in return accuracy will be so bad that the UX is gonna suck!”

That’s the key piece people miss when they wanna do anything with ML…that’s it’s a different problem compared to writing code because it’s not about the code anymore, it’s about having great training data!

Apple Photos seems to be using just Core ML[1] for on-device recognition and it does a pretty good job. As for Android, we plan to use tflite, but the accuracy is yet to be measured. And if customers do install our desktop app, we will be able to improve the indexes by re-indexing data with the extra bit of compute available.

We don't feel that the entire UX of a photo storage app will "suck" because of a reduced accuracy in search results, and we think that for some of us the reduced accuracy might not be a deal breaker.

[1]: https://developer.apple.com/documentation/coreml

Up until recently I’ve used Apple Photos happily since it provided a good combination of convenience plus the privacy of on-device recognition. You have a compelling product if you can convince customers you are as reliable and more trustworthy than Apple. You do face the disadvantage of not being the default option for iOS/macOS but that should be balanced by being available cross-platform in Android, Linux, Windows.

Core ML and TFlite are just tools for running ML models. Generating the models is the hard part, and that is what encryption will make more difficult.

We will resort to models that are available in the public domain.

Bingo!

To be honest, that wasn't a concern with my question. I think most people on HN understand this aspect. My question was more about how you improve your models when you don't have the same feedback mechanisms as non-privacy preserving apps. Google can look at your photos and see what photos fail and collect the biased statistics. In a privacy preserving version you won't be able to do this. Sure, you can on an internal dataset, but then there are lots of questions about that dataset's bias and if it is representative of the real world. I mean how many people think ImageNet is representative of real world images? A surprising number.

As someone else who works on systems like these, I agree training data is the whole problem. However you can use some techniques like homomorphic encryption and gradient pooling to collect training data from client code while remaining end-to-end encryption. It's hard, but it's not impossible.

Really? Have we had a revolution in homomorphic encryption such that it can be used for anything other than 1-million-times-slower proofs-of-concept?

I know IBM has released something lately, but given the source..

Does anyone use HE for the type of ML application you are describing?

So I guess there is more to the question that I'm asking.

> Our accuracy will not match that offered by services who index your data on their servers. But there's a trade off between user experience and privacy here,

I think most people here understand that[0]. We are on Hacker News after all and not Reddit or a more general public place. The concern isn't that you are worse. The concern is that your product has to advance and get better over time. That mechanism is unclear and potentially concerning. The answer to this is the answer to how you ensure continued privacy.

You talk about the "push files/thumbnails for indexing" and this is what is most concerning to me and at the heart of my original question. How are you collecting those photos for _your_ training set? Obviously this isn't just ImageNet (dear god I hope not). Are you creating your own JFT-300M? Where are those photos being sourced from? What's the bias in that dataset? Obviously there are questions about the model too (CNNs and Transformers have different types of biases and see images differently). But that's a bigger question of training methods and that gets complicated and nuanced fast. Obviously we know there is going to be some distillation going on.

There's a lot of concerns here and questions that won't really get asked of people that aren't pushing privacy based apps. But the biggest question is how you get feedback into your model and improve it. Non-privacy preserving apps are easier in this respect because you know what (real world) examples you're failing on. But privacy preserving methods don't have this feedback mechanism. We know homomorphic encryption isn't there yet and we know there are concerns with federated learning (images can be recreated from gradients). So the question is: how are you going to improve your model in a privacy preserving method?

[0] I think people also understand that on device NNs are going to be worse than server side NNs since there's a huge difference in the number of parameters and throughput between these and phone hardware can only do so much.

> how are you going to improve your model in a privacy preserving method

We will not improve our models with the help of user-data and will resort to only pre-trained models that are available in the public domain.

This is one of your best replies in the whole thread.

Yes to this. Prove it as well.

Why is it such a great reply? They didn't really answer my question.

I liked the clarity of response. Public models, not user data seems a clear answer to your question?

Not really. In fact it might suggest something I'm specifically more worried about. Datasets that we use in research aren't really appropriate in production. They have a lot of biases that we don't exactly care about in research but you do in production that can also get you into a lot of political and cultural trouble. So really if they are going to just use public datasets and not create their own then I expect a substantially low performance, potential trouble ahead, and I'm concerned about who is running their machine learning operations.

Appreciate the detail here. Given your relevant experience sounds like something that the devs need to address.

Being in the ML community I have a lot of criticisms of it. There are far too many people, especially in production, that think "just throw a deep neural net at it and it'll work." There is far more to it than that. We see a lot of it[0]

[0] https://news.ycombinator.com/item?id=28252634

Wow fascinating. What do you ideally want to see in terms of datasets enabled by user data?

Having vendors vacuum up my data is sub-optimal from a privacy/ownership standpoint. I'm curious how to enable models without giving away my data. Open source models owned by society? Numerai style training (that I don't understand) https://numer.ai/ ?

Datasets are actually pretty hard to create. You can see several papers specifically studying ImageNet[0] including some on fairness and how labels matter. There's also Google's famous private JFT-300M dataset[1]. JFT was specifically made with heavy tails in the distribution to better help study these areas, which is specifically the problem we're interested with here and one that is not solved in ML. Even with more uniform datasets like CIFAR there are still many features that are noisy in the latent space. This is often one of the issues with doing facial recognition and why there's issues with people with darker skin. Even if you have the same number of dark skinned people as light skinned you may be ignoring the fact that cameras often do not have high dynamic ranges and so albedo and that dynamic range play a bigger role that simply "1M white people and 1M black people". There's tons of effects like this that add up quickly (this is just an easy to understand example and one that's more near the public discourse). You can think back to how Google's image search at one point showed black people if you searched gorilla. On one hand you can think "oh got a dark color humanoid" or you can think "oh no... dear god...". That's not a mistake you want to make, even if we understand why the model made it. It is also hard to find these mistakes, especially because the specifics of them aren't shared universally across cultures because this mistake has to do with historical context.

This is still an unsolved problem in ML. Not only do we have dataset biases (as discussed above) but models can also exaggerate these biases. So even if you get a perfectly distributed dataset your model can still introduce problems.

But in either case, we don't have the same concerns in research as we have in production. While there are people researching these topics most of us are still trying to just get good at dealing with large data (and tails) in the first place. Right now the popular paradigm is "throw more data at the model." There are nuances and opinions to this why this may not be the best strategy and why we should be focusing on other aspects (opinions being key here).

Either way, "using publicly available datasets" is an answer that suggests 1) they might not understand these issues and 2) the model is going to have a ton of bias because they're just using off the shelf models. I want some confidence that these people actually understand ML instead of throwing a neural net at the problem and hitting go.

> I'm curious how to enable models without giving away my data.

Our best guess right now is homomorphic encryption. But right now this is really slow and not as accurate. There's federated learning but this has issues too. Remember, we can often reconstruct images from the dataset if we have the trained model[2]. You'll see in this reference that while the reconstructions aren't perfect, they are more than satisfactory. So right now we should probably rule out federated learning.

> Open source models owned by society?

Actually models aren't the big issue. Google and Facebook have no problem sharing their models because that isn't their secret sauce. The secret sauce is the data (like Google's proprietary JFT-300M) and the training methods (though most of the training methods are public as well as few are able to actually reproduce due to not having millions of dollars in compute).

I hope this accurately answers your questions and further expands on the reasoning behind my concerns (and specifically why I don't think the responses to me are sufficient).

[0] https://image-net.org/about.php

[1] https://arxiv.org/abs/1707.02968 (personally it bugs me that this dataset is proprietary and used in their research. Considering how datasets can allow for gaming the system I think this is harmful to the research space. We shouldn't have to just trust them. I don't think Google is being nefarious, but that's 300M images and mistakes are pretty easy to make).

[2] https://arxiv.org/abs/2003.14053

godelski, I really appreciate such a thoughtful response to my curiosity.

Looking at this while better understanding the problem, I wonder what features I really want for my own photo library. Thinking of iOS photos. Matching people together seems hard. But grouping photos by GPS location or date is trivial. So we have to get clear on what features are important for home photo libraries.

I can now see how the idea of "use public libraries = solution" falls short. It neither presents a viable solution or demonstrates rigorous understanding.

Hey, that's what HN is about. You got experts in very specific niches and we should be able to talk to each other in detail, right? That's the advantage of this place as opposed to somewhere like Reddit. Though expanding size we face similar issues.

These are good points about GPS and other metadata. I didn't really think about that when thinking about this problem, but every album I create is pretty much a combination of GPS and temporally based (though I create this with friends). But I think you're right in suggesting that there are likely _simple_ ways to group some things that aren't currently being done.

> I can now see how the idea of "use public libraries = solution" falls short. It neither presents a viable solution or demonstrates rigorous understanding.

ML is hard. But everyone sells it as easy. But then again, if it was easy why would Google and Facebook pay such a high rate for researchers? There's a lot of people in this space and so it is noisy. But I think if you have a pretty strong math background you start to be able to pick out the signal from the noise better and see that there is a lot more to the research than getting SOTA results on benchmark datasets.

You can run algorithms locally and still violate privacy by uploading private facts derived from the data with algorithms. Saying you won’t hold “indexes” doesn’t begin to cover it.

Well, it does begin to cover it. Do you have to be so strident?

What do you think is meant by indexes?

But that will mean that for every version of the algorithms, it have to read all the photos since 15 years ago... my phone battery will die soon.

And if I need to have other kind of client... like a nas to do that... Why I need the cloud?

> phone battery will die soon

Indexing will be opt-in. You will be able to run the indexing only on your desktop client for instance.

> Why I need the cloud?

So that you don't have to manage your own storage infrastructure? But if you would like to do that, then there are self-hosted alternatives that will better serve your use case.

Agree with the above poster. I don't care about algorithms. I want algorithms. But I want algorithms that only work for me. Screw off everyone else.

Apple used to sell this. Then they stopped.

Those "algorithms" can run locally, on a NAS or a desktop, generate the metadata and make it available to you only on your mobile.

I can see myself paying for such software if it was mature enough.

Synology Photos is one such solution already for example.

I have Synology, actually. Is Synology Photos trustable?

The software with these features is called Synology Moments. I use it and I mostly love it, at the very least as a backup for my Google Photos.

My experience is that it works great, provided that you're on your local network. When away from home or traveling, less so. Maybe I could configure things better to alleviate that, I don't know, but I haven't managed to yet.

Sharing is less convenient. Trying to share a photo on-platform is a terrible experience for the receiver with multiple slow redirects, so much so that generally if you're on mobile it's easier/better to just download the photo to your device and share the photo directly. The Moments android app has a flow for doing this, which is nice. It also makes a certain amount of sense: the alternative would be others connecting to your NAS online, which is always going to be less nice than just connecting to Google photos.

The search capabilities are pretty decent. It can recognize people and tag them appropriately. It can recognize some things. In some ways, I prefer searching it over searching Google Photos. But again, only if you're on your local network with your NAS.

--

Edit: see aborsy's response to me below. Looks like I'm a version behind. Maybe on-platform photo sharing is better now, I'll update the software and check it out

Yeah, new version is about the same.

If you want to check it out, here's a couple photos from when I picked some peppers the other day:

https://ojensen5115.quickconnect.to/mo/sharing/pgdYsVEqu

In DSM 7, it’s called Synology Photos!

Thanks for the heads up! Looks like I have an update to install :)

For at-home NAS, is Synology the best for recreating Google services?

I've had Synology for years and I have used their Photos and Momemts app.

It's pretty dang hard to recreate a Google service.It's great for backup and have control over the photos - but dang it's slow....if I need something real quick, I usually go to google photos...even when I'm home. Maybe I need to upgrade to a NAS w/ faster processor, I don't know.

I've turned off the Google Photos facial recognition stuff because of privacy, but dang I miss the convenience. Moments has their own but it's not a good.

Google photos I can easily search for a city or text or an object and it pops up quickly.

It's the best I've found so far. They have a number of apps (docs, drive, moments, etc), and I wouldn't say they are as good as Google right now, but they are quite workable.

In Love my synology. The differentiator between NAS devices is not the hardware but the software.

i have one myself and i would say its the best out of all the alternative nas's out there. you pay a bit extra but its worth it considering how easy it is to setup. i also paid a bit more extra for the plus model so i could run docker which in turns gives you a huge selection of other apps over the built in apps or the synocommunity apps

https://github.com/awesome-selfhosted/awesome-selfhosted

> Those "algorithms" can run locally

But I don't want my GPU burning away running them when they could run much more efficiently and out of mind in the cloud.

Then you aren't the target audience?

Am I the only one who never realized you can search "museum" and see your museum photos?

Now that you've mentioned it, yes, I'd like to try that. But as a counterpoint to your argument, I've never needed it, and I suspect that a lot of people may not actually be getting the same value propositions that you're getting.

On the other hand, Google Photos is Google Photos. But it's often a mistake to compete directly with an established product. New ideas tend to win by transcending the competition.

I propose that if this Show HN turns into a product, it will be because it does something people didn't realize they wanted. Maybe that's privacy. I don't know.

I use it all the time - it's the killer feature of google photos. The premise is that if you come back from vacation with 300 photos, it's unlikely that you (the average non photography-nerd user) are going to sit there and tag them all. If in a few years you want to find "that photo of me you took on the beach in north carolina", with a quick search you can.

There are annoying limitations though, probably because the original team moved on and it's in maintenance stage. Using my example above, google photos has no idea what the "outer banks" are (which is where the beach photos were taken in north carolina) and returns no results. It also has trouble parsing out entities from search terms, so "north carolina beach maggie" isn't going to find pictures of Maggie on the beach in North Carolina (which you'd think they could really fix given that, well, they're google). Finally, there's no way (that I know of) to jump from search results to your full timeline; let's say that "north carolina beach" gets me a bunch of beach pictures from January 2015 (yeah, it was cold), but doesn't have _the_ picture from the trip that I know I want - there's no direct way to click to January 2015 from the results, which really sucks. (Instead you have to go back out of results and use their fiddly scroll to get there.)

Yeah, it's a killer feature, but I really wish they had some sort of a documented "search API".

Instead of natural language search, where I have no idea whether it understood me, I wish I could do (modifying your example):

"North Carolina" "Maggie Thomson" "Tom Morgan" -beach 2018

for all photos in NC, with Maggie and Tom, not in a beach from 2018

and even better, if it could tell me the number of results that would show up if we removed each keyword above.

I guess it's a tough problem, even for Google :(

> there's no direct way to click to January 2015 from the results, which really sucks. (Instead you have to go back out of results and use their fiddly scroll to get there.)

It's amusing how people's insights can turn myopic. Search in photos is the killer feature, and it even solves the problem that you have.

If you realize that you need to see photos from January 2015, don't try to scroll back in your photos feed. Just do a second search for "January 2015".

That's worse than click-to-this-photo-in-context though. Maybe I have 4000 photos from January 2015, so it doesn't help to search for the month.

To an extent it does. For example “Seattle night in July” shows me night pictures taken in Seattle the one July I was there a few years ago.

I try to use it often but it works pretty poorly and I always have to scroll through years of photos to look for what I need.

For me the killer feature of Google photos are: - Free storage of photos (hence why I'll move after I run out of free space) - Tagging faces - Sharing albums

It's a great idea that works in a limited way. Getting that next 30% is going to take awhile nevermind natural language queries.

There's more you can do honestly. Search and and assign people so you can find picture with just them. This also works for pets. People, pets , objects, place, etc. Hell, I searched the car I use to drift and it showed up. It's really neat.

The search is really quite fun to play with, and very useful! I also like searching on the map and seeing where I’ve taken photos. Especially if I’m looking for one particular photo, it’s fun to zoom in from the world map

Thanks for pointing that out. I actually had the opportunity to sync my iPhone photos to Google Photos, but opted to decline. This made me reconsider; cheers.

Why would this feature , that is also apart of Photos, make you reconsider ?

As much as I like apple / iCloud / my iPhone, I do like the idea of seeing all the places on a map that I’ve traveled with my lovely wife Emily. We’re hoping to go to the Seychelles if the next three months work out at my contract gig.

I like the idea of being able to type “water” and see a bunch of water bottles mixed in with all the water-y places we’ve visited.

What sealed the deal was to see it on a map. I typed “water” into Photos just now, and it did a pretty good job. But there’s something peculiar about being able to look at a pin and say “I’ve been at that pin.”

Just a silly thing. But it costs me nothing to get it, so I want it.

Yes but I’m saying that’s a feature in Photos right now. So long as the photo has location data , you can see it on the Map in Places on the albums tab.

Thank you! I did finally figure out what you were pointing out. Apparently there is a “places” album, as you say.

For some reason, it only has 40 places, whereas I have 9,987 photos. I definitely have photos from Cancun, so I wonder if the location data somehow got stripped, possibly when I got a new iPhone around 4 months ago… though that doesn’t make much sense to me.

Anyway, I just wanted to say thanks for pointing out that the thing I wanted already exists on iOS, even if it didn’t have a pin on Cancun. I’ll check the exif data someday, perhaps, or sync to Google photos and see if it pops up.

Cheers!

I use this feature occasionally, but it also seems to be pretty bad for the searches I try. For example, if I search for 'dog', I do indeed get pictures back that contain my dog. However, there are a ton of false negatives -- that is to say, the 'dog' search doesn't show me all of the photos that most definitely and very clearly have my dog in them.

And it's not just dogs. Specific people, locations (before I turned of geotagging on my photos), scenery (mountains, outdoors), etc.

Sometimes this search is nice, but it's not good enough that I can really rely on it.

We need to make this stuff local again, that will be the real competitor to big corp Foo... no servers, no end-to-end, no service cost, no ads, no privacy issues, no random revokation of accounts without recourse, just one end - the users. We can have face detection etc locally if people want it... cycles, it's going to happen eventually.

>lets me search my library using text search for anything

This is untrue, and actually one of the reasons I hope a strong competitor to Google Photos comes along soon. The search function is, for whatever reason, heavily censored and perhaps even biased in some circumstances. Worse, it is completely useless. For example, the query "fat" returns nothing, despite the fact that my gallery is filled with drawing reference photos that includes plus-sized people. "Black people" returns photos of non-black people, and (infamously, and perhaps for related reasons re: the shortcomings of Google's image recognition and tagging algorithm) "gorilla" returns 0 results. "Red shirt" returns an image of a blue decorative screen; "comic" returns anime and webpage screenshots; "woman" returns multiple photos consisting entirely of groups of men.

The situation is dire.

Think of it from Google's POV. Imagine if the tabloids found out about a situation of someone searching for 'fat' in the search bar and then it coming back with pictures of themselves or their friends - that could cause some serious controversy.

Well, this gets to the heart of one of the issues with the current approach to AI. Statistical consensus doesn't always align with a user's personal view or desires. I don't know how you solve the problem; my issue is that Google doesn't seem to know, either, but they insist that they do.

To think that someone can just throw their photos in s3 assumes people are ops, devils, or devs. That’s a small slice of the population. What about everyone else?

I also mention Dropbox. I haven't used it for a while though

Besides search another feature of Google Photos that I would need is automatically inclusion of photos in shared albums based upon who is in them. Some examples:

I have an album shared with my parents which photos of my daughter are automatically added to.

I have an album shared with my daughter which photos of our dog is automatically added to.

I also like the collages, slideshows, movies and this day x years ago photos which Google Photos automatically creates and notifies me of.

You're willing to pay the price of those algorithms and the Google ecosystem. Others are not.

I'm excited to review this project. Thanks to the creators.

This has come at a perfect moment ... as, this weekend, I'm literally downloading my entire Google photos archive (one year at a time) to my local harddrive and figuring out a way forward.

I'm done with Google after a 'straw breaking the camels back' moment with their payment system.

For me the features that make Google photo, Google photo are:

* it's free and comes by default with an Android phone.

* it just works.

If you can make an effortless way to get online backups of my photos at a reasonable price while regaining privacy, then I'll switch in a heartbeat without a single thought about any of those ML-based moat features Google has crammed in their service.

I want none of those features.

I want automatic backup, easy sharing, and accessibility from all devices.

Personally I'd find the pure storage and basic categories suitable. I dislike almost all the algorithms. Especially "memories" and shit like that.

Simple and reliable backup and reasonably speedy browsing is what I need.

> If I wanted to just store my photos I'd throw them in a S3 bucket or Dropbox or something.

Neither of those give you any privacy unless you do the encryption yourself in which case you have to build something to access them unencrypted. Have you checked out what the service actually does?

Wouldn't a mega encripted folder make sense for the average person?

Let's say you store your photos in Dropbox but inside an encrypted folder. What would you have to do to view the photos? Unless there client you encrypt your files with has a photo viewer, you'd have to download the pictures and decrypt them to look at them. The whole thing becomes very inconvenient very quickly.

https://mega.io/ is client side. It works like protonmail you decript to view.

On top of this, good algorithms should be run if it is possible to do it in a privacy friendly way.

>So you're going to implement algorithms then?

Jeesh, that's easy.

You encrypt the algorithms too.

I don't want Google at all in my life, so I think this product seems very attractive. But of course it depends on the user, what they value.