Oh my god thank you so much for doing this !

I think its better (for me as a user) if you don't boil things down to a score as different people expect different things when talking privacy. It would help if you could simply highlight the potential problematic clauses in different privacy statements along with some reason why it might be problematic.

I don't agree. I just looked in my password manager, and I have roughly ~220 accounts across the web. If I want to go through that list and see which website rank well and which rank poorly, and I want to do that in under two hours, that gives about 30 seconds per service.

In other words, giving a single score plus a two-sentence highlight is probably about the right amount of information.

How about a compromise - not a score for each policy but for "each" individual, or tranche of similarly concerned individuals. I go through a list of privacy options (maybe just once) and the point at which my "okay" becomes "not okay" determines my score. Then each policy simply passes or fails based on my score. And if you want more detail, the list of failure items for a given policy can be bulleted.

Or make the rank adjustable to some personal criteria that matches different privacy expectations.

This would also be helpful in determining how to weight (or not) user feedback in the training portion. I just tried it out (the 10 questions) and there were at least a few I thought, "huh, I know some others would disagree with me on this" because I value X and they value Y more.

Having scores that weight X more than Y would give me more accurate scores, while seemingly also giving other people more accurate scores at the same time.

A good compromise would be a chrome extension that shows a 1-10 score. You click the extension to see clauses

Not for everyone, as thecleaner pointed out. You are assuming your requirements are universal among other users. Also, you are assuming the policies can be simplified to a weighted average of their parts, which is not necessarily the case.

well, just checked and i already see the majority of the web gathering around "C". so really, we can argue both ways about this score thing...

The scoring definitely needs some work – I think some factors should have more weight. Also services' data vary a lot so it's difficult to come up with a good measure for everyone. Ex: I try to take into account whether the service has had any recent data breach, so it penalizes a lot if it has but also scores rather low if it hasn't; privacy policies' length vary wildly and I think that also plays a large role... It needs some tweaking but I think with some improvements I'll reach a more accurate scoring

For a site trying to fight for privacy, don't you think it would be better to not use Google Analytics to track the people who visit your site?

Yes, definitely. I mention it in Guard's own privacy policy, I don't like using it either, but reasons are: (a) it's the simplest and as far as I know one of the few free analytics tools available, (b) not having a measure of the website activity will make me effectively blind and unable to make decisions, (c) I don't send any personally identifiable event (and, for this matter, I don't send any events apart from page loaded events). I'm also open to suggestions to replace GA.

You could just not use user-level analytics; I run the infrastructure for PrivacySpy.org, and using CloudFlare's aggregate analytics has served us perfectly well.

Have you considered hosting your own Matomo server? You could just scrape logs or use a Javascript tracker. But the data will stay local to you.

+1 for Matomo, it's a no brainer for any 'privacy service' to stay far away from Google Analytics.

I’m on mobile, so I can’t check if you’re already using this technique, but you could always use the anonimize IP function. This way the last 3 numbers of the IP data will not be send through the Google Analytics script. More info: https://www.jeffalytics.com/gdpr-ip-addresses-google-analyti...

How did you create a data set large and accurate enough to be useful in training a model?

Some friends run an AI bootcamp and helped me finding the initial set of users to help me with labelling. Initial labelled data was generated mostly through them, both manually labelling and with the approach described in https://useguard.com/experiment Also, the model I'm using relies heavily in transfer learning and achieves very reasonable results with few labelled items (the paper in which the technique is described actually maintains that with only 100 labelled examples they reached comparable results to using 10x that data in models that use older approaches)

What paper is this? Is it the UDA paper?

Important work, and this seems to be doing a decent job already. Cheers. One thing about the teaching: some sentences don’t have anything to do with privacy, so there might be a button to train the AI on that.

Looks cool. It's a small point, but not pluralising words when the number is 1 shows attention to detail (e.g. 1 scandals for instagram).

Thanks for the heads up!

> It understands that "we don't share your data with third parties" is privacy friendly while "we may share your data with anyone" is a potential threat.

Does it understand "we don't share your data with just any old third parties", or "we're not like our competitors who may share your data with anyone"?

Not OP, but curious, are those quotes actually from privacy policies or just hypotheticals?

With enough sites / privacy policies out there, even hypotheticals will end up in a real policy at some point (if not already). Is there something significant in the distinction or is it just curiosity? If the latter, I also share it. :)

How do we know it isn't just people doing the analysis if we can't actually use the AI ourselves?

I'm most probably publishing a paper later this year detailing the process. Also, 80 pages of my thesis would have loved this wouldn't have involved AI to make the whole thing simpler :)

Very nice! The fact you mentioned Tinder's T+C on Twitter got my attention.

Does your work integrate pretrained LMs like BERT or GPT2?

Yes, but not Transformer-based like these two, rather LSTM-based like ULMFiT

Interesting! Did you have to make any substantive changes to the ULMFiT approach to make it work for this problem? Did you use the fastai implementation, or write something from scratch?

(Disclaimer: I'm a co-author on the ULMFiT paper.)

off topic, but i saw in one of your fast.ai videos, you were running an autohotkey script. Made my day !

Great work - would love to read your thesis if it’s available online?

>>”along with other interesting stuff like recent data breaches”

Do you have an integration with HaveIBeenPwned?

> Happy to answer Qs about the tech behind [...]

What tech is involved to get something like this on the web?

Hey... can you reach out to me... I'm kevin ... at datastreamer.io (trying to hide that from spam but I think you can avoid that).

I'm working on something similar at the moment for a client.

Right now I'm just starting out but I've built a privacy policy classifier that is an RNN classifier based on TensorFlow that is just an 'is this a privacy policy' classifier.

I have about 650MB of privacy policies at the moment which I fetched via a crawler. I'm just about to classify the rest of them.

I'm trying to automate the whole thing so that we have a full workflow.

Anyway... ping me to discuss.