The GDPR allows for reasonable usecases like this. The principles are pretty broad and only stifling once you get into the realm of sharing data between organisations [0]
Such uses are "Appropriate", but you would be expected to maintain this information securely and you would not be allowed to use it for some other surreptitious purpose?
This actually enables further innovations for instance you can log a cars behaviour without a user fearing you might sell this information on to their insurance company.
What upsets people about GDPR is it does away with the notion of data being the holders property and enforces obligations to the subjects of that data.
Of course in this day and age where "data is the new oil" this is upsetting for some.
At the end of the day however this demonstrates how regulation, apart from propping up inneficiet markets also serves to protect the public from rampant commercialisation of every aspect of their lives!
[0] https://united-kingdom.taylorwessing.com/globaldatahub/artic...
The GDPR is rather ambiguous about use cases like this. That's part of the problem. Different national regulators are going to have to decide what they consider reasonable and how much they want to enforce. Unless and until they issue authoritative guidance, everyone is basically in the dark on what this really means, and the penalties for misjudgement could be severe if the regulator strongly disagreed with your interpretation.
As for sharing data between organisations, one of the more unwise (IMHO) aspects of the GDPR is that when it comes to the right to erasure, the obligations to notify other parties to which the data has been supplied have guard conditions about reasonable efforts, but there are no such conditions at all on the immediate obligations to delete the data. If you're not covered by one of the acceptable cases (where the scope is, as noted above, not entirely clear yet) then you must be able to delete the data on demand, regardless of the practicality or cost of doing so.
It allows for simple logs. It does not allow for simple immutable logs, because your customers must always be able to force you to forget about them. There's nothing about sharing data between organisations: the GDPR requires — with certain very minor exceptions — that someone holding personally-identifying information delete it on request.
As an aside, this:
> This actually enables further innovations for instance you can log a cars behaviour without a user fearing you might sell this information on to their insurance company.
… isn't really true: the logger can still sell it, it just may not.
> That’s wrong. Data protection legislation expressly forbids
I'm sorry, I assume that English is not your first language. 'Can' means 'is able to'; 'may' means 'is allowed to.' The GDPR does not allow selling something, but it's still perfectly possible for someone to do so. People break the law all the time.
I raised the issue of can vs. may because it's germane to the issue of immutability. It is possible to build a system in which it is mathematically impossible (within arbitrarily-determined limits) to alter data; it is not possible to design a system in which it is possible to prevent someone from telling someone else something he knows. It's odd to me that the GDPR bans the former, desirable system.
GDPR has expectations that Data Controllers take reasonable steps to protect a data subject. Theres no expectation that you anticipate all mathematical possibilities...
When user asks to be forgotten, simply overwrite all the identifying information with generated data. Backups work the same way, but depending on where and how those are stored it could get expensive.
We're seperating personally indentifying information out of our normal backups (held warm -> cold over time), and keeping PII backups offsite as warm data, so we can delete/ammend relatively pain-free.
Yes. GDPR compliance will cost money and I'm not trivialising that. The degree of panic it's causing baffles me though. Sure, it requires design up front. Oh woe. But regulatory compliance always does. Ref. previous rending of hair and gnashing of teeth in the run-ups to Glass–Steagall, HIPAA, Sarbanes–Oxley, and so on.
I think it's a combination of two things. On the one hand, people are justifiably worried whether or not their design and implementation of necessary changes will be fully correct. The cost of getting that wrong will be huge, and laws are generally not known for being clean and unambiguous. On the other hand, you also have people panicking because their user-hostile practices are no longer legal, and so they'll lose money.
And it's a sad state of affairs.
~12 years ago, in 2005, Kim Cameron first published his 7 laws of identity[1]. Follow those and you'll ace GDPR. Instead we've been directing the sharpest, brightest and smartest minds on the planet at squeezing revenue from advertising, consequences be damned.
[1] https://www.identityblog.com/stories/2005/05/13/TheLawsOfIde...
Maybe that should be true, but it isn't.
As a personal example, let's look at one of my own businesses that is not even remotely in the aggressive advertising or other data mining space. The only PII we keep is basic records of how our systems are used, who our customers are, what financial things happened, email correspondence, and so on. Nothing even remotely controversial. Nothing that more or less any online business wouldn't have to keep, often due to legal requirements.
And yet in at least one case we're still worried, because having read the actual GDPR, it is far from clear that things like our backup and logging systems aren't technically in violation. We don't know exactly how we would react if a customer did suddenly request deletion of everything we held on them, because we don't know exactly what our legal obligations would be in that situation.
Clearly we couldn't just delete everything, because we'd be required to keep a lot of that data for legal and accounting reasons. So if nothing else, we'd then have to spend hundreds or even thousands of pounds on taking expert legal advice, and then potentially hundreds or thousands more to update our systems to remove whatever we had to remove.
This is a side business run by a few people, part-time, like many other small or microbusinesses all over Europe. We're not dealing with anything like the scale of personal data of the Facebooks and Googles of the world. Apparently our carefully designed systems, which only collect a reasonable amount of data for legitimate purposes and take care to handle it securely, are already doing significantly better than some of the biggest data hoarding organisations in the world.
And yet, the effort and overheads required to address GDPR could pose an existential threat to the business. Tell me how that isn't disproportionate and how it's all clear and been thought through.
As you say, you must absolutely hold onto transaction history for tax purposes, at least! That said, unless you're selling arms or something (if compliance with the law is ambiguous go see a lawyer) you must honour an individual's right to be forgotten. You can do this without deleting records of transactions with that individual. You would however note that you can no longer identify the buyer because of such request x on date y. That should be as simple as deleting PII and then ammending the transaciton record with a comment.
Your biggest challenge here is how to handle the customer record. Do you delete it outright, and point the buyer field in the transaction record at a single GDPR ghost customer, or do you overwrite the customer data with random, non-PII values (I've always gone with the former)?
If your architecture is solid then these change requests should form part of ongoing BAU, no? Submit change request, develop, test, deploy...
Instant EU VAT law violation if you do this without retaining the required proof of customer location for at least 7 years. Formal tax investigation(s) to follow if you're big enough to be interesting and doing this systematically, possibly by up to 28 different member states' tax authorities depending on how much tax revenue each suspects that you have avoided paying them. Good luck. :-)
If your architecture is solid then these change requests should form part of ongoing BAU, no? Submit change request, develop, test, deploy...
And again I come back to my recurring theme that for many smaller organisations, almost every term you mentioned in that paragraph might as well be written in a foreign language. Small organisations don't have dedicated staff or established formal processes for running a lot of their day-to-day activities, never mind things like regulatory compliance.
You just made the log no longer simple. Now writing a log line isn't a matter of writing bytes to disk; it's a matter of looking up PII in one table, reading a key, then writing the key and the non-PII to disk. If the definition of PII gets more expansive … whoops.
> The degree of panic it's causing baffles me though. Sure, it requires design up front. Oh woe. But regulatory compliance always does.
You answered your question. The compliance costs, and it's for what is IMHO a non-right. I do not, and no-one has, a right to demand that my IP activity be erased. I do not, and no-one has, a right to change history.
Much of the rest of the GDPR is wonderful stuff, but the so-called 'right to be forgotten' is Orwellian.
But that's just my interpretation.
Can anyone, with citation, please shed some light on this. What's fair use, in broad terms, when it comes to simple HTTP logs or similar structures?
That’s generally not identifying a single person anymore, and usually good enough for anything else.
> That’s generally not identifying a single person anymore, and usually good enough for anything else.
'Usually'? Even if true (highly doubtful), that's not the same as 'always.' The whole purpose of logs is to be truthful accounts of pertinent data. A full IP address is a pertinent datum.
I'm going to step up on my soapbox and assert that any law which forbids me from indelibly recording that 192.0.2.17 requested /all-your-records-are-belong-to-us is a bad law.
It does require you to treat the information as PII, which is going to give you some hassles, but you are not banned from recording it.
If I want to record that a particular address accessed my system forever, that is my right.
Interestingly, the GDPR exempts records required for legal compliance. So it's okay to hold onto data for the law's purposes, but not my own? That's a bit one-sided.
In the general case (e.g. hosting a simple website where you strip user identifiers and the last octet out of the nginx logs) it becomes no PII then.
I don't really know about relational data, where a true "delete" would destroy someone's database. Somebody told me that, if hard deletions aren't technically feasible, then soft-deleting (or marking as archived or whatever) is acceptable. Not too sure I believe them though -- simply replacing the sensitive data with garbage and keeping the relations seems like a better idea.
Just also remember to record that you shouldn’t restore that persons data to the pre-forgotten state.
The European courts haven’t weighed in, but they’re not unreasonable, and the UK has given specific guidance:
https://ico.org.uk/media/for-organisations/documents/1475/de...
huehehue was interested in what the GDPR might mean by erasure and deletion. This is the ICO's opinion about it. It'd be a surprise if they suddenly interpret erasure and deletion differently: There would have to be something in the GDPR that suggests that this isn't good enough.
We can already achieve that result by removing records from the live DB and ignoring the whole backup situation.
So what has this separate encryption process achieved?
Even if your business does have the understanding and technical ability to do it, it adds a whole extra layer of complexity and unreliability to what used to be simple, plain text logs -- exactly the kind of information you probably need to access quickly and reliably if you're in the middle of fixing a major fault, for example.
But do you need to keep them forever? If you delete logs after 30 days you are unlikely to be impacted anyways.
We also have backups going back on a staged basis, more frequent backups from the recent past, less frequent going back further. This has been useful more than once for retrieving older information that someone had accidentally modified or deleted and not noticed immediately, so is also demonstrably useful for legitimate business purposes.
Both of these appear to be at risk of conflict with the right to erasure under GDPR, in that for example old backups of emails will inevitably contain customer correspondence that can't readily be isolated.
The trouble is that the definition of PII could be interpreted so broadly as to include almost anything useful ever logged on a server, because if a log record references any data that could be associated with a specific individual, including in combination with other data, then it counts. Given what we already know about de-anonymisation of supposedly anonymous data sets, even just based on quite simple patterns and correlations in the data, any approach based on pseudonymisation is likely to be simplistic and open to challenge.
In any case, what is reasonable for a business to want to do with that sort of data? Can we analyse which content has been most popular on a web site over the past week/month/year/decade? Can we analyse which content a particular paying customer has been accessing, in order to promote a new plan as they approach a limit or suggest a more cost-effective one if they aren't using what they're paying for right now? Can we analyse access patterns for an account over a period of time to detect shared use of an account contrary to our terms? There are plenty of legitimate business purposes for which you might want to know the entire history of access for a particular account and which do not violate the privacy of the user in any unjustified or unfairly exploitative way, but again, any method based on log pseudonymisation will immediately fail to satisfy those requirements.
Nothing about this discussion is even remotely as clear and straightforward as some in this discussion are suggesting.
Most businesses with personal data also have invoices. Since you don’t have to destroy those, simply note their request there.
Other kinds of personal data like marketing leads and medical data should be much easier to see how to forget.
If someone sends you something in the post telling you to delete their data, it should not be difficult to simply keep that record available when you need to restore an unrelated machines backup, unless you have incompetent IT.
This is part of the reason many GDPR consultants suggest applying an IT standard like ISO so that it’s clear you already know what backups are for and how to use it.
A single machine/database with all your business data on it mixed with personal data and the only record of requests from subjects is criminally irresponsible.
And to then restore to a point in time before the software or hardware bugs occurred, no?
> A single machine/database with all your business data on it mixed with personal data and the only record of requests from subjects is criminally irresponsible.
Um what? There are ton of good reasons to stick to a single DB, it’s not unreasonable at all. The main one being that it massively simplifies ensuring data integrity compared to a distributed system. Also it’s just easier for a small team to manage.
For example:
If someone sends you an email asking to be deleted, why would you store that email in the DB and nowhere else? Your team would read the email then update the database.
After a crash, if you then restored a backup, why wouldn't you be able to review your last days email (since the last backup) for any updates? Of course you would. Even small teams could.
"Who's IT?" -- Small and micro-businesses across Europe
This is part of the reason many GDPR consultants suggest applying an IT standard like ISO so that it’s clear you already know what backups are for and how to use it.
"We can't even afford a GDPR consultant. And who's IT?" -- Small and micro-businesses across Europe
A single machine/database with all your business data on it mixed with personal data and the only record of requests from subjects is criminally irresponsible.
Again, this is simply detached from the reality of many, many small and microbusinesses, community organisations, non-profits, etc.
It's absurd to think that every organisation that has reasonable need to handle personal data has a dedicated IT team, the people and budget to drop a few thousand on professional advice, the budget to buy extra equipment that isn't otherwise needed, etc.
It's also absurd to think that without those things, an organisation can't take reasonable precautions to protect that data or prevent its exploitation for illegitimate purposes.
Unfortunately the GDPR (as often happens with EU legislation) makes little allowance for any of this and thus imposes absurdly disproportionate overheads on smaller organisations.
But the photographer that hosts her website and help forms on wix[1] isn't going to have any problems. The local restaurant using viewtouch PoS and a facebook page isn't going to have any problems[2].
[1]: https://support.wix.com/en/article/general-data-protection-r...
[2]: https://www.facebook.com/business/news/facebooks-commitment-...
Very few real people will have any problems, because the GDPR is fundamentally about tradeoffs: If you have a lot of personal data, you need to protect it and inform the subject what you do with it. If you can't afford to protect it and think that the subject wouldn't want you doing what you're doing with their data, then you probably shouldn't have that personal data.
And process bugs/defects. And human error. Human error such as, for example, 'delete all of Mr. Smith's records.'
Translucent Databases covers this.
TL;DR: Various clever ways to encrypt records, organized for fast & easy retrieval, uncoverable if the key is lost (or deleted).
Also: We should always encrypt all (demographic) data at rest.
IIRC zeroing the last one or two groups of IPv4 and the /56 or /48 of the IPv6 address block should be fine.
Alternatively you can use a short hash function (xxhash32 cut to 16bits for IPv4 or pure xxhash64) to effectively destroy most of the data but still get a good pseudo-identifier.
If you can identify someone then it probably is. That’s for example if you have a web form with their details filled in— just make sure they know you’re keeping their IPv6 address for the avoidance of doubt.
If you can give it to someone like their ISP, and they can identify someone for you, then it might be: especially if you have a relationship with that ISP where you do that.
If not, then probably not.
And there's the whole question of what counts as identifying information. My understanding is that IP addresses do — so an immutable plaintext server log is now illegal.
Privacy is really, really important. But expanding the definition of privacy too far is just too much.