bloodearnest

Indeed it does, we are well aware of this, and write about it extensively.

The leader of the OpenSAFELY project produced the Goldacre review for the UK Government, which discusses this risk in detail. Here's a quote from the executive summary:

  Knowing the approximate date range in which someone had a medical intervention, their 
  approximate age, and their approximate location is often enough to re-identify someone in a 
  pseudonymised dataset, and then – illegally – to see everything else in their record. Women 
  face particular concerns: knowing someone’s approximate age, approximate location, and the 
  approximate time at which they had children can also often be enough to make a confident 
  unique match; this is the kind of information that will be known by someone at the school 
  gate, or a colleague. This is not to say that health data users are untrustworthy: but the 
  system must be resilient to untrustworthy users; and it is well documented that other large 
  administrative national datasets are sometimes misused.[1]

That's why there are multiple layers of security within the OpenSAFELY system. Other layers include: multiple authentication required from separate orgs to view outputs, various limits on the outputs you can view, full public audit trail of all code run, and all outputs viewed, separation between the systems used to run code and to view outputs (so compromise of one system doesn't affect the other). And finally, you cannot release an output from a secure backend until it has been checked for potential disclosivity by 2 ONS[2] trained reviewers.

Like all computer system, you cannot remove all risk from the system. But you can reduce it significantly, and make all activity public and auditable. And it's a damn sight better than current practice for medical data research, tbh. We try as much as possible to provide data-minimisation, in that you only have the minimal set of data you need answer your question. Whereas, in the UK at least, most of the time, researchers just get given access to everything.

[1]https://www.gov.uk/government/publications/better-broader-sa...

[2] Office of National Statistics, UK equivalent of the US Census Bureau

EDIT: fixed a reference

FWIW, the highest precision of location available is UK's MSOA[1] regions, which is more like your whole neighbourhood than your street.

And whilst that level of data is available, it is not allowed to be released from the OpenSAFELY system at that level, it must be aggregated.

[1] https://geoportal.statistics.gov.uk/search?collection=Datase...

edit: added a missing word

I don't think so - do you have a link?

Also, the OpenSAFELY team are actually all honorary employees of NHS England as part of the Information Governance for the project. Thus AIUI we are beholden to their policies and obligations regarding FOIA.

And nearly everything except the patient data is public anyway!

Also, the actual data is held by the GP system suppliers, who have their own FOIA obligations.

Nice Tuesday morning surprise to find your work on the front page of HN!

I am one of engineering team working on OpenSAFELY at the Bennett Institute.

Feel free to ask me any questions about the design or implementation of the system. We take patient privacy very seriously - it's something of a crusade for us!

Some more information

- all the researcher code is at https://github.com/opensafely

- all the platform code is at https://github.com/opensafely-core

- high level architecture overview talk by myself and my colleague Becky at PyCon UK '22 https://www.youtube.com/watch?v=L55mq5wi3Cc

EDIT: formatting

Some good points, but just to clarify snaps current behaviour re libssl et al:

Snap makes a distinction between a minimal set of critical core system dependencies, and application dependencies.

So on every system with a snap, there is also a 'core' snap, which has the base root fs, including glic, libssl, bash, etc. This core snap is updated regularly by its publisher (just Canonical's ubuntu core snap right now, but nothing stopping some other core snap). So a lot of critical updates are applied by default, as per normal distro policy, but just in a better way (rollback, transactional, qa, etc).

The application snap is only responsible for keeping its own dependencies up to date, it uses the base libraries from the core snap. It's distinct from docker in this way, it tries to define a line between core and app dependencies, to get the best of both worlds. I.E. critical components are updated ASAP by the disro security team, but apps are free to use a specific version of a library (e.g gnome, qt) that they want to use, rather than being stuck on the distro maintained version. Which, of course, they can still use if they want :)

It does push the burden on updating libraries to the app vendor, but on the flip side, snaps run in a very confined environment, which mitigates the scope of most CVEs there might be in an older library.

A lot of careful trade-offs considered and involved in the design - we'll see they're the right trade offs or not :)

I think readlines() is slower in python 3 due to unicode decoding. Python 2 did no decoding, readlines() returns python 2's str type, just bytes. You'd have to decode yourself (or decode the whole file in open()). Python 3 readlines() does decoding into unicode code points by default.

Do a benchmark after the file has been opened with an explicit encoding, to see if I'm right. If the files already been decoded, there shouldn't be as big a difference.