daamien

OP is one of 7 PostgreSQL core team members. That's kind of his job to massively overthink this :)

Hi !

I'm the main developer of this extension. Happy to answer any question you have about this project and anonymization in general!

Pseudonymizing functions are just one way to mask the data.

There are many other masking functions that will actually anonymize the data.

And the extension does not force you to respect the foreign keys.

It's really up to you to decide how you want to implement your masking policy

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained. This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

Both approach are possible

* Static Masking will destroy the authentic data once for all

* Dynamic Masking will only alter the data the "masked users". Regular users will continue to view the real data.

Both techniques have their own advantage depending on your context.

This is called "Privacy By Default"

https://postgresql-anonymizer.readthedocs.io/en/stable/priva...

Of course there's no perfect solution for anonymizing a dataset...

The extension offers a large panel of masking functions : some are pseudonymizing functions but others are more destructive. For instance there's large collection of fake data generators ( names, address, phones, etc. )

It's up to the database administrator or the application developer to decide which columns need to be masked and how it should be masked.

In some use cases, pseudonymization is enough and others anonymization is required....

Version 2.0 was released a few days ago. It's a complete rewrite in Rust.

Performance should be better than with v1.x

This called "Masking Data Wrappers"

https://postgresql-anonymizer.readthedocs.io/en/stable/maski...

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained.

This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

PostgreSQL Anonymizer developer here : the problem you are describing here only affects version 1.x

Version 2.0 was released a few days ago with a branch new masking system that does not block database migrations.

Actually the GDPR "just" requires to protect the data against "reasonably likely" attacks.

« To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly.

To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. »

https://www.privacy-regulation.eu/en/r26.htm

See PEV2 : https://github.com/dalibo/pev2

This is still an issue :

https://postgresql.verite.pro/blog/2018/08/27/glibc-upgrade....

... and with dozens of other data sources:

https://wiki.postgresql.org/wiki/Foreign_data_wrappers

PostgreSQL does this out of the box with :

  CREATE EXTENSION file_fdw;
  CREATE SERVER import FOREIGN DATA WRAPPER file_fdw;
  CREATE FOREIGN TABLE foo (
    col1 text,
    col2 text,
    ...
  ) SERVER import 
  OPTIONS ( filename '/path/to/foo.csv', format 'csv' );
  SELECT col1 FROM foo WHERE col2='x';

direct link to the PDF : http://www.postgresql.org/message-id/attachment/43080/Postgr...

Temporary index and Hypothetical index are two different things.

Temporary index is a real index that's usually dynamically created by the optimizer to speed up a query. But it's kind of useless because most of the time the cost of creating a one-time index exceed the cost of not using it. Afaik only DB2 as this feature

http://www.centerfieldtechnology.com/PDFs/DB2%20Temp%20Index...

An hypotetical index is just a way to "trick" the planner into thinking that there's an index and see what the query plan would be if that index really existed. The cost of creating this is close to zero.

Hi !

I'm involved in both projects. Here' s some insight :

OPM and POWA are not related. They were born at different times and developed by two different R&D teams at DALIBO, a French PostgreSQL company. The fact that they've been released publicly almost at the same time is merely a coincidence

OPM is designed to help a DBA to manage a very large number of servers. It provides answers to the questions : "Is there a problem somewhere on my PostgreSQL servers ? Which instances need to improvements ?"

POWA is more focused showing the "real-time" activity of one specific instance. It provides answers to questions like : "Which queries are slowing down my server right now ?"

If you're familiar with Oracle products, the difference between OPM and POWA is more-or-less the difference between Oracle AWR and Oracle Grid Control.

In a nutshell, if you only have one or two PostgreSQL servers in production, then POWA is easier to install and probably enough for you. If you want more stats and an overview of your servers, then OPM is a better choice.

Both projects are autonomous and moving fast. We're just pushing code on github without direct economic goals or marketing plans :-) In the long run we'll see which one finds its user base and how they evolve.

That being said, we also have other guys working on other similar PostgreSQL tools, in particular :

pgBadger ( http://dalibo.github.io/pgbadger/ ) is a log analyzer which provides rich reports of the database activity. Basically it answers to the question "Which queries should be optimized ?"

pgCluu ( http://pgcluu.darold.net/ ) is a audit/reporting tools, more focused on system and resource utilization. It answers to questions such as "What's the load on my CPUs ? Is my memory swapping ?"

pg_activity ( https://github.com/julmon/pg_activity/ ): is like linux "top" command but for PostgreSQL queries.

pg_stat_kcache ( https://github.com/dalibo/pg_stat_kcache ) will gather statistics about reads done by the filesystem layer

So basically regarding PostgreSQL monitoring, we're just moving in every possible directions at the same time... Some say it's a "waste of time", we prefer the term "Darwinian Approach" :P

fixed.