Show HN: A tool to seed your dev database with real data

Re:Show HN: A tool to seed your dev database with real data by gregwebs on Wednesday April 27, 2022 @09:40PM (#31187911)

Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.

[ Parent | on HN ]

Re:Show HN: A tool to seed your dev database with real data by jve on Thursday April 28, 2022 @03:19AM (#31189627)

That was the first thing I checked whether it supports data masking. And it does, via transforms, as evoxmusic already pointed out.

However, there may be times when data masking must be nuanced. Suppose some random email/domain pair is bad and you would rather replace all "example.com" domain instances with "fake.com", and not "random1.com", "random2.com", etc (for ML, 3rd party random analysis). Out of the box I don't see it is provided, HOWEVER I see that you can write custom transformer: https://github.com/Qovery/replibyte/tree/main/examples/wasm and fulfill your needs.

Excellent :)

[ Parent | on HN ]

Re:Show HN: A tool to seed your dev database with real data by evoxmusic on Thursday April 28, 2022 @01:39AM (#31189118)

That's why RepliByte uses S3 to store transformed data. Then the real production data never leave the production environment. https://github.com/Qovery/replibyte/blob/main/docs/DESIGN.md

[ Parent | on HN ]

Re:Show HN: A tool to seed your dev database with real data by gregwebs on Thursday April 28, 2022 @03:39AM (#31189769)

The doc refers to S3 as an intermediary- it is then going to be loaded in a different DB where all these issues exist. The intermediary adds one more place where data could be leaked.

[ Parent | on HN ]

Re:Show HN: A tool to seed your dev database with real data by closeparen on Thursday April 28, 2022 @12:51AM (#31188856)

And not using the chaos of the real world to harden your software before it hits production probably violates their expectations of correctness. It's a real and interesting tension! Software that helps people walk this line well is valuable.

[ Parent | on HN ]

Re:Show HN: A tool to seed your dev database with real data by pyr0hu on Thursday April 28, 2022 @01:53AM (#31189176)

Cannot agree more. There are tons of edgecases that only happens with the chaotic blob that we call production data. Tools like this or anyother that helps anonymizing data is soo useful for debugging issues

[ Parent | on HN ]