News for hackers. Stuff that matters.
Re:Show HN: Neosync – Open-Source Data Anonymization for Pos... by NortySpock on Wednesday May 22, 2024 @03:25PM (#40445144)
So, how does one create synthetic relational data? Do you just crank out a list of synthetic customers, assign IDs, create between 0 and 3 synthetic orders per person, and between 0 and 3 order line entries per order?
Re:Show HN: Neosync – Open-Source Data Anonymization for Pos... by blopker on Wednesday May 22, 2024 @03:33PM (#40445282)
This is somewhat framework dependent, but factory_boy supports connecting factories together via SubFactory. There's a real-world example I'm building [0]. See where "author = SubFactory(UserFactory)". I'd imagine there are similar ways to do this for Rails and others too.

[0]: https://github.com/totem-technologies/totem-server/blob/main...

Re:Show HN: Neosync – Open-Source Data Anonymization for Pos... by edrenova on Wednesday May 22, 2024 @03:36PM (#40445324)
we're actually working on this right, can see the PR here -> https://github.com/nucleuscloud/neosync/pull/1832/files

it's a combination of creating a random number of records for foreign keys i.e 1 customer - create between 2 and 5 transctions. Working on giving you control over that, and handling referential integrity with table constraints (foreign keys, unique constraints, etc.)

ML based approaches typically are not very good at this and struggle with handling things like referential integrity. So a more "procedural" or imperative way is slightly better. The ideal is a combination of both.

Re:Show HN: Neosync – Open-Source Data Anonymization for Pos... by beeboobaa3 on Wednesday May 22, 2024 @03:36PM (#40445331)
Pretty much, yeah. Use normal distribution so you can get some outliers.