Re:Show HN: We scaled Git to support 1 TB repos
by WorldMaker on Tuesday December 13, 2022 @01:10PM
(#33973108)
Something that I've experienced from many years in enterprise software: 90% of enterprise software is about versioning data in some way. SharePoint is half as complicated as it is because it has be a massive document and data version manager. (Same with Confluence and other competitors.) "Everyone" needs deep audit logs for some likely overlap of SOX compliance, PCI compliance, HIPAA compliance, and/or other industry specific standards and practices. Most business analysts want accurate "point in time" reporting tools to revisit data as it looked at almost any point in the past, and if you don't build it for them they often build it as ad hoc file cabinets full of Excel export dumps for themselves.
The wheels of data versioning just get reinvented over and over and over again, with all sorts of slightly different tools. Most of the job of "boring CRUD app development" is data version management and some of the "joy" is how every database you ever encounter is often its own little snowflake with respect to how it versions its data.
There have been times I've pined for being able to just store it all in git and reduce things to a single paradigm. That said, I'd never actually want to teach business analysts or accountants how to use git (and would probably spend nearly as much time building custom CRUD apps against git as against any other sort of database). There are times though where I have thought for backend work "if I could just checkout the database at the right git tag instead needing to write this five table join SQL statement with these eighteen differently named timestamp fields that need to be sorted in four different ways…".
Reasons to version data are plenty and most of the data versioning in the world is ad hoc and/or operationally incompatible/inconsistent across systems. (Ever had to ETL SharePoint lists and its CVC-based versioning with a timestamp based data table? Such "fun".) I don't think git is necessarily the savior here, though there remains some appeal in "I can use the same systems I use for code" two birds with one stone. Relatedly, content-addressed storage and/or merkle trees are a growing tool for Enterprise and do look a lot like a git repository and sometimes you also have the feeling like if you are already using git why build your own merkle tree store when git gives you a swiss army knife tool kit on top of that merkle tree store.
The wheels of data versioning just get reinvented over and over and over again, with all sorts of slightly different tools. Most of the job of "boring CRUD app development" is data version management and some of the "joy" is how every database you ever encounter is often its own little snowflake with respect to how it versions its data.
There have been times I've pined for being able to just store it all in git and reduce things to a single paradigm. That said, I'd never actually want to teach business analysts or accountants how to use git (and would probably spend nearly as much time building custom CRUD apps against git as against any other sort of database). There are times though where I have thought for backend work "if I could just checkout the database at the right git tag instead needing to write this five table join SQL statement with these eighteen differently named timestamp fields that need to be sorted in four different ways…".
Reasons to version data are plenty and most of the data versioning in the world is ad hoc and/or operationally incompatible/inconsistent across systems. (Ever had to ETL SharePoint lists and its CVC-based versioning with a timestamp based data table? Such "fun".) I don't think git is necessarily the savior here, though there remains some appeal in "I can use the same systems I use for code" two birds with one stone. Relatedly, content-addressed storage and/or merkle trees are a growing tool for Enterprise and do look a lot like a git repository and sometimes you also have the feeling like if you are already using git why build your own merkle tree store when git gives you a swiss army knife tool kit on top of that merkle tree store.