Dat – A git-like tool for large datasets

Re:Dat – A git-like tool for large datasets by coherentpony on Friday April 25, 2014 @11:09PM (#7650264)

I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores.

I could be missing something here so anybody more knowledgeable should chime in.

[ Parent | on HN ]

Re:Dat – A git-like tool for large datasets by platz on Friday April 25, 2014 @11:12PM (#7650270)

It might not be optimal; however I wonder if most of the primary sources this is intended to target (i.e. govt) prefers to use textual data?

[ Parent | on HN ]

Re:Dat – A git-like tool for large datasets by mistursinistur on Friday April 25, 2014 @11:23PM (#7650300)

Nope. Any really large government data is not text -- it's mostly in older open binary formats (e.g. weather data in GRIB from NOAA) or expensive proprietary binary formats (e.g. healthcare data in SAS from CMS).

[ Parent | on HN ]

Re:Dat – A git-like tool for large datasets by coherentpony on Friday April 25, 2014 @11:16PM (#7650283)

>I wonder if most of the primary sources this is intended to target (i.e. govt) ...

The second sentence on their website appears to state otherwise: "As a team we have a bias towards supporting scientific + research data use cases."

Edit: Well, on further reading they do go on to say, "What we're building isn't quite ready for prime time yet, but if you want to play around...", so perhaps I should not jump the gun :)

[ Parent | on HN ]