News for hackers. Stuff that matters.

A comment on Figure out who's leaving the company: dump, diff, repeat.

Re:Figure out who's leaving the company: dump, diff, repeat by simonw on Friday February 09, 2024 @12:43AM (#39311782)
If you're going to run something like this, I thoroughly recommend using Git for it.

You can have your cron do something like this:

    curl https://internal.corp/employees.txt > employees.txt
    git add employees.txt
    git commit -m "Automated: $(date -u)" || exit 0
The || exit 0 should ensure no errors even if there is nothing to commit

Now you have a commit history of every change made to that source of information - just run "git log" to view it.

I run this kind of thing on scheduled GitHub Actions all the time, see https://simonwillison.net/2020/Oct/9/git-scraping/

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by svat on Friday February 09, 2024 @01:00AM (#39311875)
I do something similar but instead of `|| exit 0` I use `--allow-empty` on the `git commit`. I don't mind the empty commits this creates, as they let me know that there was a successful automated run that happened to be empty, rather than having failed to run for whatever reason.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by michaelcampbell on Friday February 09, 2024 @09:48AM (#39315328)
Thanks for this, much more "intent revealing" than my (up to today) standard practice of `... || true` to keep my `set -e` from killing my script for this "error-but-not-really"
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by MarkSweep on Friday February 09, 2024 @12:57AM (#39311848)
Maybe add a ‘| sort’ in there for determinism. But yeah, git is an underrated database for this type of small scale data.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by hiAndrewQuinn on Friday February 09, 2024 @01:15AM (#39311942)
I used your exact technique to start generating a diffable archive for the Finnish easy language news broadcast. It's been a huge help in gathering high quality comprehensible input for me, thanks!

https://github.com/hiAndrewQuinn/selkouutiset-scrape/

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by itronitron on Friday February 09, 2024 @01:19AM (#39311968)
A colleague of mine would do a sort of temporal-network analysis of this data to see which people either enjoyed working with (or for) each other or which did not, based on how people would switch groups over time.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by cutemonster on Saturday February 10, 2024 @03:01PM (#39329674)
Did s/he use the analysis results for anything? (If you know)
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by itronitron on Sunday February 11, 2024 @04:57AM (#39333886)
I'm fairly certain they used it to gauge future project opportunities as our work was often multi-disciplinary across departments. It was probably also useful for validating people's stories if they came up. For example, if someone wanted to switch into your group 'A' from group 'C' and they had been working on a project with X, Y, and Z, then you could get a strong sense of why that might feel that way.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by jacquesm on Friday February 09, 2024 @12:47AM (#39311799)
That's clever, thank you! I will definitely use this.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by GauntletWizard on Friday February 09, 2024 @01:04AM (#39311893)
I've been seriously considering using Git for all sorts of oft-changing-but-rarely-majorly data. Lists of books in my bookshelves.

The other problem is that I sorta want transactional-database features on top of these things. Git does this well. I also want fast indexing on parts. Git does not do this well. I am considering writing a "standard" for the dumping of sqlite to git, so that I can just delegate this out; Any transaction can be expressed as a git commit, and I can run both at once for both the durability and the reasonable indexing; The sqlite database can be re-created and reindexed whenever, and it also sorta works for backups...

Definitely just spinning my wheels, though. We'll see where databases take us next.

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by rrrrrrrrrrrryan on Friday February 09, 2024 @03:43AM (#39312760)
> oft-changing-but-rarely-majorly data

I think you're referring to SCDs, and there are plenty of well-defined ways to track these within relational databases:

https://en.wikipedia.org/wiki/Slowly_changing_dimension

Why git?

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by jarofgreen on Friday February 09, 2024 @03:15AM (#39312609)
In your git & sqlite setup, I'm not sure which way round you are thinking - which is the SSOT (Single Source Of Truth) and which is the handy cache.

I've been working on a tool that treats the git repo as the SSOT then lets you dump out all kinds of formats for data work including a sqlite DB. I haven't had as much time as I would like on it but it's at https://pypi.org/project/DataTig/

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by ElectricalUnion on Friday February 09, 2024 @02:39AM (#39312425)
Would fossil fit this bill?

Somewhat git-compatible, based on sqlite3.

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by JonChesterfield on Friday February 09, 2024 @08:03AM (#39314402)
Specifically the fossil repo is a sqlite database.

I have a cron script stashing whatever is currently in an obsidian vault into a fossil repo. There's a fossil addremove command that makes that very easy. Thus distributed backups of said vault.

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by Zambyte on Friday February 09, 2024 @12:10PM (#39317287)
What does "somewhat git-compatible" mean? Can it sometimes use existing git repos, or is the mental model close enough to pick up without learning much?
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by ElectricalUnion on Friday February 09, 2024 @12:39PM (#39317764)
> is the mental model close enough to pick up without learning much?

I would say git core concepts are pretty similar to fossil concepts, but actual plumbing implementation details are pretty distinct.

The major difference that I remember from a day-to-day "git porcelain" perspective is that rebases and other types of history rewriting are very discouraged.

For a Rosetta stone of somewhat comparable commands: https://fossil-scm.org/home/doc/trunk/www/gitusers.md

> Can it sometimes use existing git repos

You can, but it's kinda a lot of really slow busywork. And you lose some of the not-file-dvcs features of Fossil, but it is possible.

This page explains how: https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub...

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by rofrol on Friday February 09, 2024 @01:28AM (#39312015)
Maybe this https://github.com/dolthub/dolt
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by GauntletWizard on Friday February 09, 2024 @02:40AM (#39312437)
Dolt is really close and yet just doesn't feel right; I don't want to "commit" between transactions, I want every transaction to be a commit.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by timsehn on Friday February 09, 2024 @11:22AM (#39316421)
You can do this with a setting:

https://docs.dolthub.com/sql-reference/version-control/dolt-...

[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by GauntletWizard on Friday February 09, 2024 @12:59PM (#39318047)
That's awesome. I will have to try that out.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by GauntletWizard on Saturday February 10, 2024 @10:32PM (#39332447)
If anyone is coming in late, I have discovered https://immudb.io/ , which claims to be a lot of what I'm looking for. Transactional SQL database, indexed, yet immutable data and verifiable chains of transactions.
[ Parent | on HN ]
Re:Figure out who's leaving the company: dump, diff, repeat by bandie91 on Saturday February 10, 2024 @02:18PM (#39329324) 

  if git is-there-anything-to-commit; then git commit ...; fi

  'is-there-anything-to-commit' is aliased to '! [ -n "$(git status -s --porcelain --untracked-files=no)" ]'
[ Parent | on HN ]