News for hackers. Stuff that matters.
Re:How I think I want to drop modern Python packages into a ... by linkdd on Thursday September 17, 2020 @02:40PM (#24508371)
Deduplication should happen at the filesystem level.

Databases do it, Git does it, ZFS does it.

Ok the first two examples aren't filesystems but they can store files, so not totally irrelevant?

Re:How I think I want to drop modern Python packages into a ... by zwieback on Thursday September 17, 2020 @02:49PM (#24508473)
Yeah, agree kind of. But having to resort to duplicating binaries over and over is a sign something isn't right. Not that it's an easy thing to solve.
Re:How I think I want to drop modern Python packages into a ... by linkdd on Thursday September 17, 2020 @02:59PM (#24508578)
It depends, binaries are just data. You reference that data with a path on the filesystem. In a programming language it would be assigning a value to a variable. For static values, the compiler/interpreter can optimize and do deduplication. And you expect it to do so because you don't want a global variable holding the value 1 to do the optimization yourself.

Relying on abstractions to simplify the architecture is a sign something might be right. Simply because simple things are less error-prone IMHO.