News for hackers. Stuff that matters.
Re:The semver trick (2019) by hinkley on Thursday December 26, 2024 @02:51PM (#42517292)
> The cause of the difficulty was the large number of crates using types from these libraries in their public API.

This has bitten me on the ass so many times. Exposing third party types means you have to upgrade the library simultaneously across the entire application, even if the language allows you to load different versions of the same library in different parts of the app. And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint.

I’d much rather my coworkers use libraries instead of NIH, but you have to be very careful how you use them. It gives me a little sympathy for microservices, but that’s often just swapping in a devil you don’t know.

Re:The semver trick (2019) by do_not_redeem on Thursday December 26, 2024 @03:09PM (#42517412)
Microservices only solve this because you have to serialize objects on the wire, no? You could accomplish the same within an app or lib by accepting a String and calling `serde_json::from_str` on it, avoiding the problems of microservices. Or more realistically, define your own struct and convert it to/from the dependency type at the interface boundary.
Re:The semver trick (2019) by hinkley on Thursday December 26, 2024 @03:20PM (#42517481)
Process boundaries tend to increase friction to exposing internal implementation details like this. They don’t fix the problem but they do discourage it.
Re:The semver trick (2019) by mgaunard on Thursday December 26, 2024 @03:09PM (#42517410)
Using third-party libraries is usually a bad idea, and must be done so very carefully. If you do, onboard it onto your own build system, and pin it to a version you're going to provide support for and ensure compatibility with.

You should never have multiple versions of the same code in objects that end up linked together, that's a recipe for ODR.

Re:The semver trick (2019) by progval on Thursday December 26, 2024 @03:30PM (#42517549)
Concrete example: the Arrow Arrow format is designed to allow passing arrays across languages and libraries without conversion; so many public functions/methods (and FFIs) will have Arrow arrays in their signature.

However, the Rust Arrow library bumps its major version every few months because it changes methods and auxiliary types, without changing the memory layout of arrays. But this means you can't have two dependencies (eg. an ORC parser and a CSV serializer) that depend on different versions of Arrow if you want to pass arrays from one dependency's functions to the other's.

And vendoring like you mention won't help, because the Rust compiler wouldn't recognize both vendored Arrow crates as compatible.

Re:The semver trick (2019) by mgaunard on Thursday December 26, 2024 @05:48PM (#42518455)
Use a single version of the arrow library, and build everything against that one. I don't see where the problem is.

I don't use Rust or third-party build or packaging systems -- I usually recommend that people don't do so either, but I understand that cargo is part of the appeal of Rust.

I'd say just build whatever you need to do, this way you won't be limited by arbitrary restrictions others have decided.

Re:The semver trick (2019) by progval on Thursday December 26, 2024 @05:59PM (#42518523)
I can't always do that because dependency A depends dependency B that depends on arrow 52, and dependency C depends on arrow 53.

And sure, I can soft-fork dependency B to add support for arrow 53, and dependency A to depend on my soft-fork of dependency B, but it quickly becomes messy. Especially when in parallel I send unrelated patches to A and/or B.

Re:The semver trick (2019) by mgaunard on Friday December 27, 2024 @05:26AM (#42521195)
your problem is caring about what other people have decided. Surely metadata is irrelevant and can be changed/overridden.

You onboard a version and make sure everything you need can build against it, patching software as needed.

Re:The semver trick (2019) by hinkley on Friday December 27, 2024 @11:54AM (#42523755)
Security patches say no, doesn’t work that way.
Re:The semver trick (2019) by hinkley on Friday December 27, 2024 @11:55AM (#42523766)
Sounds like a team of misanthropes. How can you be that blind to DevEx?
Re:The semver trick (2019) by mgaunard on Saturday December 28, 2024 @07:46AM (#42530689)
For the best developer experience, you need to write custom tools.
Re:The semver trick (2019) by hinkley on Saturday December 28, 2024 @01:38PM (#42533300)
Eh. Software is full of cliches and idioms. I shouldn’t be writing my own logging library and I sure as fuck shouldn’t be writing my own security library.

Also don’t write your own billing system unless you’re a finance company.

Re:The semver trick (2019) by eastbound on Thursday December 26, 2024 @03:39PM (#42517621)
CVEs are detected on third party libraries. This is why you must upgrade.
Re:The semver trick (2019) by sethammons on Thursday December 26, 2024 @02:57PM (#42517332)
> And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint.

this is why I'm not on team monorepo. Let other teams vendor their dependencies and upgrade on their own time. Just don't go trying to release N versions. Measure who is on an old version and work with them to update. But again, this slow team allows the other N-1 teams to adopt the new version and provide value to the org.

for those of you who are pro monorepo and have worked in one with dozens of teams in different orgs who have competing priorities and can't upgrade a dependency in lock-step, how have you solved this?

Re:The semver trick (2019) by osigurdson on Thursday December 26, 2024 @03:27PM (#42517528)
Does monorepo confer that all projects use the same dependencies? Binary dependencies seem orthogonal to how source code is managed.
Re:The semver trick (2019) by adastra22 on Thursday December 26, 2024 @04:05PM (#42517813)
Usually, yes. Monorepo means all dependencies are checked into the same repo. Even external dependencies (when source available) are checked in.
Re:The semver trick (2019) by osigurdson on Friday December 27, 2024 @01:26AM (#42520318)
Usually we are using package managers and just referencing packages. However, even in the hypothetical where I am downloading binaries and checking them in, I still don't see the relationship to the monorepo. I can have a monorepo with 100 folders representing 100 programs, each of these can have completely separate dependencies, languages, etc.

Dependency versions matter when they are all part of the same program or library. Then you have the diamond dependency issue and related problems. However, none of this is related maintaining code in a monorepo. One program can be maintain in 10 different repos (impractical but possible) and still have the diamond dependency issue.

Re:The semver trick (2019) by hinkley on Saturday December 28, 2024 @01:40PM (#42533315)
I’ve done monorepo, separate compilation units before. It has its own set of issues but it cuts down on circular dependencies and makes more sense than having sixty repositories checked out. Those are hell.
Re:The semver trick (2019) by mgaunard on Thursday December 26, 2024 @03:15PM (#42517452)
1. ensure you have continuous integration against both stable and latest versions of everything, with the stable set of all transitive dependencies defined on a per-project basis, and what gets used for releases.

2. people are expected to fix the latest build whenever they have spare cycles. The incentive is that it is part of the pipeline for code reviews and can only be bypassed by the project owner. This means you'll need to support both stable and latest concurrently (bumping stable is also an option to simplify the requirement).

3. if you added conditional support for old stable/new latest and latest was eventually made stable, you can refactor that code.

Re:The semver trick (2019) by frizlab on Thursday December 26, 2024 @02:59PM (#42517347)
Wouldn’t subtrees fix this instead of a mono-repo? Or even (gasp) submodules?
Re:The semver trick (2019) by osigurdson on Thursday December 26, 2024 @03:28PM (#42517535)
You mentioned sub modules - you are now on call until someone else mentions them again.
Re:The semver trick (2019) by frizlab on Thursday December 26, 2024 @04:41PM (#42518055)
I’m fine with that :)
Re:The semver trick (2019) by dallasg3 on Thursday December 26, 2024 @03:22PM (#42517492)
I was looking into Paket for this on .NET.

https://github.com/fsprojects/Paket

Re:The semver trick (2019) by alexchamberlain on Thursday December 26, 2024 @04:33PM (#42517995)
If you have a single deployable(1), I think you need to have a person/team responsible for the whole. They need to be trusted to work on any of the code to make the whole build, without overstepping into micromanaging the whole code base. It's a tough balance.

(1) it has to be said I don't have much experience with mono-builds, but having brought together many services ( micro or not) from different teams and backgrounds, I wish for a little more coordination and governance.