Re:Show HN: Buckaroo – A decentralized C++ package manager
by Firecracker on Friday February 01, 2019 @02:33PM
(#19057620)
Counteracting some of the comments below:
My experience using buckaroo in a large project over the past year has been extremely positive, and I'd say addresses most of the problems raised here. It's far and away better than anything else I've used.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by DoctorOetker on Friday February 01, 2019 @04:39PM
(#19058942)
I don't read any content in your comment?
Your experience (using buckaroo ...) adresses most of the problems raised here? You could perhaps share at least how your experience adressess some of these problems raised here: how does your experience address the hidden telemetry for example?
Re:Show HN: Buckaroo – A decentralized C++ package manager
by ferdek on Friday February 01, 2019 @04:39PM
(#19058933)
I've made an account just to upvote this.
I've finished reading manual on Nix and just started with documentation on packaging. I want to have total control on dependencies, together with compiler versions and standard library implementations (effectively cross compiling everything x86_64 -> x86_64). From Buckaroo, Conan and Nix, only Nix gets this right...
Re:Show HN: Buckaroo – A decentralized C++ package manager
by IshKebab on Friday February 01, 2019 @02:15PM
(#19057440)
So many people have tried and failed to do this. I'm skeptical that these guys have succeeded.
The main problem seems to be that because there is no standard C++ build system, every library uses a different one. Wrangling all those together without requiring the library authors to do anything is extremely difficult.
Hell many C++ libraries still use autotools or hand-written Makefiles. I don't really see how it is even possible to automatically download and build those projects.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by entelechy on Friday February 01, 2019 @02:47PM
(#19057766)
One of the authors here.
Yes you are totally right, as build-systems are not standardized it is quite challenging.
However we have seen more than 300 ports back in 2017 [1] and saw many emerging patterns.
This enabled us to write rules to transpile and optimize 100s of build-systems to buck using
buildinfer [2] [3].
We hope that more sane decisions will be made regarding build-systems in new projects.
We discovered analyzing over 300 projects that a lot of complexity is not needed and often a non-turing complete language for describing the build is sufficient. In fact we found that over 90% of projects can be described solely by using glob expressions.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by ghthor on Friday February 01, 2019 @04:58PM
(#19059139)
That's so freaking cool! I'm not to surprised that 90% can be globs. Were there any projects that you saw that were polyglots? Maybe some type of library that had built in diagnostics and metrics providing prebuilt html or interactive webpages?
Re:Show HN: Buckaroo – A decentralized C++ package manager
by entelechy on Friday February 01, 2019 @05:13PM
(#19059257)
May projects use some document generators or tools to collect metrics. However some build-systems that were really interesting were OpenCV and OpenSSL.
OpenSSL buildsystem is over 6k lines of perl scripts which communicate over pipes.
OpenCV uses cmake, python and prolog for it's build. Until this day, I have no clue why there is prolog involved.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by shin_lao on Friday February 01, 2019 @02:25PM
(#19057539)
It's indeed going to be very, very, hard to retrofit a package manager into C++.
You could structure something around CMake.
One immediate issue is compilation flags and platforms management. You may need to use some custom flags, for example, you need to compile against a specific architecture. You want those flags to be passed to all libraries uniformly.
But the horrible problem is quickly the compatibility matrix.
In other words, if you are making available, let's say you are using alib v1 and blib v2, you need to check the two libraries compile, link, and work together nicely. So when you upgrade one of the libraries, you need to check that compatibility again.
It's an important problem because if you're working on production software, you need to be able to build all versions and maintain them. And maybe you will have to upgrade only one library. You can't always be "current" everywhere.
So very quickly, using a package manager becomes more painful than having the libraries management manually with a couple of custom scripts.
I'll be happily be proven wrong by a package manager (and we tried a lot).
Re:Show HN: Buckaroo – A decentralized C++ package manager
by dcuthbertson on Friday February 01, 2019 @05:21PM
(#19059334)
I can't say I'd be a fan of a CMake based package manager. While CMake has been a great tool under Linux environments (I've used it for C/C++ builds, and in mixed C & Go build environments), I've also experienced horrors when CMake is used to generate Visual Studio solution and project files.
I mostly work in Windows development. CMake scripts created by a professional build engineer caused a lot of havoc. It injected itself into the project files in such an insidious fashion, that it was nearly impossible to tell if dependencies were correct. It completely broke Intellisense, making code navigation an exercise in frustration.
I was not happy, and wound up ripping out the scripts and creating new solution and project files that actually supported the needs of the developers.
i will say that in my time using conan to attempt to solve for compiling an application for x86 and a few flavors of ARM, this is NOT a silver bullet.. a huge fraction of third party libraries we tried to pull in needed to have their recipes modified to build correctly.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by entelechy on Friday February 01, 2019 @03:16PM
(#19058062)
Afaik Conan requires a server and you cannot install from arbitrary sources. Furthermore Conan prefers to download precompiled libraries. Unless you run your own artifactory, this may be problematic for people for people concerned about security or ABI compatibility.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by cbuq on Friday February 01, 2019 @03:29PM
(#19058216)
Just to break down your complaints.
1. Conan does not require a server. If a project contains a conanfile.py with build instructions, you can also run `conan create` from a project's source repository (no server required).
2. You can easily force build dependencies with the `--build *` flag
Re:Show HN: Buckaroo – A decentralized C++ package manager
by adgasf on Friday February 01, 2019 @04:31PM
(#19058849)
Interesting. Regarding (1), does this mean you have to download the dependency from GitHub by hand, then put it into the cache before you can "install" it?
Re:Show HN: Buckaroo – A decentralized C++ package manager
by adgasf on Friday February 01, 2019 @06:36PM
(#19059838)
I think this misses the point though. You shouldn't have to "prepare" all of the dependencies you might require since this could be such a large graph.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by rienbdj on Friday February 01, 2019 @02:43PM
(#19057728)
This is the chicken - egg problem of C++ package managers. one approach is to try to wrap all build systems (Conan); the other is to port to a common build system (vcpkg, Hunter, Buckaroo). The wrapping approach makes it very hard to inspect the build and customize things. The porting approach makes it hard to get things moving. I think the porting approach will prove better in the long run.
What we really need are automated porting tools to fix this mess!
Re:Show HN: Buckaroo – A decentralized C++ package manager
by dana321 on Friday February 01, 2019 @02:35PM
(#19057648)
linuxbrew / homebrew does a pretty good job at downloading packages and compiling them.
The wildcard is cross-platform. vcpkg doesn't seem to be 100% there and has many bugs for example the folders where includes and libraries are kept changes based on platform.
The other problem is static vs dynamic linking, as well as conflicts between different combinations of cmt libs etc. it gets very messy quickly.
Its almost like you're spending more time trying to get all the libraries you want working together than actually coding.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by rienbdj on Friday February 01, 2019 @02:37PM
(#19057666)
The solution in my experience (also the approach at Google) is to always build from source but cache the build artefacts. This is how Buckaroo works too (although it is Buck instead of Bazel).
Lastly I'd like to note that Buck and Bazel are converging to Skylark for describing builds. The differences between Buck and Bazel should become smaller over time
Re:Show HN: Buckaroo – A decentralized C++ package manager
by spacksmack on Friday February 01, 2019 @03:34PM
(#19058267)
Obligatory Spack reference: Spack is also worth checking out if you want to build everything from source. A lot of the national labs use spack now because you can specify all the dependencies for a tool (eg, g++8, boost1.68, opempi3) and it will build out any of the missing tools/compilers for you in one command. It doesn't solve the build system stuff like buckaroo, but it works with whatever build stuff I want to do (cmake, make, sh).
Re:Show HN: Buckaroo – A decentralized C++ package manager
by MereInterest on Friday February 01, 2019 @02:59PM
(#19057882)
Could you verify what you mean by an include source statement? As you describe its usage, it seems identical to the already existing #include statement, which does a text inclusion of another file.
> Single Compilation Unit (SCU) is a computer programming technique [...] The technique can be applied to an entire program or to some subset of source files; when applied to an entire program, it is also known as a unity build.
Who came up with that? I would say this "technique" doesn't even justify having a wikipedia page. In modern c++ you can be glad if you have a c++ unit that doesn't need to recompile on change because of all the header-only stuff. No one would include .cpp-files by choice.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by c-smile on Friday February 01, 2019 @05:28PM
(#19059381)
In C/C++ static declarations are local to the compilation unit. So you will have two separate storages for foo - these will be two independent variables.
That's how C and C++ work.
And if you will have this (no static):
int foo = 45;
then linker will complain - "multiple 'foo' entities" as non-static entities have global visibility.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by DoctorOetker on Friday February 01, 2019 @04:42PM
(#19058961)
perhaps we don't as much need a specific package manager implementation as standard(s) for package maintainers so different package manager implementations can ingest the compliant packages?
Re:Show HN: Buckaroo – A decentralized C++ package manager
by simfoo on Friday February 01, 2019 @02:23PM
(#19057528)
The Buckaroo workflow looks like this:
# Create your project file
$ buckaroo init
# Install dependencies
$ buckaroo add github.com/buckaroo-pm/boost-thread@branch=master
# Run your code
$ buck run :my-app
And yet another development tool that wants to hijack my workflow. No thanks
Conan.io already does decentralization right and is well on the way on becoming the defacto standard package manager for C++. And all of that without constraining my workflows
Re:Show HN: Buckaroo – A decentralized C++ package manager
by simfoo on Friday February 01, 2019 @02:54PM
(#19057832)
It does not require a server. It supports arbitrary remotes (like git does) to exchange packages, but it works without one as well (just publishing packages to the local cache)
Re:Show HN: Buckaroo – A decentralized C++ package manager
by simfoo on Friday February 01, 2019 @03:58PM
(#19058526)
It is supported. You create the packages using "conan create" from any place with a recipe, which puts the packages into your local conan data directory (called the cache).
Any other recipe can then depend on any package that is already in the cache.
Re:Show HN: Buckaroo – A decentralized C++ package manager
by simfoo on Saturday February 02, 2019 @02:47AM
(#19061755)
No, conan create builds a package from a recipe. Provided you cloned sources form a git remote, you can locally build the conan package without conan ever hitting your network.
You can even just export any recipes into your local cache without manually building the package and then let conan do the work when you require the package as a dependency (using --build all)
Re:Show HN: Buckaroo – A decentralized C++ package manager
by adgasf on Saturday February 02, 2019 @06:17AM
(#19062448)
I didn't realize that a recipe was distinct from a package. In most systems there is only one concept: packages.
So to see if I have this right:
With Conan, to use a package from GitHub, I must download the recipe and add it to my local cache ("cache" seems like the wrong term here, maybe "registry" would be better?) and then I can install it?
What about the transitive dependencies? Do I need to repeat the process for each of those?
Re:Show HN: Buckaroo – A decentralized C++ package manager
by simfoo on Saturday February 02, 2019 @03:42PM
(#19065580)
There are no conan packages on Github, usually you can find recipes along (or separate) the sources.
Yeah, the term conan "cache" may be a bit misleading, as it is not only a cache but really your "local remote". As soon as a recipe is in your cache, you can depend on it from your own recipes/conan builds. The actual building of the package may happen during the registration of the recipe or during resolve time when you depend on a certain package configuration. Any transitive dependencies will be resolved and, in case of --build missing, the packages will be built automatically according to the recipe.
So no, you do not need to build any dependency packages (including transitive dependencies)
Re:Show HN: Buckaroo – A decentralized C++ package manager
by entelechy on Sunday February 03, 2019 @10:46AM
(#19070179)
But all those dependency packages will be resolved against a conan server not against the origin eg. github.
Furthermore I'd need to put every version of the package into the conan cache.
This is impractical if a complex dependency graph as it would require the user to solve a SAT problem just to determine which packages and versions to put into the conan cache to avoid pulling from the conan server.
https://github.com/LoopPerfect/buckaroo/wiki/Installation#te...
https://github.com/LoopPerfect/buckaroo/wiki/Telemetry
We gather this data so we can improve Buckaroo and it's ecosystem.
My experience using buckaroo in a large project over the past year has been extremely positive, and I'd say addresses most of the problems raised here. It's far and away better than anything else I've used.
Your experience (using buckaroo ...) adresses most of the problems raised here? You could perhaps share at least how your experience adressess some of these problems raised here: how does your experience address the hidden telemetry for example?
I've finished reading manual on Nix and just started with documentation on packaging. I want to have total control on dependencies, together with compiler versions and standard library implementations (effectively cross compiling everything x86_64 -> x86_64). From Buckaroo, Conan and Nix, only Nix gets this right...
The main problem seems to be that because there is no standard C++ build system, every library uses a different one. Wrangling all those together without requiring the library authors to do anything is extremely difficult.
Hell many C++ libraries still use autotools or hand-written Makefiles. I don't really see how it is even possible to automatically download and build those projects.
However we have seen more than 300 ports back in 2017 [1] and saw many emerging patterns. This enabled us to write rules to transpile and optimize 100s of build-systems to buck using buildinfer [2] [3]. We hope that more sane decisions will be made regarding build-systems in new projects. We discovered analyzing over 300 projects that a lot of complexity is not needed and often a non-turing complete language for describing the build is sufficient. In fact we found that over 90% of projects can be described solely by using glob expressions.
[1] https://hackernoon.com/lessons-learned-from-porting-300-proj...
[2] https://hackernoon.com/announcing-buildinfer-for-c-3dfa3eb15...
[3] https://buildinfer.loopperfect.com/
OpenSSL buildsystem is over 6k lines of perl scripts which communicate over pipes.
OpenCV uses cmake, python and prolog for it's build. Until this day, I have no clue why there is prolog involved.
You can read some bits about it in our initial announcement of buildinfer: https://hackernoon.com/announcing-buildinfer-for-c-3dfa3eb15...
You could structure something around CMake.
One immediate issue is compilation flags and platforms management. You may need to use some custom flags, for example, you need to compile against a specific architecture. You want those flags to be passed to all libraries uniformly.
But the horrible problem is quickly the compatibility matrix.
In other words, if you are making available, let's say you are using alib v1 and blib v2, you need to check the two libraries compile, link, and work together nicely. So when you upgrade one of the libraries, you need to check that compatibility again.
It's an important problem because if you're working on production software, you need to be able to build all versions and maintain them. And maybe you will have to upgrade only one library. You can't always be "current" everywhere.
So very quickly, using a package manager becomes more painful than having the libraries management manually with a couple of custom scripts.
I'll be happily be proven wrong by a package manager (and we tried a lot).
I mostly work in Windows development. CMake scripts created by a professional build engineer caused a lot of havoc. It injected itself into the project files in such an insidious fashion, that it was nearly impossible to tell if dependencies were correct. It completely broke Intellisense, making code navigation an exercise in frustration.
I was not happy, and wound up ripping out the scripts and creating new solution and project files that actually supported the needs of the developers.
(Edited to remove typos and improve grammar)
so i mean what e.g. conan does is acknowledge this and require that package creators provide "recipes" that tell conan how to build the project.
buckaroo seems to take a similar approach: https://github.com/LoopPerfect/buckaroo/wiki/Creating-a-Pack...
i will say that in my time using conan to attempt to solve for compiling an application for x86 and a few flavors of ARM, this is NOT a silver bullet.. a huge fraction of third party libraries we tried to pull in needed to have their recipes modified to build correctly.
1. Conan does not require a server. If a project contains a conanfile.py with build instructions, you can also run `conan create` from a project's source repository (no server required).
2. You can easily force build dependencies with the `--build *` flag
What we really need are automated porting tools to fix this mess!
The wildcard is cross-platform. vcpkg doesn't seem to be 100% there and has many bugs for example the folders where includes and libraries are kept changes based on platform.
The other problem is static vs dynamic linking, as well as conflicts between different combinations of cmt libs etc. it gets very messy quickly.
Its almost like you're spending more time trying to get all the libraries you want working together than actually coding.
[1] https://buckbuild.com/files-and-dirs/buckconfig.html#cache
Lastly I'd like to note that Buck and Bazel are converging to Skylark for describing builds. The differences between Buck and Bazel should become smaller over time
https://spack.io/
> just one file, download here...
Later on:
> You'll need Buck on your system
Then, on Buck's site:
> Buck requirements: Java 8, Apache Ant, Python 2.7, ...
Well, that does not seems too honest, would be best to warn upfront about all these dependencies.
So in order to include some library I'll put
in my main.c file. And that png.c may look as: And that would be it.C/C++ preprocessor is enough for configuration.
#include source "path.c"; compiles and includes path.c as a new compilation unit ( https://en.wikipedia.org/wiki/Single_Compilation_Unit ) .
So if a-file.c and b-file.c are both have
then these two files can be successfully included as But this: will produce the error "foo is already defined"This small feature will allow 99% of existing libraries to be included this way - without any makefile, GUP, GIP and the rest of the zoo.
Now I can put in Microsoft VC++ this
to include the library into final binary.#include source is just a generalization of the idea.
> as a new compilation unit ( https://en.wikipedia.org/wiki/Single_Compilation_Unit )
> Single Compilation Unit (SCU) is a computer programming technique [...] The technique can be applied to an entire program or to some subset of source files; when applied to an entire program, it is also known as a unity build.
Who came up with that? I would say this "technique" doesn't even justify having a wikipedia page. In modern c++ you can be glad if you have a c++ unit that doesn't need to recompile on change because of all the header-only stuff. No one would include .cpp-files by choice.
That's how C and C++ work.
And if you will have this (no static):
then linker will complain - "multiple 'foo' entities" as non-static entities have global visibility.not so, static declaration are local for the compilation unit so you may have as many different foo's as files you have in your project.
Conan.io already does decentralization right and is well on the way on becoming the defacto standard package manager for C++. And all of that without constraining my workflows
Afaik this is not supported by conan
Any other recipe can then depend on any package that is already in the cache.
You can even just export any recipes into your local cache without manually building the package and then let conan do the work when you require the package as a dependency (using --build all)
So to see if I have this right:
With Conan, to use a package from GitHub, I must download the recipe and add it to my local cache ("cache" seems like the wrong term here, maybe "registry" would be better?) and then I can install it?
What about the transitive dependencies? Do I need to repeat the process for each of those?
Yeah, the term conan "cache" may be a bit misleading, as it is not only a cache but really your "local remote". As soon as a recipe is in your cache, you can depend on it from your own recipes/conan builds. The actual building of the package may happen during the registration of the recipe or during resolve time when you depend on a certain package configuration. Any transitive dependencies will be resolved and, in case of --build missing, the packages will be built automatically according to the recipe.
So no, you do not need to build any dependency packages (including transitive dependencies)
This is impractical if a complex dependency graph as it would require the user to solve a SAT problem just to determine which packages and versions to put into the conan cache to avoid pulling from the conan server.