https://learn.microsoft.com/en-us/dotnet/core/deploying/trim...
https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
which get reinvented all the time, like in dotnet with "trimming" or in JS with "tree-shaking".
C/C++ compiler have been doing that since before dot net was a thing, same for rust which does that since it's 1.0 release (because it's done by LLVM ;) )
The reason it gets reinvented all the time is because while it's often quite straight forward in statically compiled languages it isn't for dynamic languages as finding out what actually is unused is hard (for fine grained code elimination) or at lest unreliable (pruning submodules). Even worse for scripting languages.
Which also brings use to one area where it's not out of the box, if you build .dll/.so in one build process and then use them in another. Here additional tooling is needed to prune the dynamic linked libraries. But luckily it's not a common problem to run into in Rust.
In general most code size problems in Rust aren't caused by too huge LOC of dependencies but by an overuse of monopolization. The problem of tons of LOC in dependencies is one of supply chain trust and review ability more then anything else.
*monomorphization, in case anyone got confused
It seems to me in a strict sense the problem of eliminating dead code may be impossible for code that uses some form of eval(). For example, you could put something like eval(decrypt(<encrypted code>,key)), for a user-supplied key (or otherwise obfuscated); or simply eval(<externally supplied code>); both of which could call previously dead code. Although it seems plausible to rule out such cases. Without eval() some of the problem seems very easy otoh, like unused functions can simply be removed!
And of course there are more classical impediments, halting-problem like, which in general show that telling if a piece of code is executed is undecidable.
( Of course, we can still write conservative decisions that only cull a subset of easy to prove dead code -- halting problem is indeed decidable if you are conservative and accept "I Don't Know" as well as "Halts" / "Doesn't Halt" :) )
The comment you're replying to is talking about not pulling in dependencies at all, before compiling, if they would not be needed.
This is an extreme example, but the same thing happens very often at a smaller scale. Optional functionality can't always be removed statically.
let uri = get_uri_from_stdin();
networking_library::make_request(uri);
let uri: Uri<HTTP> = get_uri_from_stdin().parse()?;
If you want to mix schemes you would need to be able to handle all schemes; you can either go through all variations (through the same generics) you want to test or just just accept that you need a full URI parser and lose the generic.
let uri: Uri<FTP or HTTP or HTTPS> = parse_uri(get_uri_from_stdin()) or fail;If your library cannot parse FTP, either you enable that feature, add that feature, or use a different library.
Rust does have a feature flagging system for this kind of optional functionality though. It's not perfect, but it would work very well for something like curl protocol backends though.
it's also not always a fair comparison, if you include tokio in LOC counting then you surely would also include V8 LOC when counting for node, or JRE for Java projects (but not JDK) etc.
[0] https://clang.llvm.org/docs/ClangCommandLineReference.html#c...
It is also done on the cousin Android, and available as free beer on GraalVM and OpenJ9.
The others no longer matter, out of business.
And since we are always wrong unless proven otherwise,
https://www.graalvm.org/jdk21/reference-manual/native-image/...
https://www.graalvm.org/latest/reference-manual/native-image...
The other nice thing is that bytecode is easy to modify, so if you have a library that has some features you know you don't want, you can just knock it out and bank the savings.
The analogy I use is cooking a huge dinner, then throwing out everything but the one side dish you wanted. If you want just the side-dish you should be able to cook just the side-dish.
I just don't listen. Things should be easy. Rust is easy. Don't overthink it
Sure, don't overthink it. But underthinking it is seriously problematic too.
I wouldn't say completely. People still sometimes struggle to get this to work well.
Recent example: (Go Qt bindings)
It should be easy to build and deploy profiling-aware builds (PGO/BOLT) and to get good feedback around time/instructions spent per package, as well as a measure of the ratio of each library that's cold or thrown away at build time.
I'll note that it isn't just PGO/BOLT style optimizations. Largely, it is not that at all, oddly.
Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a house where every room had a different substructure under it. That, largely, seems to be the general approach we use to building software. The idea being that you can namespace a room away from other rooms and not have any care on what happens there.
This gets equally frustrating when our metrics for determining the safety of something largely discourages inaction on any dependencies. They have to add to it, or people think it is abandoned and not usable.
Note that this isn't unique to software, mind. Hardware can and does go through massive changes over the years. They have obvious limitations that slow down how rapidly they can change, of course.
Well, it's not required to trim code that you can prove unreachable, true. But I was thinking about trying to measure if a given library really pulls it's non-zero weight, and how much CPU is spent in it.
A library taking "too much time" for something you think can be done faster might need replacement, or swapping for a simple implementation (say the library cares about edge cases you don't face or can avoid).
My point on PGO/BOLT not being relevant was more that I see people reaching for libraries to do things such as add retries to a system. I don't think it is a terrible idea, necessarily, but it can be bad when combined with larger "retries plus some other stuff" libraries.
Now, fully granted that it can also be bad when you have developers reimplementing complicated data structures left and right. There has to be some sort of tradeoff calculation. I don't know that we have fully nailed it down, yet.
I'm not sure what the problem is here.
Are you after pinning dependencies to be sure they didn't change? Generally I want updating dependencies to fix bugs in them.
Are you after trusting them through code review or tests? I don't think there's shortcuts for this. You shouldn't trust a library, changing or not, because old bugs and new vulnerabilities make erring on both sides risky. On reviewing other's code, I think Rust helps a bit by being explicit and fencing unsafe code, but memory safety is not enough when a logic bug can ruin your business. You can't avoid testing if mistakes or crashes matter.
Examples: Google's Guava for the migration department. Apache Commons would be a good example of how not to make life painful for users there.
For sweeping features, Log4j introduced some pretty terrible security concerns.
Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.
For example, you can't split up a module into foo.rs containing `Foo` and bar.rs containing `Bar`, both in module 'mymod' in such a way that you can `use mymod::Bar and foo.rs is never built/linked.
My point is the granularity of the package/mod encourages course-grained deps, which I argue is a problem.
yesn't, you can use feature flags similar to `#if` in C
but it's also not really a needed feature as dead code elimination will prune out all code functions, types, etc. you don't use. Non of it will end up in the produced binary.
Historically Rust wanted that foo.rs to be renamed foo/mod.rs but that's no longer idiomatic although of course it still works if you do that.
in rust crates are semantically one compilation unit (where in C oversimplified it's a .h/.c pair, and practically rustc will try to split it in some more units to speed up build time).
the reason I'm pointing this out is because many sources of "splitting a module across files" come from situations where 1 file is one compilation unit so you needed to have a way to split it (for organization) without splitting it (for compilation) in some sitation
What? I don't know about Go, but this certainly isn't true in Rust. Rust has great support for fine-grained imports via Cargo's ability to split up an API via crate features.
It's a terrible idea because you're trying to reinvent section splitting + `--gc-sections` at link time, which rust (which the article is about) already does by default.
Things like --gc-sections feels like a band-aid, a very practical and useful band-aid, but a band-aid none the less. You're building a bunch of things you don't need, then selectively throwing away parts (or selectively keeping parts).
IMO it all boils down to the granularity. The granularity of text source files, the granularity of units of distribution for libraries. It all contributes to a problem of large unwieldy dependency growth.
I don't have any great solutions here, its just observations of the general problem from the horrifying things that happen when dependencies grow uncontrolled.
If each layer of “package abstraction” is only 50% utilised, then each layer multiplies the total size by 2x over what is actually required by the end application.
Three layers — packages pulling in packages that pull their own dependencies — already gets you to 88% bloat! (Or just 12% useful code)
An example of this is the new Windows 11 calculator that can take several seconds to start because it loads junk like the Windows 10 Hello for Business account recovery helper library!
Why? Because it has currency conversion, which uses a HTTP library, which has corporate web proxy support, which needs authentication, which needs WH4B account support, which can get locked out, which needs a recovery helper UI…
…in a calculator. That you can’t launch unless you have already logged in successfully and is definitely not the “right place” for account recovery workflows to be kicked off.
But… you see… it’s just easier to package up these things and include them with a single line in the code somewhere.
If anything, the 1980s is when the idea of fully reusable, separately-developed software components first became practical, with Objective-C and the like. In fact it's a significant success story of Rust that this sort of pervasive software componentry has now been widely adopted as part of a systems programming language.
It worked great, but it took diligence, it also forces you to interact with your deps in ways that adding a line to a deps file does not.
It feels like torture until you see the benefits, and the opposite ... the tangled mess of multiple versions and giant transitive dependency chains... agony.
I would prefer to work in shops that manage their dependencies this way. It's hard to find.
Being able to change a dependency very deep and recompile the entire thing is just magic though. I don't know if I can ever go back from that.
Kids today don't know how to do that anymore...
Compared to Rust where my experience with protobuf lib some time ago was that there is a choice of not 1 but even 3 different libraries, one of which doesn't support services, another didn't support the syntax we had to support, and the third one was unmaintained. So out of 3 choices no single one worked.
Compared that to Maven, where you have only one official supported choice that works well and well maintained.
It's even more pronounced with the main Java competitor: .Net. They look at what approach won in Java ecosystem and go all in. For example there were multiple ORM tools competing, where Microsoft adopted the most popular one. So it's even easier choice there, well supported and maintained.
That's still consolidation, and it also needs time.
Even in Rust crates like hashbrown or parkinglot have been basically subsumed in the standard library.
Besides consolidation point, I still think that "barrier to entry" point is still valid -- if it's more effort to even publish a library, its author is probably more committed already.
Alternatively, for some project it might be enough to only depend on stuff provided by Debian stable or some other LTS distro.
But I think the best defense against this problem at the moment is to be extremely defensive/protective of system dependencies. You need to not import that random library that has a 10 line function. You need to just copy that function into your codebase. Don’t just slap random tools together. Developing libraries in a maintainable and forward seeking manner is the exception not the rule. Some ecosystems exceed here, but most fail. Ruby and JS is probably one of the worst. Try upgrading a Rails 4 app to modern tooling.
So… be extremely protective of your dependencies. Very easy to accrue tech debt with a simple library installation. Libraries use libraries. It becomes a compounding problem fast.
Junior engineers seem to add packages to our core repo with reckless abandon and I have to immediately come in and ask why was this needed? Do you really want to break prod some day because you needed a way to print a list of objects as a table in your cli for dev?
That’s literally the JS module system? It’s how we do tree shaking to get those bundle sizes down.
Part of the issue I have with the dependency bloat is how much effort we currently go through to download, distribute, compile, lint, typecheck, whatever 1000s of lines of code we don't want or need. I want software that allows me to build exactly as much as I need and never have to touch the things I don't want.
Why, in principle, wouldn't the same algorithms work before distribution?
For that matter, check out the `auditwheel` tool in the Python ecosystem.
For example, you have a function calling XML or PDF or JSON output functions depending on some output format parameter. That's three very different paths and includes, but if you don't know which values that parameter can take during runtime you will have to include all three paths, even if in reality only XML (for example) is ever used.
Or there may be higher level causes outside of any analysis, even if you managed a dynamic one. In a GUI, for example, it could be functionality only ever seen by a few with certain roles, but if there is only one app everything will have to be bundled. Similar scenarios are possible with all kinds of software, for example an analysis application that supports various input and output scenarios. It's a variation of the first example where the parameter is internal, but now it is external data not available for an analysis because it will be known only when the software is actually used.
Go and C# (.NET) are counterexamples. They both have great ecosystems and just as simple and effective package management as Rust or JS (Node). But neither Go or C# have issues with dependency hell like Rust or even more JavaScript, because they have exceptional std libs and even large frameworks like ASP.NET or EF Core.
A great std lib is obviously the solution. Some Rust defenders are talking it down by giving Python as counter example. But again, Go and C# are proving them wrong. A great std lib is a solution, but one that comes with huge efforts that can only be made by large organisations like Google (Go) or Microsoft (C#).
Python's standard library is big. I wouldn't call it great, because Python is over 30 years old and it's hard to add things to a standard library and even harder to remove them.
I’m still hoping we can get a decently typed argparse with a modern API though (so much better for tiny scripts without deps!)
Argument parsing, in partucular, is a great place to start realizing that you can implement what you need without adding a dozen dependencies
Don't disagree with the principle, there are a lot of trivial pythong deps, but rolling your own argument parsing is not the way
If youve never thought about it, it might seem like you need an off-the-shelf dependency. But as programmers sometimes we should think a bit more before we make that decision.
It's a very different story with heavyweight dependencies like Numpy (which include reams of tests, documentation and headers even in the wheels that people are only installing to be a dependency of something else, and covers a truly massive range of functionality including exposing BLAS and LAPACK for people who might just want to multiply some small matrices or efficiently represent an image bitmap), or the more complex ones that end up bringing in multiple things completely unrelated to your project that will never be touched at runtime. (Rich supports a ton of wide-ranging things people might want to do with text in a terminal, and I would guess most clients probably want to do exactly one of those things.)
Compared to go and c#, Rust std lib is mostly lacking:
- a powerful http lib
- serialization
But Rust approach, no Runtime, no GC, no Reflection, is making it very hard to provide those libraries.
Within these constraints, some high quality solutions emerged, Tokio, Serde. But they pioneered some novel approaches which would have been hard to try in the std lib.
The whole async ecosystem still has a beta vibe, giving the feeling of programming in a different language. Procedural macros are often synonymous with slow compile times and code bloat.
But what we gained, is less runtime errors, more efficiency, a more robust language.
TLDR: trade-offs everywhere, it is unfair to compare to Go/C# as they are languages with a different set of constraints.
All those AFAIR need 3rd party packages:
Regex, DateTime, base64, argument parsing, url parsing, hashing, random number generation, UUIDs, JSON
I'm not saying it's mandatory, but I would expect all those to be in the standard library before there is any http functionality.
As some of the previous commenters said, when you focus your language to make it easy to write a specific type of program, then you make tradeoffs that can trap you in those constraints like having a runtime, a garbage collector and a set of APIs that are ingrained in the stdlib.
Rust isn't like that. As a system programmer I want none of them. Rust is a systems programming language. I wouldn't use Rust if it had a bloated stdlib. I am very happy about its stdlib. Being able to swap out the regex, datetime, arg parsing and encoding are a feature. I can choose memory-heavy or cpu-heavy implementations. I can optimize for code size or performance or sometimes neither/both.
If the trade-offs were made to appease the easy (web/app) development, it wouldn't be a systems programming language for me where I can use the same async concepts on a Linux system and an embedded MCU. Rust's design enables that, no other language's design (even C++) does.
If a web developer wants to use a systems programming language, that's their trade-off for a harder to program language. The similar type safety to Rust's is provided with Kotlin or Swift.
Dependency bloat is indeed a problem. Easy inclusion of dependencies is also a contributing factor. This problem can be solved by making dependencies and features granular. If the libraries don't provide the granularity you want, you need to change libraries/audit source/contribute. No free meals.
A feature present on every language that has those in the stdlib.
The de facto standard regex library (which is excellent!) brings in nearly 2 MB of additional content for correct unicode operations and other purposes. The same author also makes regex-lite, though, which did everything we need, with the same interface, in a much smaller package. It made it trivial to toss the functionality we needed behind a trait and choose a regex library appropriately in different portions of our stack.
Regex is not 3rd party (note the 'rust-lang' in the URL):
In theory they should reduce it because you wouldn’t make proc macros to generate code you don’t need…right? How much coding time you save with macros compared to manually implementing them?
Rust has other challenges it needs to overcome but this isn't one.
I'd put Go behind both C#/F# and Rust in this area. It has spartan tooling in odd areas it's expected to be strong at like gRPC and the serialization story in Go is quite a bit more painful and bare bones compared to what you get out of System.Text.Json and Serde.
The difference is especially stark with Regex where Go ships with a slow engine (because it does not allow writing sufficiently fast code in this area at this moment) where-as both Rust and C# have top of the line implementations in each which beat every other engine save for Intel Hyperscan[0].
[0]: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa... (note this is without .NET 9 or 10 preview updates)
I don't think that's why. Or at least, I don't think it's straight-forward to draw that conclusion yet. I don't see any reason why the lazy DFA in RE2 or the Rust regex crate couldn't be ported to Go[1] and dramatically speed things up. Indeed, it has been done[2], but it was never pushed over the finish line. My guess is it would make Go's regexp engine a fair bit more competitive in some cases. And aside from that, there's tons of literal optimizations that could still be done that don't really have much to do with Go the language.
Could a Go-written regexp engine be faster or nearly as fast because of the language? Probably not. But I think the "implementation quality" is a far bigger determinant in explaining the current gap.
A large stdlib solves the problems the language is focused on. For C# and Go that is web hosts.
Try using them outside that scope and the dependencies start to pile in (Games, Desktop) or they are essentially unused (embedded, phones, wasm)
If you're willing to constrain yourself to 2D games, and exclude physics engines (assume you just use one of the Box2D bindings) and also UI (2D gamedevs tend to make their own UI systems anyway)... Then your best bet in the C# world is Monogame (https://monogame.net/), which has lots of successful titles shipped on desktop and console (Stardew Valley, Celeste)
Depends. There is Godot Script. Seeing how it comes with a game engine.
But original claim was
> actually dotnet also does not need too many dependencies for games and desktop apps.
But general purpose programming language has very little to gain from including a niche library even if it's the best in business. Imagine if C++ shipped with Unreal.
Are you really trying to compare serde to rendering engines?
As for the generic things, I think C# is the only mainstream language which has small vectors, 3x2 and 4x4 matrices, and quaternions in the standard library.
They've got SIMD-accelerated methods for calculating 3d projection matrices. No other ecosystem is even close once you start digging into the details.
But I agree that graphics is often overlooked in std libs. However that’s a bit of a different beast. Std libs typically deal with what the OS provides. Graphics is its own world so to speak.
As for Wasm: first, that’s a runtime issue and not a language issue. I think GC is on the roadmap for Wasm. Second, Go and C# obviously predate Wasm.
In the end, not every language should be concerned with every use case. The bigger question is whether it provides a std lib for the category of programs it targets.
To take a specific example: JS isn’t great at efficiently and conveniently generating dynamic HTML. You can go far without (or minimal) dependencies and some clever patterns. But a lot of pain and work hours would have been saved if it had something that people want to use out of the box.
Even ignoring that, those are just common formats. They don't tell you what a particular web server is doing.
Take a few examples of some Go projects that either are web servers or have them as major components like Caddy or Tailscale. Wildly different types of projects.
I guess one has to expand "web server" to include general networking as well, which is definitely a well supported use case or rather category for the Go std lib, which was my original point.
You seem to have a very different definition of "web server" to me.
Maybe it's a language community thing.
You don't consider games, desktop and mobile applications big use cases, each being multi billion industries?
I don't know man, I feel like you're arguing in bad faith and are intentionally ignoring what the athrowaway3z said: it works there because they're essentially languages specifically made to enable web development . That's why their standard lib is plenty for this domain.
I can understand that web development might be the only thing you care about though, it's definitely a large industry - but the thesis of a large standard lib solving the dependency issue really isnt true, as (almost) every other usecase beyond web development shows.
Edit: after rereading this I feel like I may have come across sarcastic, I was legitimately impressed a guess without looking it up would peg the ratio that closely. It was off topic as a response too. So I'll add that rust never would have an asynch as good as tokio, or been able to have asynch in embedded as with embassy, if it hadn't opted for batteries excluded. I think this was the right call given its initial focus as a desktop/systems language. And it is what allowed it to be more than that as people added things. Use cargo-deny, pin the oldest version that does what you need and doesn't fail cargo deny. There are several hundred crates brought in by just the rust lang repo, if you only vet things not in that list, you can save some time too.
I don't think the dependency issue can be solved by a good std lib, but it certainly can be mitigated as some languages show.
I think JS is a very pronounced case study here.
That's part of it, but it also solves the problem of vetting. When I use a Go stdlib I don't have to personally spend time to vet it like it do when looking at a crate or npm package.
In general, Go & Rust packages on github are high quality to begin with, but there is still a pronounced difference between OS packages and what is approved to be part of the language's own stdlib.
It's nice to know thousands of different companies already found the issues for me or objected to them in reviews before the library was published.
They also have a lot narrower scope of use, which means it is easier to create stdlib usable for most people. You can't do it with more generic language.
This is an organizational problem, of course. Rust can be different things to different people, without necessarily forcing one group to compromise overmuch. But some tension is probably inevitable.
That depends on the language. In an interpreted language (including JIT), or a language that depends on a dynamically linked runtime (ex c and c++), it isn't directly included in your app because it is part of the runtime. But you need the runtime installed, and if your app is the only thing that uses that runtime, then the runtime size is effectively adds to your installation size.
In languages that statically link the standard library, like go and rust, it absolutely does impact binary size, although the compiler might use some methods to try to avoid including parts of the standard library that aren't used.
What you say is true enough for external-runtime languages and Go, though TinyGo is available for resource-constrained environments.
The no_std Rust only has core but this is indeed a library of code, and freestanding C does not provide such a thing = freestanding C stdlib provides no functions, just type definitions and other stuff which evaporates when compiled.
Two concrete examples to be going along with: Suppose we have a mutable foo, it's maybe foo: [i32; 40]; (forty 32-bit signed integers) or in C maybe they're int foo[40];.
In freestanding C that's fine, but we're not provided with any library code to do anything with foo, we can use the core language features to write it outselves, but nothing is provided.
Rust will happily foo.sort_unstable(); this is a fast custom in-place sort, roughly a modern form of introspective sort written for Rust by its creators and because it's in core, that code just goes into your resulting embedded firmware or whatever.
Now, suppose we want to perform a filter-map operation over that array. In C once again you're left to figure out how to write that in C, in Rust foo impl IntoIterator so you can use all the nice iterator features, the algorithms just get baked into your firmware during compilation.
Or you have ultra-fine-grained modules, and rely on existing tree-shaking systems.... ?
The more pressing issue with dependencies is supply chain risks including security. That's why larger organizations have approval processes for using anything open source. Unfortunately the new crop of open source projects in JS and even Go seem to suffer from "IDGAF about what shit code from internet I am pulling" syndrome.
Unfortunately granularity does not solve that as long as your 1000 functions come from 1000 authors on NPM.
Functions are defined by AST structure and are effectively content addressed. Each function is then keyed by hash in a global registry where you can pull it from for reuse.
So, what's is the compiler doing that he doesnt remove unused code?
For example you know you will never use one of the main functions in the parsing library with one of the arguments set to "XML", because you know for sure you don't use XML in your domain (for example you have a solid project constraint that says XML is out of scope).
Unfortunately the code dealing with XML in the library is 95% of the code, and you can't tell your compiler I won't need this, I promise never to call that function with argument set to XML.
doc_format = get_user_input() parsed_doc = foolib.parse(doc_format)
You as the implementer might know the user will never input xml, so doc_format can't be 'xml' (you might even add some error handling if the user inputs this), but how can you communicate this to the compiler?
And you always run it in a data center that uses a specific locale, and only UTC time zone, and very few simple types. But all this can only be known at runtime, except maybe types if the compiler is good.
And even going beyond "debug", plenty of libraries ship features that are downright unwanted by consumers.
The two famous recent examples are Heartbleed and Log4shell.
It's effectively an end-run around the linker.
It used to be that you'd create a library by having each function in its own compilation unit, you'd create a ".o" file, then you'd bunch them together in a ".a" archive. When someone else is compiling their code, and they need the do_thing() function, the linker sees it's unfulfiled, and plucks it out of the foolib.a archive. For namespacing you'd probably call the functions foolib_do_thing(), etc.
However, object-orientism with a "god object" is a disease. We go in through a top-level object like "foolib" that holds pointers to all its member functions like do_thing(), do_this(), do_that(), then the only reference the other person's code has is to "foolib"... and then "foolib" brings in everything else in the library.
It's not possible for the linker to know if, for example, foolib needed the reference to do_that() just to initialise its members, and then nobody else ever needed it, so it could be eliminated, or if either foolib or the user's code will somehow need it.
> Go and Rust, for example, encourage everything for a single package/mod to go in the same file.
I can say that, at least for Go, it has excellent dead code elimination. If you don't call it, it's removed. If you even have a const feature_flag = false and have an if feature_flag { foobar() } in the code, it will eliminate foobar().
It also happens to be an object, but that's just because python is a dynamic language and libraries are objects. The C++ equivalent is foolib::do_thing(); where foolib is not an object.
I'm not convinced that happens that often.
As someone working on a Rust library with a fairly heavy dependency tree (Xilem), I've tried a few times to see if we could trim it by tweaking feature flags, and most of the times it turned out that they were downstream of things we needed: Vulkan support, PNG decoding, unicode shaping, etc.
When I did manage to find a superfluous dependency, it was often something small and inconsequential like once_cell. The one exception was serde_json, which we could remove after a small refactor (though we expect most of our users to depend on serde anyway).
We're looking to remove or at least decouple larger dependencies like winit and wgpu, but that requires some major architectural changes, it's not just "remove this runtime option and win 500MB".
My feeling is that Python scores fairly well in this regard. At least it used to. I haven't been following closely in recent years.
Just so you can multiply matrices or something.
I’m just not convinced that it’s worth the pain to avoid installing these packages.
You want speedy matrix math. Why would you install some second rate package just because it has a lighter footprint on disk? I want my dependencies rock solid so I don’t have to screw with debugging them. They’re not my core business - if (when) they don’t “just work” it’s a massive time sink.
NumPy isn’t “left pad” so this argument doesn’t seem strong to me.
A better design is to make it easy you to choose or hotswap your BLAS/LAPACK implementation. E.g. OpenBLAS for AMD.
Edit: To be clear, Netlib (the reference implementation) is almost always NOT what you want. It's designed to be readable, not optimized for modern CPUs.
A lot of the bloat comes from functionality that can be activated via flags, methods that set a variable to true, environment variables, or even via configuration files.
If you want to disable certain runtime features, you'd do so with feature flags.
I think that is much more of a problem in ecosystems where it is harder to add dependencies.
When it is difficult to add dependencies, you end up with large libraries that do a lot of stuff you don't need, so you only need to add a couple of dependencies. On the other hand, if dependency management is easy, you end up with a lot of smaller packages that just do one thing.
It’s getting hard to take these conversations seriously with all of the hyperbole about things that don’t happen. Nobody is producing Rust binaries that hit 500MB or even 50MB from adding a couple simple dependencies.
You’re also not ending up with mountains of code that never gets called in Rust.
Even if my Rust binaries end up being 10MB instead of 1MB, it doesn’t really matter these days. It’s either going on a server platform where that amount of data is trivial or it’s going into an embedded device where the few extra megabytes aren’t really a big deal relative to all the other content that ends up on devices these days.
For truly space constrained systems there’s no-std and entire, albeit small, separate universe of packages that operate in that space.
For all the doom-saying, in Rust I haven’t encountered this excessive bloat problem some people fret about, even in projects with liberal use of dependencies.
Every time I read these threads I feel like the conversations get hijacked by the people at the intersection of “not invented here” and nostalgia for the good old days. Comments like this that yearn for the days of buying paid libraries and then picking them apart anyway really reinforce that idea. There’s also a lot of the usual disdain for async and even Rust itself throughout this comment section. Meanwhile it feels like there’s an entire other world of Rust developers who have just moved on and get work done, not caring for endless discussions about function coloring or rewriting libraries themselves to shave a few hundred kB off of their binaries.
This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible.
Today we pile libraries on top of libraries on top of libraries. Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running. Who knows or cares what all 'foolib' contains.
At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.
In some cases the languages make this worse. Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Adding optional functionality can get ugly when it would require creating new modules, but if you only want to use a tiny part of the module, what do you do?
The only real solution I can think of to deal with this long term is ultra-fine-grained symbols and dependencies. Every function, type, and other top-level language construct needs to declare the set of things it needs to run (other functions, symbols, types, etc). When you depend on that one symbol it can construct, on demand, the exact graph of symbols it needs and dump the rest for any given library. You end up with the minimal set of code for the functionality you need.
Its a terrible idea and I'd hate it, but how else do you address the current setup of effectively building the whole universe of code branching from your dependencies and then dragging it around like a boat anchor of dead code.