This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.
>
> This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.
I'm the programmer at Backblaze that made the copies on purpose, I wrote some extra code to do this, and it's meant to help us debug certain things. Yes they are identical, the installer only ships with one copy of the executable, the installer then makes the copies on purpose. I get to explain this from time to time. :-)
In Windows when you want to know what is going on behind the scenes, you can bring up Task Manager and look at the different names of the different processes that are running. On the Macintosh this is called Activity Monitor, same sort of thing. The different names for the executables are for different "threads" which have different roles. Backblaze is multi-threaded to get higher performance.
The parent coordination process is called "bztransmit". But when doing the actual transmission it spawns the bztrans_thread01, bztrans_thread02, bztrans_thread03, etc.
So BEFORE I made multiple copies of the executables with different names, a customer would say "bztransmit is hung" or "bztransmit is using up too much memory". There was very little visibility into this. But now that I made multiple copies with different executable names, when the same customer says "bztrans_thread03 is hung". Or they say "bztransmit is using too much memory". We immediately have narrowed down what to look at.
Here is a screenshot showing what "Chrome" looks like to me in Windows, and how it compares to how Backblaze's bztransmit looks like to me in Windows: https://i.imgur.com/KOJHJ9Q.jpg In that screenshot, you can see there is the "main thread", and "worker threads". Meanwhile chrome is just one big list of processes all named the same thing (see the screenshot). I prefer the Backblaze system, but I understand it upsets some customers that prefer the chrome experience.
That's it. It's not some huge mystery.
One question asked here was do we know you can launch the same executable twice? Yes, and we do that. The bztrans_thread05 is launched for thread 05, thread 25, thread 45, thread 65, etc. It's THREAD_NUMBER mod 20. Here is what it looks like to hit 500 Mbits/sec upload speeds, this isn't photoshopped, it's a real screenshot on my development computer: https://i.imgur.com/hthLZvZ.gif
Another question is: why not use hard links or symbolic links? That's the only real optimization possible here, everything else was on purpose. The answer is not an excuse, it's just an explanation if you are curious. The software we develop at Backblaze is cross platform, so what we like to do is make the most general form first that will always work, then if customers complain or we want to refine it we special purpose code in per file system or per platform. The most general thing to do is make full copies. We could then go on to make links on the Mac WHEN POSSIBLE and the equivalent on Windows WHEN POSSIBLE, but it never became a large priority. The reason I can't use one technology is we support several file systems, and not all of them are the same or support the same technology for links.
Every feature we have is the result of prioritizing it over working on other things. Until recently, we did not have a lot of funding or an infinite supply of programmers, so we had to choose what order to implement each feature in. I'm not saying we got all the priorities correct, or that we did things in the correct order. I don't really even think there is one correct order. For example, some individual home user customers prefer saving 180 MBytes of their valuable boot SSD space over me implementing single sign on for our corporate customers. On the other hand some corporate customers DEMANDED single sign on or they wouldn't purchase the product at all. They are both correct, but there is only one of me, so we made some judgement calls and left the multiple copies and worked on single sign on. Some customers were happy, some are miserable.
We do have open client recs for both Windows and Mac programmers, so if you would like to make a good salary, full benefits, and help us out, come join us! :-)
Consider also that having multiple distinct binaries interferes with (dis)allowing Backblaze to reach the internet with process-based firewalls like Little Snitch, because each copy needs to be configured separately.
Some tools, like Docker, have a function for collecting relevant diagnostics. Perhaps it would be useful to you to migrate towards a solution like that rather than asking the user to identify a misbehaving process on their own.
Thanks, that was super exiting for us. After 14 years, I claim (and this is controversial) that we're no longer a startup and now we're just a mid-sized publicly traded company. :-)
> process-based firewalls like Little Snitch, because each copy needs to be configured separately
Yeah, that was actually a surprise and unfortunate. What the Mac architect (one of my business partners) and I think is that now that it is nice and stable, we might go down to 1 or 2 bztrans_thread executables, and one bztransmit. That seems like a better tradeoff where we waste much MUCH less disk space, and it is only 3 executables to allowlist in Little Snitch, and it achieves basically what we want now that it's stable and working well.
Originally there were 10 threads MAXIMUM, and we made 10 copies. And each copy was linking with shared libaries so it was only 10 MBytes of disk space which nobody noticed. Then Windows lost their friggin' minds with one of their releases and forced us to link statically which bloated it way up to 5 or 10 MBytes per executable. Then we went to 20 threads maximum and the whole thing was silly. When we went to 100 threads maximum we said "enough" and went to mod 20 for re-using executable names.
By the way, ALL OF THIS could be avoided if Microsoft and Apple provided an API to set the name displayed in Task Manager/Activity Monitor. Maybe that's a security issue, I don't know. But frankly wouldn't it be SUPER TOTALLY USEFUL if chrome displayed the current web page loaded in the process name of each and every chrome process? Then you would know which one to kill when something goes sideways.
The problem is process lists should be showing the true state of the computer. It wouldn't be a good idea to hide the actual executable name. But it sounds like it could be useful to add another column for "label", so that threads could set a label and offer more insight on the process list.
Yeah, that would work really well. When you look at the "services" control panel in Windows, there are two columns. One column is "Name" of the service, and another column is a longer explanation with the column header "Description". I put a small description in there for bzserv (our service) plus a URL to our company website. I think this is just being polite, customers who don't recognize what "bzserv" is can immediate find more information on it.
A clean solution would be to have stub executables that link to a central libClang.dll/libLLVM.dll (-DCLANG_LINK_CLANG_DYLIB=ON), but this is not supported under Windows because a DLL can export at most 2^16 symbols. Some work would need to be invested to make process launching on Windows work differently than on other platforms, but then disk space is not that much of a problem.
(Also note that in this case you don't need symlinks per se, just identical binaries with hardlinks. Each executable could inspect argv[0] and figure out what it's invoked as, and behave accordingly.)
What I don't understand is why can't they just bundle everything into a single DLL, then make whatever stub executables they want just call the exported main() in that DLL. I think that might be what MSYS2's copy already does, though I haven't checked. That DLL wouldn't need to export anything else, just the main() function that each .exe stub would forward to. And it can handle everything else internally if it so desires.
Symlinks have been supported since Vista, though by default creating a symlink does indeed require Admin rights. Hardlinks can be created by anybody and are extensively used by Windows itself.
I'm not aware of even Windows 10 or Windows 11 being able to create Symlinks on FAT32 and exFAT, but I could be wrong? But it doesn't matter, the code as written is cross platform, it works on Apple Journaled File Systems, APFS, exFAT, etc. We then compile it for Macintosh, compile it for Windows, and compile it for Linux.
We can then spend extra time and carefully detect each filesystem and each platform and then make the optimization if we can. And this is a valid criticism that we have not done this yet. But no matter what we need this general code that will always work FIRST, what the links are is a space optimization to save valuable SSD space when it is possible.
Or are you saying your heuristic for determining the underlying engineering quality changes its output based on 1 bit flip? Identical executables is awful engineering, but a few bytes different (out of 90MB) is great engineering?
Really? You genuinely don't see why I would think a case with >99.99% identical executables might be relevant to a case with 100% identical executables?
With 100% identical executables, there does not appear to be a good reason to have multiple copies under any circumstance.
With anything < 100% identical, well, maybe there’s a good reason to have multiples. Who knows? Id probably give someone the benefit of the doubt and figure there was some engineering challenge that made it faster/easier to do it that way.
So yes, 100% identical is completely different than almost 100% identical.
You can instead configure LLVM’s build system to build a single dynamic library and have the tools link to it, and this eliminates all of the duplication. However, it apparently comes with a “substantial performance penalty” [1] due to the nature of dynamic linking. (This actually surprises me, and I wonder whether it’s only referring to the inability to do LTO, or whether even LTO-less static linking is faster. Aside from the startup time issue.)
A theoretical alternative would be to build all of the tools into a single executable, à la Busybox, where the combined executable would inspect argv[0] to figure out which tool’s code should be run. That way you could statically link the LLVM libraries without duplicating them in multiple executables. LLVM’s build system does not support this. I think it would be nice if it did, but it would be nontrivial to implement.
I think the latter is also the case (though to a much lesser extent) on x64. One of the unfortunate features of x64 is it lacks direct 64-bit jumps, so every jump to an external library ends up being an indirect call. (In fact, with a potential memory load on top of that.) This was kind of surprising for me when I learned it too; it doesn't apply to x86.
They're completely (!) different? And you're saying this despite the fact that the comment I replied to was discussing cases where one could "just execute one binary n times (with different argv[0] if they'd like)"... which is something you can do with different executables just as well as with identical ones? It's not just a little different but completely different? So different that not only you don't see any similarity, but you also cannot fathom why I might think there's some similarity?!
Context is important. For example, chimpanzee DNA is 99% identical to human DNA. Without context you could make the argument, “They are almost identical!” and be correct. But in the context of discussing the ability to fly to the moon and return safely, the two genomes are completely different.
But with computer code, you very much can (for example) easily replace all the executables with a combined one that gets hardlinked with different names, such that they behave differently depending on argv[0]. They don't even need to be 100% identical for us to be able to do this; it works just as well with 1% identical code—you just end up with bigger combined binaries. This is the Busybox approach (notice it combines executables that have hardly anything in common), and it's in fact exactly what Clang already sometimes does (like on MSYS2); one would think they could take the same approach here. This is also precisely the context of the comment I replied to was saying, right? This is the context I was replying to. It's so confusing to me that you claim I'm missing context when I was in fact addressing the context I saw directly—and it was the other comments that were not.
If they're 99% the same, it's generously easy to assume that there's a material difference. That assumption is completely nonsensical if they're 100% identical.
So no, for the sake of every bit of context in this conversation, it does not make sense that you'd bring up an unrelated scenario of "similar" binaries.
Needless to say it doesn't instill confidence in the quality of software engineering that I rely on for disaster recovery.