It was a pretty much settled argument 10 years ago, even before the era of LLVM lifters, but post-LLM the standard of care practice is often full recompilation and execution.
I think there's a lot of historical evidence that doesn't support this position. For instance, Internet Explorer was generally agreed by all to be a much weaker product from a security perspective than its open source competitors (Gecko, WebKit, etc).
Nobody was defending IE from a security perspective because it was closed source.
The difference between obscurity and a secret (password, key, etc) is the difference between less then a year to figure it out and a year or more to figure it out.
There is a surprising amount of software out there with obscurity preventing some kind of "abuse" and in my experience these features are not that strong, but it takes someone like me hours to reverse engineer these things, and in many cases I am the first person to do that after years of nobody else bothering.
Is an unbreakable security mechanism
with
Improves security
anything that complicates an attacker improves security, at least grossly. That said, then there might be counter effects that make it a net loss or net neutral.
I love the Rob Joyce quote that explained why TAO was so successful: "In many cases we know networks better than the people who designed and run them."
It could be interleaved in any of the many analytics tools in there too.
You have to trust the client in E2E encryption. There's literally no way around that. You need to trust the client's OS (and in some cases, other processes) too.
Vastly easier than spotting a clever bugdoor in the source code of said app.
Not having the source code hasn't stopped people from finding exploits in Windows (or even hardware attacks like Spectre or Meltdown). Having source code didn't protect against Heartbleed or log4j
I'd conclude it comes down to security culture (look how things changed after the Trustworthy Computing initiative, or OpenSSL vs LibreSSL) and "how many people are looking" -- in that sense, maybe "many eyes [do] make bugs shallow" but it doesn't seem like "source code availability" is the deciding factor. Rather, "what are the incentives" -- both on the internal development side and the external attacker side
But it's still possible. And analyzing source code is still hard.
Does this rewording reflect it's meaning?
"You don't actually need code to evaluate security, you can analyze a binary just as well."
Because that doesn't sound correct?
But that's just my first pass, at a high level. Don't wanna overinterpret until I'm on surer ground about what the dispute is. (i.e. don't want to mind read :) )
Steelman for my current understanding is limited to "you can check if it writes files/accesses network, and if it doesn't, then by definition the chats are private and its secure", which sounds facile. (presumably something is being written to somewhere for the whole chat thing to work, can't do P2P because someone's app might not be open when you send)
Whether the original comment knows it or not, Stallman greatly influenced the very definition of Source Code, and the claim being made here is very close to Stallman's freedom to study.
>"You don't actually need code to evaluate security, you can analyze a binary"
Correct
>"just as well"
No, of course analyzing source code is easier and analyzing binaries is harder. But it's still possible (feasible is the word used by the original comment)
>Steelman for my current understanding is limited to "you can check if it writes files/accesses network, and if it doesn't, then by definition the chats are private and its secure",
I didn't say anything about that? I mean those are valid tactics as part of a wider toolset, but I specifically said binaries, because it maps one to one with the source code. If you can find something in the source code, you can find it in the binary and viceversa. Analyzing file accesses and networks, or runtime analysis of any kind, is going to mostly be orthogonal to source code/binary static analysis, the only difference being whether you have a debug map to source code or to the machine code.
This is a very central conflict of Free Software, what I want to make clear is that Free Software refuses to study closed source software, not because it is impossible, but because it is unjustly hard. Free Software never claims it is impossible to study closed source software, it claims that source code access is a right, and they prefer rejecting to use closed source software, and thus never need to perform binary analysis.
I was referring to source code to binary maps, these are files that map binary locations to source code locations. In C (gcc/gdb) these are debug objects, they are also used in gdb style debuggers like Python's pdb, Java's jdb. They also exist in js/ts when using minifiers or react, so that you are able to debug in production.
I'm not paying for the tokens I use, so I just choose whatever is the most performant model OpenAI offers. I find LLMs to be highly capable, struggling very little even against fairly obnoxious obfuscation.
My use cases have ranged from malware analysis to adding new features to complicated EOL enterprise software without access to the source code.
I've done a lot of manual reverse engineering. In many cases you can genuinely 100x your productivity using these tools. Tasks like matching decompilation are an especially good fit for LLMs.
The way they analyze binaries now is by using textual interfaces of command tools, and the tools used are mostly the ones supported by Foundation Models at training time, mostly you can't teach it new tools at inference, they must be supported at training. So most providers are focused on the same tools and benchmarking against them, and binary analysis is not in the zeitgeist right now, it's about production more than understanding.
Also MCP is not an Agent protocol, it's used in a different category. MCP is used when the user has a chatbot, sends a message, gets a response. Here we are talking about the category of products we would describe as Code Agents, including Claude Code, ChatGPT Codex, and the specific models that are trained for use in such contexts.
The idea is that of course you can tell it about certain tools in inference, but in code production tasks the LLM is trained to use string based tools such as grep, and not language specific tools like Go To Definition.
My source on this is Dax who is developing an Open Source clone of Claude Code called OpenCode
You say they’re better with the tools they’re trained on. Maybe? But if so not much. And maybe not. Because custom tools are passed as part of the prompt and prompts go a long way to override training.
LLMs reason in text. (Except for the ones that reason in latent space.) But they can work with data in any file format as long as they’re given tools to do so.
https://youtu.be/VsTbgYawoVc?si=6ZE83umppNCz9h-a&t=1021
"The models today, they are tuned to call specific tools. We've played with a lot of tools, you can hand it a bunch of tools it's never seen before and it just doesn't call them. There's something to the post-training process being catered to certain sets of tools. So anthropic, cloud 4, cloud3.7 before that, those models are the best at calling tools from a programming standpoint. They'll actually keep trying and going for it. Other models like Gemini2.5 can be really good, but it doesn't really call tools very eagerly. So we are in the state right now where we kind of have to provide the set of tools that the model expects.
I don't think that'll always be the case, but we've given it a bunch of LSP tools. I've played with giving, say, giving it 'go to definition' and 'find references', and it just doesn't use them. I mean you can get it to use them if you ask it to, but it doesn't default to kind of thinking that way, I think that'll change."
He then goes on to theorize it's the System Prompt, so open models like Llama where you can customize the system prompt might have an advantage. (I think API models still have a prebaked prompt, not sure). Additionally, even when you control the prompt, he argues there's a (soft) limit to the amount of tools it can handle.
Personally, I think a common error with LLMs is conflating what is technically possible and what works in practice. In this case the argument is that custom tools and MCPs are possible, but it's limited in the sense that "you often need to explicitly tell them to use such tool, you can only have a small amount of custom tools", when you compare it to system prompt specified tools, and tools in the training set which are fine tuned to, it's a whole different category, the native tools are just capable of way much more autonomy.
A similar error I've seen is conflating the context length to the capacity to remember. That a model has a 1M token window means that it could remember something, but it would be a categorical mistake to claim or depend on the model remembering stuff in a 1M token conversation.
There's a lot of nuance in these discussions.
In contrast, I've seen coding agents figure out extremely complex systems problems that are clearly outside of their training set - by using tools and interacting with complex environments, and reasoning and figuring it out.
Plus, "tools" can be multi-layered. You give an agent a "bash" tool, and voila, it has access to every piece of software ever written. So I don't think any of these arguments apply in the slightest to the question of de-compiling code.
This is the original comment I was responding to, we were talking about state of the art Agentic models. So not generalizing to other scenarios.
>Sure you and they aren't skeptics and acknowledge this will get better
I think this is a common bipartisan trap where you lose a lot of nuance. And it's imprecise, you don't know whether I'm a skeptic or not. It's like reading a nuanced opinion and trying to see if they are Republican so you can agree or Democrats so you can disagree.
>Plus to blame the LLM when they haven't optimized the system prompt is IMHO quite silly - it's kind of like "did you read the instructions you were giving it?"
I think the context here is that when using agentic tools like Claude Code, you don't control the system prompt. You could write your own prompts and use naked API calls, but that's always more expensive, (because it's subsidized), and I'm not sure what the quality of that is.
The bottom line is that API calls, where you can fully control the system prompt, are more expensive. And using your OpenAI/Anthropic subscription has a fixed cost. So in that context they don't control the system prompt.
Even in cases where you could control the system prompt and use the API, there's the fact that some models (the state of the art) are fine tuned for specific tool use, so they have a bias towards fine tuned tool use. The claim is not that they are "incapable of doing X thing" it's that it's a bias towards the usage that was known at train-time or fine-tune time, instead of at inference which is much weaker. Nuance.
>Instead of blaming the model intrinsically.
Again, not blaming or being a skeptic here, just analyzing the state of the art and its current weakness, it's likely that these things are going to be improved in the next generation, this is going to move fast, if you conflate any criticism of the tools with "skepticism" you are going to miss the nuance.
>In contrast, I've seen coding agents figure out extremely complex systems problems that are clearly outside of their training set - by using tools and interacting with complex environments, and reasoning and figuring it out.
Yeah for sure, I'll give you a concrete example on this point where we agree. I made a model download a webdriver for a browser and taught it to use the webdriver to open the site, take screenshots, and evaluate how it looks visually, in addition to actually clicking buttons and navigating it. This is a great improvement when the traditional approach is just to generate frontend code and trust that it works (which to be fair, sometimes works great, but you know, it's better if it see that.)
And it works, until it doesn't and I have to remind it that it can do that. It's just a bias. If they would have trained the model with WebDriver tool access, the model would use it much more (and perhaps they are already doing that and we will see it in the next model.)
The main thesis is that instructions taught at train time 'work better' than at fine-tune time which in turn are stronger than 'inference'. To be very specific during inference tool use is much more likely immediately after mentioning it, it might be stronger more consistently at the system prompt, (but it competes with other system prompt instructions and it's still inference based). To say nothing of the costs associated with adding to inference tokens, compared to essentially free training/finetune biases. I don't think anyone disagrees that stuff you teach the model during training has better quality and less cost than stuff you teach at inference.
I think playing around with logit biases is an underrated tool to increase and control frequency of certain tools, but it doesn't seem that's being used much in this generation of vibecode tools, the interface is almost entirely textual (with some /commands starting to surface). Maybe the next generation will have the option to configure some specific parameters instead of entirely relying on textual prompting.
Like, it's assuredly harder for an agent than having access to the code, if only because there's a theoratical opportunity to misunderstand the decompile.
Alternatively, it's assuredly easier for an agent because given execution time approaches infinity, they can try all possible interpretations.
Devin is also going very strong, but it's a bit quieter and growing in enterprises (and pretty sure it uses Claude Opus 4.5 and possibly Claude Code itself). In fact Clawdbot/Moltbot/OpenClaw was itself created with devin.
The big difference is the autonomy these models have (Devin more than Claude Codes), Cursor was meant to work in an IDE and that was a huge strength during the 12 months that the models still weren't strong enough to work autonomously, but they are getting to the point where that's becoming a weakness. Models like Devin are getting a slower acceleration but higher top speed advantage. My chips are on Devin
And yes you can analyze binary blobs for backdoors and other security vulnerabilities, but it's a lot easier with the source code.
The claim Stallman would make (after punishing you for using Open Source instead of Free Software for an hour) is that Closed Software (Proprietary Software) is unjust. but in the context of security, the claim would be limited to Free Software being capable of being secure too.
You may be able to argue that Open Source reduces risk in threat models where the manufacturer is the attacker, but in any other threat model, security is an advantage of closed source. It's automatic obfuscation.
There's a lot of advantages to Free Software, you don't need to make up some.