News for hackers. Stuff that matters.
Re:LLM4Decompile: Decompiling Binary Code with LLM by potatoman22 on Sunday March 17, 2024 @07:03AM (#39733524)
It's interesting the 6b model outperforms the 33b model. I wonder if it means the 33b model needs more training data? It was pretrained on ~1 million C programs, compared to DeepSeek-Coder, which was trained on 2 trillion tokens, which is a few orders of magnitude more data.

I'm also curious about how this compares to non-LLM solutions.

Re:LLM4Decompile: Decompiling Binary Code with LLM by mattashii on Sunday March 17, 2024 @11:13AM (#39734977)
> on ~1 million C programs, compared to [...] 2 trillion tokens, which is a few orders of magnitude more data.

Is that comparable like that? This would assume that the average C program of the set is orders (plural) of magnitude less than 2m tokens in size, which could indeed be true but sounds like an optimistic assumption.

Re:LLM4Decompile: Decompiling Binary Code with LLM by Der_Einzige on Sunday March 17, 2024 @11:35AM (#39735156)
This has been the dynamics with LLMs for awhile. The majority of LLMs are massively undertrained. 7b models are the least "undertrained" mainstream models we have, hence why they have proliferated so much among the LLM fine-tuning community.
Re:LLM4Decompile: Decompiling Binary Code with LLM by albertan017 on Monday March 18, 2024 @07:19AM (#39742548)
Yes, it's not easy to train a 33B model. An interesting point is, naive fine-tuning, which means if one followed the standard way to fine-tune the model. Training a larger model is tricky, not only the data amount matters, everything like data cleaning, learning rate, and decays will affect the final performance.