News for hackers. Stuff that matters.
Re:Ask HN: Is there any software you only made for your own ... by boricj on Friday July 05, 2024 @10:38AM (#40883197)
Over the past two years I've been working on tooling that allows me to delink programs back into object files. What started out as a bunch of Jython scripts is nowadays a full-blown Ghidra extension that can export working object files from a program selection in two mouse clicks. I'm using it as part of a video game decompilation project, but it also enables a whole bunch of other use cases I've documented on my blog.

It's not that it is meant for my use only (any capable reverse-engineer familiar with Ghidra should be able to pick it up and use it) nor that it will never see the light of the day (it's open-source). However, it is such an esoteric capability and outright heresy according to computer sciences that I'm having a hard time just finding people who can wrap their heads around the concept, let alone people who would actually want to use this. Simply put, it's so far out there that I don't even know where or how I could even advertise it in the first place, which makes it de facto for my own use only.

A couple of people did end up reaching out to me in the last couple of weeks about it, so it might be on the cusp of sprouting a very tiny user base. Still, I've made it for my own use and until very recently I've never expected anybody else would use it.

If someone wants to check out the dark magic: https://github.com/boricj/ghidra-delinker-extension (disclaimer: might give nightmares to linker developers).

Re:Ask HN: Is there any software you only made for your own ... by vintagedave on Friday July 05, 2024 @11:08AM (#40883429)
I know a linker developer, I'm going to send this to him :)

Not sure about heresy according to computer science. Sure, it's not intended, but it's a very clever thing to be able to do.

Re:Ask HN: Is there any software you only made for your own ... by boricj on Friday July 05, 2024 @01:51PM (#40884663)
Delinking by itself isn't a heresy (no more than disassembly or decompilation), but what I do with it definitively is. Ripping out MIPS code from a PlayStation video game and shoving it into a Linux program, dismembering a x86 Linux program and turning it into a native Windows program...

It's when you get creative and throw ABIs out the window in order to create some cursed chimeras that this really becomes heresy.

Re:Ask HN: Is there any software you only made for your own ... by nrr on Friday July 05, 2024 @02:12PM (#40884826)
One person's heresy is another person's sickos.png. This sounds exactly like my favorite sort of object code vivisection.
Re:Ask HN: Is there any software you only made for your own ... by liamYC on Friday July 05, 2024 @01:53PM (#40884684)
Technology like this will create huge selection pressures against desktop apps if it becomes easy for people to reverse engineer, remove payment mechanisms, and then freely distribute. Wouldn’t you think?
Re:Ask HN: Is there any software you only made for your own ... by boricj on Friday July 05, 2024 @03:08PM (#40885213)
Delinking isn't a miracle technology in that regard, you still need to put in the work to reverse-engineer the artifact.

It does allow a couple of nifty tricks, like pervasive binary patching (if the program is chunked into relocatable object files, then you're no longer constrained by the original program memory layout when patching/replacing stuff). It's also useful for decompilation projects, where you can reimplement a program one piece at a time until you no longer have binary pieces left and still create a fully working program at each step (you don't even need perfectly matching decompilation since the linker will mend stuff back together anyway).

Re:Ask HN: Is there any software you only made for your own ... by pmontra on Friday July 05, 2024 @02:46PM (#40885062)
Freely redistribute, I don't know. If a program can extract code from another one, a program can detect that code. It looks similar to virus signatures. A company with some IP would run the detector on the software of competitors.
Re:Ask HN: Is there any software you only made for your own ... by fragmede on Sunday July 07, 2024 @04:30AM (#40896061)
The tools and know-how to remove payment mechanisms from binaries have existed basically as long as binaries themselves have.
Re:Ask HN: Is there any software you only made for your own ... by jart on Friday July 05, 2024 @12:48PM (#40884225)
Something like that really deserves to be written in C so you don't need to install Ghidra to use it. The way I imagine it working is if I want the function `foo` then I'd say `objsuck -ffoo -ofoo.o prog.elf` and it'd look at the elf symbol table to find the `.size` of `foo` and copy that symbol into the .o. I guess you would then need to use xed to disassemble the opcodes to see what other symbols it jumps into or calls, and grab those too, along with any memory references, and then emit relocations. Overall I support this, since it'd be the easiest way to expropriate content from open source codebases whose source code is too byzantine to let me extract one teensy tiny little feature without the bloat. To me this is perfectly normal. You'd also be smart to only support COFF if the person uploads the binary to a hosted service you control and pays you money. In fact this would be even better if it could generate .s files from the object content, so no one would get triggered by binaries.
Re:Ask HN: Is there any software you only made for your own ... by boricj on Friday July 05, 2024 @02:07PM (#40884796)
> Something like that really deserves to be written in C so you don't need to install Ghidra to use it.

I think there's the Witchcraft Compiler Collection if you want a freestanding option [1], although I haven't looked into it too closely.

The problem is that object files are made up of section bytes, a symbol table and a relocation table. You can't just rip bytes out of a program and call it a day, you need to recreate all that information in order to actually delink code into relocatable object files.

Doing that isn't a trivial problem, it requires a lot of analysis and metadata, especially if you don't have debugging symbols or symbols at all. Leveraging Ghidra allows me to concentrate on the specifics of delinking, which can get very tricky depending on the platform (MIPS in particular is a nightmare to deal with).

I'm also trying to solve delinking in general and not just for one platform/ISA pair, so reinventing the wheel for every architecture out there is a nonstarter in that context.

[1] https://github.com/endrazine/wcc

Re:Ask HN: Is there any software you only made for your own ... by jart on Saturday July 06, 2024 @05:16AM (#40889202)
It's trivial if you can compile the binary with `cc -static -r` and you don't need to extract dwarf data too. If relocations are stripped, you're fine so long as you can count on st_size being present (i.e. doesn't contain handwritten assembly) and you're able to parse the machine language. If symbols are stripped, then it simply can't be solved. It's just reverse engineering at that point.
Re:Ask HN: Is there any software you only made for your own ... by boricj on Saturday July 06, 2024 @09:01AM (#40890195)
If you reduce the problem statement to an ELF x86 program written in C, with a symbol table and a complete relocation table (not just the one you get when dynamically linking), then sure it's trivial, you have almost all the information you need to make an object file (issues like switch jump tables can still crop up in that case). If you don't have that relocation table from `cc -r` however, you'll run into problems.

Without this relocation table on hand, you'll have to recreate it in order to make the section bytes relocatable again. This means analyzing code/data and identifying relocation spots, like you've said. But that `0x00400000` integer constant within an instruction or a word data, is it referring to the function at that address or is it the TOSTOP flag value? Who knows, but each time you get it wrong you'll corrupt four bytes in that object file.

I'm dealing with one rather gnarly scenario, which is a PlayStation video game without any source code, symbols [1] or linker map, just a bag of bytes in an a.out-like format. The MIPS architecture also happens to be an absolute nightmare to delink code from (one of the many pitfalls for example is the interaction between HI16/LO16 relocation pairs, branch delay slots and linkers with a peephole optimizer).

I've been at it for two years and I've only recently managed to pull it off on the entire game code [2]. Writing out the object file when you have the program bytes, the symbol table and the relocation tables is the easy part. Writing an analyzer that recreates a missing relocation table for the 80% of easy cases isn't too difficult. Squashing out the remaining 20% of edge cases is hard. All it takes is one mistake that affects a code path that's taken for some very exotic undefined behavior to occur in the delinked code.

Delinking with a missing relocation table (and without manually annotating the relocation spots yourself) is a thing that looks easy at first glance, but is deceptively hard to nail all of the edge cases. I'd gladly be proven wrong, but if you do have the full, original relocation table on hand then you're cheating with `cc -r` on code you just built yourself. Almost no real-world artifact spotted in the wild one would care about is ever built with that flag.

[1] I did end up recovering lots of data out of a leftover debugging symbols file from an early build later on, but that's a story for another time.

[2] Note that I'm working on top of a Ghidra database that contains symbols, type definitions and references, so the bulk of analysis is actually performed upstream of my tooling. Even then, the MIPS relocation synthesizer is a thousand lines of absolute eldritch horrors, but I do acknowledge that the x86 relocation synthesizer I have is quite tame in comparison.

Re:Ask HN: Is there any software you only made for your own ... by jart on Monday July 08, 2024 @11:10PM (#40912196)
> the MIPS relocation synthesizer is a thousand lines of absolute eldritch horrors

Wow I only really know amd64, arm64, and i8086. What is it about MIPS that makes it so evil?

Re:Ask HN: Is there any software you only made for your own ... by boricj on Tuesday July 09, 2024 @03:21AM (#40913383)
That would warrant an entire blog post to describe all the pitfalls [1], but I'll condense it down to the highlights.

On MIPS, loading a pointer is classically done in two instructions, a LUI and an ADDIU, which forms a HI16/LO16 relocation pair I need to identify precisely in order to delink code. I'm using Ghidra's references in my analyzers, but these are attached to only one instruction operand, typically a register or an immediate constant.

So my MIPS analyzer has to traverse the graph of register dependencies for a reference within an instruction and find which two instructions are the relocation pair. It's trickier than it sounds because references can have an addend that's baked in the immediate constants (so we can't just search for the pattern of the address bits inside the instructions) and complex memory access patterns inside large functions can create large graphs (ADDU in particular generates two branches to analyze, one per source register). It's bad enough that I have one method inside my analyzer in particular that is recursive and takes six arguments, four of which rotate right one step at each iteration.

But that graph traversal can't be done in reverse program order, because there are instruction patterns that can terminate the graph traversal too early with the right mix of branches, instruction sequencing and register reuse. I've had to integrate code flow analysis to figure out which parent code block has to be actually considered during the register graph traversal.

But the most evil horror is the branch delay slot. One particular peephole optimizer consists of vacuuming up a branch target instruction inside a branch delay slot and shift the branch target one instruction forward, which effectively shortens the execution flow by one instruction. It also duplicates the instruction, which is catastrophic if it had a HI16 relocation because now we have LO16 relocations with multiple HI16 parents, which can't be represented by object file formats. I have to detect and undo that optimization on the fly by shifting the branch targets one instruction back, which I accomplish by adjusting the relocation addends for the branches.

I've only written relocation analyzers for x86 and MIPS so far. I don't know what other horrors are lurking inside other architectures, but I expect that all RISC architectures with split relocations will require some form of that register graph traversal and code flow analysis [2]. What I do know is that my MIPS relocation analyzer [3] is probably the most algorithmically complex piece of code I've ever written so far, one that I've rewritten a half-dozen times over two years due to all the edge cases that kept propping up. I also had to create an extensive regression test suite to keep the analyzer from breaking down in subtle ways every time I need to modify it. I expect that there are still edge cases to fix in there that I haven't encountered yet.

[1] I've written about some of them here, but it's far from the whole story: https://boricj.net/tenchu1/2024/05/15/part-10.html

[2] That piece of code is split off in its own Java class: https://github.com/boricj/ghidra-delinker-extension/blob/mas...

[3] In case you're curious: https://github.com/boricj/ghidra-delinker-extension/blob/mas... (remember that the register graph and code flow bits are split off inside another class)

Re:Ask HN: Is there any software you only made for your own ... by cxr on Monday July 15, 2024 @05:46PM (#40971740)
> Something like that really deserves to be written in C so you don't need to install Ghidra to use it.

By that logic, it should be written in JS so you don't need to install a C build toolchain for it.